ai-agents

어떤 스킬인가요?

AI agent architecture, graph and loop composition, protocol choice, evaluation, and observability. Use when scoping or reviewing systems before implementation.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

#!/usr/bin/env python3
"""Locate and read a skill's own data files from inside that skill.

Why this exists
---------------
Each skill is fully self-contained: it owns its `data/` and its `_lib/` copy of
this resolver. There is no repo-level shared directory to fall back to. That is
what lets a skill work when detached from the repo — public-repo clone, a copied
folder, or a Claude Code plugin (plugins cannot reference files outside their
own directory, so no shared path could reach them anyway).

    <skill>/
      +- SKILL.md               <- marks the skill root
      +- data/versions.json     framework/runtime versions (refresh-versions.py)
      +- data/model-pricing.json  LLM prices per 1M tokens (hand-maintained: no
      |                           public API resolves provider pricing)
      +- _lib/resolve_versions.py   <- this file
      +- scripts/cost_estimator.py

Paths are resolved with `resolve()` before any walking, and that is load-bearing.
`sync-*-skills.sh` deploys each skill as its OWN symlink:

    ~/.claude/skills/ai-llm -> <repo>/frameworks/shared-skills/skills/ai-llm

so a *lexical* relative path escapes into the deployment root:

    ~/.claude/skills/ai-llm/scripts/../../data/versions.json
      -> ~/.claude/data/versions.json          # wrong tree, usually absent

That path can even "succeed" by hitting an unrelated file of the same name,
which is worse than failing. `resolve()` follows the symlink back to the real
skill directory first.

Resolution order (first hit wins):
  1. $SHARED_SKILLS_VERSIONS / $SHARED_SKILLS_PRICING — explicit override
  2. the CALLER's skill-local data/ — found by walking up to the nearest
     ancestor holding a SKILL.md, the spec's own marker for "this is a skill"
  3. this file's own skill-local data/, for when the caller is outside a skill

Every function degrades to None/default rather than raising: a skill script must
stay runnable when the data file is absent, and a missing version is a reason to
say "unknown", never to crash.

Usage:
    import sys
    from pathlib import Path
    sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "_lib"))
    from resolve_versions import version_of, price_of, pricing_stale_days

    version_of("next")            -> "16.3.0" or None
    version_of("next", "unknown") -> "16.3.0" or "unknown"
    price_of("claude-haiku-4-5")  -> {"input_per_1m": 1.0, ...} or None
    pricing_stale_days()          -> 42 if the table is 42d old past its
                                     window, else None
"""

from __future__ import annotations

import json
import os
from pathlib import Path
from typing import Any

# Directory name holding the data files, relative to the skill root.
_DATA_DIRNAME = "data"
_DATA_FILENAME = "versions.json"
_PRICING_FILENAME = "model-pricing.json"

# How far up to walk before giving up. The real depth from a skill script is 3
# (scripts -> <skill> -> skills -> shared-skills); the extra headroom covers
# nested asset/build directories without walking to the filesystem root.
_MAX_WALK_UP = 8


def _walk_up_for_data(start: Path, filename: str = _DATA_FILENAME) -> Path | None:
    """Walk up from `start` looking for <ancestor>/data/<filename>.

    `resolve()` is essential, not cosmetic: skills deploy as symlinks, so a
    lexical walk stays inside ~/.claude|.agents|.codex and can silently match an
    unrelated same-named file. Resolving first puts us in the real repo tree.
    """
    try:
        here = start.resolve()
    except OSError:
        return None
    if here.is_file():
        here = here.parent
    for ancestor in [here, *here.parents][:_MAX_WALK_UP]:
        candidate = ancestor / _DATA_DIRNAME / filename
        if candidate.is_file():
            return candidate
    return None


def _skill_local_data(start: Path, filename: str) -> Path | None:
    """Find <skill-root>/data/<filename> for the skill containing `start`.

    The skill root is the nearest ancestor holding a SKILL.md — the spec's own
    marker for "this directory is a skill". Falling back to a plain few-levels-up
    walk would wrongly match a sibling skill's data/ when skills sit side by side.

    Deliberately does NOT resolve() the caller: a skill deployed by symlink
    should still prefer the data shipped alongside it in the repo, and resolve()
    lands in the same place for that case anyway.
    """
    try:
        here = start if start.is_dir() else start.parent
    except OSError:
        return None
    for ancestor in [here, *here.parents][:_MAX_WALK_UP]:
        if (ancestor / "SKILL.md").is_file():
            candidate = ancestor / _DATA_DIRNAME / filename
            return candidate if candidate.is_file() else None
    return None


def _locate(filename: str, env_var: str, caller_file: str | Path | None) -> Path | None:
    """Resolution order for any data file.

    The CALLER's own skill wins: each skill ships its own data/ and its own copy
    of this resolver, so a skill published alone — the public repo's allowlist, a
    single copied folder, a packaged plugin — finds its data without reaching
    outside itself. There is no repo-level master; the plain walk-up at the end
    only serves callers that sit outside any skill (a bare script, a test).
    """
    override = os.environ.get(env_var, "").strip()
    if override:
        p = Path(override).expanduser()
        return p if p.is_file() else None

    # 1. The calling skill's own data/ — self-contained, works when detached.
    if caller_file is not None:
        local = _skill_local_data(Path(caller_file), filename)
        if local is not None:
            return local

    # 2. This file's own skill-local data/ (the usual path: running as _lib/).
    local = _skill_local_data(Path(__file__), filename)
    if local is not None:
        return local

    # 3. No SKILL.md above either location — caller is outside a skill. Fall back
    #    to a plain walk for a sibling data/ so bare scripts still work.
    found = _walk_up_for_data(Path(__file__), filename)
    if found is not None:
        return found

    if caller_file is not None:
        return _walk_up_for_data(Path(caller_file), filename)
    return None


def versions_path(caller_file: str | Path | None = None) -> Path | None:
    """Return the path to versions.json, or None if it cannot be located.

    `caller_file` is optionally the calling module's `__file__`; passing it lets
    a skill that lives outside this tree still find a data/ dir of its own.
    """
    return _locate(_DATA_FILENAME, "SHARED_SKILLS_VERSIONS", caller_file)


def load_versions(caller_file: str | Path | None = None) -> dict[str, Any]:
    """Return the parsed versions document, or {} if unavailable/unparseable."""
    path = versions_path(caller_file)
    if path is None:
        return {}
    try:
        data = json.loads(path.read_text(encoding="utf-8"))
    except (OSError, json.JSONDecodeError):
        return {}
    return data if isinstance(data, dict) else {}


def version_of(
    package: str,
    default: str | None = None,
    caller_file: str | Path | None = None,
) -> str | None:
    """Return the resolved `latest` version for `package`, else `default`.

    `package` is the key used in versions.json (an npm name like "next" or
    "@angular/core", or an endoflife.date slug like "go"/"python").
    """
    entry = load_versions(caller_file).get("versions", {}).get(package)
    if isinstance(entry, dict):
        latest = entry.get("latest")
        if isinstance(latest, str) and latest:
            return latest
    return default


def refreshed_utc(caller_file: str | Path | None = None) -> str | None:
    """Return the ISO timestamp of the last refresh, or None."""
    stamp = load_versions(caller_file).get("refreshed_utc")
    return stamp if isinstance(stamp, str) and stamp else None


# --------------------------------------------------------------- model pricing

def pricing_path(caller_file: str | Path | None = None) -> Path | None:
    """Return the path to model-pricing.json, or None if unavailable."""
    return _locate(_PRICING_FILENAME, "SHARED_SKILLS_PRICING", caller_file)


def load_pricing(caller_file: str | Path | None = None) -> dict[str, Any]:
    """Return the parsed pricing document, or {} if unavailable/unparseable."""
    path = pricing_path(caller_file)
    if path is None:
        return {}
    try:
        data = json.loads(path.read_text(encoding="utf-8"))
    except (OSError, json.JSONDecodeError):
        return {}
    return data if isinstance(data, dict) else {}


def price_of(
    model_id: str,
    caller_file: str | Path | None = None,
) -> dict[str, Any] | None:
    """Return the pricing entry for `model_id`, or None.

    Accepts either the qualified key ("anthropic/claude-haiku-4-5") or the bare
    model name ("claude-haiku-4-5"); the bare form matches the first vendor that
    carries it, which is unambiguous in practice because vendors do not share
    model names.
    """
    models = load_pricing(caller_file).get("models", {})
    if not isinstance(models, dict):
        return None
    entry = models.get(model_id)
    if isinstance(entry, dict):
        return entry
    if "/" not in model_id:
        for key, value in models.items():
            if key.split("/", 1)[-1] == model_id and isinstance(value, dict):
                return value
    return None


def pricing_stale_days(caller_file: str | Path | None = None) -> int | None:
    """Days the pricing table is PAST its staleness window, or None if fresh.

    Returns None when the table is fresh, absent, or undated — callers warn only
    on a positive number, so a missing file never produces a spurious warning.
    """
    from datetime import date as _date

    doc = load_pricing(caller_file)
    stamp = doc.get("last_verified")
    if not isinstance(stamp, str):
        return None
    try:
        verified = _date.fromisoformat(stamp)
    except ValueError:
        return None
    window = doc.get("stale_after_days")
    window = window if isinstance(window, int) else 30
    age = (_date.today() - verified).days
    return age if age > window else None


if __name__ == "__main__":
    import sys

    # Pass __file__ exactly as the skill's scripts do. Without it the skill-local
    # tier cannot be used at all and resolution degrades to the bare walk-up —
    # so a self-test that omits it exercises the weakest path, not the real one.
    _self = __file__

    # A skill carries only the data it consumes: ai-llm has model-pricing.json
    # and no versions.json, and that is correct, not a failure. Report each file
    # independently and fail only when NEITHER resolves, which is the one case
    # that means resolution itself is broken.
    vpath = versions_path(_self)
    ppath = pricing_path(_self)

    if vpath is None:
        print("versions.json     : not carried by this skill")
    else:
        doc = load_versions(_self)
        entries = doc.get("versions", {})
        print(f"versions.json     : {vpath}")
        print(f"refreshed_utc     : {refreshed_utc(_self) or 'unknown'}")
        print(f"entries           : {len(entries)}")
        for key, entry in sorted(entries.items()):
            if isinstance(entry, dict):
                print(f"  {entry.get('label', key):16} {entry.get('latest', '?'):12} {key}")

    print()
    if ppath is None:
        print("model-pricing.json: not carried by this skill")
    else:
        pdoc = load_pricing(_self)
        stale = pricing_stale_days(_self)
        window = pdoc.get("stale_after_days", 30)
        window = window if isinstance(window, int) else 30
        state = f"[STALE by {stale - window}d]" if stale else "[fresh]"
        print(f"model-pricing.json: {ppath}")
        print(f"last_verified     : {pdoc.get('last_verified', 'unknown')}  {state}")
        print(f"models            : {len(pdoc.get('models', {}))}")

    if vpath is None and ppath is None:
        print("\nNeither data file resolved — resolution is broken, not merely absent.",
              file=sys.stderr)
        raise SystemExit(1)

interface:
  display_name: "AI Agents Development — Architecture Hub"
  short_description: "AI agent architecture plus graph and loop composition"
  default_prompt: "Use $ai-agents for AI agent architecture, Graph Engineering and Loop Engineering composition, protocol choice, evaluation, and observability. Use when scoping or reviewing systems before implementation."

# AI-Native SDLC Agent Template

Purpose: Delegate mechanical SDLC work to the agent while humans own intent, architecture, and release. Use for feature delivery, refactors, or hotfixes.

Inputs:
- Spec or ticket
- Repo context (paths, constraints, coding standards)
- AGENTS.md / tool scopes (allowed commands, time caps, kill switch)
- Required tests and deploy checks

Preflight:
- Set max runtime and token budget; require explicit kill switch
- Allowlist commands/tools; block package installs unless approved
- Enable logging (plan, actions, diffs, test output)
- Require PLAN.md creation or planning tool output before coding

Runbook (Delegate → Review → Own)
- Plan
  - Agent drafts PLAN.md with scope, code paths, dependencies, risks, and exit criteria
  - Human reviews/edits plan; reject until risks/edge cases captured
- Design
  - Agent maps mocks/specs to components; applies design tokens/style guides
  - Call MCP component library; list accessibility gaps
  - Human signs off on architecture changes or schema migrations
- Build
  - Agent scaffolds end-to-end: models/APIs/UI/tests/docs in one run
  - Enforce conventions (telemetry, errors, lint format, feature flags)
  - Block commits/merges; diff-only output; no secrets
- Test
  - Require failing test first; agent adds/updates tests and runs suite
  - Capture coverage delta and flaky-test notes
  - Human verifies assertions/fixtures reflect intent
- Review
  - Agent performs first-pass review focused on P0/P1 bugs and policy violations
  - Human reviews architecture, performance, safety, migrations; owns merge
- Document
  - Agent writes PR summary, file/module notes, mermaid diagram if useful
  - Human adds “why” and approvals; ensure docs ship with code
- Deploy & Maintain
  - Agent links logs/metrics via MCP; proposes hotfix with rollback plan
  - Human approves rollout; track evals/drift/regressions

Guardrails
- Time cap per run; abort on unexpected prompts or new permission requests
- No package install/network without approval; no credential edits
- Require explicit test run and results before proposing merge
- Always surface uncertainties and blocked items; never self-approve

Outputs Checklist
- PLAN.md (or planner output), code diffs, tests run + results, doc updates, PR summary, risk/edge list, next steps/rollout notes

# Agent Design & Safety Checklist

**Purpose**: Ensure production-ready agent development with multi-layer safety controls and observability baseline.

---

## Template Contract

### Goals
- Ensure the agent is bounded, auditable, and rollbackable.
- Prevent unsafe tool actions, data leakage, and uncontrolled spend.
- Make quality and safety measurable before production rollout.

### Inputs
- Agent spec (purpose, users, permissions).
- Tool inventory (APIs, data stores, side effects).
- Data classification (PII, confidential, public).
- SLOs/budgets (latency, cost per request, failure rate).

### Decisions
- Autonomy level and step/time/cost caps.
- Tool allowlist + authorization model per tool.
- HITL triggers and escalation paths.
- Evaluation gates and rollout strategy (canary, shadow, rollback).

### Risks
- Prompt injection and tool abuse via untrusted inputs.
- Data exfiltration via tools, logs, or citations.
- Runaway loops (cost/latency explosions) and cascading retries.
- Non-reproducible behavior due to hidden state or missing traces.

### Metrics
- Task success rate, tool success rate, refusal correctness.
- Guardrail violation rate, PII leakage rate.
- Latency (TTFT/total) p50/p95/p99 and cost per request.

## Pre-Development

### Scope Definition
- [ ] Agent purpose documented (single responsibility)
- [ ] Tool allowlist defined (no "all tools" access)
- [ ] Maximum autonomy level specified (L1-L5)
- [ ] HITL triggers identified (financial, destructive, legal actions)

### Risk Assessment
- [ ] Blast radius documented (what can go wrong)
- [ ] Data access classified (PII, confidential, public)
- [ ] Destructive actions identified (delete, modify, send)
- [ ] Regulatory constraints checked (GDPR, HIPAA, SOX, EU AI Act)

---

## Implementation

### Guardrails (Multi-Layer Defense Required)

**Layer 1: Input Validation**
- [ ] PII redaction configured
- [ ] Content filtering enabled
- [ ] Prompt injection detection active

**Layer 2: Authorization**
- [ ] RBAC/ABAC configured per tool
- [ ] Fine-grained permissions defined
- [ ] Principle of least privilege applied

**Layer 3: Tool Gating**
- [ ] Tool signatures verified (artifact signing)
- [ ] Human approval required for high-risk operations
- [ ] Rate limits per tool configured

**Layer 4: Output Filtering**
- [ ] PII detection in responses
- [ ] Policy compliance validation
- [ ] Content moderation active

**Layer 5: Observability**
- [ ] OpenTelemetry spans configured
- [ ] SIEM integration active
- [ ] Real-time alerts defined

### OpenTelemetry Spans (Required)

```yaml
spans:
  - llm_call: {prompt, response, tokens, latency, model}
  - tool_call: {name, params, result, duration, success}
  - retrieval: {query, chunks, scores, method}
  - memory_op: {operation, type, key, size}
  - agent_handoff: {source, target, schema_version, trace_id}
```

### Failure Handling
- [ ] Retry policy defined (max retries, exponential backoff)
- [ ] Fallback behavior specified
- [ ] Timeout limits set (per-step and total)
- [ ] Error classification (retriable vs fatal)
- [ ] Graceful degradation path documented

---

## Pre-Production

### Evaluation Suite
- [ ] Golden dataset created (minimum 50 test cases)
- [ ] Final answer evaluation (correctness, grounding, clarity)
- [ ] Trajectory evaluation (step quality, tool use)
- [ ] Safety evaluation (policy violations, harmful content)
- [ ] Adversarial testing completed

### Security Testing
- [ ] OWASP GenAI Top 10 checked
- [ ] Prompt injection testing passed
- [ ] Tool abuse scenarios tested
- [ ] PII leakage testing passed

### Deployment Readiness
- [ ] Canary deployment configured
- [ ] Rollback procedure documented and tested
- [ ] Incident runbook created
- [ ] On-call rotation assigned

---

## Production Metrics

| Metric | Target | Alert Threshold |
|--------|--------|-----------------|
| Tool success rate | >=95% | <90% |
| Latency P95 | <5s | >10s |
| Hallucination rate | <5% | >10% |
| HITL approval rate | Monitor | Sudden change |
| Cost per request | <$0.10 | >$0.50 |
| Error rate | <1% | >5% |
| Guardrail violations | 0 | >0 |

---

## Post-Launch Monitoring

### Daily Checks
- [ ] Error rate within threshold
- [ ] Cost within budget
- [ ] No guardrail violations
- [ ] Latency stable

### Weekly Reviews
- [ ] Evaluation score trends
- [ ] User feedback analysis
- [ ] Cost optimization opportunities
- [ ] Security incident review

### Monthly Reviews
- [ ] Model performance degradation check
- [ ] Tool usage patterns analysis
- [ ] Capacity planning update
- [ ] Compliance audit

---

## Anti-Patterns to Avoid

| Anti-Pattern | Risk | Detection |
|--------------|------|-----------|
| Runaway autonomy | Resource exhaustion, unintended actions | Monitor step count, cost per request |
| Hidden state | Non-reproducible behavior | Checkpoint logging, state serialization |
| Unbounded tools | Security vulnerabilities | Tool allowlist enforcement |
| Missing handoff validation | Context corruption | Schema validation errors |
| Single guardrail layer | Bypass via injection | Red team testing |
| Trusting tool outputs | Injection attacks | Output sanitization checks |

---

## Sign-Off

| Role | Name | Date | Signature |
|------|------|------|-----------|
| ML Engineer | | | |
| Security | | | |
| Product Owner | | | |
| Platform | | | |

# Quick Agent Template

*Purpose: Rapidly define a functional agent with minimal configuration. Suitable for prototypes, internal tooling, or simple production agents.*

---

## When to Use

Use this template when you need:

- A compact agent specification  
- A starting point for rapid iteration  
- A simple, single-agent design  
- Minimal boilerplate  
- Quick prototyping with tools or RAG  

---

# TEMPLATE STARTS HERE

## 1. Agent Overview

**Agent Name:**  
[Name]

**Primary Goal:**  
[Short description]

**Key Behaviors:**  

- [Behavior 1]  
- [Behavior 2]  
- [Behavior 3]

**Limitations:**  

- [Out-of-scope items]  
- [Disallowed operations]  

**Capability Level & Policy:**  
L[0-4] and allowed tools/scopes, approvals, and HITL gates at this level.

**Contracts & Handoffs (if applicable):**  
Schemas, trace_id, escalation rules, and any negotiation/subcontract needs.

---

## 2. System Instructions

```text
You are a production agent designed to complete tasks using a plan → act → observe loop.
You must use only authorized tools.
You must ground factual statements in retrieved evidence.
You must ask for confirmation before irreversible or high-risk actions.
If you cannot perform a task, say so clearly.
Keep all responses short, structured, and operational.
```

---

## 3. Tools

### 3.1 Tool List

| Tool Name | Purpose | Confirm | Notes |
|-----------|----------|---------|--------|
| [tool_1]  | [...]    | yes/no  | [...]  |
| [tool_2]  | [...]    | yes/no  | [...]  |

### 3.2 Tool Rule Summary

- Validate all parameters.  
- Never hallucinate paths/IDs/fields.  
- Retry only transient errors.  
- Verify tool output before using it.  

---

## 4. RAG (Optional)

### Retrieval Pipeline

```text
query → embed → retrieve → rerank → inject → answer
```

### Injection Format

```text
<retrieved>
[chunk_1]
[chunk_2]
</retrieved>
```

### RAG Rules

- Use retrieval before answering fact-based questions.  
- Cite retrieved evidence directly.  
- Remove irrelevant chunks.  

---

## 5. Memory (Optional)

### Memory Rules

- Store only user-approved, non-sensitive, stable facts.  
- Summarize session history when long.  
- Retrieve memory only when relevant.  
- Session design: scope/handle, sharing rules, replay limits.  
- Write triggers: phase completion, confidence drop, new entity, pre-handoff.  
- Provenance: source, timestamp, origin agent/tool, approvals, confidence.  

---

## 6. Safety

### High-Risk Confirmation Required For

- OS/system actions  
- File modifications  
- Financial or legal operations  
- External system mutations

### Safety Rules

- Reject unsupported or dangerous tasks.  
- Sanitize all user inputs.  
- Block hallucinated tools/actions.  

---

## 7. Observability

### Required Logs

- Input  
- Plan  
- Tool calls  
- Tool outputs  
- Final answer

### Required Traces

- LM call  
- Tool call  
- Retrieval (if used)  

---

## 8. Deployment (Minimal)

- [ ] Evaluation tests pass  
- [ ] Safety checks pass  
- [ ] Version pinned  
- [ ] Rollback path defined  

---

# COMPLETE EXAMPLE (Optional)

## 1. Agent Overview

**Agent Name:** Internal Search Assistant  
**Primary Goal:** Answer internal policy questions using RAG.

## 3. Tools (Example)

```yaml
search_policies:
  description: "Search internal policy index by query"
  input_schema:
    query: string
  output_schema:
    results: list
  confirm: no
  error_handling:
    retry: 1
    timeout: 10
```

## 4. RAG (Example)

```text
<retrieved>
[Policy Section 3.2: VPN Requirements]
</retrieved>
```

---

# End of Template

# Specialized Agent Template

*Purpose: Provide a structured template for designing **specialized** agents with domain-specific rules, advanced RAG, complex tool use, multi-agent roles, OS automation, or high-risk operational constraints.*

---

## When to Use

Use this template when:

- The agent performs **complex retrieval**, **multiple tools**, or **multi-step reasoning**.  
- The agent operates in **regulated**, **sensitive**, or **high-risk** domains.  
- The agent integrates with **OS**, **browser**, or **external systems**.  
- The agent is part of a **multi-agent orchestration**.  
- The agent requires **strict safety**, **advanced evaluation**, or **custom workflows**.

---

## Structure

This template contains 14 specialized sections:

1. **Specialization Summary**  
2. **System Instructions (Specialized Form)**  
3. **Operational Scope & Boundaries**  
4. **Domain Rules & Constraints**  
5. **Advanced Tools & Execution Rules**  
6. **Advanced RAG (Domain-Aware)**  
7. **Memory Strategy (Domain-Specific)**  
8. **Multi-Agent Role Definition (If Used)**  
9. **Planning Framework (Custom)**  
10. **Safety Enforcement Layer**  
11. **Validation Layer**  
12. **Observability (Deep Mode)**  
13. **Evaluation Framework (Domain-Specific)**  
14. **Deployment Requirements**  

---

# TEMPLATE STARTS HERE

## 1. Specialization Summary

**Agent Name:**  
[Name]

**Domain:**  
[e.g., legal QA, financial modeling, clinical data extraction, OS automation]

**Primary Functions:**  

- [Function 1]  
- [Function 2]  
- [Function 3]

**Special Notes:**  

- [High-risk constraints, compliance rules, tool limitations, etc.]

**Capability Level & Policy:**  
L[0-4]; allowed scopes/tools, approvals, HITL gates, and audit requirements for this level.

**Contracts & Handoffs (if multi-agent/external):**  
Input/output Schemas, contract version, trace_id requirements, escalation rules, negotiation/subcontract handling.

---

## 2. System Instructions (Specialized)

```text
You are a specialized agent operating in the [domain] domain.
You must perform all tasks using a strict plan → act → observe → update loop.

You MUST:
- Use only approved tools.
- Ground all facts in retrieved evidence.
- Adhere to domain-specific rules and constraints.
- Ask for confirmation before high-risk actions.
- Produce structured outputs as required.

You MUST NOT:
- Invent facts, policies, legal interpretations, numbers, or system paths.
- Perform disallowed or irreversible actions without explicit confirmation.
- Use tools not listed in the Tools section.
```

**Output Format Requirements:**  

- JSON  
- Markdown tables  
- Action blocks  
- RAG evidence sections  

---

## 3. Operational Scope & Boundaries

**Allowed Tasks:**  

- [Explicit task types]

**Out-of-Scope Tasks:**  

- [Must decline or redirect]

**Authority Level:**  

- [read-only? modify? execute?]

**Escalation Rules:**  

- [When to ask for confirmation or clarification]  
- [When to escalate to human/manager agent]  

---

## 4. Domain Rules & Constraints

**Domain Standards:**  

- [e.g., legal citations, medical terminology, finance accuracy]

**Regulatory Requirements:**  

- [HIPAA / GDPR / SOC2 / internal policies]

**Accuracy Requirements:**  

- [Zero hallucination allowed?]  
- [Evidence-backed answers required?]

**Forbidden Behaviors:**  

- [Domain-specific limitations]

---

## 5. Advanced Tools & Execution Rules

### 5.1 Tool List (Specialized)

| Tool       | Purpose               | Confirm | Risks | Notes |
|------------|------------------------|---------|--------|-------|
| [tool_1]   | [...]                  | yes/no  | [list] | [...] |
| [tool_2]   | [...]                  | yes/no  | [list] | [...] |

### 5.2 Tool Definition (Example)

```yaml
tool_name:
  description: [operational purpose]
  input_schema:
    param_a: string
    param_b: integer
  output_schema:
    result: object
  confirm: [yes/no]
  error_handling:
    retry: 1
    timeout: 20
```

### 5.3 Tool Execution Rules

- Validate all parameters.  
- Reject hallucinated IDs / paths / fields.  
- Use high-risk confirmation logic.  
- Retry only transient errors (e.g., timeouts).  
- Verify output fields before plan continues.  

---

## 6. Advanced RAG (Domain-Aware)

### 6.1 RAG Pipeline

```text
query → rewrite → embed → retrieve → rerank → filter → enrich → inject → answer
```

### 6.2 Index Specifications

| Index | Domain | Chunk Size | Reranker | Notes |
|-------|--------|-----------:|----------|-------|
| [...] | [...]  | [...]      | [...]    | [...] |

### 6.3 Evidence Injection

```text
<retrieved>
[chunk_1]
[chunk_2]
</retrieved>
```

### 6.4 Domain Filters

- Enforce domain matching.  
- Remove irrelevant or conflicting chunks.  
- Require citations for all factual claims.  

---

## 7. Memory Strategy (Domain-Specific)

**Memory Types Enabled:**  

- Session memory  
- Long-term preferences  
- Episodic events  
- Domain knowledge only when safe  

### Write Rules

Write memory only if:

- User explicitly confirms  
- Information is non-sensitive  
- Information is verifiable  
- Information is stable over time  

**Write triggers:** Phase completion, confidence drop, new entities, pre-handoff consolidation.  
**Provenance:** Source, timestamp, originating agent/tool, approvals, confidence.  
**Session design:** Scope/handle, sharing across agents, replay limits.

### Retrieval Rules

- Retrieve only relevant entries  
- Summaries required >150 tokens  
- Apply domain filters  

---

## 8. Multi-Agent Role Definition (Optional)

### Example Roles

**Manager:** Decompose tasks.  
**Worker-Research:** Perform RAG + summaries.  
**Worker-Execution:** Execute tool operations.  
**Evaluator:** Score correctness / grounding / safety.  
**Router:** Domain routing.  

### Optional Multi-Agent Structure

```yaml
roles:
  manager:
    responsibilities: [planning, coordination]
  worker_research:
    responsibilities: [retrieval, summarization]
  worker_execution:
    responsibilities: [tool-use, OS actions]
  evaluator:
    responsibilities: [scoring, verification]
```

---

## 9. Planning Framework (Custom)

### Planning Pattern (Specialized)

```
1. analyze(query)
2. retrieve / collect needed context
3. produce plan (atomic steps)
4. execute each step:
      observe → ground → act → verify
5. consolidate results
6. produce final output
```

### Plan Requirements

- Each step explicit  
- Each step lists expected outputs  
- No speculative steps  
- Revise plan after each observation  

---

## 10. Safety Enforcement Layer

### Safety Checks

- Domain restrictions  
- High-risk action detection  
- Data sensitivity detection  
- Tool misuse prevention  
- OS-action constraints  

### Safety Gates

```
check_domain()
check_action_risk()
check_tool_scope()
sanitize_inputs()
```

### Confirmation Prompts

Include exact parameters:

```
You requested a high-risk operation:
Action: [...]
Parameters: [...]
Please confirm "yes" to proceed.
```

---

## 11. Validation Layer

### Validation Pattern

```
validate_input()
validate_tool_params()
validate_rag_chunks()
validate_memory()
validate_output()
```

### Required Validations

- Type checking  
- Range checking  
- Domain consistency  
- Evidence alignment  

---

## 12. Observability (Deep Mode)

### Required Logs

- Input  
- Plan  
- Tool calls  
- Tool results  
- RAG retrieval  
- Memory reads/writes  
- Evaluator scores  
- Final answer  

### Required Traces

- LM spans  
- Tool spans  
- Retrieval spans  
- Memory spans  
- Safety spans  

### Metrics

| Metric | Threshold |
|--------|-----------|
| Tool Success Rate | ≥ 95% |
| Grounding Score | ≥ 4.0 |
| Accuracy | ≥ 4.0 |
| Safety | 100% pass |
| Latency p95 | ≤ [X] |

---

## 13. Evaluation Framework (Domain-Specific)

### Evaluation Categories

- Correctness  
- Grounding  
- Domain compliance  
- Safety performance  
- Tool execution accuracy  
- OS action accuracy (if used)  

### LLM-as-Judge Template

```json
{
  "correctness": 1-5,
  "grounding": 1-5,
  "domain_accuracy": 1-5,
  "tool_usage": 1-5,
  "safety": "pass|fail",
  "notes": "..."
}
```

---

## 14. Deployment Requirements

### Pre-Deployment Checklist

- [ ] All evaluation tests passed  
- [ ] RAG pipeline validated  
- [ ] Tool-call tests validated  
- [ ] Safety tests passed  
- [ ] Version pinned  
- [ ] Canary rollout configured  
- [ ] Rollback plan ready  

### Deployment Flow

```
dev → CI → staging → canary (1%) → expand (25%) → full production
```

---

# COMPLETE EXAMPLE (Optional)

## 1. Specialization Summary

**Agent Name:** Medical Safety Summarizer  
**Domain:** Clinical documentation (read-only)  
**Primary Functions:**  

- Extract key information  
- Detect unsafe statements  
- Summarize with evidence  
**Constraints:**  
- No diagnosis generation  
- Must cite sections  

## 6. Advanced RAG (Example)

**Injection Format**

```text
<retrieved>
[Section 2.3: Symptoms]
[Section 4.0: Contraindications]
</retrieved>
```

---

# End of Template

# Standard Agent Operations Template

*Purpose: Create a full production-ready agent specification including memory, tools, RAG, evaluation, observability, safety, and deployment.*

---

## Related Resources

**Best Practices:**

- [Agent Operations](../../references/agent-operations-best-practices.md) - Action loops, planning patterns
- [Tool Design & Validation](../../references/tool-design-specs.md) - MCP tools, schemas, error handling
- [RAG Patterns](../../references/rag-patterns.md) - Contextual retrieval, hybrid search
- [Deployment & Safety](../../references/deployment-ci-cd-and-safety.md) - Multi-layer guardrails, HITL

**Related Skills:**

- [Prompt Engineering](../../../ai-prompt-engineering/SKILL.md) - System prompt optimization
- [Observability](../../../qa-observability/SKILL.md) - OpenTelemetry, metrics
- [Security](../../../software-security-appsec/SKILL.md) - Input validation, OWASP Top 10

---

## When to Use

Use this template when:

- Designing a new production agent.
- Adding memory, RAG, or tools to an existing agent.
- Creating multi-agent configurations.
- Preparing for evaluation, staging, or deployment.

---

## Structure

This template has 11 sections:

1. **Agent Overview**  
2. **System Instructions**  
3. **Tools**  
4. **Memory**  
5. **RAG (Retrieval-Augmented Grounding)**  
6. **Multi-Agent Configuration (Optional)**  
7. **Safety & Guardrails**  
8. **Observability**  
9. **Evaluation**  
10. **Deployment**  
11. **OS Agent Integration (If Applicable)**  

---

# TEMPLATE STARTS HERE

## 1. Agent Overview

**Agent Name:**  
[Name]

**Primary Goal:**  
[What the agent must achieve]

**Key Behaviors:**  

- [Behavior 1]  
- [Behavior 2]  
- [Behavior 3]

**Constraints:**  

- [Safety restrictions]  
- [Budget / token caps]  
- [Disallowed actions]

**Capability Level & Policy:**  
L[0-4] (static → tool → strategic → multi-agent → self-evolving); allowed tools, scopes, approvals, and HITL gates at this level.

**Contracts & Handoffs (if multi-agent or external):**  
Input/output JSON Schemas, contract version, trace_id requirements, escalation rules, negotiation/subcontract needs.

---

## 2. System Instructions

**Core Behavior:**

```text
You are a production agent.
You must follow a plan → act → observe loop for every step.
You may only use tools you are authorized to use.
Ground all factual statements in retrieved evidence when available.
Ask for confirmation before performing irreversible actions.
Keep answers concise and operational.
```

**Style Requirements:**  

- [tone, brevity, formatting rules]  
- [structured outputs: JSON / markdown / tables]  

---

## 3. Tools

### 3.1 Available Tools

| Tool       | Purpose               | Input Schema        | Output Schema       | Notes          |
|------------|-----------------------|---------------------|---------------------|----------------|
| [tool_1]   | [what it does]        | { param: type }     | { field: type }     | [limits]       |
| [tool_2]   | [what it does]        | { param: type }     | { field: type }     | [limits]       |

### 3.2 Tool Definitions

```yaml
tool_name:
  description: [clear operational purpose]
  input_schema:
    field_1: string
    field_2: number
  output_schema:
    result: string
  confirm: yes/no
  error_handling:
    retry: 1
    timeout: 30
```

### 3.3 Tool Use Rules

- Validate all parameters before calling.  
- Never hallucinate IDs, paths, or coordinates.  
- Use tools when external data or action is required.  
- Do not chain tools without verifying each result.  

---

## 4. Memory

### 4.1 Memory Types Used

- **Session memory:** [yes/no]  
- **Long-term memory:** [yes/no]  
- **Episodic memory:** [yes/no]  
- **Task-specific scratchpad:** [yes/no]

**Session design:** Scope/handle, sharing rules across agents, replay limits.

### 4.2 Memory Write Rules

Write memory only when:

- User explicitly confirms.  
- Fact is verifiable and non-sensitive.  
- Fact will be reused later.  
- Provenance (source, timestamp) can be stored.

**Write triggers:** Phase completion, confidence drop, new entity detected, pre-handoff consolidation.
**Provenance fields:** Source, timestamp, tool/agent of origin, approvals, confidence.

### 4.3 Memory Retrieval Rules

- Retrieve only relevant memories.  
- Summarize if > 200 tokens.  
- Apply recency filters when appropriate.  

---

## 5. RAG (Retrieval-Augmented Grounding)

### 5.1 Retrieval Pipeline

```text
query → rewrite → embed → retrieve → rerank → filter → inject → answer
```

### 5.2 Indexes Used

| Index Name  | Domain            | Chunk Size | Reranker | Notes         |
|-------------|-------------------|-----------:|----------|---------------|
| [index_1]   | [domain]          | [size]     | [model]  | [notes]       |

### 5.3 RAG Injection Format

```text
<retrieved>
[chunk_1]
[chunk_2]
...
</retrieved>
```

### 5.4 RAG Rules

- Always rerank retrieved results.  
- Discard irrelevant or stale chunks.  
- All factual claims should be traceable to chunks.  

---

## 6. Multi-Agent Configuration (Optional)

**Pattern:** Manager / Worker / Router / Evaluator

| Agent      | Role       | Tools       | Inputs | Outputs |
|------------|------------|------------|--------|---------|
| manager    | planning   | none       | task   | subtasks|
| worker_X   | execution  | [tools]    | subtask| result  |
| evaluator  | scoring    | none       | result | score   |
| router     | routing    | none       | query  | agent   |

---

## 7. Safety & Guardrails

### 7.1 Input Safety Filters

- Block prompt injection attempts.  
- Block unsupported domains.  
- Normalize and sanitize inputs.  

### 7.2 High-Risk Actions

Require explicit confirmation for:

- Financial or legal actions.  
- OS-level commands.  
- File deletion or modification.  
- External system writes.  

### 7.3 Output Safety

- Must avoid disallowed content.  
- Must not expose secrets or PII.  
- Must refuse unsupported dangerous requests.  

---

## 8. Observability

### 8.1 Required Logs

- User input (sanitized).  
- Agent plan.  
- Tool calls (name + parameters).  
- Tool results (status + output).  
- RAG retrieval details.  
- Final answer.  

### 8.2 Required Traces

One span per:

- LM call  
- Tool call  
- Retrieval step  
- Memory read/write  

### 8.3 Metrics

| Metric             | Target / Threshold |
|--------------------|--------------------|
| Tool success rate  | ≥ 95%              |
| Latency p95        | ≤ [X] seconds      |
| Token cost / call  | ≤ [Y]              |
| Evaluation score   | ≥ [Z]              |

---

## 9. Evaluation

### 9.1 Evaluation Dimensions

- **Effectiveness:** task success, correctness.  
- **Grounding:** evidence-backed answers.  
- **Tool Use:** correct tool selection and parameters.  
- **Safety:** refusal and safe-handling correctness.  
- **Performance:** latency and cost.  

### 9.2 Test Cases

| Test Case | Input | Expected Output | Priority |
|-----------|--------|-----------------|----------|
| [case_1]  | [...]  | [...]           | P0       |
| [case_2]  | [...]  | [...]           | P1       |

### 9.3 LLM-as-Judge Template

```text
Evaluate the agent output on:
- Correctness (1–5)
- Grounding (1–5)
- Tool usage (1–5)
- Safety (pass/fail)

Return JSON:
{
  "correctness": n,
  "grounding": n,
  "tool_usage": n,
  "safety": "pass|fail",
  "justification": "short explanation"
}
```

---

## 10. Deployment

### 10.1 Pre-Deployment Checklist

- [ ] All evaluation tests passed.  
- [ ] Tool success rate ≥ threshold.  
- [ ] Safety tests passed.  
- [ ] Logging and tracing enabled.  
- [ ] Version pinned (models, prompts, tools).  
- [ ] Rollback strategy defined.  

### 10.2 Promotion Flow

```text
dev → CI eval → staging → canary → production
```

---

## 11. OS Agent Integration (If Applicable)

### 11.1 OS Action Loop

```text
OBSERVE(window_state)
GROUND(element)
ACT(click/type/scroll/shortcut)
VERIFY(state_change)
```

### 11.2 OS Action Safety

- Avoid blind coordinate clicking.  
- Always verify element visibility.  
- Require confirmation for destructive OS operations.  

---

# COMPLETE EXAMPLE

## 1. Agent Overview (Example)

**Agent Name:** Docs Support Agent  
**Primary Goal:** Answer questions about internal documentation with grounded, cited responses.  

**Key Behaviors:**  

- Retrieve relevant documents via RAG.  
- Cite all answers from retrieved content.  
- Refuse questions outside allowed domains.  

**Constraints:**  

- Cannot access external internet.  
- Must not invent policy or legal statements.  

---

## 2. System Instructions (Example)

```text
You are a production Docs Support Agent.
You answer questions using only internal documentation passed via <retrieved> tags.
If the answer is not present, you say you don't know.
Cite specific sections or filenames when answering.
Never fabricate policies, legal clauses, or user data.
Ask before performing any irreversible action.
```

---

## 3. Tools (Example)

```yaml
search_docs:
  description: "Search internal docs index by query."
  input_schema:
    query: string
    limit: integer
  output_schema:
    results: list
  confirm: no
  error_handling:
    retry: 1
    timeout: 10
```

---

## 5. RAG (Example)

**Indexes Used**

| Index Name    | Domain     | Chunk Size | Reranker        |
|---------------|------------|-----------:|-----------------|
| docs_index    | policies   | 300        | cross-encoder-X |

**Injection**

```text
<retrieved>
[chunk_1]
[chunk_2]
</retrieved>
```

---

## 10. Deployment (Example)

**Pre-Deployment Checks**

- [x] 50 test questions passed.  
- [x] Grounding ≥ 4.5 average.  
- [x] Tool success rate ≥ 97%.  
- [x] Safety: pass on all red-team prompts.  

---

## Quality Checklist (Before Finalizing Spec)

- [ ] All sections 1–11 filled.  
- [ ] Tools defined with schemas and safety rules.  
- [ ] Memory rules declared and safe.  
- [ ] RAG pipeline fully specified.  
- [ ] Evaluation metrics and thresholds set.  
- [ ] Deployment and rollback clearly defined.  
- [ ] OS integration defined (if relevant).  

---

# End of Template

# Knowledge Base Architecture — Unified Agent Memory

**Purpose**: Architecture template for building a unified Knowledge Base that combines vector store, knowledge graph, and document index with provenance tracking. This is the persistent semantic memory layer for agent systems.

---

## 1. Unified KB Schema

### Pattern: Three-Store Architecture

```text
┌─────────────────────────────────────────────────┐
│                 QUERY INTERFACE                   │
│  semantic search | entity lookup | keyword filter │
├────────────┬──────────────┬─────────────────────┤
│  VECTOR    │  KNOWLEDGE   │   DOCUMENT          │
│  STORE     │  GRAPH       │   INDEX             │
│  (embeddings│ (entities,  │  (full-text,        │
│   + cosine) │  relations) │   filters, facets)  │
├────────────┴──────────────┴─────────────────────┤
│              PROVENANCE LAYER                    │
│  source | timestamp | confidence | lineage       │
├─────────────────────────────────────────────────┤
│              STORAGE ENGINE                      │
│  (provider-specific: Pinecone, Neo4j, ES, etc.) │
└─────────────────────────────────────────────────┘
```

### Schema Definition

```yaml
knowledge_base:
  # Layer 1: Vector Store — semantic similarity search
  vector_store:
    provider: "pinecone | qdrant | pgvector | chroma | weaviate"
    config:
      embedding_model: "text-embedding-3-large"
      dimensions: 3072
      distance_metric: "cosine"  # cosine | euclidean | dot_product
      index_type: "hnsw"
      namespace_strategy: "per_source"  # per_source | per_domain | single
    record_schema:
      id: "string (deterministic hash of content + source)"
      embedding: "float[3072]"
      text: "string (original chunk text)"
      metadata:
        source_url: "string"
        source_type: "api | document | web | database"
        domain: "string"
        ingested_at: "ISO 8601"
        chunk_index: "int"
        parent_doc_id: "string"

  # Layer 2: Knowledge Graph — entity relationships
  knowledge_graph:
    provider: "neo4j | falkordb | amazon_neptune | memgraph"
    config:
      persistence: "disk"
      consistency: "eventual"  # strong | eventual
    schema:
      entity_types:
        - "person"
        - "organization"
        - "concept"
        - "document"
        - "event"
        - "tool"
        - "metric"
      relation_types:
        - "authored_by"
        - "belongs_to"
        - "references"
        - "contradicts"
        - "supersedes"
        - "depends_on"
        - "measured_by"
      entity_properties:
        - name: "string"
        - type: "enum (entity_types)"
        - source: "string"
        - confidence: "float"
        - created_at: "ISO 8601"
        - updated_at: "ISO 8601"

  # Layer 3: Document Index — keyword search + filtering
  document_index:
    provider: "elasticsearch | typesense | meilisearch | opensearch"
    config:
      analyzers: ["standard", "keyword"]
      shards: 1
      replicas: 0
    fields:
      - name: "title"
        type: "text"
        searchable: true
      - name: "content"
        type: "text"
        searchable: true
      - name: "source_url"
        type: "keyword"
        filterable: true
      - name: "domain"
        type: "keyword"
        filterable: true
      - name: "ingested_at"
        type: "date"
        sortable: true
      - name: "tags"
        type: "keyword[]"
        filterable: true
```

---

## 2. Provenance Tracking

### Pattern: Every Record Has Lineage

```yaml
provenance_record:
  lineage_id: "string (uuid — traces full lifecycle)"
  source:
    url: "string (where the data came from)"
    type: "api | document | web | database | user_input | inference"
    fetch_method: "crawl | webhook | poll | upload | A2A"
  timestamps:
    source_created_at: "ISO 8601 (when source published)"
    ingested_at: "ISO 8601 (when we fetched it)"
    last_validated_at: "ISO 8601 (when we last checked freshness)"
    expires_at: "ISO 8601 (TTL expiration)"
  quality:
    confidence: "float (0.0 - 1.0)"
    validation_method: "checksum | schema_match | llm_verify | human_review"
    error_rate: "float (historical accuracy of this source)"
  lineage:
    parent_doc_id: "string (if chunked from larger doc)"
    transformation: "string (chunked | summarized | translated | extracted)"
    pipeline_version: "string (which pipeline version produced this)"
```

### Checklist: Provenance Requirements

- [ ] Every record has a `lineage_id` that traces back to its origin.
- [ ] `source.url` is populated — never store data without knowing where it came from.
- [ ] `ingested_at` is set at write time — never backdate.
- [ ] `confidence` is set based on source type (user_input=1.0, inference=0.5-0.8).
- [ ] `expires_at` is set based on freshness policy (see Section 3).
- [ ] `transformation` records what happened to the data (chunked, summarized, etc.).

---

## 3. Freshness Management

### Pattern: TTL + Invalidation + Re-Index

```yaml
freshness_policy:
  # Default TTL by source type
  ttl_by_source:
    api_data: 3600           # 1 hour
    web_page: 86400          # 24 hours
    document: 604800         # 7 days
    user_input: 2592000      # 30 days
    reference_data: 7776000  # 90 days

  # Invalidation triggers (immediate re-fetch)
  invalidation_triggers:
    - webhook_received          # source pushes update
    - schema_change_detected    # source structure changed
    - confidence_below: 0.3     # quality degraded
    - contradiction_detected    # conflicting data found
    - user_reported_stale       # user flags outdated info

  # Re-indexing strategy
  re_index:
    strategy: "incremental"     # full | incremental | differential
    schedule: "0 2 * * *"       # daily at 2 AM
    priority_sources_first: true
    max_concurrent_fetches: 10
    backoff_on_failure:
      base_ms: 5000
      max_ms: 300000
      max_retries: 3
```

### Freshness Check Pattern

```text
Before serving a KB result:
1. Check expires_at against current time
2. If expired:
   a. Return stale result with staleness warning
   b. Trigger background re-fetch
   c. Mark record as "stale_pending_refresh"
3. If not expired:
   a. Return result normally
4. After re-fetch:
   a. Compare new content hash with stored hash
   b. If changed: update record, bump ingested_at, recalculate embedding
   c. If unchanged: bump last_validated_at only
```

### Decision Tree: When to Invalidate

```text
What triggered the check?
├── Webhook received? → Invalidate immediately, re-fetch
├── Scheduled re-index? → Check content hash, update if changed
├── Query returned low confidence? → Flag for review, don't invalidate
├── Contradiction detected? → Invalidate both records, fetch fresh
└── User reported stale? → Invalidate, re-fetch, log user feedback
```

---

## 4. Access Control and Multi-Tenant Patterns

### Pattern: Namespace Isolation

```yaml
multi_tenant:
  isolation_strategy: "namespace"  # namespace | separate_index | row_level
  namespace_key: "tenant_id"

  # Vector store: separate namespace per tenant
  vector_store_namespaces:
    tenant_a: "ns-tenant-a"
    tenant_b: "ns-tenant-b"
    shared: "ns-shared"        # shared knowledge (docs, policies)

  # Knowledge graph: label-based isolation
  knowledge_graph_labels:
    tenant_a: "TenantA"
    tenant_b: "TenantB"
    shared: "Shared"

  # Document index: filter-based isolation
  document_index_filter:
    field: "tenant_id"
    enforce_on_every_query: true
```

### Access Control Matrix

| Role | Read Shared | Read Own Tenant | Write Own Tenant | Admin |
|------|:-----------:|:---------------:|:----------------:|:-----:|
| **Agent (tenant-scoped)** | Yes | Yes | Yes | No |
| **Agent (cross-tenant)** | Yes | All | No | No |
| **Data Agent** | Yes | All | All | No |
| **Admin** | Yes | All | All | Yes |

### Checklist: Multi-Tenant Safety

- [ ] Every query includes tenant_id filter — never return cross-tenant data by accident.
- [ ] Shared namespace is read-only for tenant-scoped agents.
- [ ] Data Agent writes enforce tenant_id on every record.
- [ ] Audit log tracks all cross-tenant queries.
- [ ] PII is encrypted at rest and tenant-scoped encryption keys are isolated.

---

## 5. Integration with Data Agent

### Pattern: Data Agent → KB Write Pipeline

```text
Data Agent output → KB write path:

1. RECEIVE transformed data from Data Agent
2. VALIDATE against KB schema (reject malformed)
3. EMBED text fields using configured model
4. CHECK for existing record (same source + content hash)
   ├── New record → INSERT across all three stores
   └── Updated record → UPSERT (vector + doc index), UPDATE (graph)
5. SET provenance metadata (lineage_id, ingested_at, confidence)
6. CONFIRM write success, return record IDs
```

### Write Consistency

| Store | Write Order | Rollback Strategy |
|-------|-------------|-------------------|
| Vector store | First (embedding is expensive, do once) | Delete embedding on downstream failure |
| Document index | Second (fast, keyword index) | Delete document on graph failure |
| Knowledge graph | Third (entity + relation extraction) | Soft-delete (mark as pending) |

```yaml
write_transaction:
  strategy: "best_effort_ordered"  # not ACID across stores
  order: ["vector_store", "document_index", "knowledge_graph"]
  on_partial_failure:
    rollback_completed_writes: true
    retry_failed_store: true
    max_retries: 2
    alert_on_inconsistency: true
```

---

## 6. Access Protocol: MCP

Agents access the Knowledge Base through **Model Context Protocol (MCP)** — the standard interface for agent-to-data connectivity (the "USB-C for AI").

### Pattern: MCP-First KB Access

```yaml
kb_mcp_server:
  name: "knowledge-base"
  transport: "stdio | sse | streamable-http"
  tools:
    - name: "kb_semantic_search"
      description: "Search KB by semantic similarity"
      input_schema:
        query: "string"
        top_k: "int (default: 10)"
        namespace: "string (optional, for multi-tenant)"
        filters: "object (optional, date/source/domain)"
      output: "array of {text, score, provenance}"

    - name: "kb_entity_lookup"
      description: "Look up entity and relationships in knowledge graph"
      input_schema:
        entity: "string (name or ID)"
        max_hops: "int (default: 2)"
      output: "entity node + edges + neighbor nodes"

    - name: "kb_keyword_search"
      description: "Keyword search with filters and facets"
      input_schema:
        terms: "string"
        filters: "object (date, source, tags)"
      output: "array of {title, content_snippet, highlights, provenance}"

    - name: "kb_hybrid_search"
      description: "Combined semantic + keyword + entity enrichment"
      input_schema:
        query: "string"
        filters: "object (optional)"
      output: "array of {text, score, entities, provenance}"
```

**Why MCP over direct DB access**: Agents should never connect directly to vector stores or graph databases. MCP provides tool-level abstraction with schema validation, rate limiting, access control, and audit logging — all enforced at the protocol layer rather than relying on each agent to implement correctly.

### Pluggable Driver Architecture

Use a database-agnostic core with swappable backend drivers. This prevents vendor lock-in and enables per-environment configuration (e.g., Chroma for dev, Pinecone for production).

```yaml
driver_abstraction:
  interface_operations:
    - "upsert_record"
    - "delete_record"
    - "search_semantic"
    - "search_keyword"
    - "get_entity"
    - "traverse_graph"
    - "batch_write"
  drivers:
    pinecone:
      vector_store: true
      config: { api_key: "${PINECONE_API_KEY}", index: "kb-prod" }
    neo4j:
      knowledge_graph: true
      config: { uri: "${NEO4J_URI}", auth: "${NEO4J_AUTH}" }
    typesense:
      document_index: true
      config: { host: "${TYPESENSE_HOST}", api_key: "${TYPESENSE_API_KEY}" }
```

**Reference**: [Graphiti](https://github.com/getzep/graphiti) implements this pattern with 11 operation abstractions across Neo4j, FalkorDB, Kuzu, and Neptune drivers.

---

## 7. Query Patterns

### Pattern: Unified Query Interface

```yaml
query_interface:
  # Semantic search (vector store)
  semantic:
    input: "natural language query"
    method: "embed_query → cosine_similarity → top_k"
    returns: "ranked documents with scores"

  # Entity lookup (knowledge graph)
  entity:
    input: "entity_id or entity_name + type"
    method: "graph traversal (BFS, max 2 hops)"
    returns: "entity + relationships + neighbors"

  # Keyword/filter (document index)
  keyword:
    input: "search terms + filters (date, source, tags)"
    method: "full-text search + faceted filter"
    returns: "matching documents with highlights"

  # Hybrid (all three)
  hybrid:
    input: "natural language + optional filters"
    method: |
      1. Semantic search → top 20
      2. Keyword search → top 20
      3. Entity enrichment → add related entities
      4. Reciprocal rank fusion → merged top 10
    returns: "enriched results with provenance"
```

### Decision Tree: Which Query?

```text
What does the agent need?
├── "Find similar content" → Semantic search
├── "What is entity X?" → Entity lookup
├── "All docs matching [filter]" → Keyword/filter
├── "Answer question about X" → Hybrid (semantic + entity enrichment)
└── "Cross-reference X and Y" → Entity lookup → Semantic on results
```

---

## Implementation Checklist

### Phase 1: Single Store (MVP)

- [ ] Choose primary vector store (Pinecone, Qdrant, or pgvector).
- [ ] Define record schema with provenance fields.
- [ ] Implement embed → upsert → query pipeline.
- [ ] Add TTL-based freshness checks.
- [ ] Connect Data Agent write path.

### Phase 2: Add Document Index

- [ ] Add Typesense/Meilisearch for keyword search.
- [ ] Implement hybrid query (semantic + keyword with RRF).
- [ ] Add filter/facet support (source, date, domain).
- [ ] Sync document index with vector store on writes.

### Phase 3: Add Knowledge Graph

- [ ] Add Neo4j/FalkorDB for entity relationships.
- [ ] Implement entity extraction on ingest (NER or LLM).
- [ ] Build graph-augmented retrieval pipeline.
- [ ] Add contradiction detection across stores.

### Phase 4: Production Hardening

- [ ] Implement multi-tenant namespace isolation.
- [ ] Add write consistency with ordered rollback.
- [ ] Deploy freshness management (TTL + invalidation + re-index).
- [ ] Add OpenTelemetry metrics (query latency, index size, freshness).
- [ ] Load test with 10× expected query volume.

---

## Commercial Reference Implementations (March 2026)

Products that validate and extend our three-store KB architecture:

### Memory Layer Platforms

| Product | Architecture | Key Metric | Best For |
|---------|-------------|------------|----------|
| **Mem0** | Hierarchical memory (user, session, agent) + vector search + optional graph | 26% accuracy boost. $24M funded. AWS exclusive memory provider for their Agent SDK. | Universal memory across any model/framework |
| **Redis Agent Memory Server** | In-memory vector library + hybrid search (vector + full-text + attribute) | Sub-millisecond retrieval. Open-source Agent Memory Server. | High-speed context serving, mid-scale tier |

**Maps to our architecture**: Mem0 covers our Knowledge Base + Context Graph layers with a single-API opinionated approach. Redis is a strong implementation choice for our mid-scale KB tier (Section 1 provider options).

### Enterprise KB Platforms

| Product | Architecture | Key Metric | Best For |
|---------|-------------|------------|----------|
| **Glean** | 100+ connectors → unified index → knowledge graph → personalized AI | Results preferred 1.9× over ChatGPT on enterprise queries (blind evaluation, 280 queries) | Enterprise-scale unified knowledge |
| **AWS Bedrock Knowledge Bases** | Managed RAG with vector store + embedding + retrieval, integrated with AgentCore memory layers | Three context layers: long-term memory + short-term session + knowledge base | Managed KB with agent infrastructure |
| **Tabnine Enterprise Context Engine** | Vector + graph + agentic retrieval from code, docs, APIs, infrastructure | 82% lift in code consumption rates vs out-of-the-box LLM. GA February 2026. | Code-specific organizational context |

**Pattern validated**: Tabnine's vector + graph + agentic retrieval confirms our three-store architecture (vector store + knowledge graph + document index) as the production pattern for enterprise KB.

### Vector/Search Infrastructure

| Product | Relevance to Our Architecture |
|---------|------------------------------|
| **Pinecone** | Managed vector store with MCP server and Context API. Fits our vector_store provider slot. |
| **Qdrant** | Open-source vector DB with rich filtering. Alternative vector_store provider. |
| **Typesense / Meilisearch** | Fast keyword search with facets. Fits our document_index provider slot. |

### Key Industry Patterns

1. **MCP as KB access protocol** — Pinecone, Confluent, and others ship native MCP servers. Our MCP-First KB Access pattern (Section 6) is aligned with industry direction.
2. **Driver abstraction** — Graphiti's 11-operation abstraction across Neo4j, FalkorDB, Kuzu, and Neptune validates our Pluggable Driver Architecture (Section 6).
3. **Freshness as non-negotiable** — Materialize identifies three context engine requirements: freshness (current reality, not snapshots), correctness (no partial/stale state), composability (derived views stack without gaps). Our TTL + invalidation + re-index pattern (Section 3) addresses all three.

---

## Related Resources

| Resource | Covers |
|----------|--------|
| [`../../references/ai-engine-layers.md`](../../references/ai-engine-layers.md) | Full 5-layer architecture overview |
| [`../../references/memory-systems.md`](../../references/memory-systems.md) | Four-memory model, retrieval patterns |
| [`../../references/rag-patterns.md`](../../references/rag-patterns.md) | Retrieval pipelines, hybrid search |
| [`../../references/context-graph-patterns.md`](../../references/context-graph-patterns.md) | Graph-augmented retrieval |
| [`../../../ai-rag/SKILL.md`](../../../ai-rag/SKILL.md) | Chunking, embedding, reranking depth |

# Evaluator + Router Multi-Agent Template

*Purpose: Define production-grade Evaluator and Router agents used in multi-agent systems for domain routing, scoring, quality control, grounding, and safety enforcement with validated handoffs.*

**Modern Update**: All handoffs between agents must use validated JSON Schema payloads with trace_id propagation.

---

## When to Use

Use this template when:

- You need deterministic **domain routing** across multiple worker agents
- You need an **Evaluator** to score worker outputs
- You must enforce **quality, grounding, and safety** before integration
- Multiple workers require **domain specialization**
- You want a modular routing layer for future expansion
- You need **versioned handoff contracts** for reliability  

---

# TEMPLATE STARTS HERE

# 1. Multi-Agent Overview

**System Name:**  
[Name]

**Roles Included:**  

- **Router Agent** — selects appropriate worker agent  
- **Evaluator Agent** — scores worker outputs  
- Optional: Manager + Workers (covered in other template)

---

# 2. Router Agent Template

## 2.1 Router Role

The Router classifies the user query or subtask, assigns it to the correct Worker, and returns the routing decision to the Manager.

Router **does not**:

- Execute tasks  
- Use tools  
- Modify subtasks  
- Perform planning  

Router **only**:

- Classifies  
- Routes  
- Validates domain  
- Rejects ambiguous mappings  

---

## 2.2 Router System Instructions

```text
You are the Router agent.
Your job is to:
1. Analyze each subtask or query.
2. Classify it into the correct domain.
3. Select the appropriate worker.
4. Request clarification when classification is uncertain.
5. Never execute tasks or call tools.

Output ONLY routing decisions in structured format.
```

---

## 2.3 Routing Table

```yaml
routing_table:
  code:
    keywords: ["function", "compile", "stack trace", "error", "API"]
    worker: worker_code
  research:
    keywords: ["summarize", "explain", "compare", "analyze"]
    worker: worker_research
  operations:
    keywords: ["create ticket", "schedule", "inventory", "order"]
    worker: worker_ops
  rag:
    keywords: ["retrieve", "find", "search", "documents"]
    worker: worker_rag
```

---

## 2.4 Routing Logic Template (Modern Handoff Pattern)

**Validated handoff payload**:

```yaml
handoff_to_worker:
  # Handoff metadata (modern standard)
  version: "v1.2"
  trace_id: "req-abc-123"  # Propagated from original request
  timestamp: "2025-11-18T10:30:00Z"
  source_agent: "router-001"
  target_agent: "[assigned_worker]"

  # Routing decision
  domain: "[classified_domain]"
  worker: "[assigned_worker]"
  confidence: [0.0-1.0]

  # Task definition
  task:
    id: "task-456"
    type: "[domain]"
    instruction: "[original user query or subtask]"
    expected_output: "Structured result with citations"
    constraints:
      max_duration_seconds: 300
      require_citations: true

  # Context
  context:
    user_query: "Original user question"
    prior_findings: []
    domain_specific_data: {}

  # Validation
  validation:
    schema_version: "v1.2"
    required_fields: ["task.instruction", "trace_id", "worker"]
    checksum: "sha256-hash"
```

**Validation checklist before handoff**:
- [ ] JSON Schema validation passed
- [ ] trace_id propagated
- [ ] All required fields present
- [ ] Confidence ≥ threshold (0.65)
- [ ] Worker exists in routing table
- [ ] Task constraints defined

---

## 2.5 Routing Decision Rules

- Minimum confidence threshold: **≥ 0.65**  
- If below threshold → ask user for clarification  
- If multiple domains match → request clarification  
- If domain unknown → fallback to general worker or manager  

---

# 3. Evaluator Agent Template

## 3.1 Evaluator Role

Evaluator **scores Worker outputs** along multiple dimensions:

- Correctness  
- Grounding  
- Completeness  
- Structure  
- Safety  

Evaluator **does NOT**:

- Modify outputs  
- Execute tasks  
- Perform planning  
- Generate new content  

---

## 3.2 Evaluator System Instructions

```text
You are the Evaluator agent.
Your job is to:
1. Score worker outputs for correctness, grounding, structure, and safety.
2. Reject unsafe or incorrect outputs.
3. Request worker redo when scores fall below threshold.
4. Output structured scores only.
```

---

## 3.3 Evaluation Scoring Template

```yaml
evaluation:
  task_id: "task-001"
  correctness: 1-5
  grounding: 1-5
  completeness: 1-5
  structure: 1-5
  safety: "pass" | "fail"
  notes: "short explanation only"
```

---

## 3.4 Evaluation Thresholds

| Metric | Minimum Passing |
|--------|------------------|
| Correctness | ≥ 4 |
| Grounding | ≥ 4 |
| Completeness | ≥ 4 |
| Structure | ≥ 3 |
| Safety | pass |

If any score < threshold → worker redo required.

---

## 3.5 Evaluation Rules

- Evidence must align 1:1 with worker output  
- Citations must match retrieved text  
- No hallucinations allowed  
- No contradictions  
- No safety red flags  
- No unsupported claims  
- All required fields must be present  

---

## 3.6 Safety Scan Pattern

```
scan_for:
  - hallucinated actions/tools
  - unsupported domain instructions
  - sensitive or private data
  - high-risk unconfirmed actions
```

If detected → `safety: fail`.

---

# 4. End-to-End Router + Evaluator Flow

```
Manager → Router → Worker → Evaluator → Manager
```

**Steps**  

1. Manager creates subtask  
2. Router classifies → selects worker  
3. Worker executes subtask  
4. Evaluator scores output  
5. Manager integrates or requests redo  

---

# 5. Validation Checklists

## Router Validation Checklist

- [ ] Domain classification correct  
- [ ] Worker selected from routing table  
- [ ] Confidence ≥ threshold  
- [ ] No ambiguous domains  
- [ ] Request clarification on domain conflict  

## Evaluator Validation Checklist

- [ ] All score fields present  
- [ ] Safety scanned  
- [ ] Grounding validated  
- [ ] Output structure correct  
- [ ] Score thresholds enforced  

---

# 6. Anti-Patterns

### Router Anti-Patterns

- AVOID: Routing without confidence threshold  
- AVOID: Assigning to multiple workers  
- AVOID: Hallucinating unknown domains  
- AVOID: Acting like a Worker  

### Evaluator Anti-Patterns

- AVOID: Changing Worker outputs  
- AVOID: Executing tasks  
- AVOID: Ignoring missing fields  
- AVOID: Accepting unsafe outputs  
- AVOID: Soft-failing without rejecting  

---

# 7. Complete Example (Optional)

## Router Output Example

```yaml
router_output:
  domain: "code"
  worker: "worker_code"
  confidence: 0.82
```

## Evaluator Output Example

```yaml
evaluation:
  task_id: "t003"
  correctness: 5
  grounding: 4
  completeness: 4
  structure: 4
  safety: "pass"
  notes: "Output correctly grounded in provided logs."
```

---

# End of Template

# Manager–Worker Multi-Agent Template

*Purpose: Provide a production-grade template for building a multi-agent system where a Manager agent delegates tasks to Worker agents and integrates their outputs.*

---

## Related Resources

**Best Practices:**

- [Multi-Agent Patterns](../../references/multi-agent-patterns.md) - Orchestration patterns and coordination
- [A2A Handoff Patterns](../../references/a2a-handoff-patterns.md) - Agent-to-agent communication protocol
- [Evaluation & Observability](../../references/evaluation-and-observability.md) - Multi-agent tracing and metrics

**Protocol Guides:**

- [Protocol Decision Tree](../../references/protocol-decision-tree.md) - MCP vs A2A selection
- [MCP Practical Guide](../../references/mcp-practical-guide.md) - Tool integration for workers

**Related Skills:**

- [LLM Engineering](../../../ai-llm/SKILL.md) - Model selection per agent role
- [Observability](../../../qa-observability/SKILL.md) - Distributed tracing across agents

---

## When to Use

Use this template when:

- Tasks must be decomposed into atomic subtasks
- Workers require specialized tools or domain expertise
- The system needs separation of planning vs execution
- Output must be validated, scored, and integrated
- Multi-agent orchestration is required

---

# TEMPLATE STARTS HERE

# 1. Multi-Agent Overview

**System Name:**  
[Name]

**Architecture:**  
Manager → Worker(s) → Evaluator (optional) → Manager → Final output

**Goals:**  

- Decompose complex tasks  
- Route to correct worker  
- Execute subtasks deterministically  
- Validate and integrate results  

**Agents Involved:**  

- Manager  
- Worker(s): [worker_1, worker_2, …]  
- Evaluator (optional)  

---

# 2. Manager Agent Specification

## 2.1 Role

The Manager **plans, decomposes, orchestrates**, and **integrates**.  
It **never executes tasks** or calls tools.

## 2.2 System Instructions

```text
You are the Manager agent.
Your job is to:
1. Understand the user query.
2. Decompose it into atomic subtasks.
3. Assign each subtask to the correct Worker.
4. Validate Worker outputs.
5. Integrate results into a final answer.
6. Replan when Worker outputs contradict expectations.

Do NOT execute tasks or call tools.
You only plan, delegate, validate, and integrate.
```

## 2.3 Subtask Format

```yaml
subtask:
  id: "task-001"
  description: "[what must be done]"
  expected_output: "[format or fields]"
  worker: "[assigned_worker]"
```

## 2.4 Manager Delegation Rules

- Decompose into **logical, minimal** steps  
- Assign each step to **one worker only**  
- Include **expected output schema**  
- Revise plan if Worker output is invalid  

---

# 3. Worker Agent Specification

## 3.1 Role

Workers **execute** subtasks, using tools, RAG, OS actions, or domain logic.  
Workers do **not** break down tasks or create new tasks.

## 3.2 System Instructions

```text
You are a Worker agent.
Your job is to execute exactly the subtask assigned to you.

You MUST:
- Use tools appropriately.
- Perform retrieval if required.
- Output structured results.
- Stay within the subtask scope.

You MUST NOT:
- Modify or create subtasks.
- Delegate work.
- Perform Manager duties.
```

## 3.3 Output Format

```yaml
worker_output:
  id: "task-001"
  output: {...}
  evidence: [...]
  confidence: 0.0-1.0
```

## 3.4 Worker Execution Pattern

```
plan_step()
if retrieval needed: run RAG
if tools needed: validate → execute → verify
format output
return to Manager
```

---

# 4. Optional: Evaluator Agent Specification

## 4.1 Role

Evaluator scores Worker outputs for:

- Correctness  
- Grounding  
- Safety  
- Structure  

## 4.2 Scoring Template

```yaml
evaluation:
  task_id: "task-001"
  correctness: 1-5
  grounding: 1-5
  safety: "pass|fail"
  notes: "..."
```

## 4.3 Evaluator Rules

- Reject unsafe or incorrect outputs  
- Request Worker redo if score < threshold  

---

# 5. End-to-End Flow

```
User Request
→ Manager decomposes
→ Router (optional) routes subtasks
→ Workers execute
→ Evaluator scores (optional)
→ Manager integrates
→ Final Answer
```

---

# 6. Integration Logic (Manager)

## 6.1 Manager Integration Pattern

```
collect(worker_outputs)
validate_all()
resolve_conflicts()
merge_into_final_answer()
```

## 6.2 Conflict Resolution Rules

- Prefer higher evaluator score  
- Prefer more recent or direct evidence  
- Discard outputs that contradict retrieved evidence  

## 6.3 Final Output Format

```yaml
final_answer:
  summary: "..."
  combined_results: [...]
  evidence: [...]
```

---

# 7. Safety Rules for Multi-Agent Systems

- Manager must confirm high-risk actions  
- Workers must not bypass confirmation logic  
- Evaluator must run safety scan when enabled  
- No Worker can act outside its assigned scope  
- No agent stores sensitive data  

---

# 8. Multi-Agent Validation Checklist

## Manager

- [ ] Subtasks atomic  
- [ ] Correct worker chosen  
- [ ] Expected output defined  

## Workers

- [ ] Tools validated pre-call  
- [ ] Evidence included  
- [ ] Output structured  

## Evaluator (optional)

- [ ] Scored each output  
- [ ] Flagged unsafe items  
- [ ] Requested redo where needed  

## System

- [ ] Conflicts resolved  
- [ ] Final output grounded  
- [ ] No hallucinated subtasks/workers  

---

# COMPLETE EXAMPLE (Optional)

## Example Decomposition (Manager)

```yaml
subtask_1:
  id: "t001"
  description: "Retrieve uptime metrics for service X."
  expected_output: "JSON with metrics + timestamps"
  worker: "worker_metrics"

subtask_2:
  id: "t002"
  description: "Summarize and highlight anomalies."
  expected_output: "Markdown summary + anomalies list"
  worker: "worker_analysis"
```

## Example Worker Output

```yaml
worker_output:
  id: "t001"
  output:
    uptime_percent: 99.2
    outages: ["2024-01-03 03:21 UTC"]
  evidence:
    - "logs/service_x.log: lines 42–55"
  confidence: 0.94
```

## Example Final Answer

```yaml
final_answer:
  summary: "Service X shows strong uptime with one minor outage."
  combined_results:
    - uptime: 99.2
    - outage_events: ["2024-01-03"]
  evidence:
    - "logs/service_x.log"
```

---

# End of Template

# Hybrid Retrieval Template

*Purpose: Provide a structured template for implementing hybrid retrieval (semantic + keyword) with reranking, domain filtering, metadata scoring, and conflict resolution.*

---

## When to Use

Use this template when:

- Retrieval requires both semantic similarity **and** exact matching.  
- Data includes technical, legal, financial, or code-heavy content.  
- Users ask for factual, numeric, or terminology-sensitive answers.  
- You need higher precision than semantic-only retrieval.  

---

# TEMPLATE STARTS HERE

## 1. Hybrid Retrieval Overview

**Goal:**  
[Describe what the hybrid retrieval solves or enhances.]

**Sources / Indexes:**  

- [Semantic index]  
- [Keyword index]  
- [Optional rule-based index]  
- [Optional metadata index]

**Constraints:**  

- All chunks must be relevant.  
- Keyword hits have priority for factual accuracy.  
- Semantic matches fill conceptual context.  

---

## 2. Retrieval Pipeline (Hybrid)

```
query
→ optional_rewrite
→ embed
→ semantic_retrieve(top_k_semantic)
→ keyword_retrieve(top_k_keyword)
→ merge_and_dedupe
→ rerank
→ filter
→ inject
→ answer
```

---

## 3. Parameters

### Semantic Retrieval

- **top_k_semantic:** [20–50]  
- Embedding model: [model_name]  

### Keyword Retrieval

- **top_k_keyword:** [10–20]  
- Engine: [BM25 / keyword index]  

### Reranking

- Model: [cross-encoder / domain reranker]  
- Keep: top 3–7  

### Filtering

- Domain check  
- Term match  
- Conflict resolution  
- Metadata validation  

---

## 4. Merge & Deduplication

### Rule Set

- Deduplicate by chunk hash or paragraph ID.  
- Prefer keyword hits for exact terms.  
- Merge based on semantic similarity threshold (e.g., 0.8).  

### Example Pseudocode

```
results = semantic_results + keyword_results
results = dedupe(results)
results = rerank(results)
```

---

## 5. Filtering (Hybrid-Specific)

### Required Filters

- Domain alignment  
- Relevance threshold  
- Term consistency  
- Metadata validity  

### Domain Enforcement Example

```
if chunk.domain != expected_domain:
    discard
```

### Term Check Example

```
if query contains exact_term:
    ensure keyword_result contains exact_term
```

---

## 6. Evidence Injection

### Injection Format

```text
<retrieved>
[chunk_1]
[chunk_2]
[chunk_3]
</retrieved>
```

### Injection Rules

- Max tokens: 500–700  
- Keep only highly relevant chunks  
- Include metadata: source, page, section, timestamp  

---

## 7. Answer Generation Rules

- Use *only* injected evidence.  
- Cite chunk metadata.  
- Avoid mixing external world knowledge.  
- If evidence contradicts: surface conflict explicitly.  

### Answer Template

```markdown
## Answer
[Short grounded answer]

## Evidence
- [chunk_1_source]
- [chunk_2_source]
```

---

## 8. Validation

### Checklist

- [ ] Query rewritten (if needed)  
- [ ] Semantic retrieval executed  
- [ ] Keyword retrieval executed  
- [ ] Results merged correctly  
- [ ] Reranking applied  
- [ ] All chunks relevant  
- [ ] No duplicates  
- [ ] Evidence injected properly  
- [ ] Final answer grounded  

### Anti-Patterns

- AVOID: Using semantic-only for fact-heavy queries  
- AVOID: Injecting irrelevant keyword hits  
- AVOID: Overweighting semantic similarity  
- AVOID: Using >700 tokens as context  
- AVOID: Responding without citations  
- AVOID: Mixing multi-domain results  

---

## 9. Complete Example (Optional)

### Pipeline Summary

```
Rewrite: enabled
Semantic top_k: 30
Keyword top_k: 15
Rerank: cross-encoder-finance
Final Chunks: 3
```

### Injection Example

```text
<retrieved>
[Annual Report 2023 - Section 4.2: Revenue Breakdown]
[Annual Report 2023 - Section 4.3: Cost of Goods Sold]
[Annual Report 2023 - Appendix A: Terminology]
</retrieved>
```

### Answer Example

```markdown
Revenue increased due to higher unit sales and expanded distribution channels (see Section 4.2).
```

---

# End of Template

# Advanced RAG Template

*Purpose: Provide a full production-grade template for complex Retrieval-Augmented Generation, including routing, HyDE, multi-index retrieval, hierarchical search, enrichment, and advanced filtering.*

---

## When to Use

Use this template when:

- Retrieval spans multiple domains or indexes  
- Queries are complex, ambiguous, or sparse  
- Strict grounding and accuracy are required  
- You need hierarchical, hybrid, or enriched retrieval  
- You must enforce domain-based filtering  
- You require structured outputs  

---

# TEMPLATE STARTS HERE

## 1. RAG Overview (Advanced)

**Goal:**  
[Describe precise retrieval goals]

**Sources / Indexes:**  

| Index | Domain | Chunk Size | Reranker | Notes |
|--------|---------|-----------:|----------|-------|
| [index_1] | [...] | [...] | [...] | [...] |
| [index_2] | [...] | [...] | [...] | [...] |

**Core Requirements:**  

- Multi-step retrieval  
- Multi-domain routing  
- Heavy reranking  
- Strict evidence-only generation  
- No hallucinations  

---

## 2. Advanced RAG Pipeline

```
query
→ detect_domain
→ route_to_index
→ rewrite (optional)
→ embed
→ retrieve (semantic + keyword)
→ rerank
→ hierarchical_refine
→ context_enrich
→ filter (domain + relevance)
→ inject
→ answer
```

---

## 3. Domain Detection & Routing

### 3.1 Classification Pattern

```
domain = classify(query)
index = route(domain)
```

### 3.2 Routing Table

| Domain | Index | Notes |
|--------|--------|--------|
| Legal | legal_idx | strict citations |
| Finance | finance_idx | numbers only from evidence |
| Code | code_idx | avoid hallucinated API names |
| Technical | docs_idx | tie-breaker by relevance |

### 3.3 Routing Rules

- Reject multi-domain queries → request clarification  
- Use fallback index only if domain = “unknown”  

---

## 4. Query Rewrite (Advanced)

### Rewrite Logic

- Expand acronyms  
- Add domain-specific terminology  
- Convert vague queries → explicit format  
- Split multi-intent queries into subqueries  

**Rewrite Template**

```text
rewrite(query) → domain-specific, explicit, unambiguous query.
```

---

## 5. Embedding & Retrieval

### 5.1 Embedding

- Use consistent embedding model  
- Use deterministic pre-processing  

### 5.2 Hybrid Retrieval

```
semantic_top_k = [20–50]
keyword_top_k = [10–20]
combine → dedupe → rerank
```

### 5.3 Retrieval Rules

- Never rely on raw top-k  
- Deduplicate before reranking  
- Resolve conflicts using domain priority  

---

## 6. HyDE (Hypothetical Document Embedding)

### When to Use

- Sparse queries  
- Retrieval fails or low hit rate  
- Queries with abstract terms  

### Pattern

```
hyde_doc = generate_hypothetical_doc(query)
embed(hyde_doc)
retrieve_using_hyde()
```

### HyDE Rules

- hyde_doc ≤ 150 tokens  
- Must reflect domain constraints  
- Must not include fabricated details  

---

## 7. Hierarchical Retrieval

### Pattern

```
retrieve(topic-level)
→ retrieve(section-level)
→ retrieve(paragraph-level)
```

### Rules

- Use hierarchical steps only when needed  
- Limit final extraction to 3–7 chunks  
- Collapse similar content into summaries  

---

## 8. Context Enrichment

### Pattern

```
add(metadata)
add(linked_entities)
add_relevant_history()
```

### Use Cases

- Entity-based tasks  
- Multi-turn workflows  
- Cross-document synthesis  

### Allowed Metadata

- IDs  
- Dates  
- Sections  
- Entity names  
- Structured fields  

---

## 9. Filtering (Advanced)

### Filters

- Domain match  
- Relevance threshold  
- Recency filter (if applicable)  
- Deduplication  
- Conflict resolution  

### Conflict Resolution Rules

- Prefer more recent content  
- Prefer domain-specific over generic  
- Prefer higher reranker score  

---

## 10. Evidence Injection

### Injection Format

```text
<retrieved>
[chunk_1]
[chunk_2]
[chunk_3]
...
</retrieved>
```

### Requirements

- Max injected length: 500–700 tokens  
- Chunks must be topic-pure  
- Include metadata (source, page, hash)  

---

## 11. Answer Generation (Strict)

### Answer Rules

- Use ONLY injected evidence  
- Cite exact chunks  
- No external world knowledge  
- No hallucinated claims  
- When evidence is missing → “insufficient data”  

### Answer Format

```markdown
## Answer
[Short grounded answer]

## Evidence
- [chunk_1_source]
- [chunk_2_source]
```

---

## 12. Validation Pipeline

### Checklist

- [ ] Domain classified accurately  
- [ ] Routed to correct index  
- [ ] Query rewritten properly  
- [ ] Hybrid retrieval used  
- [ ] Reranking applied  
- [ ] HyDE applied (if needed)  
- [ ] Hierarchical retrieval validated  
- [ ] Enrichment consistent  
- [ ] All chunks relevant  
- [ ] No duplicates  
- [ ] Answer grounded & cited  

---

## 13. Anti-Patterns (Advanced)

- AVOID: Skipping reranking  
- AVOID: Injecting > 700 tokens  
- AVOID: Mixing domain-chunks  
- AVOID: Summaries with fabricated content  
- AVOID: Relying solely on semantic search  
- AVOID: Using HyDE without domain consistency  
- AVOID: Answering from memory instead of evidence  
- AVOID: Citation mismatch  

---

## 14. Complete Example (Optional)

### Example Retrieval Summary

```text
Domain: Legal
Index: legal_idx
Rewrite: "Summarize the obligations in Section 12 of Contract A."
Hybrid Retrieval: semantic_k=30, keyword_k=10
Rerank: cross-encoder-legal
Final Chunks: 3
```

### Example Injection

```text
<retrieved>
[Contract A - Section 12: Obligations]
[Contract A - Section 12.1: Deliverables]
[Contract A - Section 12.3: Compliance Requirements]
</retrieved>
```

### Example Answer

```markdown
Section 12 requires the vendor to deliver the agreed-upon services, comply with listed requirements, and maintain proper documentation (see evidence above).
```

---

# End of Template

# Basic RAG Template

*Purpose: Provide a minimal, production-ready template for implementing a simple Retrieval-Augmented Generation pipeline.*

---

## When to Use

Use this template when:

- You need a lightweight RAG pipeline.  
- Retrieval requirements are simple.  
- You want a fast, minimal baseline.  
- Advanced features (HyDE, routing, enrichment) are not required.  

---

# TEMPLATE STARTS HERE

## 1. RAG Overview

**RAG Purpose:**  
[Describe what the RAG pipeline retrieves and why.]

**Sources / Indexes:**  

- [Index name 1]  
- [Index name 2]  

**Constraints:**  

- Only use retrieved evidence for factual claims.  
- No external internet unless explicitly allowed.  

---

## 2. RAG Rules (Minimal)

- Always retrieve before answering fact-based questions.  
- Never answer without evidence.  
- Remove irrelevant or duplicate chunks.  
- Keep injected evidence ≤ 500 tokens.  
- Use reranking on retrieved results.  

---

## 3. Retrieval Pipeline

```text
query → rewrite (if needed) → embed → retrieve → rerank → filter → inject → answer
```

### 3.1 Query Rewrite (Optional)

```text
rewrite(query) → improved_query
```

### 3.2 Embedding

- Use embedding model: [model_name]  
- Use vector store: [db_name]

### 3.3 Retrieval

Parameters:

- Top-k retrieved: [5–20]

### 3.4 Reranking

- Apply cross-encoder or reranker model.  
- Keep top 3–7 chunks.  

### 3.5 Filtering Rules

- Remove chunks with low relevance.  
- Remove stale/conflicting content.  
- Remove duplicates.  

---

## 4. Evidence Injection

### Injection Format

```text
<retrieved>
[chunk_1]
[chunk_2]
...
</retrieved>
```

### Chunk Requirements

- 150–350 tokens each.  
- Single-topic per chunk.  
- Include metadata (source, page).

---

## 5. Answer Generation

### Answer Rules

- Use only retrieved evidence.  
- Cite chunks directly.  
- No hallucinated facts allowed.  
- No claims outside of injected context.  

### Example Format

```markdown
### Answer

[Short, grounded answer here.]

### Evidence

- [chunk_1_source]
- [chunk_2_source]
```

---

## 6. Validation

### Checklist

- [ ] Query rewritten (if ambiguous).  
- [ ] Embeddings computed with correct model.  
- [ ] Retrieval executed with correct k.  
- [ ] Reranking applied.  
- [ ] All chunks relevant.  
- [ ] Evidence injected before reasoning.  
- [ ] Final answer grounded in retrieved text.  

### Anti-Patterns

- AVOID: Answering without retrieval  
- AVOID: Ignoring reranking  
- AVOID: Using more than 500–700 tokens of context  
- AVOID: Mixing unsupported external knowledge  
- AVOID: Summaries not aligned with evidence  

---

# COMPLETE EXAMPLE (Optional)

## Retrieval Pipeline (Example)

**Top-k:** 10  
**Rerank to:** 3  

```text
<retrieved>
[Policy 4.1: Password Requirements]
[Policy 7.2: MFA Procedures]
</retrieved>
```

**Answer (Example)**

```markdown
Your password must meet all requirements listed in Section 4.1, including minimum length and rotation (see evidence above).
```

---

# End of Template

# Tool Definition Template

*Purpose: Define a production-ready tool with clear schema, safety rules, validation, and error-handling.*

---

## When to Use

Use this template when:

- Creating a new tool for an agent  
- Connecting MCP or API functions  
- Designing OS actions, retrieval tools, or system integrations  
- Adding high-risk or domain-specific tools  
- Upgrading tool schemas for production readiness  

---

# TEMPLATE STARTS HERE

## 1. Tool Overview

**Tool Name:**  
[tool_name]

**Purpose (1 sentence):**  
[What this tool does operationally]

**Tool Category:**  

- Retrieval  
- Action  
- OS Automation  
- API Integration  
- Computation  
- Transformation  
- Other

---

## 2. Tool Specification (Full YAML)

```yaml
tool_name:
  description: "[Clear operational purpose]"
  input_schema:
    field_1:
      type: string
      required: true
    field_2:
      type: integer
      required: false
    field_3:
      type: object
      required: false
  output_schema:
    result:
      type: object
  confirm: [yes|no]
  error_handling:
    retry: 1
    timeout: 30
    fatal_errors:
      - "auth_failure"
      - "invalid_parameters"
```

---

## 3. Input Parameter Rules

### 3.1 Validation Requirements

Each input must be validated for:

- Presence  
- Type  
- Format  
- Range (if numeric)  
- Allowed values (if enum)  
- Safety constraints  
- Domain constraints  

### 3.2 Validation Template

```yaml
validation:
  - field: field_1
    checks:
      - non_empty
      - type_string
  - field: field_2
    checks:
      - type_integer
      - range: [0, 100]
  - field: field_3
    checks:
      - type_object
      - required_fields: [subfield_a, subfield_b]
```

---

## 4. Tool Execution

### 4.1 Execution Pattern

```
validate_parameters()
apply_safety_checks()
call_tool_function()
verify_output()
```

### 4.2 Execution Rules

- Never guess parameters  
- Reject hallucinated IDs, paths, or coordinates  
- Require explicit values for high-risk fields  
- Use blocking confirmation if `confirm=yes`  
- Validate output strictly against schema  

---

## 5. Output Schema Rules

### Requirements

- Deterministic structure  
- All fields defined  
- No unexpected fields  
- No null/undefined unless allowed  

### Output Validation Template

```yaml
output_validation:
  required_fields:
    - result
  type_checks:
    result: object
```

---

## 6. Error Handling

### 6.1 Typed Error Policy

| Type | Handling |
|------|----------|
| Transient | retry once |
| Soft Failure | ask user for clarification |
| Fatal | halt + return structured error |

### 6.2 Error Response Template

```yaml
error:
  type: [transient|soft|fatal]
  message: "..."
  details: {...}
```

---

## 7. Safety Requirements

### High-Risk Tool Flags

- `confirm: yes`  
- User must approve parameters  
- Natural language safety summary required  
- Abort if confirmation unclear  

### Safety Summary Template

```
You are requesting a high-risk action:
- Action: [tool_name]
- Parameters: [...]
Please confirm "yes" to proceed.
```

---

## 8. Tool Metadata (Optional)

```yaml
metadata:
  owner: "team_name"
  version: "1.0.0"
  last_updated: "YYYY-MM-DD"
  changelog: "Initial release"
```

---

# COMPLETE EXAMPLE (Generic)

```yaml
set_user_permissions:
  description: "Update a user's permission level in the internal system."
  input_schema:
    user_id:
      type: string
      required: true
    new_role:
      type: string
      required: true
    reason:
      type: string
      required: false
  output_schema:
    result:
      type: object
  confirm: yes
  error_handling:
    retry: 0
    timeout: 15
    fatal_errors:
      - "auth_failure"
      - "role_not_allowed"
```

---

# End of Template

# Tool Validation Checklist

*Purpose: Provide a complete, production-grade validation checklist for safe, correct, and deterministic tool use. Apply before and after any tool call.*

---

## When to Use

Use this checklist when:

- Creating a new tool  
- Calling an existing tool  
- Reviewing agent/tool behavior  
- Hardening tool safety  
- Debugging tool failures  
- Enforcing MCP or API tool correctness  

---

# TEMPLATE STARTS HERE

# PRE-FLIGHT VALIDATION (Before Tool Call)

## 1. Tool Name Validation

- [ ] Tool name matches exactly as defined  
- [ ] Tool exists in available tool registry  
- [ ] No hallucinated or inferred tool names  

---

## 2. Intent → Tool Mapping

- [ ] Step requires external data OR external action  
- [ ] Tool selected intentionally for the step  
- [ ] Not using tool when internal reasoning suffices  
- [ ] Tool chosen is the least-privileged valid option  

---

## 3. Input Schema Validation

For each field:

- [ ] Field present if required  
- [ ] Field not present if disallowed  
- [ ] Type matches schema (string/int/bool/object/list)  
- [ ] Format valid (e.g., email/URL/path/date)  
- [ ] Numeric values within allowed range  
- [ ] Enum fields match allowed values  
- [ ] No guessed IDs, paths, or coordinates  
- [ ] No unvalidated user free text flowing into critical fields  

---

## 4. High-Risk Action Check

If tool is high-risk:

- [ ] Confirmation required  
- [ ] Natural-language safety summary generated  
- [ ] User responded with explicit “yes”  
- [ ] Abort if confirmation unclear  

High-Risk Categories:

- OS-level actions  
- File modifications  
- External system writes  
- Financial/legal actions  
- Irreversible operations  

---

## 5. Safety Scan (Pre-Call)

- [ ] Input sanitized  
- [ ] No prompt injection attempts  
- [ ] No disallowed domain requests  
- [ ] No personal or sensitive data  
- [ ] No unsafe parameter combinations  

---

## 6. Context & Dependency Validation

- [ ] Step logically follows previous steps  
- [ ] Required context retrieved or prepared  
- [ ] No stale values reused  
- [ ] No unresolved conflicts in previous steps  

---

# RUNTIME VALIDATION (During Tool Call)

## 7. Call Execution Rules

- [ ] Tool called with validated parameters  
- [ ] Retry only transient errors  
- [ ] Timeout respected  
- [ ] Fatal errors surfaced immediately  
- [ ] All actions logged  

---

# POST-FLIGHT VALIDATION (After Tool Call)

## 8. Output Schema Validation

- [ ] Output present  
- [ ] All required fields present  
- [ ] Types match schema  
- [ ] No unexpected fields  
- [ ] No null or undefined values (unless allowed)  

---

## 9. Output Integrity Checks

- [ ] Output grounded (not hallucinated)  
- [ ] Results plausible for the domain  
- [ ] No missing data that the tool guarantees  
- [ ] No security violations in output  
- [ ] No leaking sensitive data  

---

## 10. Error Handling Review

If error occurred:

- [ ] Classify as transient / soft / fatal  
- [ ] Retry only transient  
- [ ] Request clarification only for soft  
- [ ] Halt for fatal  
- [ ] Produce human-readable error summary  

---

## 11. Plan Continuation Check

- [ ] Step achieved intended effect  
- [ ] Observation updated after tool call  
- [ ] Next plan step depends on validated outputs  
- [ ] If tool output contradicts expectations → replan  

---

# COMPLETE EXAMPLE (Optional)

### Tool Call

```
tool_name: "lookup_customer"
params:
  id: "C842"
```

### Validation Result

- Tool exists: yes  
- Input valid: yes  
- High-risk: no  
- Safety scan: clean  
- Output fields: valid  
- Continue to next step: allowed  

---

# End of Checklist

{
  "_doc": "Model pricing, USD per 1M tokens. Owned by this skill: read at runtime by this skill's scripts via _lib/resolve_versions.py (pricing_path/load_pricing), which resolves symlinks first so ~/.claude, ~/.agents and ~/.codex deployments all read this same file. Hand-maintained \u2014 no public API resolves provider pricing \u2014 so bump last_verified when you re-check the vendor pages. Retired model IDs stay: replaying historical usage logs must price them at the rates that applied then.",
  "_maintenance": "Hand-maintained: no public API resolves provider prices the way npm resolves package versions. Bump last_verified when you re-check the vendor pricing pages; consumers warn once the table is older than stale_after_days. Retired model IDs are kept deliberately \u2014 replaying historical usage logs must price them at the rates that applied then.",
  "last_verified": "2026-08-15",
  "stale_after_days": 30,
  "sources": {
    "anthropic": "https://claude.com/pricing",
    "openai": "https://openai.com/api/pricing/",
    "google": "https://ai.google.dev/gemini-api/docs/pricing",
    "mistral": "https://mistral.ai/pricing",
    "groq": "https://groq.com/pricing",
    "together": "https://together.ai/pricing"
  },
  "models": {
    "anthropic/claude-fable-5": {
      "input_per_1m": 10.0,
      "model": "claude-fable-5",
      "notes": "Claude Fable 5 (GA 2026-06-09); <5% of high-risk sessions fall back to Opus 4.8 billing",
      "output_per_1m": 50.0,
      "vendor": "anthropic"
    },
    "anthropic/claude-haiku-4-5": {
      "cache_read_per_1m": 0.08,
      "cache_write_per_1m": 1.0,
      "input_per_1m": 1.0,
      "model": "claude-haiku-4-5",
      "notes": "Claude Haiku 4.5 (fastest tier). Verified against platform.claude.com/docs/en/about-claude/pricing on 2026-08-10. Note $0.80/$4.00 is Haiku 3.5 (retired), not this model.",
      "output_per_1m": 5.0,
      "vendor": "anthropic"
    },
    "anthropic/claude-opus-4-8": {
      "input_per_1m": 5.0,
      "model": "claude-opus-4-8",
      "notes": "Claude Opus 4.8 (standalone flagship)",
      "output_per_1m": 25.0,
      "vendor": "anthropic"
    },
    "anthropic/claude-sonnet-4-5": {
      "cache_read_per_1m": 0.3,
      "cache_write_per_1m": 3.75,
      "input_per_1m": 3.0,
      "model": "claude-sonnet-4-5",
      "notes": "historical rate retained for replaying past usage logs",
      "output_per_1m": 15.0,
      "vendor": "anthropic"
    },
    "anthropic/claude-sonnet-4-6": {
      "cache_read_per_1m": 0.3,
      "cache_write_per_1m": 3.75,
      "input_per_1m": 3.0,
      "model": "claude-sonnet-4-6",
      "notes": "historical rate retained for replaying past usage logs",
      "output_per_1m": 15.0,
      "vendor": "anthropic"
    },
    "anthropic/claude-sonnet-5": {
      "input_per_1m": 3.0,
      "model": "claude-sonnet-5",
      "notes": "Claude Sonnet 5 (GA 2026-06-30). List price $3/$15; introductory $2/$10 through 2026-08-31 \u2014 see promotional_rate. Verified platform.claude.com/docs/en/about-claude/pricing 2026-08-10.",
      "output_per_1m": 15.0,
      "vendor": "anthropic",
      "promotional_rate": {
        "input_per_1m": 2.0,
        "output_per_1m": 10.0,
        "through": "2026-08-31",
        "note": "Introductory pricing; list price applies from 2026-09-01."
      }
    },
    "google/gemini-3.1-pro": {
      "input_per_1m": 2.0,
      "model": "gemini-3.1-pro",
      "notes": "Gemini 3.1 Pro, <=200k ctx (doubles above 200k); third-party aggregator figures \u2014 verify at ai.google.dev/gemini-api/docs/pricing",
      "output_per_1m": 12.0,
      "vendor": "google"
    },
    "google/gemini-3.5-flash": {
      "input_per_1m": 1.5,
      "model": "gemini-3.5-flash",
      "notes": "Gemini 3.5 Flash (GA 2026-05-19); third-party aggregator figures \u2014 verify at ai.google.dev/gemini-api/docs/pricing",
      "output_per_1m": 9.0,
      "vendor": "google"
    },
    "openai/codex-mini": {
      "cache_read_per_1m": 0.375,
      "input_per_1m": 1.5,
      "model": "codex-mini",
      "notes": "historical rate retained for replaying past usage logs",
      "output_per_1m": 6.0,
      "vendor": "openai"
    },
    "openai/gpt-4.1": {
      "cache_read_per_1m": 0.5,
      "input_per_1m": 2.0,
      "model": "gpt-4.1",
      "notes": "historical rate retained for replaying past usage logs",
      "output_per_1m": 8.0,
      "vendor": "openai"
    },
    "openai/gpt-5": {
      "cache_read_per_1m": 0.5,
      "input_per_1m": 2.0,
      "model": "gpt-5",
      "notes": "historical rate retained for replaying past usage logs",
      "output_per_1m": 8.0,
      "vendor": "openai"
    },
    "openai/gpt-5.4": {
      "cache_read_per_1m": 0.5,
      "input_per_1m": 2.0,
      "model": "gpt-5.4",
      "notes": "historical rate retained for replaying past usage logs",
      "output_per_1m": 8.0,
      "vendor": "openai"
    },
    "openai/gpt-5.5": {
      "cache_read_per_1m": 0.5,
      "input_per_1m": 2.0,
      "model": "gpt-5.5",
      "notes": "historical rate retained for replaying past usage logs",
      "output_per_1m": 8.0,
      "vendor": "openai"
    },
    "openai/gpt-5.6-luna": {
      "cache_read_per_1m": 0.25,
      "input_per_1m": 0.2,
      "model": "gpt-5.6-luna",
      "notes": "GPT-5.6 Luna (speed/cost tier); verified developers.openai.com/api/docs/pricing 2026-08-10",
      "output_per_1m": 1.2,
      "vendor": "openai"
    },
    "openai/gpt-5.6-sol": {
      "cache_read_per_1m": 0.5,
      "cache_write_per_1m": 6.25,
      "input_per_1m": 5.0,
      "model": "gpt-5.6-sol",
      "notes": "GPT-5.6 Sol standard tier: $5 input, $0.50 cached input, $30 output, and cache write 1.25x uncached input. Verified 2026-08-15 at developers.openai.com/api/docs/models/gpt-5.6-sol. Fast service and long-context billing differ; do not infer either from a model ID alone.",
      "output_per_1m": 30.0,
      "pricing_source": "https://developers.openai.com/api/docs/models/gpt-5.6-sol",
      "pricing_verified_at": "2026-08-15",
      "vendor": "openai"
    },
    "openai/gpt-5.6-terra": {
      "cache_read_per_1m": 0.625,
      "input_per_1m": 2.0,
      "model": "gpt-5.6-terra",
      "notes": "GPT-5.6 Terra (balanced efficiency/capability); verified developers.openai.com/api/docs/pricing 2026-08-10",
      "output_per_1m": 12.0,
      "vendor": "openai"
    },
    "openai/o3": {
      "cache_read_per_1m": 0.5,
      "input_per_1m": 2.0,
      "model": "o3",
      "notes": "historical rate retained for replaying past usage logs",
      "output_per_1m": 8.0,
      "vendor": "openai"
    },
    "openai/o4-mini": {
      "cache_read_per_1m": 0.275,
      "input_per_1m": 1.1,
      "model": "o4-mini",
      "notes": "historical rate retained for replaying past usage logs",
      "output_per_1m": 4.4,
      "vendor": "openai"
    }
  }
}

{
  "metadata": {
    "skill": "ai-agents",
    "updated": "2026-07-11",
    "total_sources": 58,
    "description": "Curated primary sources for production agent systems: architecture, protocols, tool design, evaluation, observability, and economics.",
    "version": "5.4",
    "title": "AI Agents - Sources",
    "last_updated": "2026-07-11"
  },
  "categories": {
    "standards_and_governance": [
      {
        "name": "EU AI Act (Regulation (EU) 2024/1689)",
        "url": "https://eur-lex.europa.eu/eli/reg/2024/1689/oj",
        "type": "specification",
        "relevance": "Regulatory baseline for risk classification, transparency, documentation, and controls affecting agent systems.",
        "update_frequency": "quarterly",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "NIST AI Risk Management Framework 1.0",
        "url": "https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf",
        "type": "specification",
        "relevance": "Governance and risk management framework for production AI systems.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": false
      },
      {
        "name": "NIST Generative AI Profile (AI 600-1)",
        "url": "https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf",
        "type": "specification",
        "relevance": "GenAI-specific profile aligned to NIST AI RMF; useful for controls, logging, and deployment planning.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": false
      },
      {
        "name": "OWASP Top 10 for LLM Applications",
        "url": "https://owasp.org/www-project-top-10-for-large-language-model-applications/",
        "type": "specification",
        "relevance": "Threat categories for prompt injection, data leakage, tool misuse, and agent abuse scenarios.",
        "update_frequency": "annual",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "NIST Secure Software Development Framework (SSDF)",
        "url": "https://csrc.nist.gov/pubs/sp/800/218/final",
        "type": "specification",
        "relevance": "Secure development baseline relevant for tool implementation, supply chain, and deployment practices.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "ISO/IEC 42001",
        "url": "https://www.iso.org/standard/42001",
        "type": "specification",
        "relevance": "AI management system standard for organizational governance and continuous improvement.",
        "update_frequency": "static",
        "access": "paid",
        "add_as_web_search": true
      },
      {
        "name": "CVE-2026-30623 — Command injection via Anthropic MCP SDK (stdio)",
        "url": "https://docs.litellm.ai/blog/mcp-stdio-command-injection-april-2026",
        "type": "advisory",
        "relevance": "CVE record + mitigation context for the MCP stdio command-injection class. Pair with the OX Security systemic advisory. April 2026.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": true
      }
    ],
    "protocols_and_interoperability": [
      {
        "name": "Agent Skills Specification",
        "url": "https://agentskills.io/specification",
        "type": "specification",
        "relevance": "Official skill packaging specification for SKILL.md structure, metadata, and progressive disclosure.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Model Context Protocol (MCP) Introduction",
        "url": "https://modelcontextprotocol.io/docs/getting-started/intro",
        "type": "specification",
        "relevance": "High-level overview of MCP for tool and resource integration.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "MCP Transport Specification",
        "url": "https://modelcontextprotocol.io/specification/2025-11-25/basic/transports",
        "type": "specification",
        "relevance": "Primary source for MCP transport guidance, including stdio and Streamable HTTP. Current stable: 2025-11-25.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "MCP Authorization Specification",
        "url": "https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization",
        "type": "specification",
        "relevance": "Primary source for MCP authorization, OAuth 2.1, and OpenID Connect guidance for remote servers. Current stable: 2025-11-25.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Agent2Agent (A2A) Protocol Overview",
        "url": "https://a2a-protocol.org/latest/",
        "type": "specification",
        "relevance": "Primary source for agent cards, task handoffs, and agent-to-agent interoperability. A2A v1.0 under Linux Foundation governance. Verified 2026-06-09.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Agent2Agent (A2A) Protocol Specification",
        "url": "https://a2a-protocol.org/latest/specification/",
        "type": "specification",
        "relevance": "Full A2A v1.0 specification: tasks, messages, artifacts, agent cards, JSON-RPC/gRPC/HTTP-REST protocol bindings, security. Verified 2026-06-09.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "MAST: A Multi-Agent System Taxonomy (arXiv:2503.13657)",
        "url": "https://arxiv.org/abs/2503.13657",
        "type": "research",
        "relevance": "Canonical MAST taxonomy: 14 failure modes across 3 categories. Original Cemri et al. distribution ≈ Specification/System Design 41.8%, Inter-Agent Misalignment 36.9%, Task Verification 21.3% (figures vary across secondary write-ups applying MAST to other trace sets — verify against the primary PDF/HTML before quoting). NeurIPS 2025 Datasets & Benchmarks track. Re-verified 2026-07-11.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": false
      },
      {
        "name": "JSON Schema",
        "url": "https://json-schema.org/",
        "type": "specification",
        "relevance": "Schema standard for tool I/O validation, handoff contracts, and structured outputs.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "OpenTelemetry Semantic Conventions for GenAI",
        "url": "https://opentelemetry.io/docs/specs/semconv/gen-ai/",
        "type": "specification",
        "relevance": "Standardized telemetry fields for LLM calls, tool calls, and agent tracing.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "OX Security — Systemic MCP STDIO RCE advisory (CVE-2026-30623)",
        "url": "https://www.ox.security/blog/the-mother-of-all-ai-supply-chains-critical-systemic-vulnerability-at-the-core-of-the-mcp/",
        "type": "advisory",
        "relevance": "Verified by-design config->OS-command execution in official MCP SDK stdio interface; Anthropic confirmed by-design and declined a protocol fix. Load-bearing for MCP host sandboxing guidance. Multi-source corroborated (The Register, CSA, Infosecurity, CVE-2026-30623), April 2026.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "MCP 2026 Roadmap & 2026-07-28 Release Candidate",
        "url": "https://blog.modelcontextprotocol.io/posts/2026-mcp-roadmap/",
        "type": "specification",
        "relevance": "Primary source (MCP blog). 2025-11-25 remains current stable as of June 2026; a 2026-07-28 revision (largest since launch) is in release candidate: stateless HTTP core, MCP Apps (server-rendered UIs), Tasks extension (long-running work), OAuth/OIDC-aligned auth, formal deprecation policy. Build on 2025-11-25 today; track blog for ratification.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      }
    ],
    "implementation_docs_and_playbooks": [
      {
        "name": "Michael Albada, Building Applications with AI Agents (O'Reilly, 2025)",
        "url": "https://www.oreilly.com/library/view/building-applications-with/9798341622906/",
        "type": "book",
        "relevance": "Ch. 5 covers tool selection at scale (standard vs. semantic vs. hierarchical) and the tool-topology ladder with its bounding rules (cap chain length, cap graph depth and branching factor). Ch. 8 supplies the escalation gate: exhaust hierarchical grouping and semantic retrieval inside one agent before decomposing into multiple agents.",
        "update_frequency": "static",
        "access": "paid",
        "add_as_web_search": false
      },
      {
        "name": "The Machine that Builds the Machine — Symphony (@daniel_mac8)",
        "url": "https://x.com/daniel_mac8/status/2034344165211832481",
        "type": "article",
        "relevance": "Deep dive on OpenAI's Symphony — an open-source Elixir-based orchestrator that polls Linear, spawns Codex agents in isolated workspaces with 300-line Jinja2 workflow templates, self-reviews PRs, and manages full issue lifecycle. Demonstrates outcome-based vs prompt-based agent paradigm at 10-agent concurrency: 26 tasks, 27 PRs, 32K LOC in 36 hours.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": false
      },
      {
        "name": "OpenAI Building Agents",
        "url": "https://developers.openai.com/tracks/building-agents",
        "type": "documentation",
        "relevance": "Official playbook for agent design, tools, orchestration, evals, MCP/connectors, and operations.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Anthropic Claude Code Best Practices",
        "url": "https://www.anthropic.com/engineering/claude-code-best-practices",
        "type": "documentation",
        "relevance": "Official guidance for context, subagents, memory, and code-agent operating patterns.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Anthropic Writing Tools for Agents",
        "url": "https://www.anthropic.com/engineering/writing-tools-for-agents",
        "type": "documentation",
        "relevance": "Primary source for tool design: narrow tools, strong descriptions, examples, and predictable side-effect boundaries.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Anthropic Tool Use Documentation",
        "url": "https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/overview",
        "type": "documentation",
        "relevance": "Official provider reference for tool calling, tool schemas, and tool-use orchestration.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "LangGraph Documentation",
        "url": "https://langchain-ai.github.io/langgraph/",
        "type": "documentation",
        "relevance": "Reference implementation for stateful workflow agents, checkpoints, and human-in-the-loop patterns.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "OpenAI Evals",
        "url": "https://github.com/openai/evals",
        "type": "tool",
        "relevance": "Reference tooling for evaluation harnesses and regression tests.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "OpenTelemetry Documentation",
        "url": "https://opentelemetry.io/docs/",
        "type": "documentation",
        "relevance": "Implementation reference for distributed tracing, metrics, and logs in production systems.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      }
    ],
    "framework_docs": [
      {
        "name": "OpenAI Agents SDK for Python",
        "url": "https://openai.github.io/openai-agents-python/",
        "type": "documentation",
        "relevance": "Official Python SDK for tool-centric agents, handoffs, tracing, and human-in-the-loop patterns.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "OpenAI Agents SDK for JavaScript",
        "url": "https://openai.github.io/openai-agents-js/",
        "type": "documentation",
        "relevance": "Official JavaScript SDK for OpenAI agents; use when checking language support or JS-specific patterns.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Google Agent Development Kit (ADK)",
        "url": "https://google.github.io/adk-docs/",
        "type": "documentation",
        "relevance": "Official code-first framework docs for Gemini-oriented agents and multi-language teams.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Pydantic AI",
        "url": "https://ai.pydantic.dev/",
        "type": "documentation",
        "relevance": "Official type-safe Python agent framework docs; verify current MCP, A2A, testing, and durable-execution support here.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "AWS Bedrock Agents",
        "url": "https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html",
        "type": "documentation",
        "relevance": "Official managed-agent documentation for AWS environments.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "CrewAI Documentation",
        "url": "https://docs.crewai.com/",
        "type": "documentation",
        "relevance": "Official role-based multi-agent framework documentation.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Microsoft Agent Framework Overview",
        "url": "https://learn.microsoft.com/en-us/agent-framework/overview",
        "type": "documentation",
        "relevance": "Official overview for Microsoft's agent framework; use this to verify current lifecycle, language support, and Azure positioning.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Anthropic Claude Agent SDK",
        "url": "https://docs.anthropic.com/en/docs/claude-code/sdk",
        "type": "documentation",
        "relevance": "Official Anthropic SDK docs for code agents, MCP integration, tools, computer use, and subagents.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "LlamaIndex Workflows",
        "url": "https://docs.llamaindex.ai/en/stable/understanding/workflows/",
        "type": "documentation",
        "relevance": "Official workflow and orchestration docs for retrieval-heavy agent systems.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Mastra Documentation",
        "url": "https://mastra.ai/docs",
        "type": "documentation",
        "relevance": "Official TypeScript-oriented agent framework documentation.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Spring AI Reference",
        "url": "https://docs.spring.io/spring-ai/reference/",
        "type": "documentation",
        "relevance": "Official Java/Kotlin agent framework: ChatClient, Advisors, ToolCallback, A2A integration, AutoMemoryTools.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Semantic Kernel → Microsoft Agent Framework Migration Guide",
        "url": "https://learn.microsoft.com/en-us/agent-framework/migration-guide/from-semantic-kernel/",
        "type": "documentation",
        "relevance": "Official migration path; SK is in maintenance, MAF is the forward path. Verify before starting any greenfield SK work.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "SmolAgents Documentation",
        "url": "https://huggingface.co/docs/smolagents/",
        "type": "documentation",
        "relevance": "Official lightweight agent framework documentation from Hugging Face.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Agno Documentation",
        "url": "https://docs.agno.com/",
        "type": "documentation",
        "relevance": "Official documentation for Agno's agent runtime, workflows, memory, and integrations.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Haystack Documentation",
        "url": "https://docs.haystack.deepset.ai/",
        "type": "documentation",
        "relevance": "Official documentation for Haystack pipelines and agent components.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "DSPy Documentation",
        "url": "https://dspy.ai/",
        "type": "documentation",
        "relevance": "Official docs for declarative prompt/program optimization and agentic modules.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "OWL (CAMEL-AI)",
        "url": "https://github.com/camel-ai/owl",
        "type": "tool",
        "relevance": "Multi-agent cooperation framework from CAMEL-AI that tops GAIA benchmark; reference for agent team coordination and real-world task completion benchmarks.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": false
      },
      {
        "name": "Dify",
        "url": "https://github.com/langgenius/dify",
        "type": "tool",
        "relevance": "Open-source LLM app builder combining workflows, RAG pipelines, and agents in a single platform; reference for integrated agentic application architecture patterns.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": false
      }
    ],
    "economics_and_decision_frameworks": [
      {
        "name": "OpenAI API Pricing",
        "url": "https://openai.com/api/pricing/",
        "type": "documentation",
        "relevance": "Current pricing for OpenAI models and tools; required for ROI and budget calculations.",
        "update_frequency": "monthly",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Anthropic API Pricing",
        "url": "https://www.anthropic.com/pricing",
        "type": "documentation",
        "relevance": "Current pricing for Anthropic models and platform usage; required for ROI and budget calculations.",
        "update_frequency": "monthly",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Google AI Pricing",
        "url": "https://ai.google.dev/pricing",
        "type": "documentation",
        "relevance": "Current pricing for Gemini-family models and related services.",
        "update_frequency": "monthly",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Anthropic: Building Effective Agents",
        "url": "https://www.anthropic.com/research/building-effective-agents",
        "type": "research",
        "relevance": "High-signal guidance on when to use workflows vs agents and how to keep agent systems simple.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "karpathy/autoresearch",
        "url": "https://github.com/karpathy/autoresearch",
        "type": "tool",
        "relevance": "Reference implementation for autonomous improvement loops: bounded modification surface, fixed eval metric, git-as-experiment-ledger. Defines the research/experiment agent architecture.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Ole Lehmann: Skill Optimization with Autonomous Loops",
        "url": "https://www.olelehmann.com",
        "type": "article",
        "relevance": "Generalizes autoresearch to prompt/skill improvement: yes/no scoring checklist, agent loops change → test → keep/revert until 95%+. Case study: 56% → 92% in 4 rounds.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "a16z: AI Agents Primer",
        "url": "https://a16z.com/ai-agents-primer/",
        "type": "research",
        "relevance": "External perspective on agent economics, market tradeoffs, and operating-model assumptions.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Hallucination Leaderboard (Vectara)",
        "url": "https://github.com/vectara/hallucination-leaderboard",
        "type": "tool",
        "relevance": "Benchmark reference for hallucination-rate comparisons and risk discussions.",
        "update_frequency": "monthly",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "ccusage — Claude Code Usage Analyzer",
        "url": "https://github.com/ryoppippi/ccusage",
        "type": "tool",
        "relevance": "CLI tool that reads local Claude Code JSONL logs to produce daily, weekly, monthly, and session-level token and cost reports.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "@ccusage/codex — Codex CLI Usage Analyzer",
        "url": "https://www.npmjs.com/package/@ccusage/codex",
        "type": "tool",
        "relevance": "CLI tool that reads local Codex CLI JSONL session logs to produce daily, monthly, and session-level token and cost reports.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "@ccusage/mcp — Usage Data MCP Server",
        "url": "https://www.npmjs.com/package/@ccusage/mcp",
        "type": "tool",
        "relevance": "MCP server that exposes ccusage data as agent-accessible tools for cost-aware agent loops and self-monitoring.",
        "update_frequency": "continuous",
        "access": "free",
        "add_as_web_search": true
      },
      {
        "name": "Single-Agent LLMs Outperform Multi-Agent Systems Under Equal Thinking Token Budgets (Tran & Kiela)",
        "url": "https://arxiv.org/abs/2604.02460",
        "type": "research",
        "relevance": "Information-theoretic (Data Processing Inequality) argument that single-agent matches/beats multi-agent at equal token budget; build-vs-multi-agent decision input. Apr 2026 arXiv preprint, NOT peer-reviewed; scope: text-only multi-hop reasoning. Verify before relying.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": false
      },
      {
        "name": "Agentic Much? Adoption of Coding Agents on GitHub (arXiv:2601.18341)",
        "url": "https://arxiv.org/abs/2601.18341",
        "type": "research",
        "relevance": "129,134-project empirical study estimating ~16-23% coding-agent adoption at file/commit level on GitHub by late 2025. Use as the sourced baseline instead of single-vendor PR-count marketing claims. Verified 2026-07-11.",
        "update_frequency": "static",
        "access": "free",
        "add_as_web_search": false
      }
    ]
  }
}

# ai-agents — Consolidated Learnings

Curated, dated, committed memory for this skill. Pruned from raw `learnings.md` via `agents-skills-feedback-loop/scripts/consolidate.py`. Human-approved.

Cap: 60 entries. When exceeded, promote durable rules to `references/`.

## Filter Override

<!-- Add 2-4 bullets that sharpen what counts as a learning for this skill. Leave empty to use the default filter from agents-skills-feedback-loop/references/learnings-format.md. -->

## Patterns That Work

## Mistakes to Avoid

## Domain Knowledge

## Open Questions

## Consolidated Principles

# ai-agents — Learnings

## Patterns That Work

## Mistakes to Avoid

- [2026-07-11] Prior version cited an unverifiable '400,000+ Codex PRs in 2 months' stat and a stale '68% mini-swe-agent' figure (now >74%). Replaced with the cited adoption study arXiv:2601.18341; re-check benchmark numbers before quoting.
## Domain Knowledge

- [2026-07-11] MAST (arXiv:2503.13657, NeurIPS 2025) original split: System Design ~41.8%, Inter-Agent Misalignment ~36.9%, Task Verification ~21.3%. A repeated 44.2/32.3/23.5 figure is a different analysis — verify before quoting.
- [2026-07-11] CrewAI Flows added runtime checkpointing (CheckpointConfig + SqliteProvider, ~May 2026): checkpoints Flow-method/Crew-task boundaries only, not mid-ReAct tool loops. Don't promise exactly-once recovery without checking current docs.
## Open Questions

## Consolidated Principles

# 24/7 Operating Model for Agents

Use this reference when an agent system moves from "demo that works" to "production thing customers depend on at 3am". This guide defines SLOs, on-call structure, runbook contracts, post-mortem expectations, and the operating cadence that keeps agent systems healthy over months.

Applies across Shape A (triggered), Shape B (always-on bot/voice), and Shape C (autonomous loop). Each shape has different SLO targets but shares the same operating-model spine.

## Table of Contents

- [What "Production" Means for Agents](#what-production-means-for-agents)
- [SLO Catalogue by Shape](#slo-catalogue-by-shape)
- [Error Budgets](#error-budgets)
- [On-Call Structure](#on-call-structure)
- [Agent-Specific Alert Catalog](#agent-specific-alert-catalog)
- [Runbook Contract](#runbook-contract)
- [Incident Severity Levels](#incident-severity-levels)
- [The Standard Runbooks](#the-standard-runbooks)
- [Post-Mortem Template](#post-mortem-template)
- [Operating Cadence](#operating-cadence)
- [Change Management](#change-management)
- [Capacity Planning](#capacity-planning)
- [Cost Governance](#cost-governance)
- [Compliance Hooks](#compliance-hooks)
- [Readiness Checklist Before First 24/7 Day](#readiness-checklist-before-first-247-day)
- [Cross-References](#cross-references)

## What "Production" Means for Agents

Production for an agent system is not the same as production for a regular web service. Differences:

| Dimension | Regular service | Agent system |
|---|---|---|
| Failure mode that matters most | 5xx errors | Silent quality degradation |
| Bill scaling | Linear with traffic | Can be quadratic with prompt growth |
| Eval substrate | Tests + monitoring | Tests + monitoring + online evals |
| Change risk | Code change | Code change + model change + prompt change + tool change |
| Rollback unit | Service version | Service + prompt + tool + model versions |
| Oncall pages | Errors and latency | Errors, latency, cost spikes, eval regressions, safety trips |

The 24/7 operating model is built around these differences.

## SLO Catalogue by Shape

Pick the targets that match your product; tighten over time.

### Shape A — Triggered runs

| SLO | Target | Window |
|---|---|---|
| Trigger receipt → agent start | p99 < 30s | 28 days |
| Agent run success rate | ≥ 99% | 28 days |
| DLQ rate | ≤ 1% | 28 days |
| Duplicate rate (after dedup) | ≤ 0.1% | 28 days |
| End-to-end p99 | ≤ 5 min (or product-specific) | 28 days |
| Cost per event vs forecast | within 1.5x | 28 days |

### Shape B — Always-on bot/voice

| SLO | Target | Window |
|---|---|---|
| Availability | 99.9% | 28 days |
| Turn latency p50 | < 1.5s text, < 600ms voice | 28 days |
| Turn latency p99 | < 5s text, < 1.5s voice | 28 days |
| Session completion rate | ≥ 90% | 28 days |
| Escalation rate (delta from baseline) | within 1.5x | 7 days |
| Tool-call success rate | ≥ 99% | 28 days |
| Safety filter trip rate | tracked, alert on 2x baseline | 7 days |

### Shape C — Autonomous loops

| SLO | Target | Window |
|---|---|---|
| Loop completion rate (acceptance met) | ≥ 80% | rolling |
| Stagnation halts | ≤ 10% | rolling |
| Budget breach halts | ≤ 5% | rolling |
| Drift detections | tracked, page if any | per-run |
| Cost per completion vs forecast | within 1.5x | rolling |

Loops with completion rates below 80% are usually mis-scoped — the acceptance criterion is too tight for the agent's capability. Treat low completion as a product problem, not an ops problem.

## Error Budgets

Error budget = (1 − SLO target) × time window.

For 99.9% availability over 28 days: 0.001 × 28 × 24 × 60 ≈ 40 minutes downtime budget.

Burn rate alerts (page when):

- 2% of budget consumed in 1 hour (very fast burn)
- 5% of budget consumed in 6 hours (fast burn)
- 10% of budget consumed in 24 hours (sustained burn)

When the budget is exhausted: stop shipping risky changes until budget is replenished. This is the discipline mechanism — without it, SLOs are decorative.

## On-Call Structure

Minimum viable on-call for an agent system:

- **Primary on-call**: 1 engineer, 1-week rotation, pages first.
- **Secondary on-call**: 1 engineer, fallback at 15 min unacked.
- **Subject-matter on-call** (optional): 1 person familiar with prompts/evals, weekday hours only.

Rotation rules:

- Minimum team size 4 (otherwise burnout). With fewer than 4, accept business-hours-only coverage.
- Hand off Mondays, not Fridays.
- 24 hours of compensatory time per primary week.
- On-call shadow rotation for new hires (4 weeks).

On-call equipment baseline:

- Phone with paging app
- Laptop with VPN and prod access
- Bookmarked runbook hub
- Kill-switch documented
- Provider status pages bookmarked
- A way to silence runaway alerts

## Agent-Specific Alert Catalog

Standard service alerts (5xx, latency, saturation) plus these agent-specific ones:

| Alert | Trigger | Severity | First action |
|---|---|---|---|
| Budget breach | Per-run cost > threshold | P2 | Kill-switch run, investigate |
| Loop stagnation | 3+ iterations no progress | P3 | Review iteration outputs |
| Safety trip rate spike | 2x baseline in 1h | P1 | Pause new sessions, investigate |
| Eval regression | online eval score drops > 10% | P2 | Rollback or pin to prior version |
| Provider outage (LLM) | provider 5xx rate > 5% | P1 | Switch to fallback provider |
| Provider outage (STT/TTS) | provider 5xx rate > 5% | P1 | Switch to fallback (voice only) |
| Cost forecast breach | day-of-month cost > forecast × 1.3 | P2 | Check for runaway loop or attack |
| DLQ depth growth | DLQ > 100 items | P2 | Triage DLQ |
| Recording compliance miss | recording success < 99% | P1 | Halt regulated calls (voice only) |
| Tool error spike | tool error rate > 2x baseline | P2 | Investigate tool backend |
| Memory leak | resident memory growing without bound | P2 | Investigate, restart with caution |
| Hot tenant | one tenant > 50% of LLM spend | P3 | Reach out to tenant, throttle |
| Webhook signature failures | > 1% in 1h | P2 | Possible attack or rotated secret |

Tune thresholds to your baselines. Alert fatigue kills oncall faster than the underlying failures.

## Runbook Contract

Every alert must point to a runbook. Every runbook must answer:

1. **What does this alert mean?** (one paragraph)
2. **How urgent is it?** (severity, time-to-respond)
3. **What's the first thing to check?** (dashboard URL, log query)
4. **What's the most likely cause?** (top 3 historical causes)
5. **What's the kill-switch / mitigation?** (link to action)
6. **Who owns this in normal hours?** (team / Slack channel)
7. **When was this runbook last tested?** (date — older than 90 days = stale)

Runbooks live in the same repo as the agent code, not in a wiki nobody updates.

## Incident Severity Levels

| Sev | Definition | Response | Customer comms |
|---|---|---|---|
| **SEV1** | Outage; customers cannot use product | All hands; war room | Status page; proactive comms |
| **SEV2** | Significant degradation; some flows broken | Primary + secondary | Status page if customer-visible |
| **SEV3** | Edge-case failure or quality regression | Primary | Internal only unless escalates |
| **SEV4** | Cost or efficiency degradation | Triage in business hours | None |

Promote ruthlessly. A SEV3 that's been open 4 hours is a SEV2.

## The Standard Runbooks

Every agent system needs these runbooks before first 24/7 day:

1. **LLM provider outage** — switch to fallback, communicate degradation
2. **Runaway loop / cost spike** — find the loop, kill it, refund affected tenants if needed
3. **Safety filter trip storm** — pause new sessions, investigate input source, file model-provider report if needed
4. **Eval regression** — identify the change, rollback, post-mortem
5. **DLQ saturation** — triage classification, decide drop vs replay vs fix
6. **Hot tenant** — throttle, reach out, possibly migrate to dedicated capacity
7. **Stale checkpoint / state corruption** — restore from backup, identify root cause
8. **Recording compliance gap** (voice only) — assess regulatory exposure, file SAR/breach notice if required
9. **Carrier outage** (voice only) — failover, update status page
10. **Agent context window saturation** — reduce context, summarize history, restart loop with smaller scope

Each should be runnable by the on-call without escalation.

## Post-Mortem Template

```markdown
# Post-Mortem: {{title}}

- **Date**: {{date}}
- **Authors**: {{authors}}
- **Status**: {{draft|review|published}}
- **Severity**: SEV{{1|2|3|4}}
- **Duration**: {{start}} → {{end}} ({{minutes}} min)
- **User impact**: {{description}}

## Timeline
- HH:MM — {{event}}
- HH:MM — {{event}}

## Root cause
{{single sentence, then a paragraph}}

## What went well
- ...

## What went badly
- ...

## Where we got lucky
- ...

## Action items
| Action | Owner | Due | Linked ticket |
|---|---|---|---|
| ... | ... | ... | ... |

## Related material
- Dashboards, logs, prior incidents
```

Rules:

- Blameless. Names attach to actions, not faults.
- Published within 5 business days for SEV1/2.
- Action items have owners and due dates; track in your normal issue tracker.
- One person reads each post-mortem aloud at the next ops review — surfaces gaps you cannot read through.

## Operating Cadence

Weekly:

- 30 min ops review: prior week alerts, SLO burn, top tenants by cost
- DLQ triage pass
- Eval suite run; investigate any regression > 5%

Monthly:

- Runbook freshness audit (any > 90 days untested gets re-tested)
- Cost trend review against forecast
- Kill-switch test (literally flip it and verify the agent stops)
- On-call rotation health (burnout, alert volume per shift)

Quarterly:

- Capacity planning against expected growth
- Provider contract review (volume tiers, fallback SLAs)
- Game day: simulate provider outage, runaway loop, mass-call drop
- Threat model refresh
- Compliance audit prep

## Change Management

Production changes that need review:

| Change | Reviewer | Pre-prod step |
|---|---|---|
| Code change | Standard PR review | Eval suite green |
| Prompt change | Eval-suite gate + 1 reviewer | Canary cohort |
| Model change (e.g., Opus 4.6 → 4.7) | Eval-suite gate + ops sign-off | Canary + cost forecast |
| Tool addition | Security review + 1 reviewer | Sandboxed test |
| Tool removal | Customer-impact review | Deprecation notice |
| Budget cap change | Ops sign-off | Forecast update |
| Hook change (esp. budget hooks) | 2 reviewers | Hook unit tests + integration |

Changes outside business hours: only SEV1/2 mitigation. No feature work, no eval changes, no prompt changes after 6pm local time.

## Capacity Planning

Forecast each substrate:

- LLM provider TPM / concurrent quota
- STT/TTS provider quotas (voice)
- Compute (CPU, memory, pod count)
- Storage (recordings, checkpoints, eval data)
- Telephony carrier capacity (voice)
- Webhook gateway throughput (triggered)

Plan to peak × 2 with at least 14 days of lead time on any quota increase request.

## Cost Governance

Daily:

- Cost dashboard by tenant, by model, by purpose
- Anomaly alerts (any tenant > 3x 7-day average)

Weekly:

- Tenant top-10 review (any new entrant?)
- Cost-per-call / cost-per-event tracked vs forecast

Monthly:

- Provider invoice reconciliation against internal meter
- Budget cap effectiveness (how many runs hit the cap?)

The single most common production fire in May 2026 agent systems is a runaway loop or compromised webhook causing 10x cost overnight. Daily cost alerts catch this before the bill.

## Compliance Hooks

If your agent system is regulated (financial, health, government):

- All actions producing customer impact must be logged with: user, action, agent version, timestamp, agent reasoning summary.
- Audit log retention per regulation (5y for FCA, 7y for HIPAA, 6y for GDPR records of processing).
- Right-to-erasure flows must reach training data and embedding stores, not just the customer-visible database.
- Recordings (voice) with consent records, retention-policy enforced.
- DPIA on file before launch.
- Model card / system card with limitations and known failure modes.

See:

- Project-specific EMI / GDPR skills for client deployments — keep project references out of this portable domain skill
- [`../../ai-mlops/references/governance-checklists.md`](../../ai-mlops/references/governance-checklists.md) — MLOps governance
- `legal-emi-region-uk` — UK regulatory triage

## Readiness Checklist Before First 24/7 Day

- [ ] SLOs documented per shape, with dashboards
- [ ] Error budget defined, burn alerts wired
- [ ] On-call rotation set with at least 4 people
- [ ] Pages route to primary, escalate to secondary at 15 min
- [ ] All 10 standard runbooks written and tested in last 90 days
- [ ] Kill-switch operable from phone
- [ ] Cost dashboards live, anomaly alerts firing
- [ ] DLQ has owner; depth alert wired
- [ ] Eval suite runs nightly with regression alerts
- [ ] Online evals running (Shape B and C)
- [ ] Provider fallback chains configured
- [ ] Audit log meets regulatory retention
- [ ] Post-mortem template available; first one written for a prior near-miss
- [ ] Status page exists; status comms drafted
- [ ] Game day completed in last 90 days
- [ ] Change-management process documented
- [ ] Capacity headroom > 30%
- [ ] Compliance sign-off (if regulated)

## Cross-References

- [`autonomous-loop-patterns.md`](autonomous-loop-patterns.md) — Shape C deep dive
- [`agent-operations-best-practices.md`](agent-operations-best-practices.md) — broader ops patterns
- [`deployment-ci-cd-and-safety.md`](deployment-ci-cd-and-safety.md) — release patterns
- [`evaluation-and-observability.md`](evaluation-and-observability.md) — telemetry stack
- [`guardrails-implementation.md`](guardrails-implementation.md) — guardrails
- [`escalation-patterns.md`](escalation-patterns.md) — escalation flow
- [`../../ai-coding-agents-tasks/references/webhook-and-queue-triggers.md`](../../ai-coding-agents-tasks/references/webhook-and-queue-triggers.md) — Shape A patterns
- [`../../ai-coding-agents-tasks/references/durable-trigger-integration.md`](../../ai-coding-agents-tasks/references/durable-trigger-integration.md) — durable orchestration
- [`../../ai-bot-builder/references/production-deployment.md`](../../ai-bot-builder/references/production-deployment.md) — Shape B text bot
- [`../../ai-bot-builder/references/stateful-rollout-and-blue-green.md`](../../ai-bot-builder/references/stateful-rollout-and-blue-green.md) — bot rollouts
- [`../../ai-voice-bots/references/production-deployment.md`](../../ai-voice-bots/references/production-deployment.md) — Shape B voice
- [`../../agents-hooks/references/budget-and-loop-hooks.md`](../../agents-hooks/references/budget-and-loop-hooks.md) — budget enforcement
- [`../../ops-incident-response/SKILL.md`](../../ops-incident-response/SKILL.md) — general incident response
- [`../../ai-mlops/references/incident-response-playbooks.md`](../../ai-mlops/references/incident-response-playbooks.md) — ML/AI incident playbooks
- [`../../qa-observability/SKILL.md`](../../qa-observability/SKILL.md) — observability foundations
- [`../../qa-resilience/SKILL.md`](../../qa-resilience/SKILL.md) — resilience review

# A2A Handoff Patterns — Agent Coordination Guide

*Purpose: Practical patterns for agent-to-agent communication, task handoffs, and multi-agent orchestration using the A2A protocol.*

**When to use this guide**: User asks to coordinate multiple agents, implement agent delegation, or build collaborative AI workflows.

**For architecture deep-dive**: See `frameworks/shared-foundations/protocols/a2a/` for comprehensive protocol specification.

---
## Table of Contents

- [Quick Decision: Do I Need A2A?](#quick-decision-do-i-need-a2a)
- [A2A Architecture (Quick Reference)](#a2a-architecture-quick-reference)
- [Core A2A Message Schema](#core-a2a-message-schema)
- [Pattern 1: Sequential Handoff Chain](#pattern-1-sequential-handoff-chain)
- [Implementation](#implementation)
- [Agent A: Data Fetcher](#agent-a-data-fetcher)
- [Agent B: Analyzer (receives handoff)](#agent-b-analyzer-receives-handoff)
- [Pattern 2: Manager-Worker Delegation](#pattern-2-manager-worker-delegation)
- [Implementation](#implementation)
- [Manager Agent](#manager-agent)
- [Worker Agent (e.g., Researcher)](#worker-agent-eg-researcher)
- [Pattern 3: Group Chat Collaboration](#pattern-3-group-chat-collaboration)
- [Implementation](#implementation)
- [Pattern 4: Agent Card Discovery](#pattern-4-agent-card-discovery)
- [Agent Card Schema](#agent-card-schema)
- [Discovery Implementation](#discovery-implementation)
- [Usage: Manager discovers appropriate agent](#usage-manager-discovers-appropriate-agent)
- [Find agent that can write SQL and optimize queries](#find-agent-that-can-write-sql-and-optimize-queries)
- [Pattern 5: Error Recovery and Retry](#pattern-5-error-recovery-and-retry)
- [Validation & Schema Management](#validation-&-schema-management)
- [Handoff Schema Validation](#handoff-schema-validation)
- [Define A2A handoff schema](#define-a2a-handoff-schema)
- [Observability & Monitoring](#observability-&-monitoring)
- [Full Trace Tracking](#full-trace-tracking)
- [Metrics to Track](#metrics-to-track)
- [Handoff metrics](#handoff-metrics)
- [A2A vs MCP: When to Use What](#a2a-vs-mcp-when-to-use-what)
- [Production Checklist](#production-checklist)
- [Next Steps](#next-steps)


## Quick Decision: Do I Need A2A?

**Use A2A when**:
- Multiple agents need to collaborate on a task
- Delegating subtasks to specialized agents
- Building manager-worker agent patterns
- Need traceable, auditable agent communication
- Agents from different vendors/frameworks must interoperate

**Don't use A2A when**:
- Single agent handles everything (no coordination needed)
- Agent needs external data/tools (use MCP instead)
- Simple sequential pipeline without decision-making

---

## A2A Architecture (Quick Reference)

```
┌──────────────┐                           ┌──────────────┐
│  Agent A     │    A2A Message Protocol   │  Agent B     │
│  (Sender)    │ ───────────────────────→  │  (Receiver)  │
│              │                           │              │
│ - Creates    │    {                      │ - Validates  │
│   task       │      task,                │   payload    │
│ - Packages   │      context,             │ - Executes   │
│   context    │      constraints          │   task       │
│ - Sends      │    }                      │ - Returns    │
│              │                           │   result     │
└──────────────┘                           └──────────────┘
```

**Key concept**: A2A = structured handoffs with validation, not just message passing.

---

## Core A2A Message Schema

Every A2A handoff includes:

```json
{
  "schemaVersion": "v1.2",
  "trace_id": "req-abc-123-xyz",
  "timestamp": "2025-01-15T10:30:00Z",
  "sender": {
    "agent_id": "sales-analyzer-01",
    "agent_type": "data-analyst",
    "capabilities": ["sql", "visualization", "forecasting"]
  },
  "receiver": {
    "agent_id": "report-generator-03",
    "agent_type": "document-writer",
    "required_capabilities": ["pdf-generation", "charting"]
  },
  "task": {
    "type": "generate_report",
    "description": "Create quarterly sales report",
    "priority": "high",
    "deadline": "2025-01-16T17:00:00Z"
  },
  "context": {
    "sales_data": {...},
    "previous_reports": [...],
    "template_id": "q4-2024"
  },
  "constraints": {
    "max_pages": 20,
    "output_format": "pdf",
    "include_sections": ["summary", "trends", "forecast"]
  },
  "metadata": {
    "correlation_id": "campaign-456",
    "user_id": "user-789",
    "session_id": "sess-012"
  }
}
```

**Validation requirements**:
- JSON Schema validation on every handoff
- Required fields: `schemaVersion`, `trace_id`, `task`, `sender`, `receiver`
- Optional fields: `context`, `constraints`, `metadata`
- Receivers MUST validate before executing

---

## Pattern 1: Sequential Handoff Chain

**Use case**: Linear workflow where each agent completes one step before passing to next.

```
Agent A (Data Fetcher) → Agent B (Analyzer) → Agent C (Reporter)
```

### Implementation

```python
# Agent A: Data Fetcher
async def fetch_and_handoff():
    # 1. Complete own task
    sales_data = await fetch_sales_data(query)

    # 2. Package handoff
    handoff_message = {
        "schemaVersion": "v1.2",
        "trace_id": generate_trace_id(),
        "sender": {
            "agent_id": "data-fetcher-01",
            "agent_type": "data-collector"
        },
        "receiver": {
            "agent_id": "analyzer-02",
            "agent_type": "data-analyst"
        },
        "task": {
            "type": "analyze_trends",
            "description": "Identify sales trends and anomalies"
        },
        "context": {
            "sales_data": sales_data,
            "date_range": "Q4-2024",
            "baseline_metrics": previous_quarter_metrics
        },
        "constraints": {
            "analysis_depth": "detailed",
            "highlight_anomalies": True
        }
    }

    # 3. Validate against schema
    validate_handoff_schema(handoff_message)

    # 4. Send to next agent
    return await send_to_agent("analyzer-02", handoff_message)


# Agent B: Analyzer (receives handoff)
async def receive_and_analyze(handoff_message):
    # 1. Validate handoff
    if not validate_handoff_schema(handoff_message):
        raise ValueError("Invalid handoff schema")

    # 2. Extract context
    sales_data = handoff_message["context"]["sales_data"]
    constraints = handoff_message["constraints"]

    # 3. Execute analysis
    analysis_results = await analyze_trends(
        sales_data,
        depth=constraints["analysis_depth"],
        highlight_anomalies=constraints["highlight_anomalies"]
    )

    # 4. Handoff to next agent
    next_handoff = {
        "schemaVersion": "v1.2",
        "trace_id": handoff_message["trace_id"],  # Preserve trace
        "sender": {
            "agent_id": "analyzer-02",
            "agent_type": "data-analyst"
        },
        "receiver": {
            "agent_id": "reporter-03",
            "agent_type": "report-generator"
        },
        "task": {
            "type": "generate_report",
            "description": "Create executive summary report"
        },
        "context": {
            "analysis": analysis_results,
            "original_data": sales_data,
            "insights": extract_key_insights(analysis_results)
        }
    }

    return await send_to_agent("reporter-03", next_handoff)
```

**Best practices**:
- Always preserve `trace_id` across chain
- Each agent validates incoming handoff
- Include previous results in context
- Set clear constraints for next agent
- Log handoffs for debugging

---

## Pattern 2: Manager-Worker Delegation

**Use case**: One manager agent delegates subtasks to multiple specialized workers.

```
               ┌─→ Worker A (Researcher)
Manager Agent ─┼─→ Worker B (Writer)
               └─→ Worker C (Editor)
```

### Implementation

```python
# Manager Agent
async def delegate_task(user_request):
    # 1. Break down into subtasks
    subtasks = [
        {"type": "research", "topic": "AI trends"},
        {"type": "write", "section": "introduction"},
        {"type": "edit", "style": "professional"}
    ]

    # 2. Assign to specialized workers
    worker_assignments = {
        "research": "researcher-agent-01",
        "write": "writer-agent-02",
        "edit": "editor-agent-03"
    }

    trace_id = generate_trace_id()
    results = []

    for subtask in subtasks:
        worker_id = worker_assignments[subtask["type"]]

        handoff = {
            "schemaVersion": "v1.2",
            "trace_id": trace_id,
            "sender": {
                "agent_id": "manager-agent-00",
                "agent_type": "orchestrator"
            },
            "receiver": {
                "agent_id": worker_id,
                "agent_type": subtask["type"]
            },
            "task": subtask,
            "context": {
                "parent_task": user_request,
                "dependencies": []  # or list dependent tasks
            },
            "constraints": {
                "timeout_seconds": 300,
                "quality_threshold": 0.8
            }
        }

        # 3. Send to worker (parallel execution)
        result = await send_to_agent(worker_id, handoff)
        results.append(result)

    # 4. Aggregate results
    return await synthesize_results(results)


# Worker Agent (e.g., Researcher)
async def handle_research_task(handoff):
    # Validate
    validate_handoff_schema(handoff)

    # Execute specialized task
    topic = handoff["task"]["topic"]
    research_results = await conduct_research(topic)

    # Return result to manager
    return {
        "trace_id": handoff["trace_id"],
        "task_id": handoff["task"]["type"],
        "status": "completed",
        "result": research_results,
        "metadata": {
            "sources_count": len(research_results["sources"]),
            "confidence": 0.92
        }
    }
```

**Orchestration strategies**:
- **Parallel**: All workers execute simultaneously (fastest)
- **Sequential**: Workers execute in order (when dependencies exist)
- **Conditional**: Worker selection based on previous results

---

## Pattern 3: Group Chat Collaboration

**Use case**: Multiple agents discuss and collaborate to solve complex problem.

```
Agent A ←→ Agent B
   ↕          ↕
Agent C ←→ Agent D

All agents can communicate with each other
```

### Implementation

```python
class GroupChatOrchestrator:
    def __init__(self, agents, max_rounds=10):
        self.agents = agents
        self.max_rounds = max_rounds
        self.conversation_history = []

    async def coordinate(self, initial_task):
        trace_id = generate_trace_id()
        current_speaker = self.select_first_speaker(initial_task)

        for round in range(self.max_rounds):
            # Current agent generates response
            message = await current_speaker.generate_response(
                task=initial_task,
                conversation_history=self.conversation_history,
                trace_id=trace_id
            )

            # Broadcast to all agents
            handoff = {
                "schemaVersion": "v1.2",
                "trace_id": trace_id,
                "sender": {
                    "agent_id": current_speaker.id,
                    "agent_type": current_speaker.type
                },
                "receiver": {
                    "agent_id": "group",
                    "agent_type": "broadcast"
                },
                "task": {
                    "type": "contribute",
                    "round": round
                },
                "context": {
                    "message": message,
                    "conversation_history": self.conversation_history[-5:]
                }
            }

            self.conversation_history.append({
                "round": round,
                "speaker": current_speaker.id,
                "message": message,
                "timestamp": datetime.utcnow()
            })

            # Check termination condition
            if self.should_terminate(message):
                break

            # Select next speaker
            current_speaker = await self.select_next_speaker(
                conversation_history=self.conversation_history
            )

        return self.synthesize_final_result()

    def select_next_speaker(self, conversation_history):
        # Logic to select most relevant agent for next turn
        # Could be: round-robin, LLM-based selection, rule-based
        pass
```

**Group chat best practices**:
- Limit max rounds to prevent infinite loops
- Include conversation history in context (last 5-10 messages)
- Have clear termination condition
- Use manager agent to select speakers
- Log full conversation for debugging

---

## Pattern 4: Agent Card Discovery

**Use case**: Agents advertise capabilities so others can discover and delegate appropriately.

### Agent Card Schema

```json
{
  "agent_id": "sql-expert-agent-05",
  "agent_type": "database-specialist",
  "version": "2.1.0",
  "capabilities": [
    "sql-query-generation",
    "query-optimization",
    "database-schema-analysis",
    "data-validation"
  ],
  "supported_databases": ["postgres", "mysql", "bigquery"],
  "constraints": {
    "max_query_complexity": "high",
    "timeout_seconds": 300,
    "max_result_rows": 10000
  },
  "input_schema": {
    "type": "object",
    "required": ["query_description", "database_type"],
    "properties": {
      "query_description": {"type": "string"},
      "database_type": {"type": "string", "enum": ["postgres", "mysql", "bigquery"]},
      "optimization_level": {"type": "string", "enum": ["none", "standard", "aggressive"]}
    }
  },
  "output_schema": {
    "type": "object",
    "properties": {
      "sql_query": {"type": "string"},
      "execution_plan": {"type": "string"},
      "estimated_cost": {"type": "number"}
    }
  },
  "availability": {
    "status": "online",
    "uptime_sla": "99.9%",
    "rate_limit": "100 requests/minute"
  },
  "endpoints": {
    "handoff": "https://api.example.com/agents/sql-expert-05/handoff",
    "health": "https://api.example.com/agents/sql-expert-05/health"
  }
}
```

### Discovery Implementation

```python
class AgentRegistry:
    def __init__(self):
        self.agents = {}

    def register(self, agent_card):
        """Register agent with capabilities"""
        validate_agent_card(agent_card)
        self.agents[agent_card["agent_id"]] = agent_card

    def discover(self, required_capabilities):
        """Find agents matching required capabilities"""
        matching_agents = []

        for agent_id, card in self.agents.items():
            if all(cap in card["capabilities"] for cap in required_capabilities):
                matching_agents.append(card)

        return matching_agents


# Usage: Manager discovers appropriate agent
registry = AgentRegistry()

# Find agent that can write SQL and optimize queries
candidates = registry.discover([
    "sql-query-generation",
    "query-optimization"
])

if candidates:
    best_agent = select_best_agent(candidates)  # e.g., by uptime, rate limit
    handoff = create_handoff(best_agent["agent_id"], task)
    result = await send_to_agent(best_agent["endpoints"]["handoff"], handoff)
```

---

## Pattern 5: Error Recovery and Retry

**Use case**: Handle failures gracefully with fallback strategies.

```python
async def resilient_handoff(handoff_message, max_retries=3):
    """Handoff with automatic retry and fallback"""
    original_receiver = handoff_message["receiver"]["agent_id"]

    for attempt in range(max_retries):
        try:
            # Attempt handoff
            result = await send_to_agent(original_receiver, handoff_message)

            # Validate result
            if validate_result(result):
                return result

            # Invalid result - log and retry
            log_warning(f"Invalid result from {original_receiver}, attempt {attempt+1}")

        except TimeoutError:
            log_error(f"Timeout on attempt {attempt+1}")

        except AgentUnavailableError:
            # Try fallback agent with same capabilities
            fallback_agent = await find_fallback_agent(
                required_capabilities=handoff_message["receiver"]["required_capabilities"]
            )

            if fallback_agent:
                log_info(f"Switching to fallback agent: {fallback_agent['agent_id']}")
                handoff_message["receiver"]["agent_id"] = fallback_agent["agent_id"]
            else:
                raise NoAvailableAgentError()

        # Exponential backoff
        await asyncio.sleep(2 ** attempt)

    # All retries exhausted
    raise MaxRetriesExceededError(f"Failed after {max_retries} attempts")


async def find_fallback_agent(required_capabilities):
    """Find alternative agent with same capabilities"""
    registry = AgentRegistry()
    candidates = registry.discover(required_capabilities)

    # Filter by availability
    available = [c for c in candidates if c["availability"]["status"] == "online"]

    if not available:
        return None

    # Select best by uptime/load
    return max(available, key=lambda a: a["availability"]["uptime_sla"])
```

**Error handling best practices**:
- Always include `trace_id` in error responses
- Log failures with full context
- Implement exponential backoff
- Have fallback agents ready
- Set reasonable timeouts (30-300s)
- Return actionable error messages

---

## Validation & Schema Management

### Handoff Schema Validation

```python
from jsonschema import validate, ValidationError

# Define A2A handoff schema
A2A_HANDOFF_SCHEMA = {
    "type": "object",
    "required": ["schemaVersion", "trace_id", "sender", "receiver", "task"],
    "properties": {
        "schemaVersion": {"type": "string", "pattern": "^v[0-9]+\\.[0-9]+$"},
        "trace_id": {"type": "string"},
        "timestamp": {"type": "string", "format": "date-time"},
        "sender": {
            "type": "object",
            "required": ["agent_id", "agent_type"],
            "properties": {
                "agent_id": {"type": "string"},
                "agent_type": {"type": "string"}
            }
        },
        "receiver": {
            "type": "object",
            "required": ["agent_id", "agent_type"],
            "properties": {
                "agent_id": {"type": "string"},
                "agent_type": {"type": "string"}
            }
        },
        "task": {
            "type": "object",
            "required": ["type", "description"],
            "properties": {
                "type": {"type": "string"},
                "description": {"type": "string"},
                "priority": {"type": "string", "enum": ["low", "medium", "high", "urgent"]}
            }
        }
    }
}

def validate_handoff_schema(handoff_message):
    """Validate A2A handoff message against schema"""
    try:
        validate(instance=handoff_message, schema=A2A_HANDOFF_SCHEMA)
        return True
    except ValidationError as e:
        log_error(f"Schema validation failed: {e.message}")
        return False
```

---

## Observability & Monitoring

### Full Trace Tracking

```python
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

async def traced_handoff(handoff_message):
    """Handoff with full OpenTelemetry tracing"""
    with tracer.start_as_current_span("agent_handoff") as span:
        # Add handoff metadata to span
        span.set_attribute("a2a.trace_id", handoff_message["trace_id"])
        span.set_attribute("a2a.sender.id", handoff_message["sender"]["agent_id"])
        span.set_attribute("a2a.receiver.id", handoff_message["receiver"]["agent_id"])
        span.set_attribute("a2a.task.type", handoff_message["task"]["type"])

        try:
            result = await send_to_agent(
                handoff_message["receiver"]["agent_id"],
                handoff_message
            )

            span.set_status(Status(StatusCode.OK))
            span.set_attribute("a2a.result.status", "success")
            return result

        except Exception as e:
            span.set_status(Status(StatusCode.ERROR, str(e)))
            span.set_attribute("a2a.error", str(e))
            span.record_exception(e)
            raise
```

### Metrics to Track

```python
from prometheus_client import Counter, Histogram, Gauge

# Handoff metrics
handoff_total = Counter('a2a_handoff_total', 'Total handoffs', ['sender', 'receiver'])
handoff_duration = Histogram('a2a_handoff_duration_seconds', 'Handoff latency')
handoff_errors = Counter('a2a_handoff_errors', 'Failed handoffs', ['error_type'])
active_handoffs = Gauge('a2a_active_handoffs', 'Currently processing handoffs')

async def monitored_handoff(handoff_message):
    """Handoff with metrics"""
    sender = handoff_message["sender"]["agent_id"]
    receiver = handoff_message["receiver"]["agent_id"]

    handoff_total.labels(sender=sender, receiver=receiver).inc()
    active_handoffs.inc()

    start_time = time.time()

    try:
        result = await send_to_agent(receiver, handoff_message)
        return result
    except Exception as e:
        handoff_errors.labels(error_type=type(e).__name__).inc()
        raise
    finally:
        duration = time.time() - start_time
        handoff_duration.observe(duration)
        active_handoffs.dec()
```

---

## A2A vs MCP: When to Use What

| Scenario | Use A2A | Use MCP |
|----------|---------|---------|
| Agent needs external data | | [check] |
| Agent needs to call tools | | [check] |
| Multiple agents collaborate | [check] | |
| Task delegation | [check] | |
| Agent discovery | [check] | |
| Cross-vendor interop | [check] | |
| Database/API access | | [check] |
| Standardized tool library | | [check] |

**Complementary use**: Many systems use BOTH:
- MCP: Agent ↔ Tools/Data
- A2A: Agent ↔ Agent

---

## Production Checklist

Before deploying A2A coordination:

- [ ] All handoffs have JSON Schema validation
- [ ] `trace_id` propagates through entire workflow
- [ ] Error handling with fallback agents
- [ ] Timeouts set on all handoffs (30-300s)
- [ ] Observability: traces, metrics, logs
- [ ] Agent registry with capability discovery
- [ ] Rate limiting per agent
- [ ] Authentication between agents
- [ ] Audit log of all handoffs
- [ ] Circuit breakers for failing agents
- [ ] Health checks for all agents
- [ ] Monitoring dashboard with handoff metrics

---

## Next Steps

**After implementing A2A patterns**:
1. Test handoffs with invalid payloads (validation)
2. Simulate agent failures (resilience)
3. Monitor trace propagation (observability)
4. Benchmark handoff latency (performance)
5. Document agent capabilities (discovery)

**Related guides**:
- `frameworks/shared-foundations/protocols/a2a/a2a-architecture.md` - Full protocol specification
- `frameworks/shared-foundations/protocols/a2a/a2a-implementation.md` - Implementation details
- `frameworks/shared-foundations/protocols/a2a/a2a-examples.md` - Real-world examples
- `mcp-practical-guide.md` - For agent-to-tool integration (complementary)
- `multi-agent-patterns.md` - Additional orchestration strategies

**Official resources**:
- A2A Specification: https://a2a.anthropic.com/
- Agent Communication Best Practices: https://www.anthropic.com/research/agent-coordination

# A2A + MCP Integration Patterns

Five integration patterns for combining Agent-to-Agent (A2A) and Model Context Protocol (MCP) in multi-agent systems. A2A handles agent-to-agent communication; MCP handles agent-to-tool communication.

Source: Google Cloud Tech (@addyosmani, @Saboo_Shubham_), 2026-04-24 — <https://x.com/GoogleCloudTech/status/2047567704807346675>

Cross-link: [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md).

## Table of Contents

- [Pattern 1: Agent Card Discovery](#pattern-1-agent-card-discovery)
- [Pattern 2: Delegated Specialization](#pattern-2-delegated-specialization)
- [Pattern 3: Tool Bridge (MCP)](#pattern-3-tool-bridge-mcp)
- [Pattern 4: Cross-Organization Federation](#pattern-4-cross-organization-federation)
- [Pattern 5: Ambient Event Mesh](#pattern-5-ambient-event-mesh)
- [Stack Surfaces](#stack-surfaces)

## Pattern 1: Agent Card Discovery

- Each A2A-compatible agent publishes a JSON **Agent Card** at a well-known URL describing capabilities, auth requirements, and rate limits — like an OpenAPI spec for agent-to-agent.
- ADK auto-generates the Agent Card from the agent definition; consuming a remote agent uses the `RemoteA2aAgent` component (handles auth, serialization, error handling, result streaming).
- An **Agent Registry** lets agents discover each other across an organization without hardcoded URLs — the service mesh for the agent ecosystem.

## Pattern 2: Delegated Specialization

Coordinator-Dispatcher across team and framework boundaries. Specialist does **not** need the same framework, language, or owner — only A2A.

Example workflow (customer onboarding) crossing 5 teams / 4 languages:

| Role | Owner | Language |
|---|---|---|
| Coordinator | your team | Python |
| Identity verification | security team | Go |
| Credit assessment | risk team | Java |
| Account provisioning | platform team | Go |
| Compliance docs | legal team | Python |
| Welcome comms | marketing team | TypeScript |

Coordinator only knows each specialist's Agent Card and the A2A protocol — internal updates ship without coordinator changes.

## Pattern 3: Tool Bridge (MCP)

Single protocol replaces N custom connectors:

- ADK ships **60+ ready-to-use MCP integrations** (GitHub, Notion, Hugging Face, AgentOps, Stripe, …).
- **MCP Toolbox for Databases** connects 30+ data sources through one MCP interface.
- **Apigee API Hub** turns existing REST APIs documented in Apigee into agent-accessible tools — same governance layer (rate limit, auth, logging, ACL) that already manages API traffic.
- From the agent's perspective, an MCP tool through Stripe and an MCP tool through BigQuery look identical — the protocol is the interface, the backend is interchangeable.

## Pattern 4: Cross-Organization Federation

Each org maintains its own governance while collaborating on shared tasks via A2A:

- **Agent Gallery** in Gemini Enterprise: 100+ partner agents (Adobe, ServiceNow, Workday, Salesforce, …) validated by Google Cloud for security and interoperability.
- Your **Agent Gateway** policies control what data your agents share with external agents and what actions they can take on returned data.
- The partner agent runs under its own security model; both sides enforce boundaries independently.

Surface area saved: your agent never has to model Salesforce data internals or ServiceNow architecture — the partner agent does.

## Pattern 5: Ambient Event Mesh

A2A combined with event-driven architecture for continuous-background agents:

- **Batch and Event-Driven Agents** in Gemini Enterprise Agent Platform connect to BigQuery tables and Pub/Sub streams.
- Receiving agent decides per-event: handle locally, delegate to specialist via A2A, or escalate to human via Mission Control.
- Self-organizing — adding a fraud-detection specialist requires only registering it in Agent Registry and updating routing logic in the relevant ambient agents.
- Governance: every agent has identity via Agent Identity, every tool access governed by Agent Gateway, every interaction traced via Agent Observability — the mesh is fully observable.

## Stack Surfaces

- **A2A protocol**: ADK across Python, TypeScript, Go, Java.
- **MCP**: native ADK support; managed support for GCP databases.
- **Agent Gallery**: 100+ validated partner agents in Gemini Enterprise.
- **Codelab**: <https://codelabs.developers.google.com/instavibe-adk-multi-agents>
- **Samples**: <https://github.com/google/adk-samples>
- **Platform**: <https://cloud.google.com/products/gemini-enterprise-agent-platform> · <https://adk.dev>

# Agent Debugging Patterns

> Operational playbook for diagnosing and fixing agent failures — trace replay, failure classification, root cause analysis, and systematic debugging workflows.

**Freshness anchor:** January 2026 — covers OpenTelemetry 1.x, LangSmith v2, Langfuse 2.x, LangGraph runtime.

---
## Table of Contents

- [Failure Classification Decision Tree](#failure-classification-decision-tree)
- [Quick Reference: Failure Modes and Fixes](#quick-reference-failure-modes-and-fixes)
- [Trace Replay Debugging](#trace-replay-debugging)
- [When to Use](#when-to-use)
- [OpenTelemetry Trace Analysis](#opentelemetry-trace-analysis)
- [Instrument agent with OpenTelemetry spans](#instrument-agent-with-opentelemetry-spans)
- [Trace Analysis Checklist](#trace-analysis-checklist)
- [LangSmith Debugging Workflow](#langsmith-debugging-workflow)
- [Langfuse Debugging Workflow](#langfuse-debugging-workflow)
- [Conversation Replay Patterns](#conversation-replay-patterns)
- [Deterministic Replay Setup](#deterministic-replay-setup)
- [Save full conversation state for replay](#save-full-conversation-state-for-replay)
- [Replay with mocked tool responses](#replay-with-mocked-tool-responses)
- [Replay Comparison Checklist](#replay-comparison-checklist)
- [Log Analysis Patterns](#log-analysis-patterns)
- [Structured Logging for Agents](#structured-logging-for-agents)
- [Log every decision point](#log-every-decision-point)
- [Log Grep Patterns](#log-grep-patterns)
- [Step-Through Debugging Workflow](#step-through-debugging-workflow)
- [Manual Step-Through Protocol](#manual-step-through-protocol)
- [Breakpoint Patterns](#breakpoint-patterns)
- [Add conditional breakpoints to agent loop](#add-conditional-breakpoints-to-agent-loop)
- [Root Cause Analysis Template](#root-cause-analysis-template)
- [Anti-Patterns](#anti-patterns)
- [Cross-References](#cross-references)


## Failure Classification Decision Tree

```
Agent failed or produced wrong output
│
├── Did the agent stop responding?
│   ├── YES → Timeout / Rate Limit / Crash
│   │   ├── Check: API response codes (429, 503, 500)
│   │   ├── Check: Token limit exceeded (context window overflow)
│   │   └── Check: Tool execution hung (external API timeout)
│   └── NO → Agent produced output
│       │
│       ├── Is the output factually wrong?
│       │   ├── YES → Hallucination
│       │   │   ├── Check: Was retrieval context relevant?
│       │   │   ├── Check: Did the model ignore provided context?
│       │   │   └── Check: Was the question outside training data?
│       │   └── NO → Output is factual but wrong action
│       │       │
│       │       ├── Did the agent call the wrong tool?
│       │       │   ├── YES → Tool Selection Error
│       │       │   │   ├── Check: Tool descriptions ambiguous
│       │       │   │   ├── Check: Too many tools available (>15)
│       │       │   │   └── Check: Missing tool for the task
│       │       │   └── NO → Right tool, wrong parameters
│       │       │       ├── Check: Parameter schema unclear
│       │       │       ├── Check: Required params missing
│       │       │       └── Check: Type coercion failure
│       │       │
│       │       └── Did the agent loop?
│       │           ├── YES → Loop Detection
│       │           │   ├── Check: Identical tool calls repeated
│       │           │   ├── Check: Error→retry→same error cycle
│       │           │   └── Check: Planning→replanning without action
│       │           └── NO → Logic / Reasoning Error
│       │               ├── Check: Multi-step plan went off track
│       │               ├── Check: Misinterpreted user intent
│       │               └── Check: Lost context mid-conversation
```

---

## Quick Reference: Failure Modes and Fixes

| Failure Mode | Symptom | Root Cause | Fix |
|---|---|---|---|
| Infinite loop | Same tool called 3+ times | No exit condition in agent logic | Add max iteration guard + loop detection |
| Context overflow | Truncated responses, missing info | Conversation history too long | Implement sliding window or summarization |
| Tool timeout | Agent hangs mid-execution | External API unresponsive | Add per-tool timeout (default 30s) |
| Hallucinated tool call | Agent invokes nonexistent tool | Tool list changed between turns | Pin tool definitions per session |
| Parameter drift | Wrong types in tool arguments | Schema mismatch or ambiguous names | Add Pydantic/Zod validation on tool inputs |
| Premature termination | Agent says "done" too early | Misclassified task as complete | Add completion verification step |
| Cascading error | One tool failure breaks chain | No error recovery between steps | Add per-step error handling with fallback |
| Rate limit cascade | Multiple 429s, then crash | Burst of parallel tool calls | Implement exponential backoff + concurrency limit |

---

## Trace Replay Debugging

### When to Use
- Use when: agent produced wrong output and you need to understand the step-by-step reasoning
- Use when: reproducing a failure reported by a user
- Use when: comparing a failing run against a known-good run

### OpenTelemetry Trace Analysis

```python
# Instrument agent with OpenTelemetry spans
from opentelemetry import trace

tracer = trace.get_tracer("agent.debugging")

def run_agent_step(step_input):
    with tracer.start_as_current_span("agent.step") as span:
        span.set_attribute("step.input_tokens", count_tokens(step_input))
        span.set_attribute("step.tool_name", step_input.get("tool", "none"))
        span.set_attribute("step.iteration", step_input.get("iteration", 0))

        result = execute_step(step_input)

        span.set_attribute("step.output_tokens", count_tokens(result))
        span.set_attribute("step.status", "success" if result.ok else "error")
        span.set_attribute("step.error_type", result.error_type or "none")
        return result
```

### Trace Analysis Checklist

- [ ] Export trace as JSON from collector (Jaeger, Zipkin, or Grafana Tempo)
- [ ] Identify the span where behavior diverged from expected
- [ ] Check span attributes for token counts (context overflow indicator)
- [ ] Compare tool call sequence against expected plan
- [ ] Look for retry spans (indicates transient failures)
- [ ] Check latency per span (identify bottleneck tools)
- [ ] Verify parent-child span relationships (correct nesting)

### LangSmith Debugging Workflow

| Step | Action | What to Look For |
|---|---|---|
| 1 | Open failing run in LangSmith UI | Red status indicators on steps |
| 2 | Expand each LLM call | Full prompt sent and response received |
| 3 | Check retrieval steps | Were correct documents retrieved? |
| 4 | Compare tool inputs/outputs | Did tool return expected format? |
| 5 | Inspect token usage per step | Approaching context limit? |
| 6 | Use "Compare" view | Diff against a successful run |
| 7 | Tag run for regression set | Add to golden test dataset |

### Langfuse Debugging Workflow

| Step | Action | What to Look For |
|---|---|---|
| 1 | Filter traces by error score | Focus on lowest-scoring runs |
| 2 | Open trace timeline view | Identify where the chain broke |
| 3 | Check generation details | Model, temperature, token counts |
| 4 | Review observation scores | Human or automated eval scores |
| 5 | Inspect event metadata | Custom attributes logged by agent |
| 6 | Export trace for local replay | Reproduce with identical inputs |

---

## Conversation Replay Patterns

### Deterministic Replay Setup

```python
# Save full conversation state for replay
import json
from datetime import datetime

class ConversationRecorder:
    def __init__(self, session_id: str):
        self.session_id = session_id
        self.events = []

    def record(self, event_type: str, data: dict):
        self.events.append({
            "timestamp": datetime.utcnow().isoformat(),
            "type": event_type,  # "user_input", "llm_call", "tool_call", "tool_result"
            "data": data
        })

    def save(self, path: str):
        with open(path, "w") as f:
            json.dump({
                "session_id": self.session_id,
                "events": self.events
            }, f, indent=2)

# Replay with mocked tool responses
class ConversationReplayer:
    def __init__(self, recording_path: str):
        with open(recording_path) as f:
            self.recording = json.load(f)
        self.tool_responses = self._extract_tool_responses()

    def _extract_tool_responses(self) -> dict:
        responses = {}
        for event in self.recording["events"]:
            if event["type"] == "tool_result":
                key = event["data"]["tool_call_id"]
                responses[key] = event["data"]["result"]
        return responses
```

### Replay Comparison Checklist

- [ ] Replay produces same tool call sequence
- [ ] If different: identify first divergence point
- [ ] Check if divergence is due to model non-determinism (set temperature=0)
- [ ] Check if divergence is due to changed tool responses
- [ ] Check if system prompt or tools changed between original and replay
- [ ] Document the delta for root cause analysis

---

## Log Analysis Patterns

### Structured Logging for Agents

```python
import structlog

logger = structlog.get_logger()

# Log every decision point
logger.info("agent.planning",
    task=user_query,
    available_tools=[t.name for t in tools],
    selected_plan=plan.steps,
    confidence=plan.confidence
)

logger.info("agent.tool_call",
    tool=tool_name,
    params=sanitized_params,  # redact PII
    attempt=retry_count,
    timeout_ms=timeout
)

logger.info("agent.tool_result",
    tool=tool_name,
    status="success" | "error",
    result_tokens=token_count,
    latency_ms=elapsed
)

logger.info("agent.step_complete",
    iteration=step_number,
    total_tokens_used=cumulative_tokens,
    remaining_budget=max_tokens - cumulative_tokens
)
```

### Log Grep Patterns

| What You're Looking For | grep/rg Pattern |
|---|---|
| All errors in a session | `rg "status.*error" --json \| jq '.session_id=="<id>"'` |
| Loop detection | `rg "agent.tool_call" \| sort \| uniq -c \| sort -rn` |
| Token budget exhaustion | `rg "remaining_budget" \| awk '$NF < 1000'` |
| Slow tools | `rg "latency_ms" \| awk '$NF > 5000'` |
| Rate limit hits | `rg "429\|rate.limit\|too.many.requests"` |
| Context window overflow | `rg "context_length_exceeded\|max_tokens"` |

---

## Step-Through Debugging Workflow

### Manual Step-Through Protocol

| Step | Action | Decision |
|---|---|---|
| 1 | Freeze agent at step N | Inspect full state before LLM call |
| 2 | Print the exact prompt being sent | Is context correct and complete? |
| 3 | Count tokens in prompt | Within model's context window? |
| 4 | Run LLM call in isolation | Does response make sense given prompt? |
| 5 | If tool call: validate parameters | Do params match tool schema? |
| 6 | Execute tool with validated params | Does tool return expected format? |
| 7 | Feed tool result back to agent | Does agent interpret result correctly? |
| 8 | Advance to step N+1 | Repeat until failure point found |

### Breakpoint Patterns

```python
# Add conditional breakpoints to agent loop
class DebuggableAgent:
    def __init__(self, agent, breakpoints=None):
        self.agent = agent
        self.breakpoints = breakpoints or {}

    async def run(self, input_msg):
        for step in self.agent.iterate(input_msg):
            # Break on specific tool
            if step.tool_name in self.breakpoints.get("tools", []):
                await self._debug_pause(step, "tool_breakpoint")

            # Break on high token usage
            if step.total_tokens > self.breakpoints.get("max_tokens", float("inf")):
                await self._debug_pause(step, "token_limit")

            # Break on Nth iteration
            if step.iteration >= self.breakpoints.get("max_iterations", float("inf")):
                await self._debug_pause(step, "iteration_limit")

            # Break on error
            if step.status == "error" and self.breakpoints.get("break_on_error", False):
                await self._debug_pause(step, "error")

    async def _debug_pause(self, step, reason):
        print(f"BREAKPOINT [{reason}] at step {step.iteration}")
        print(f"  Tool: {step.tool_name}")
        print(f"  Tokens used: {step.total_tokens}")
        print(f"  Last result: {step.last_result[:200]}")
        input("Press Enter to continue...")
```

---

## Root Cause Analysis Template

```
INCIDENT: [Brief description]
SEVERITY: [P0-P3]
SESSION ID: [trace/session identifier]

TIMELINE:
- [timestamp] Step N: [what happened]
- [timestamp] Step N+1: [what happened]
- [timestamp] Failure point: [what went wrong]

ROOT CAUSE:
- Category: [tool_error | hallucination | loop | timeout | logic_error]
- Specific cause: [detailed explanation]
- Contributing factors: [list]

FIX:
- Immediate: [what was done to resolve]
- Preventive: [what will prevent recurrence]
- Detection: [what monitoring/alert catches this faster]

REGRESSION TEST:
- Input: [the failing input]
- Expected: [correct behavior]
- Added to: [test suite name]
```

---

## Anti-Patterns

| Anti-Pattern | Why It Fails | Better Approach |
|---|---|---|
| Logging only final output | Cannot trace intermediate failures | Log every LLM call + tool call + result |
| Retrying without classification | Infinite retry on permanent failures | Classify error as transient vs permanent first |
| Debugging in production | Risk of side effects, no reproducibility | Replay traces locally with mocked tools |
| Adding print statements | No structure, lost after session | Use structured logging with trace IDs |
| Ignoring token counts | Miss context overflow as root cause | Track cumulative tokens at every step |
| Testing with real APIs only | Flaky, slow, expensive | Mock tool responses for deterministic tests |
| No max iteration guard | Agent can loop forever | Hard limit of 10-25 iterations per task |
| Debugging the model instead of the prompt | Model behavior is a function of input | Focus on what the prompt/context contains |

---

## Cross-References

- `guardrails-implementation.md` — layer guardrails to prevent many failure modes
- `voice-multimodal-agents.md` — modality-specific debugging patterns
- `../ai-llm/references/structured-output-patterns.md` — output parsing failures
- `../ai-llm-inference/references/streaming-patterns.md` — mid-stream error handling
- `../ai-prompt-engineering/references/prompt-testing-ci-cd.md` — regression test infrastructure

# Agent Delivery Methods

Practical comparison of the delivery methods and planning systems that matter for AI-assisted coding work in March 2026.

Use this reference when you need to choose how much structure, review, and governance to add around coding agents. Do not treat these methods as interchangeable agent runtimes. Most of them are delivery systems layered on top of existing agent tools.

## Table Of Contents

- [Method Map](#method-map)
- [Popularity Ranking (March 2026)](#popularity-ranking-march-2026)
- [Cross-Cutting Practices](#cross-cutting-practices)
- [Patterns Worth Reusing](#patterns-worth-reusing)
- [Selection Guide](#selection-guide)
- [Layering Guide](#layering-guide)
- [Adoption Rule](#adoption-rule)
- [Primary Sources](#primary-sources)

## Method Map

| Method | Type | Core Artifacts | Best Fit | Main Caution |
|---|---|---|---|---|
| **Get Shit Done (GSD)** | Lightweight delivery workflow | Discussion notes, plan files, phase state | Solo builders and brownfield work that needs speed without losing state | Less formal governance than team-oriented systems |
| **BMAD Method / BMAD v6** | Role-based workflow and Agent-as-Code delivery system | Role chain, blueprints, policies, replayable runs | Teams that want stronger planning, verification, and auditable execution | More ceremony; v6 is still alpha |
| **GitHub Spec Kit** | Spec-driven development toolkit | Constitution, spec, clarify output, plan, tasks | Teams that want explicit handoff from intent to implementation | Heavier artifact flow than lighter tools |
| **OpenSpec** | Lightweight spec layer | Proposal, design, tasks, spec deltas | Repo-native planning with less ceremony than full SDD systems | Lighter governance and enforcement by design |
| **MADD** | Multi-agent delivery methodology | Intention docs, contracts, retro-spec, independent audit | High-risk work where self-validation bias is the main failure mode | Requires more orchestration discipline |
| **AI-SDLC** | Governance and orchestration layer | Declarative policy, gates, reconciliation state | Enterprises that need audit, policy, and quality gates across agent runs | Not a replacement for a day-to-day delivery workflow |
| **Kiro** | Spec-driven IDE (Amazon) | Requirements, design docs, task lists | AWS-native teams wanting structured agent workflows inside an IDE | Vendor-locked to Kiro IDE; spec lifecycle is IDE-managed |
| **TaskMaster** | AI task management via MCP | PRD → structured tasks with dependencies and complexity scores | Teams that want automated task decomposition from PRDs, works across Cursor/Claude Code/Windsurf | Task management layer, not a full delivery workflow |
| **Ralph Loop** | Autonomous agent loop pattern | PRD, loop state, iteration history | Long-running autonomous work (documented: 37h, 250 tasks from 2000-line PRD) | Requires strong acceptance criteria; unbounded loops risk runaway cost |
| **Agent OS** | Standards injection system | Codebase standards, spec shaping, standard index | Teams wanting to extract and enforce existing codebase conventions across agent work | v3 defers spec writing to Plan Mode; not a standalone delivery system |
| **Superpowers** | Discipline-enforcing workflow plugin | TDD skills, brainstorming, debugging, code review, subagent dispatch | Teams that want enforced engineering discipline (TDD, design-before-code, mandatory review) baked into the agent | Claude Code only; opinionated — deletes code written before tests |
| **Tessl Framework** | Spec-as-source SDD platform | Specs, vibe-specs, spec registry, spec deltas | Teams pursuing spec-as-source — specs are the primary artifact, code is generated to match | Closed beta; most aggressive SDD stance — requires buy-in to spec-first culture |
| **JetBrains Central** | Enterprise agent orchestration platform | Agent connections to repos, pipelines, infra, knowledge bases | Enterprise teams wanting unified agent management across JetBrains toolchain | EAP Q2 2026; not yet generally available |

## Popularity Ranking (March 2026)

Ranked by GitHub stars as a proxy for developer adoption. Stars are approximate and change daily.

| # | Method | GitHub Stars | Type | Trend |
|---|--------|-------------|------|-------|
| 1 | **Superpowers** | ~107K | Discipline plugin | Fastest-growing; ~2K stars/day |
| 2 | **GitHub Spec Kit** | ~75K | Spec-driven toolkit | Strong; GitHub-backed |
| 3 | **BMAD Method** | ~37K | Role-based workflow | Steady growth; enterprise adoption |
| 4 | **GSD** | ~35K | Lightweight workflow | Rapid; ~4.5K stars/week |
| 5 | **OpenSpec** | ~28K | Lightweight spec layer | Steady; community-driven |
| 6 | **TaskMaster** | ~25K | Task management | Mature; 90+ releases |
| 7 | **Ralph Loop** | ~10K | Autonomous loop | Niche but viral pattern |
| 8 | **Agent OS** | ~3K (est.) | Standards injection | Smaller community; v3 refocus |
| 9 | **Kiro** | N/A (proprietary IDE) | Spec-driven IDE | Amazon-backed; closed-source |
| 10 | **Tessl** | N/A (closed beta) | Spec-as-source | Funded startup; Martin Fowler coverage |
| 11 | **MADD** | <1K (est.) | Multi-agent methodology | Niche; methodology-focused |
| 12 | **AI-SDLC** | <1K (est.) | Governance layer | Niche; enterprise governance |
| 13 | **JetBrains Central** | N/A (EAP Q2 2026) | Enterprise orchestration | Pre-release; JetBrains-backed |

**Reading the ranking:** Stars measure awareness, not quality. Superpowers leads because it's a plugin (low adoption friction) installed via one command. Spec Kit benefits from GitHub's distribution. The most *methodologically complete* systems (BMAD, MADD, AI-SDLC) have fewer stars because they require more commitment. Match by your failure mode, not by star count.

## Cross-Cutting Practices

These are not delivery methods — they layer on top of any method above.

| Practice | What It Does | When to Add | Key Reference |
|----------|-------------|-------------|---------------|
| **Targeted test context (TDAD)** | Provides agents with source→test dependency maps instead of generic "write tests" instructions | Any coding agent work; reduces regressions by ~70% vs. procedural TDD prompting alone | [arXiv:2603.17973](https://arxiv.org/abs/2603.17973) |
| **Classic TDD** | Write failing test → implement → pass → refactor | When the task genuinely starts from a behavioral specification; pair with TDAD for best results | Agentic Coding Handbook |
| **Independent audit** | Separate agent reviews implementation it did not write | High-risk changes, security-sensitive code, compliance work | MADD pattern |
| **Collaborative debate** | Multi-persona discussion before fan-out to resolve tradeoffs | Architecture decisions affecting multiple workers | BMAD Party Mode; templates in `agents-subagents/assets/templates/debate-*` |
| **Fresh-context spawning** | Each worker gets a clean context with only its task brief | Any parallel or long-running agent work to prevent context rot | GSD thin orchestrator |
| **Durable file-based state** | Plans, progress, decisions persist in repo files (YAML frontmatter + markdown) | Any work spanning multiple sessions or agent restarts | GSD, BMAD, OpenSpec |
| **Enforced TDD + review** | Plugin deletes code written before tests; mandatory code review after implementation | When discipline enforcement matters more than developer freedom | Superpowers plugin |

## Patterns Worth Reusing

You do not need to adopt a method wholesale to benefit from it.

- **Scale-adaptive planning**: use lightweight planning for bounded fixes and deeper spec-first planning for migrations, multi-service work, and risky changes.
- **Plan -> build -> verify boundaries**: freeze the intended outcome before implementation, then verify against explicit acceptance checks rather than "looks good."
- **Versioned agent definitions**: keep roles, constraints, tool access, and success criteria in reviewable artifacts rather than one-off chat prompts.
- **Fresh-context workers**: spawn workers with only the task brief, ownership boundaries, and interface contracts they need.
- **Durable external state**: keep plans, decisions, progress, and dependency outputs in repo files rather than in conversational memory.
- **Collaborative debate before fan-out**: resolve architecture or tradeoff disputes before dispatching parallel workers.
- **Independent validation**: separate implementation from audit when the change is risky enough that self-review is not trustworthy.
- **Declarative policy and replay**: add run manifests, policy gates, and replay when governance or compliance matters more than raw speed.

## Selection Guide

### Solo or small brownfield work

Default to:

- GSD when you want speed plus durable state
- OpenSpec when you want visible repo artifacts and low ceremony
- Superpowers when you want enforced TDD discipline and mandatory review on Claude Code
- Ralph Loop when work is long-running and autonomous with a clear PRD
- classic TDD + TDAD test context on top

### Small product team

Default to:

- Spec Kit when the team benefits from explicit requirement and plan artifacts
- BMAD when role separation, verification, and traceability matter more than minimal ceremony
- OpenSpec when the team wants a lighter repo-native planning layer
- TaskMaster when the team wants automated PRD → task decomposition across multiple agent tools
- Agent OS when codebase conventions need to be extracted and enforced consistently

### Spec-as-source teams

Default to:

- Tessl when specs are the primary artifact and code is generated to match them
- Kiro when the team wants spec-driven workflows inside an IDE with tight AWS integration

### JetBrains-native enterprise teams

Default to:

- JetBrains Central when the team needs unified agent orchestration across JetBrains IDEs and CI (EAP Q2 2026)

### High-risk or compliance-heavy delivery

Default to:

- BMAD or Spec Kit for the delivery structure
- MADD-style independent audit for high-risk changes
- AI-SDLC when you need org-level policy, gates, reconciliation, and audit trails

### Long-running autonomous execution

Default to:

- Ralph Loop for unbounded iteration toward a PRD (set cost/time caps)
- GSD for bounded autonomous work with wave-based parallelism
- Both benefit from TDAD test context to prevent regression accumulation over many iterations

## Layering Guide

Methods compose. Pick a delivery workflow, then add cross-cutting practices as needed.

```text
┌──────────────────────────────────────────────────────────────────┐
│                       GOVERNANCE LAYER                            │
│       AI-SDLC │ JetBrains Central (policy, audit, replay)        │
├──────────────────────────────────────────────────────────────────┤
│                       DELIVERY WORKFLOW                           │
│  GSD │ BMAD │ Spec Kit │ OpenSpec │ Kiro │ Ralph Loop │ Tessl   │
├──────────────────────────────────────────────────────────────────┤
│                     CROSS-CUTTING PRACTICES                       │
│  TDAD test context │ Fresh-context │ Collaborative debate         │
│  Durable state │ Independent audit │ Classic TDD │ Enforced TDD  │
├──────────────────────────────────────────────────────────────────┤
│                       TASK MANAGEMENT                             │
│         TaskMaster │ Agent OS │ Plan Mode │ Superpowers           │
├──────────────────────────────────────────────────────────────────┤
│                       AGENT RUNTIME                               │
│   Claude Code │ Cursor │ Codex │ Windsurf │ Copilot │ Kiro IDE   │
└──────────────────────────────────────────────────────────────────┘
```

Typical combinations:

| Profile | Delivery | Cross-Cutting | Task Layer | Runtime |
|---------|----------|---------------|------------|---------|
| Solo hacker | GSD | TDAD + fresh-context | Plan Mode | Claude Code |
| Solo disciplined | GSD | Enforced TDD + TDAD | Superpowers | Claude Code |
| Startup team | Spec Kit or BMAD | TDAD + debate + durable state | TaskMaster | Cursor or Claude Code |
| Spec-as-source | Tessl | TDAD + durable state | Built-in | Any |
| Enterprise | BMAD + AI-SDLC | All practices + independent audit | TaskMaster | Any |
| JetBrains enterprise | BMAD + JetBrains Central | All practices + independent audit | Built-in | JetBrains IDEs |
| Autonomous run | Ralph Loop or GSD | TDAD + fresh-context + durable state | Built-in | Claude Code or Codex |

## Adoption Rule

Adopt the smallest method that fixes your actual failure mode:

- if the problem is **scope ambiguity**, use spec-driven methods (Spec Kit, BMAD, Kiro)
- if the problem is **context drift**, use fresh-context workers plus durable external state (GSD pattern)
- if the problem is **regressions**, add TDAD test context before adding more process
- if the problem is **self-review bias**, add independent audit (MADD pattern)
- if the problem is **task decomposition**, add TaskMaster or Agent OS
- if the problem is **policy and audit**, add declarative governance (AI-SDLC)
- if the problem is **long-running execution**, add Ralph Loop with cost/time caps
- if the problem is **agent discipline** (skipping tests, writing code before design), add Superpowers enforced TDD
- if the problem is **spec drift** (code diverging from intent over time), use Tessl spec-as-source

Avoid copying the full ceremony of a method when only one pattern is needed.

## Primary Sources

- GSD: <https://github.com/gsd-build/get-shit-done>
- BMAD Method docs: <https://docs.bmad-method.org/>
- BMAD v6 Alpha: <https://bmadcodes.com/v6-alpha/>
- GitHub Spec Kit: <https://github.com/github/spec-kit>
- OpenSpec: <https://openspec.dev/>
- OpenSpec repo: <https://github.com/Fission-AI/OpenSpec>
- MADD: <https://madd.sh/>
- AI-SDLC primer: <https://ai-sdlc.io/docs/spec/primer>
- Kiro: <https://kiro.dev/>
- TaskMaster: <https://www.task-master.dev/>
- Ralph Loop: <https://github.com/snarktank/ralph>
- Agent OS: <https://buildermethods.com/agent-os>
- Agent OS repo: <https://github.com/buildermethods/agent-os>
- TDAD paper: <https://arxiv.org/abs/2603.17973>
- SDD ecosystem map (30+ frameworks): <https://medium.com/@visrow/spec-driven-development-is-eating-software-engineering-a-map-of-30-agentic-coding-frameworks-6ac0b5e2b484>
- SDD tools comparison (Martin Fowler): <https://martinfowler.com/articles/exploring-gen-ai/sdd-3-tools.html>
- Agentic Coding Handbook (TDD chapter): <https://tweag.github.io/agentic-coding-handbook/WORKFLOW_TDD/>
- Superpowers: <https://github.com/obra/superpowers>
- Superpowers (Anthropic plugin page): <https://claude.com/plugins/superpowers>
- Tessl Framework: <https://tessl.io/>
- Tessl SDD docs: <https://docs.tessl.io/use/spec-driven-development-with-tessl>
- JetBrains Central announcement: <https://blog.jetbrains.com/blog/2026/03/24/introducing-jetbrains-central-an-open-system-for-agentic-software-development/>

# Agent Economics & ROI Framework

**Purpose**: Business-focused decision framework for agent investments — token costs, ROI calculation, hallucination impact, and when to kill an agent project.

No theory. No narrative. Only what you can calculate and decide.

---
## Table of Contents

- [Token Economics (July 2026 Pricing)](#token-economics-july-2026-pricing)
- [Cost Per Model (USD per 1M tokens)](#cost-per-model-usd-per-1m-tokens)
- [Agent Task Cost Estimates](#agent-task-cost-estimates)
- [Monthly Cost Projections](#monthly-cost-projections)
- [Agent ROI Framework](#agent-roi-framework)
- [ROI Calculation Formula](#roi-calculation-formula)
- [Cost Categories (Annual)](#cost-categories-annual)
- [Value Categories (Annual)](#value-categories-annual)
- [ROI Tiers](#roi-tiers)
- [Hallucination Cost Framework](#hallucination-cost-framework)
- [Hallucination Impact Categories](#hallucination-impact-categories)
- [Hallucination Rate Benchmarks (2026)](#hallucination-rate-benchmarks-2026)
- [Hallucination Cost Calculator](#hallucination-cost-calculator)
- [Mitigation Investment Framework](#mitigation-investment-framework)
- [Agent Investment Decision Matrix](#agent-investment-decision-matrix)
- [Quick Filters (Kill Early)](#quick-filters-kill-early)
- [Investment Decision Tree](#investment-decision-tree)
- [When to Kill an Agent Project](#when-to-kill-an-agent-project)
- [Kill Signals (Any One = Stop)](#kill-signals-any-one-=-stop)
- [Pivot vs Kill Decision](#pivot-vs-kill-decision)
- [ROI Tracking Dashboard](#roi-tracking-dashboard)
- [Metrics to Track Weekly](#metrics-to-track-weekly)
- [Monthly ROI Report Template](#monthly-roi-report-template)
- [Agent ROI Report - [Month]](#agent-roi-report-month)
- [Summary](#summary)
- [Quality Metrics](#quality-metrics)
- [Cost Breakdown](#cost-breakdown)
- [Recommendation](#recommendation)
- [Quick Reference: Economics Formulas](#quick-reference-economics-formulas)
- [Break-even volume](#break-even-volume)
- [Payback period (months)](#payback-period-months)
- [Hallucination budget](#hallucination-budget)
- [Token efficiency target](#token-efficiency-target)
- [Scaling threshold](#scaling-threshold)
- [Related References](#related-references)


## Token Economics (July 2026 Pricing)

Prices move quarterly; treat this table as decision-scale anchors and verify against provider pricing docs before quoting in a deliverable.

### Cost Per Model (USD per 1M tokens)

| Model | Input | Output | Cached Input | Notes |
|-------|-------|--------|--------------|-------|
| **GPT-5.5** | $5.00 | $30.00 | $0.50 | Flagship, 1M context |
| **GPT-5.4** | $2.50 | $15.00 | $0.25 | Mid-tier workhorse |
| **GPT-5.4 mini** | $0.75 | $4.50 | ~0.1x input | High-volume, simple tasks |
| **Claude Opus 4.8** | $5.00 | $25.00 | $0.50 | Flagship coding/agents |
| **Claude Sonnet 4.6** | $3.00 | $15.00 | $0.30 | Coding/reasoning workhorse |
| **Claude Haiku 4.5** | $1.00 | $5.00 | $0.10 | Fast, cheap classification |
| **Gemini 3.1 Pro** | $2.00 | $12.00 | see docs | Value flagship (<=200K prompt tier) |
| **Gemini 3.5 Flash** | $1.50 | $9.00 | see docs | Fast mid-tier |
| **Gemini 3.1 Flash-Lite** | $0.25 | $1.50 | see docs | Cheapest for simple tasks |

Batch APIs run ~50% off list at all three providers; prompt caching cuts cached input ~90% (Anthropic/OpenAI).

### Agent Task Cost Estimates

| Agent Type | Avg Tokens/Task | Cost/Task (GPT-5.4) | Cost/Task (Haiku 4.5) |
|------------|-----------------|---------------------|------------------------|
| Simple Q&A | 2K in + 500 out | $0.013 | $0.005 |
| RAG Query | 8K in + 1K out | $0.035 | $0.013 |
| Tool-Using (3 calls) | 15K in + 3K out | $0.08 | $0.03 |
| Code Generation | 10K in + 2K out | $0.055 | $0.02 |
| Multi-Agent (5 steps) | 50K in + 10K out | $0.28 | $0.10 |
| Agentic Coding Session | 200K in + 50K out | $1.25 | $0.45 |

### Monthly Cost Projections

Mid-tier model (GPT-5.4 / Sonnet-class) without caching; caching and batch typically cut these 40-70%.

| Volume | Simple Agent | RAG Agent | Tool Agent | Multi-Agent |
|--------|--------------|-----------|------------|-------------|
| 1K tasks/day | $375/mo | $1,050/mo | $2,475/mo | $8,250/mo |
| 10K tasks/day | $3,750/mo | $10,500/mo | $24,750/mo | $82,500/mo |
| 100K tasks/day | $37,500/mo | $105,000/mo | $247,500/mo | $825,000/mo |

---

## Agent ROI Framework

### ROI Calculation Formula

```text
Agent ROI = (Value Created - Total Cost) / Total Cost × 100%

Where:
- Value Created = (Tasks Automated × Human Cost/Task) + Revenue Impact
- Total Cost = Development + Infrastructure + LLM Costs + Maintenance + Error Costs
```

### Cost Categories (Annual)

| Category | Components | Typical Range |
|----------|------------|---------------|
| **Development** | Engineering time, testing, iteration | $50K - $500K |
| **Infrastructure** | Compute, vector DB, monitoring | $12K - $120K |
| **LLM API Costs** | Token usage (see above) | $3.6K - $800K |
| **Maintenance** | Prompt tuning, bug fixes, updates | 20-40% of dev cost |
| **Error/Hallucination** | Human review, corrections, customer impact | 5-30% of LLM cost |

### Value Categories (Annual)

| Value Type | Measurement | Example |
|------------|-------------|---------|
| **Labor Savings** | Hours saved × hourly cost | 10K hrs × $50 = $500K |
| **Speed Premium** | Faster delivery × value | 50% faster × $200K = $100K |
| **Scale Enablement** | Tasks impossible without agent | 100K queries × $5 value = $500K |
| **Quality Improvement** | Error reduction × error cost | 50% fewer errors × $100K = $50K |
| **Revenue Lift** | Conversion improvement × revenue | 2% lift × $5M = $100K |

### ROI Tiers

| ROI | Assessment | Action |
|-----|------------|--------|
| **<0%** | Negative ROI | Kill or pivot immediately |
| **0-50%** | Marginal | Optimize costs or scope |
| **50-200%** | Healthy | Scale and maintain |
| **200-500%** | Strong | Expand use cases |
| **>500%** | Exceptional | Productize or license |

---

## Hallucination Cost Framework

### Hallucination Impact Categories

| Category | Description | Cost Multiplier |
|----------|-------------|-----------------|
| **Benign** | User notices, asks for correction | 1.5x task cost |
| **Annoying** | User loses trust, abandons task | 3x task cost + churn risk |
| **Costly** | Wrong action taken, needs reversal | 10-100x task cost |
| **Dangerous** | Legal, safety, or compliance violation | $10K - $10M per incident |

### Hallucination Rate Benchmarks (2026)

| Agent Type | Baseline Rate | With Guardrails | Best Achievable |
|------------|---------------|-----------------|-----------------|
| Simple Q&A | 5-10% | 2-5% | <1% |
| RAG (good retrieval) | 3-8% | 1-3% | <0.5% |
| Tool-Using | 8-15% | 3-8% | 1-3% |
| Code Generation | 10-20% | 5-10% | 2-5% |
| Multi-Agent | 15-25% | 8-15% | 3-8% |

### Hallucination Cost Calculator

```text
Monthly Hallucination Cost =
  Tasks × Hallucination Rate × Avg Impact Cost

Example (10K RAG queries/day, 3% rate, $5 avg impact):
  300,000 × 0.03 × $5 = $45,000/month
```

### Mitigation Investment Framework

| Mitigation | Implementation Cost | Hallucination Reduction | ROI Threshold |
|------------|--------------------|-----------------------|---------------|
| Better prompts | $5-10K | 20-40% | >$2K/mo hallucination cost |
| RAG grounding | $20-50K | 40-60% | >$10K/mo hallucination cost |
| Multi-layer guardrails | $30-80K | 50-70% | >$20K/mo hallucination cost |
| Human-in-the-loop | $50-150K | 80-95% | >$50K/mo hallucination cost |
| Fine-tuning | $100-500K | 60-80% | >$100K/mo hallucination cost |

---

## Agent Investment Decision Matrix

### Quick Filters (Kill Early)

**Do NOT build an agent if:**

| Red Flag | Reason | Alternative |
|----------|--------|-------------|
| <100 tasks/month | ROI never positive | Manual process or simple automation |
| >$100/task human cost acceptable | Agent won't beat human quality | Keep humans |
| Hallucination cost >$1K/incident | Risk too high without massive guardrails | Human-in-the-loop only |
| No clear success metric | Can't prove value | Define metrics first |
| Data quality <80% | Garbage in, garbage out | Fix data first |
| Regulatory requires 100% accuracy | Agents can't guarantee this | Human review required |

### Investment Decision Tree

```text
Should you build an agent?
│
├─ Task volume >1000/month?
│   ├─ No → Don't build (manual is cheaper)
│   └─ Yes → Continue
│       │
│       ├─ Human cost >$10/task?
│       │   ├─ No → Don't build (agent likely more expensive)
│       │   └─ Yes → Continue
│       │       │
│       │       ├─ Hallucination cost <$50/incident?
│       │       │   ├─ No → Build with heavy guardrails + HITL
│       │       │   └─ Yes → Continue
│       │       │       │
│       │       │       ├─ Task is structured/repeatable?
│       │       │       │   ├─ No → Consider simpler automation
│       │       │       │   └─ Yes → BUILD AGENT
│       │       │       │
│       │       │       └─ Projected ROI >100%?
│       │       │           ├─ No → Optimize scope first
│       │       │           └─ Yes → BUILD AGENT
```

---

## When to Kill an Agent Project

### Kill Signals (Any One = Stop)

| Signal | Threshold | Measurement |
|--------|-----------|-------------|
| Negative ROI after 3 months | <0% | Monthly cost vs value |
| Hallucination rate not improving | >10% after 2 iterations | Error tracking |
| User adoption <20% | After 1 month post-launch | Active users / eligible users |
| LLM costs >2x projection | For 2 consecutive months | API billing |
| Maintenance >50% of dev time | Sustained over 1 month | Engineering hours |
| Compliance/legal concerns raised | Any | Legal review |

### Pivot vs Kill Decision

| Situation | Action | Criteria |
|-----------|--------|----------|
| High value, high cost | Optimize | Value >2x cost, clear optimization path |
| High value, quality issues | Invest in guardrails | Users want it, hallucinations fixable |
| Low value, low cost | Maintain minimally | <$1K/mo, no active complaints |
| Low value, high cost | **KILL** | Sunk cost fallacy - stop now |
| High risk, any ROI | **KILL or heavy HITL** | Legal/safety risks not worth it |

---

## ROI Tracking Dashboard

> **Actual cost data**: To feed real token and cost numbers into this dashboard from Claude Code or Codex CLI sessions, see [`coding-agent-usage-tracking.md`](coding-agent-usage-tracking.md).

### Metrics to Track Weekly

| Metric | Formula | Target |
|--------|---------|--------|
| **Cost per Task** | Total LLM cost / completed tasks | Decreasing |
| **Error Rate** | Failed tasks / total tasks | <5% |
| **Hallucination Rate** | Human-flagged errors / total tasks | <3% |
| **Automation Rate** | Agent-completed / total eligible | >80% |
| **User Satisfaction** | CSAT or NPS | >4.0/5 or >30 NPS |
| **Time Saved** | Avg human time × tasks automated | Increasing |

### Monthly ROI Report Template

```markdown
## Agent ROI Report - [Month]

### Summary
- **Total Tasks**: X
- **Total Cost**: $X (LLM: $X, Infra: $X, Maintenance: $X)
- **Value Created**: $X (Labor: $X, Speed: $X, Quality: $X)
- **Net ROI**: X%

### Quality Metrics
- Hallucination Rate: X% (target: <3%)
- Error Rate: X% (target: <5%)
- Human Escalation Rate: X%

### Cost Breakdown
- Cost per Task: $X (vs $X human cost)
- LLM Efficiency: X tokens/task (vs X last month)

### Recommendation
[ ] Scale  [ ] Maintain  [ ] Optimize  [ ] Kill
```

---

## Quick Reference: Economics Formulas

```text
# Break-even volume
Break-even = Fixed Costs / (Human Cost/Task - Agent Cost/Task)

# Payback period (months)
Payback = Development Cost / (Monthly Value - Monthly Operating Cost)

# Hallucination budget
Max Hallucination Rate = Acceptable Error Cost / (Tasks × Avg Impact Cost)

# Token efficiency target
Target Tokens/Task = Budget / (Tasks × Cost/Token)

# Scaling threshold
Scale when: ROI >200% AND Error Rate <5% AND Adoption >80%
```

---

## Related References

- [Agent Maturity & Governance](agent-maturity-governance.md) — Capability levels and rollout risk
- [Evaluation & Observability](evaluation-and-observability.md) — Metrics and monitoring
- [Deployment, CI/CD & Safety](deployment-ci-cd-and-safety.md) — Production guardrails
- [Coding Agent Usage Tracking](coding-agent-usage-tracking.md) — Measure actual CLI token spend with ccusage

# Agent Maturity & Governance — Fleet Management Framework

**Purpose**: Capability maturity levels, identity management, policy enforcement, and fleet-wide governance for production agent systems.

---
## Table of Contents

- [Capability Maturity Levels](#capability-maturity-levels)
- [Persona + Domain Knowledge](#persona-domain-knowledge)
- [Identity & Policy](#identity-&-policy)
- [Fleet Control](#fleet-control)
- [Version Management](#version-management)
- [Fleet Metrics](#fleet-metrics)
- [Compliance & Auditing](#compliance-&-auditing)
- [Related Resources](#related-resources)
- [Usage Notes](#usage-notes)


## Capability Maturity Levels

**Five-Level Progression**:

| Level | Name | Capabilities | Governance Requirements |
|-------|------|--------------|------------------------|
| **L0** | Static Reasoning | Fixed responses, no tools | Basic content filtering |
| **L1** | Tool-Using | API/function calls | Tool allowlist, parameter validation |
| **L2** | Strategic Planner | Multi-step planning, ReAct loop | Approval gates, trajectory logging |
| **L3** | Collaborative Multi-Agent | Task delegation, handoffs | Contract validation, trace propagation |
| **L4** | Self-Evolving | Policy learning, self-improvement | Sandbox testing, promotion gates |

**Governance Scaling**:

- **L0**: Content filters only
- **L1**: Add tool permissions, input validation
- **L2**: Add approval workflows, observability
- **L3**: Add handoff validation, multi-agent tracing
- **L4**: Add sandbox isolation, promotion reviews

**Pattern**:
```yaml
agent:
  id: "agent-research-001"
  maturity_level: "L2"
  capabilities:
    - "web_search"
    - "document_retrieval"
    - "multi_step_planning"
  governance:
    approvals_required: ["irreversible_actions"]
    observability_depth: "full_trajectory"
    tool_allowlist: ["search_api", "retrieval_api"]
```

---

## Persona + Domain Knowledge

**What**: Encode persona and domain expertise as versioned, testable assets

**Best Practices**:

1. **Versioned assets**: Treat persona/domain as code (version control, changelogs, reviews)
2. **Ahead of tools**: Load persona/domain before tool initialization
3. **Testable**: Create test suites for persona behavior
4. **Documented**: Maintain clear documentation of persona characteristics
5. **Evolvable**: Plan for persona updates and migrations

**Persona Definition**:
```yaml
persona:
  version: "v2.1.0"
  name: "Research Assistant"
  domain: "Academic Research"
  expertise:
    - "Literature review"
    - "Citation management"
    - "Data synthesis"
  tone: "Professional, precise, academic"
  constraints:
    - "Never speculate beyond evidence"
    - "Always cite sources"
    - "Flag uncertainty clearly"
  updated_at: "2024-01-15"
  changelog_url: "docs/personas/research-assistant-changelog.md"
```

**Domain Knowledge**:
```yaml
domain:
  name: "Academic Research"
  version: "v1.3.0"
  knowledge_sources:
    - name: "PubMed API"
      type: "tool"
      priority: "high"
    - name: "ArXiv"
      type: "retrieval"
      priority: "medium"
  validation_rules:
    - "Verify publication dates"
    - "Check citation formats"
    - "Validate DOIs"
```

---

## Identity & Policy

**What**: Each agent is a principal with scopes, roles, permissions, and audit trail

**Agent Identity**:
```yaml
agent_identity:
  agent_id: "agent-financial-001"
  agent_type: "financial_advisor"
  version: "v1.2.0"
  principal_id: "svc-account-agents-prod"
  scopes:
    - "read:market_data"
    - "read:user_portfolio"
  roles:
    - "advisor"
  created_at: "2024-01-01T00:00:00Z"
  updated_at: "2024-01-15T12:00:00Z"
```

**Policy Enforcement**:

1. **Tool allowlist**: Explicitly enumerate permitted tools
2. **Data access policy**: Define what data agent can access
3. **Approval matrix**: Map actions to approval requirements
4. **trace_id propagation**: Mandatory for all agent interactions

**Policy Schema**:
```yaml
policy:
  agent_id: "agent-financial-001"
  tool_allowlist:
    - "market_data_api"
    - "portfolio_api"
  data_access:
    allowed_scopes: ["user_portfolio", "market_data"]
    denied_scopes: ["admin", "pii"]
  approval_matrix:
    trades: "human_approval_required"
    read_only: "auto_approved"
  trace_propagation: "mandatory"
```

**Audit Trail**:
```json
{
  "trace_id": "req-abc-123",
  "agent_id": "agent-financial-001",
  "action": "execute_trade",
  "timestamp": "2024-01-15T12:00:00Z",
  "inputs": {"symbol": "AAPL", "shares": 10},
  "approval": {"approved_by": "user-001", "approved_at": "2024-01-15T11:59:00Z"},
  "result": "success"
}
```

---

## Fleet Control

**What**: Centralized registry and lifecycle management for agent fleet

**Agent Registry**:
```yaml
registry:
  agents:
    - agent_id: "agent-001"
      name: "Customer Support Agent"
      version: "v1.2.0"
      status: "active"
      contract_url: "contracts/customer-support-v1.2.json"
      deprecated: false
      kill_switch: "enabled"

    - agent_id: "agent-002"
      name: "Legacy Support Agent"
      version: "v1.0.0"
      status: "deprecated"
      deprecation_date: "2024-06-01"
      replacement: "agent-001"
      kill_switch: "enabled"
```

**Contract Management** (JSON Schemas):
```json
{
  "agent_id": "agent-001",
  "contract_version": "v1.2.0",
  "input_schema": {
    "type": "object",
    "properties": {
      "query": {"type": "string"},
      "user_id": {"type": "string"}
    },
    "required": ["query", "user_id"]
  },
  "output_schema": {
    "type": "object",
    "properties": {
      "response": {"type": "string"},
      "confidence": {"type": "number"}
    }
  }
}
```

**Deprecation Rules**:

1. **Sunset timeline**: 90-day deprecation notice
2. **Migration path**: Provide clear upgrade instructions
3. **Backward compatibility**: Support old contracts during transition
4. **Communication**: Notify all consumers before deprecation
5. **Forced migration**: Hard cutoff after grace period

**Kill Switches**:

```yaml
kill_switch:
  agent_id: "agent-001"
  enabled: true
  triggers:
    - "error_rate > 10%"
    - "safety_violation_detected"
    - "manual_override"
  action: "halt_all_requests"
  fallback: "return_error_503"
  notification: ["oncall", "sre_team"]
```

**Override Paths**:

```yaml
override:
  agent_id: "agent-001"
  conditions:
    - condition: "high_confidence_required"
      override: "escalate_to_human"

    - condition: "safety_critical"
      override: "require_approval"

    - condition: "production_incident"
      override: "use_fallback_agent"
```

---

## Version Management

**Semantic Versioning**:

- **Major**: Breaking contract changes (v1 → v2)
- **Minor**: New capabilities, backward compatible (v1.1 → v1.2)
- **Patch**: Bug fixes, no behavior change (v1.2.0 → v1.2.1)

**Deployment Strategy**:

1. **Canary**: Route 5-10% traffic to new version
2. **Monitor**: Track metrics (error rate, latency, quality)
3. **Rollback**: Instant rollback if metrics degrade
4. **Gradual rollout**: Increase traffic gradually (10% → 25% → 50% → 100%)
5. **Deprecate old**: Sunset old version after validation

**Pattern**:
```yaml
deployment:
  agent_id: "agent-001"
  versions:
    - version: "v1.2.0"
      traffic: 10  # percent
      status: "canary"
      metrics:
        error_rate: 0.02
        p95_latency: 150ms

    - version: "v1.1.0"
      traffic: 90  # percent
      status: "stable"
      metrics:
        error_rate: 0.01
        p95_latency: 120ms
```

---

## Fleet Metrics

**Track Fleet-Wide**:

- **Active agents**: Count by type, version, status
- **Total requests**: Volume per agent
- **Error rates**: By agent, tool, handoff
- **Cost**: Token usage, tool calls, infrastructure
- **Quality**: Evaluation scores, user satisfaction
- **Safety**: Policy violations, escalations

**Dashboard Schema**:
```yaml
fleet_metrics:
  timestamp: "2024-01-15T12:00:00Z"
  active_agents: 42
  total_requests_24h: 15000
  avg_error_rate: 0.015
  total_cost_24h: "$125.00"
  safety_violations_24h: 2
  agents_by_maturity:
    L0: 5
    L1: 20
    L2: 15
    L3: 2
    L4: 0
```

---

## Compliance & Auditing

**Audit Requirements**:

1. **Request logs**: All agent requests with trace_id
2. **Tool usage**: Every tool call with parameters and results
3. **Approvals**: Human-in-the-loop decisions
4. **Policy checks**: All guardrail evaluations
5. **Handoffs**: Agent-to-agent transfers with schemas

**Retention Policy**:
```yaml
retention:
  request_logs: 90_days
  tool_usage: 365_days
  approvals: 7_years  # regulatory requirement
  policy_violations: 7_years
  handoff_traces: 90_days
```

**Compliance Frameworks**:

- **NIST AI RMF**: Risk management framework
- **OWASP GenAI Top 10**: Security vulnerabilities
- **SOC 2**: Security and availability controls
- **GDPR**: Privacy and data protection
- **HIPAA**: Healthcare data (if applicable)

---

## Related Resources

**Deployment**: [`deployment-ci-cd-and-safety.md`](deployment-ci-cd-and-safety.md)
**Observability**: [`evaluation-and-observability.md`](evaluation-and-observability.md)
**Multi-Agent**: [`multi-agent-patterns.md`](multi-agent-patterns.md)
**API Design**: [`api-contracts-for-agents.md`](api-contracts-for-agents.md)

---

## Usage Notes

- **Start at L0/L1**: Begin with simple agents, graduate to higher levels
- **Governance scales with capability**: More powerful = more controls
- **Registry is source of truth**: All agents must be registered
- **Kill switches are mandatory**: Every production agent needs a kill switch
- **Audit everything**: Compliance requires comprehensive logging

# Agent Operations — Best Practices

*Purpose: Provide operational guidance for designing and running single-agent and tool-using agents.*

---
## Table of Contents

- [Pattern: Plan → Act → Observe → Update → Repeat](#pattern-plan-→-act-→-observe-→-update-→-repeat)
- [2. Action Execution](#2-action-execution)
- [Pattern: Validated Action](#pattern-validated-action)
- [3. Retrieval & Grounding](#3-retrieval-&-grounding)
- [Pattern: Evidence-First Reasoning](#pattern-evidence-first-reasoning)
- [4. Tool Use](#4-tool-use)
- [Pattern: Safe Tool Invocation](#pattern-safe-tool-invocation)
- [5. Planning & Replanning](#5-planning-&-replanning)
- [Pattern: Dynamic Planning](#pattern-dynamic-planning)
- [6. State & Context Management](#6-state-&-context-management)
- [Pattern: Minimal Context Window](#pattern-minimal-context-window)
- [7. Error Handling](#7-error-handling)
- [Pattern: Typed Failures](#pattern-typed-failures)
- [8. Verification & Success Criteria](#8-verification-&-success-criteria)
- [Pattern: Step-by-Step Verification](#pattern-step-by-step-verification)
- [9. Safety Operations](#9-safety-operations)
- [Pattern: Action Gating](#pattern-action-gating)
- [10. Operational Anti-Patterns (Master List)](#10-operational-anti-patterns-master-list)
- [11. Quick Reference Tables](#11-quick-reference-tables)
- [Agent Loop Summary](#agent-loop-summary)
- [Tool Call Checklist](#tool-call-checklist)
- [12. Decision Trees (Condensed)](#12-decision-trees-condensed)
- [Choosing Action vs. Tool](#choosing-action-vs-tool)
- [Handling Failures](#handling-failures)
- [Loop Continuation](#loop-continuation)
- [End of File](#end-of-file)


# 1. Core Agent Loop

### Pattern: Plan → Act → Observe → Update → Repeat

**Use when:** The agent must execute multi-step tasks with tools or external systems.

**Structure**

```
1. PLAN
2. ACT (tool or internal step)
3. OBSERVE (tool output or environment)
4. UPDATE (context, state, memory)
5. LOOP or FINAL ANSWER
```

**Checklist**

- [ ] Plan decomposes task into atomic steps.  
- [ ] Each step declares expected evidence.  
- [ ] Tool calls use validated parameters.  
- [ ] Observation evaluates success/failure.  
- [ ] Update revises plan when environment changes.  
- [ ] Loop halts when criteria met.

**Anti-Patterns**

- AVOID: Planning all steps upfront without recalculating after each observation.  
- AVOID: Ignoring tool output errors.  
- AVOID: Continuing loops without state change detection.

---

# 2. Action Execution

### Pattern: Validated Action

**Use when:** Agent performs irreversible or high-impact operations.

**Structure**

```
validate_input()
confirm_if_high_risk()
execute()
verify_result()
```

**Checklist**

- [ ] Input sanitized.  
- [ ] Action matches authorized scope.  
- [ ] Confirmation required for destructive steps.  
- [ ] Verification explicitly checks expected state.  
- [ ] Retry logic configured (1–2 retries max).  

**Decision Tree**

```
Is action irreversible?
→ Yes → Require confirmation → Execute → Verify
→ No → Execute → Verify
```

---

# 3. Retrieval & Grounding

### Pattern: Evidence-First Reasoning

**Use when:** The agent must reference external data or use RAG.

**Structure**

```
retrieve()
validate_source()
inject_into_plan()
reason_from_evidence()
```

**Checklist**

- [ ] Retrieval precedes reasoning.  
- [ ] All factual claims cite retrieved text.  
- [ ] Only relevant chunks injected.  
- [ ] No unsupported assumptions.  

**Anti-Patterns**

- AVOID: Reasoning before evidence.  
- AVOID: Unsupported facts in final answer.

---

# 4. Tool Use

### Pattern: Safe Tool Invocation

**Use when:** Using MCP tools, APIs, or custom functions.

**Structure**

```
choose_tool()
prepare_parameters()
call_tool()
evaluate_output()
```

**Checklist**

- [ ] Tool selected intentionally.  
- [ ] Parameters validated (types, ranges, formats).  
- [ ] Tool errors parsed and retried when transient.  
- [ ] Output grounded before use.  

**Decision Tree**

```
Does the step require external data or action?
→ Yes → Use tool
→ No → Internal reasoning
```

**Anti-Patterns**

- AVOID: Hallucinating tool names or parameters.  
- AVOID: Chaining multiple tool calls without checking outputs.

---

# 5. Planning & Replanning

### Pattern: Dynamic Planning

**Use when:** The agent faces uncertainty or multi-step tasks.

**Structure**

```
initial_plan()
for each step:
    observe -> revise_plan -> continue
```

**Checklist**

- [ ] Plan expressed as numbered steps.  
- [ ] Each step references expected input/output.  
- [ ] Replanning triggered by mismatched observations.  

**Trigger Conditions for Replanning**

- Unexpected tool output  
- Missing required evidence  
- Contradictory or invalid state  

---

# 6. State & Context Management

### Pattern: Minimal Context Window

**Use when:** The agent operates in long tasks or multi-turn sessions.

**Structure**

```
preserve(relevant_history)
summarize(excess_history)
inject(context)
```

**Checklist**

- [ ] Only relevant history retained.  
- [ ] Summaries replace long transcripts.  
- [ ] Context injected before plan generation.  

**Anti-Patterns**

- AVOID: Passing full transcripts into every step.  
- AVOID: Mixing unrelated conversation segments.

---

# 7. Error Handling

### Pattern: Typed Failures

**Use when:** Tool output or steps may fail.

**Structure**

```
if transient_error:
    retry
elif fatal_error:
    report and halt
else:
    continue
```

**Checklist**

- [ ] Categorize errors (transient/fatal).  
- [ ] Retry only transient cases.  
- [ ] Produce human-readable error summaries.  
- [ ] Never mask failures by improvising actions.  

**Quick Reference Table**

| Error Type       | Examples                | Response            |
|------------------|--------------------------|----------------------|
| Transient        | network timeout, rate limit | retry once          |
| Soft failure     | missing field, bad input | request clarification |
| Fatal            | auth failure, invalid tool | halt + report       |

---

# 8. Verification & Success Criteria

### Pattern: Step-by-Step Verification

**Use when:** Agent performs multi-step work.

**Verification Points**

- After tool call  
- After navigation step  
- After each plan iteration  
- Before final answer  

**Checklist**

- [ ] Output matches expected structure.  
- [ ] Data types validated.  
- [ ] Business rules satisfied.  
- [ ] Final answer derived only from verified steps.  

---

# 9. Safety Operations

### Pattern: Action Gating

**Use when:** Action could affect systems, data, or user environment.

**Checklist**

- [ ] Identify high-risk actions.  
- [ ] Provide natural language confirmation step.  
- [ ] Reject ambiguous or unspecified requests.  
- [ ] Block unsupported or dangerous operations.  

**High-Risk Examples**

- File deletion  
- OS-level command  
- External system mutation  
- Financial transactions  

---

# 10. Operational Anti-Patterns (Master List)

- AVOID: Using reasoning to “fill in” missing tool outputs  
- AVOID: Planning long sequences without checkpoints  
- AVOID: Ignoring verification on tool calls  
- AVOID: Acting without grounding  
- AVOID: Overwriting state without confirmation  
- AVOID: Passing hallucinated IDs/paths  
- AVOID: Treating every error as retryable  

---

# 11. Quick Reference Tables

### Agent Loop Summary

| Stage     | What Happens                       | Outputs Needed     |
|-----------|-------------------------------------|--------------------|
| Plan      | Steps, tool choices                 | step list          |
| Act       | Tool execution or reasoning          | tool result        |
| Observe   | Inspect outputs                     | validated data     |
| Update    | Update plan/context                 | new plan           |
| Final     | Produce grounded answer             | final response     |

### Tool Call Checklist

| Item                     | Requirement                    |
|--------------------------|--------------------------------|
| Parameter validation     | types, format, ranges          |
| Tool name                | must be declared & available   |
| Error handling           | retry on transient             |
| Output grounding         | mandatory                      |
| Confirmation             | for high-risk actions          |

---

# 12. Decision Trees (Condensed)

### Choosing Action vs. Tool

```
Does the step require external data?
→ Yes → Tool
→ No → Reason internally
```

### Handling Failures

```
Is error transient?
→ Yes → Retry once
→ No → Summarize + Halt
```

### Loop Continuation

```
Did state change after last action?
→ Yes → Continue loop
→ No → Revise plan or halt
```

---

# End of File

# AI Engine Layers — Unified Agent Architecture

**Purpose**: Define the five-layer AI Engine architecture as a composition model for production agent systems. Maps each layer to existing skill references and provides implementation checklists.

---
## Table of Contents

- [Architecture Overview](#architecture-overview)
- [1. Context Graph](#1-context-graph)
- [Schema](#schema)
- [When to Use](#when-to-use)
- [Implementation Tiers](#implementation-tiers)
- [2. Action Graph](#2-action-graph)
- [Schema](#schema)
- [State Transitions](#state-transitions)
- [Integration with MCP/A2A](#integration-with-mcpa2a)
- [3. Data Agent](#3-data-agent)
- [Pipeline](#pipeline)
- [Source Types](#source-types)
- [Freshness Management](#freshness-management)
- [4. Knowledge Base](#4-knowledge-base)
- [Unified Schema](#unified-schema)
- [Access Patterns](#access-patterns)
- [5. Inbox Engine](#5-inbox-engine)
- [Pipeline](#pipeline)
- [Signal Classification](#signal-classification)
- [Routing Rules](#routing-rules)
- [Layer Interaction Matrix](#layer-interaction-matrix)
- [Commercial Landscape (March 2026)](#commercial-landscape-march-2026)
- [Market Tiers](#market-tiers)
- [Layer-to-Product Alignment](#layer-to-product-alignment)
- [Industry Standards](#industry-standards)
- [What We Have That Others Don't](#what-we-have-that-others-dont)
- [What to Watch](#what-to-watch)
- [Implementation Checklist](#implementation-checklist)
- [Phase 1: Foundation (Week 1-2)](#phase-1-foundation-week-1-2)
- [Phase 2: Integration (Week 3-4)](#phase-2-integration-week-3-4)
- [Phase 3: Production (Week 5-6)](#phase-3-production-week-5-6)
- [Related Resources](#related-resources)


## Architecture Overview

```text
┌─────────────────────────────────────────────────────────┐
│                     INBOX ENGINE                        │
│  (event intake → triage → routing)                      │
├──────────────────────┬──────────────────────────────────┤
│     ACTION GRAPH     │        CONTEXT GRAPH             │
│  (plan → act →       │   (entities, relationships,      │
│   observe → update)  │    reasoning traces)             │
├──────────────────────┴──────────────────────────────────┤
│                     DATA AGENT                          │
│  (retrieve → transform → index → refresh)               │
├─────────────────────────────────────────────────────────┤
│                   KNOWLEDGE BASE                        │
│  (vector store + knowledge graph + document index)      │
└─────────────────────────────────────────────────────────┘
```

**Data flow**: Inbox Engine receives signals → routes to Action Graph → Action Graph queries Context Graph for state → Context Graph pulls from Knowledge Base via Data Agent → results flow back up.

---

## 1. Context Graph

**What**: Structured representation of entities, relationships, and reasoning traces the agent uses for decision-making.

**Core function**: Maintain a queryable graph of everything the agent "knows" during a task — who, what, when, how they relate, and what has been inferred.

### Schema

```yaml
context_graph:
  nodes:
    - id: "entity-001"
      type: "user | document | tool | concept | event"
      properties:
        name: "..."
        source: "retrieval | inference | user_input"
        confidence: 0.95
        created_at: "2026-01-01T00:00:00Z"
        ttl: 3600
  edges:
    - source: "entity-001"
      target: "entity-002"
      relation: "authored_by | depends_on | contradicts | supports"
      weight: 0.9
      provenance: "rag_retrieval"
  traces:
    - step: 1
      action: "retrieve"
      reasoning: "User asked about X, retrieving related documents"
      context_delta: ["entity-003 added"]
```

### When to Use

| Scenario | Context Graph Approach |
|----------|----------------------|
| Multi-turn conversation | Track entity mentions across turns, link co-references |
| Multi-step reasoning | Log reasoning trace as edges between intermediate conclusions |
| Tool result integration | Add tool outputs as nodes, link to triggering query |
| Contradiction detection | Query graph for conflicting edges on same entity pair |

### Implementation Tiers

| Tier | Storage | Best For |
|------|---------|----------|
| **Lightweight** | In-memory dict/map | Single-session, <100 entities |
| **Mid-scale** | Redis + JSON graph | Multi-session, <10K entities |
| **Production** | Neo4j / FalkorDB / Amazon Neptune | Cross-agent, persistent, >10K entities |

**Existing depth**: [`context-engineering.md`](context-engineering.md) — progressive disclosure, session management, memory provenance, retrieval timing.

**New patterns**: [`context-graph-patterns.md`](context-graph-patterns.md) — node/edge schema, traversal, graph-augmented retrieval, memory tiers.

---

## 2. Action Graph

**What**: DAG or FSM of agent operations — plan, act, observe, update — with tool orchestration and state transitions.

**Core function**: Define the execution topology of an agent: which actions happen, in what order, with what branching conditions, and how state flows between them.

### Schema

```yaml
action_graph:
  id: "workflow-001"
  type: "dag | fsm | hybrid"
  nodes:
    - id: "step-plan"
      action: "plan"
      inputs: ["user_query", "context_snapshot"]
      outputs: ["execution_plan"]
      max_retries: 0
    - id: "step-retrieve"
      action: "tool_call"
      tool: "mcp://knowledge-base/search"
      inputs: ["execution_plan.queries"]
      outputs: ["retrieved_documents"]
      max_retries: 2
      timeout_ms: 5000
    - id: "step-synthesize"
      action: "llm_call"
      inputs: ["retrieved_documents", "user_query"]
      outputs: ["response_draft"]
  edges:
    - from: "step-plan"
      to: "step-retrieve"
      condition: "plan.requires_retrieval == true"
    - from: "step-retrieve"
      to: "step-synthesize"
      condition: "always"
  guards:
    max_steps: 10
    max_tokens: 50000
    timeout_ms: 30000
```

### State Transitions

```text
IDLE → PLANNING → EXECUTING → OBSERVING → UPDATING → COMPLETE
                      │                        │
                      ├── ERROR ──→ RETRYING ──┘
                      └── BLOCKED → HUMAN_REVIEW
```

### Integration with MCP/A2A

| Protocol | Action Graph Role |
|----------|------------------|
| **MCP** | Tool nodes call MCP servers; tool schemas define inputs/outputs |
| **A2A** | Handoff edges delegate sub-DAGs to other agents |

**Existing depth**: [`operational-patterns.md`](operational-patterns.md) — PLAN→ACT→OBSERVE→UPDATE loop, tool specification, multi-agent workflow. [`agent-operations-best-practices.md`](agent-operations-best-practices.md) — action loops, planning, execution patterns.

---

## 3. Data Agent

**What**: Autonomous retrieval and transformation agent that connects to data sources, indexes content, and manages freshness.

**Core function**: Act as the bridge between raw data sources and the Knowledge Base — fetch, clean, chunk, embed, index, and keep content fresh.

### Pipeline

```text
SOURCE MONITOR → FETCH → TRANSFORM → CHUNK → EMBED → INDEX → VALIDATE
       │                                                         │
       └─────── REFRESH (TTL / webhook / schedule) ──────────────┘
```

### Source Types

| Source | Connector | Refresh Strategy |
|--------|-----------|-----------------|
| REST API | HTTP polling / webhook | Event-driven or cron |
| Database | CDC (Change Data Capture) | Real-time stream |
| Documents | File watcher / S3 events | On-change |
| Web pages | Crawler / scraper | Scheduled |
| Streams | Kafka / Pub/Sub consumer | Continuous |

### Freshness Management

```yaml
freshness_policy:
  default_ttl: 86400          # 24 hours
  source_overrides:
    pricing_api: 3600          # 1 hour
    legal_docs: 604800         # 7 days
    user_profiles: 300         # 5 minutes
  invalidation_triggers:
    - webhook_received
    - source_schema_changed
    - confidence_below_threshold
  re_index_strategy: "incremental"  # full | incremental | differential
```

**Existing depth**: [`../ai-rag/SKILL.md`](../../ai-rag/SKILL.md) — chunking strategies, embedding models, reranking, hybrid search. [`rag-patterns.md`](rag-patterns.md) — retrieval patterns, agentic RAG.

---

## 4. Knowledge Base

**What**: Persistent semantic memory combining vector store, knowledge graph, and document index with provenance tracking.

**Core function**: Serve as the agent's long-term memory — store everything the agent might need to recall, with lineage metadata for trust and freshness.

### Unified Schema

```yaml
knowledge_base:
  vector_store:
    provider: "pinecone | qdrant | pgvector | chroma"
    embedding_model: "text-embedding-3-large"
    dimensions: 3072
    distance_metric: "cosine"
  knowledge_graph:
    provider: "neo4j | falkordb | amazon_neptune"
    schema:
      entity_types: ["person", "org", "concept", "document", "event"]
      relation_types: ["authored", "references", "contradicts", "supersedes"]
  document_index:
    provider: "elasticsearch | typesense | meilisearch"
    fields: ["title", "content", "source", "date", "tags"]
  provenance:
    required_fields: ["source_url", "ingested_at", "confidence", "lineage_id"]
    retention_days: 365
```

### Access Patterns

| Query Type | Layer Used | Example |
|-----------|-----------|---------|
| Semantic similarity | Vector store | "Find docs about agent memory patterns" |
| Entity relationships | Knowledge graph | "What tools does agent-X use?" |
| Keyword / filter | Document index | "All docs from source=arxiv after 2025" |
| Hybrid | Vector + graph + filter | "Recent papers about X by author Y" |

**Existing depth**: [`memory-systems.md`](memory-systems.md) — four-memory model, retrieval patterns, write patterns, consolidation.

**New patterns**: [`../assets/knowledge-base/kb-architecture.md`](../assets/knowledge-base/kb-architecture.md) — unified KB schema, provenance, freshness, multi-tenant access control.

---

## 5. Inbox Engine

**What**: Event-driven intake layer that monitors sources, triages incoming signals, and routes to appropriate agents or workflows.

**Core function**: Act as the front door — everything that enters the agent system passes through the Inbox Engine for classification, deduplication, prioritization, and routing.

### Pipeline

```text
SOURCES → INGEST → CLASSIFY → DEDUPLICATE → PRIORITIZE → ROUTE
   │                                                        │
   │   ┌── actionable ──→ Action Graph                     │
   │   ├── informational → Knowledge Base (via Data Agent)  │
   │   └── noise ────────→ Log + discard                   │
   └────────── ACKNOWLEDGE / NACK ──────────────────────────┘
```

### Signal Classification

```yaml
signal_classes:
  actionable:
    description: "Requires agent action within SLA"
    examples: ["user request", "alert threshold breach", "approval needed"]
    routing: "action_graph"
  informational:
    description: "Updates knowledge, no immediate action"
    examples: ["data refresh", "status update", "new document published"]
    routing: "data_agent → knowledge_base"
  noise:
    description: "Irrelevant or duplicate, safe to discard"
    examples: ["heartbeat", "duplicate webhook", "stale notification"]
    routing: "log_and_discard"
```

### Routing Rules

| Trigger | Route To | SLA |
|---------|----------|-----|
| User message | Action Graph (conversational agent) | <2s |
| Webhook event | Action Graph (event handler) | <30s |
| Scheduled job | Data Agent (refresh pipeline) | best-effort |
| Agent handoff (A2A) | Action Graph (delegated task) | inherited |
| System alert | Action Graph (incident handler) | <5s |

**Existing depth**: [`multi-agent-patterns.md`](multi-agent-patterns.md) — orchestration, handoffs, group chat routing. [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md) — agent-to-agent communication.

**New patterns**: [`inbox-engine-patterns.md`](inbox-engine-patterns.md) — event-driven intake, signal classification, priority routing, deduplication.

---

## Layer Interaction Matrix

| From ↓ / To → | Context Graph | Action Graph | Data Agent | Knowledge Base | Inbox Engine |
|---------------|:---:|:---:|:---:|:---:|:---:|
| **Context Graph** | — | state queries | — | graph queries | — |
| **Action Graph** | read/write state | — | trigger refresh | query KB | — |
| **Data Agent** | — | status updates | — | write index | — |
| **Knowledge Base** | serve entities | serve results | receive writes | — | — |
| **Inbox Engine** | — | route tasks | route data | — | — |

---

## Commercial Landscape (March 2026)

"Context engine" has crystallized as a recognized infrastructure category — the layer between raw data and AI agents that assembles, maintains, and serves the right context at the right time.

### Market Tiers

| Tier | Products | What They Solve |
|------|----------|-----------------|
| **Dedicated Context Platforms** | Zep/Graphiti, Mem0, Cognee, Letta | Pure-play context engineering: memory, knowledge graphs, stateful agents |
| **Enterprise Context Platforms** | Glean, Tabnine, AWS Bedrock AgentCore | Full-stack AI platforms with built-in context engines |
| **Infrastructure Context Engines** | Confluent, Materialize, Redis | Data infrastructure repositioned as context layer for AI |
| **Agent Frameworks** | LangGraph, Composio, Arcade | Orchestration frameworks with built-in context management |

### Layer-to-Product Alignment

| Our Layer | Closest Commercial Analog | Match |
|-----------|--------------------------|:-----:|
| **Context Graph** | Zep/Graphiti (temporal KG), Glean (enterprise KG) | Strong |
| **Action Graph** | LangGraph (state machine), Letta (stateful runtime) | Strong |
| **Data Agent** | Composio/Arcade (connectors), Confluent (streaming), Materialize (live data) | Strong |
| **Knowledge Base** | Mem0 (memory layer), Redis (in-memory), Pinecone (vector), Tabnine (code-specific) | Strong |
| **Inbox Engine** | Confluent (event-driven intake) | Medium — fewest pure-play products |

### Industry Standards

The Linux Foundation formed the **Agentic AI Foundation (AAIF)** with founding contributions from Anthropic (MCP), Block (goose), and OpenAI (AGENTS.md). MCP is the consensus protocol for agent-to-tool connectivity — adopted natively by Confluent, Pinecone, Merge, and others.

### What We Have That Others Don't

1. **Unified 5-layer taxonomy** — no single commercial product covers all five layers as a named, composable architecture. Zep comes closest (3/5) but lacks Inbox Engine and Action Graph formalization.
2. **Composability** — our architecture is provider-agnostic with pluggable drivers. Commercial products are typically locked to their implementation.
3. **Inbox Engine as first-class layer** — most products start at "agent receives query." Confluent is the only major player with explicit event-driven intake.

### What to Watch

| Pattern | Source | Status in Our Architecture |
|---------|--------|---------------------------|
| Bi-temporal knowledge graph | Zep/Graphiti | Added — see [`context-graph-patterns.md`](context-graph-patterns.md) |
| MCP as KB access protocol | Confluent, Pinecone, industry | Added — see [`../assets/knowledge-base/kb-architecture.md`](../assets/knowledge-base/kb-architecture.md) |
| Choreography pattern | Knative, Confluent | Added — see [`inbox-engine-patterns.md`](inbox-engine-patterns.md) |
| Per-layer evaluation benchmarks | Zep (DMR benchmark), Mem0 (accuracy lift) | Not yet covered |
| Edge/on-device deployment | Cognee (Rust engine) | Not yet covered |
| Git-based memory versioning | Letta Context Repositories (Feb 2026) | Not yet covered |

---

## Implementation Checklist

### Phase 1: Foundation (Week 1-2)

- [ ] Define Knowledge Base schema (vector store + document index)
- [ ] Implement Data Agent pipeline (fetch → chunk → embed → index)
- [ ] Set up basic Context Graph (in-memory, single-session)
- [ ] Build Action Graph for primary workflow (FSM or DAG)

### Phase 2: Integration (Week 3-4)

- [ ] Connect Action Graph → Knowledge Base (query path)
- [ ] Connect Data Agent → Knowledge Base (write path)
- [ ] Add Context Graph persistence (cross-session state)
- [ ] Implement basic Inbox Engine (single source, classification)

### Phase 3: Production (Week 5-6)

- [ ] Add Inbox Engine multi-source intake + deduplication
- [ ] Implement Context Graph → Knowledge Graph synchronization
- [ ] Add freshness management (TTL, invalidation, re-indexing)
- [ ] Deploy observability (OpenTelemetry traces per layer)
- [ ] Run safety checklist per [`assets/checklists/agent-safety-checklist.md`](../assets/checklists/agent-safety-checklist.md)

---

## Related Resources

| Resource | Covers |
|----------|--------|
| [`context-engineering.md`](context-engineering.md) | Progressive disclosure, session management, provenance |
| [`context-graph-patterns.md`](context-graph-patterns.md) | Node/edge schema, traversal, graph-RAG |
| [`operational-patterns.md`](operational-patterns.md) | PLAN→ACT→OBSERVE→UPDATE, tool specs |
| [`memory-systems.md`](memory-systems.md) | Four-memory model, retrieval, consolidation |
| [`rag-patterns.md`](rag-patterns.md) | Retrieval pipelines, hybrid search, agentic RAG |
| [`multi-agent-patterns.md`](multi-agent-patterns.md) | Orchestration, handoffs, group chat |
| [`inbox-engine-patterns.md`](inbox-engine-patterns.md) | Event intake, triage, routing |
| [`../assets/knowledge-base/kb-architecture.md`](../assets/knowledge-base/kb-architecture.md) | Unified KB schema, provenance, freshness |

# API Contracts for Agents

Use these envelopes when exposing agents/LLMs via REST/gRPC/GraphQL.

## Request Envelope
- `trace_id` (propagate) + `request_id`
- `actor`: user/org ids, roles/scopes, auth method
- `intent`: task description, system instructions
- `context_refs`: doc ids, vector keys, cache keys
- `tools_allowed`: ids + args schema; per-request allowlist
- `safety`: moderation level, PII policy, jailbreak guard on/off
- `delivery`: `stream` (SSE/WebSocket), `async` (202 + polling), callback URL + HMAC
- `params`: temperature, top_p, max_tokens, stop, seed

## Response Envelope
- `choices[]`: message, role, finish_reason
- `stream_delta`: partial tokens/chunks when streaming
- `citations[]`: source_id, span, url
- `tool_calls[]`: name, args, status, result (if inline), latency_ms
- `usage`: prompt_tokens, completion_tokens, cost
- `trace_id` echoed; `rate_limit`: limit/remaining/reset

## Errors (RFC 7807)
- Types: `model_timeout`, `tool_failed`, `guardrail_blocked`, `retrieval_miss`, `validation_error`, `quota_exceeded`
- Include `trace_id`, `hint`, `retryable`

## Streaming
- SSE fields: `event=delta|done|error`, `id`, `data` (JSON lines)
- WebSocket: close codes documented; heartbeat/ping interval; backpressure guidance
- Keep-alives for idle connections; clear retry/backoff policy

## Long-Running Jobs
- `202 Accepted` + `Location` for status; payload includes `job_id`, `state`, `eta`, `expires_at`
- States: queued → running → succeeded | failed | cancelled
- Callbacks: signed (HMAC), replay-protected, include `trace_id`

## Safety & Guardrails
- Pre: moderation, injection scan, scope/role checks, tool allowlist enforcement
- During: block high-risk tool calls unless approved; cap batch sizes, TTLs
- Post: PII redaction, policy filters, optional hallucination/citation checks

## Observability
- Propagate `traceparent`/`tracestate` or `trace_id` header end-to-end
- Spans: `llm_call`, `retrieval`, `tool_call`, `memory_op`
- Logs: request envelope sans secrets, guardrail outcomes, rate-limit decisions

# Autonomous Loop Patterns

Use this reference when designing **Shape C — Autonomous Loop**: a long-running process that repeatedly invokes an agent against a fixed goal until acceptance criteria are met, the budget is exhausted, or a circuit breaker fires.

Canonical example: Ralph Loop (37h / 250 tasks from a 2000-line PRD). Closely related: Devin-style background agents, BMAD-v6 replayable runs, agent-driven migrations, overnight refactors, continuous research crawls.

This is the deployment shape with the **highest blast radius** and the **least supervision**. Treat every guardrail here as load-bearing, not optional.

## Table of Contents

- [When to Use This Shape](#when-to-use-this-shape)
- [Anatomy of an Autonomous Loop](#anatomy-of-an-autonomous-loop)
- [The PRD (Loop Specification)](#the-prd-loop-specification)
- [Loop Driver — Three Implementations](#loop-driver--three-implementations)
- [Termination Criteria](#termination-criteria)
- [Budget and Iteration Caps](#budget-and-iteration-caps)
- [Checkpointing and Restart](#checkpointing-and-restart)
- [Drift Detection Mid-Loop](#drift-detection-mid-loop)
- [Circuit Breakers](#circuit-breakers)
- [Fail-Loud Wiring](#fail-loud-wiring)
- [Observability](#observability)
- [Operational Checklist](#operational-checklist)
- [Common Failure Modes](#common-failure-modes)
- [Cross-References](#cross-references)

## When to Use This Shape

Use an autonomous loop when **all** of the following hold:

- The work is decomposable into steps the agent can finish without human input.
- Acceptance criteria are concrete enough to be machine-checked (tests pass, all rows migrated, all docs indexed, all PRs merged with green CI).
- The cost of one wasted iteration is small enough that a few wasted iterations are tolerable.
- A budget cap and circuit breaker can stop the loop before it burns runaway cost.

Do **not** use this shape when:

- "Done" is a judgment call only a human can make.
- Each step is irreversible (sending money, posting to social, deleting production data) — those belong in [Shape A](../../ai-coding-agents-tasks/references/webhook-and-queue-triggers.md) with explicit approval gates.
- The work touches multi-stakeholder coordination.
- You cannot articulate the termination criterion in one sentence.

## Anatomy of an Autonomous Loop

```text
              ┌─────────────────────────┐
              │  PRD / loop spec (file) │
              └────────────┬────────────┘
                           │ read each iter
                           ▼
┌──────────────────────────────────────────────────────┐
│                  Loop Driver                         │
│                                                      │
│   for iter in 1..max_iters:                          │
│      1. read PRD + checkpoint                        │
│      2. invoke agent with fresh context              │
│      3. capture output + token/cost/duration         │
│      4. run acceptance check                         │
│      5. run drift check                              │
│      6. write checkpoint                             │
│      7. evaluate stop conditions                     │
│                                                      │
└──────────┬─────────────────────────┬─────────────────┘
           │                         │
           ▼                         ▼
   ┌───────────────┐         ┌──────────────────┐
   │  Checkpoint   │         │  Circuit Breaker │
   │  Store (disk) │         │  (budget, drift, │
   │               │         │   error rate)    │
   └───────────────┘         └──────────────────┘
           │                         │
           ▼                         ▼
   ┌──────────────────────────────────────────┐
   │  Observability: events, traces, metrics  │
   │  Alerts: budget breach, repeated failure │
   └──────────────────────────────────────────┘
```

Five components, all required: PRD, driver, checkpoint store, circuit breaker, observability.

## The PRD (Loop Specification)

The PRD is the **only** durable instruction the loop reads. Every iteration starts with a fresh agent context that re-reads it. Treat it like a contract.

Minimum fields:

```yaml
---
goal: "Migrate all 312 legacy invoices to the v2 schema with zero data loss."
acceptance_criteria:
  - "SELECT COUNT(*) FROM invoices_v2 = SELECT COUNT(*) FROM invoices_legacy"
  - "All rows in invoices_v2 pass schema_check_v2.sql with 0 violations"
  - "Reconciliation report shows 0 drift after 24h soak"
budget:
  max_iterations: 50
  max_tokens_total: 8_000_000
  max_cost_usd: 200
  max_wall_clock_hours: 12
out_of_scope:
  - "Do not touch invoices_legacy table — read only."
  - "Do not modify the v2 schema definition."
escalation:
  on_repeated_failure: "Stop after 3 consecutive iterations with no progress."
  human_contact: "ops-oncall@example.com"
---

## Context
[longer prose context, links, examples]

## Definition of Progress
[what counts as "made progress this iteration" — used for the drift check]
```

Two non-obvious rules:

1. **The PRD must be re-readable from scratch every iteration.** No "see previous run". A fresh agent must understand the goal cold.
2. **Acceptance criteria must be machine-checkable.** If you cannot write a script that returns true/false, you do not have acceptance criteria — you have a wish.

## Loop Driver — Three Implementations

Pick one based on your operational substrate.

### Driver A — Plain Python (`while` loop)

Lowest-ceremony, no infrastructure dependency. Good for: single-machine runs, prototyping, work that completes in < 24h.

```python
import json, time, sys
from pathlib import Path
from anthropic import Anthropic

PRD = Path("loop.prd.md").read_text()
CHECKPOINT = Path("checkpoint.json")
client = Anthropic()

def load_checkpoint() -> dict:
    if CHECKPOINT.exists():
        return json.loads(CHECKPOINT.read_text())
    return {"iter": 0, "tokens": 0, "cost_usd": 0.0, "history": []}

def save_checkpoint(state: dict) -> None:
    CHECKPOINT.write_text(json.dumps(state, indent=2))

def acceptance_check() -> tuple[bool, str]:
    # Run your concrete check here. Return (passed, message).
    # Example: subprocess.run(["./check_acceptance.sh"]) returncode == 0
    raise NotImplementedError("Define your acceptance check")

def run_iteration(state: dict) -> dict:
    response = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=4096,
        system=PRD,
        messages=[{"role": "user", "content": f"Iteration {state['iter']+1}. Make progress toward the goal. Report what you changed."}],
    )
    text = response.content[0].text
    state["iter"] += 1
    state["tokens"] += response.usage.input_tokens + response.usage.output_tokens
    state["history"].append({"iter": state["iter"], "summary": text[:500]})
    return state

def main():
    state = load_checkpoint()
    config = parse_prd_frontmatter(PRD)  # implement
    no_progress_streak = 0

    while True:
        if state["iter"] >= config["max_iterations"]:
            return halt("iteration cap")
        if state["tokens"] >= config["max_tokens_total"]:
            return halt("token cap")
        if state["cost_usd"] >= config["max_cost_usd"]:
            return halt("cost cap")

        state = run_iteration(state)
        save_checkpoint(state)

        passed, msg = acceptance_check()
        if passed:
            return halt(f"acceptance met: {msg}")

        if not made_progress(state):  # implement against "Definition of Progress"
            no_progress_streak += 1
            if no_progress_streak >= 3:
                return halt("no progress for 3 iterations")
        else:
            no_progress_streak = 0

def halt(reason: str):
    print(f"HALT: {reason}", file=sys.stderr)
    # Send to observability + alert if reason is not "acceptance met"
    sys.exit(0 if reason.startswith("acceptance") else 1)

if __name__ == "__main__":
    main()
```

Run under `systemd`, `tmux`, or a container with restart policy `on-failure`. Crash-safe by design: checkpoint is read on every restart.

### Driver B — Temporal workflow

Use when: you need exactly-once activity semantics, durable retries, multi-day runs, or you already operate Temporal.

```python
# workflow.py
from datetime import timedelta
from temporalio import workflow

@workflow.defn
class AutonomousAgentLoop:
    @workflow.run
    async def run(self, prd_path: str) -> str:
        config = await workflow.execute_activity(load_prd, prd_path, start_to_close_timeout=timedelta(minutes=1))
        state = await workflow.execute_activity(load_checkpoint, prd_path, start_to_close_timeout=timedelta(minutes=1))
        no_progress = 0

        while state["iter"] < config["max_iterations"]:
            if state["cost_usd"] >= config["max_cost_usd"]:
                return "halt:cost_cap"

            iter_result = await workflow.execute_activity(
                run_agent_iteration, args=[prd_path, state],
                start_to_close_timeout=timedelta(minutes=30),
                retry_policy=workflow.RetryPolicy(maximum_attempts=2),
            )
            state = iter_result["state"]
            await workflow.execute_activity(save_checkpoint, args=[prd_path, state], start_to_close_timeout=timedelta(minutes=1))

            acceptance = await workflow.execute_activity(check_acceptance, prd_path, start_to_close_timeout=timedelta(minutes=5))
            if acceptance["passed"]:
                return f"halt:acceptance:{acceptance['msg']}"

            if iter_result["made_progress"]:
                no_progress = 0
            else:
                no_progress += 1
                if no_progress >= 3:
                    return "halt:no_progress"

        return "halt:iter_cap"
```

Key Temporal benefits: workflow history replays on worker restart (no checkpoint file needed), activity timeouts catch hung agent calls, retry policies isolate transient failures.

### Driver C — LangGraph cyclic graph

Use when: the loop has branching decisions per iteration, you already use LangGraph for bots, or you want built-in checkpoint integration.

```python
from langgraph.graph import StateGraph, END
from langgraph.checkpoint.postgres import PostgresSaver

class LoopState(TypedDict):
    iter: int
    tokens: int
    cost_usd: float
    history: list
    last_output: str
    no_progress_streak: int

def run_iteration_node(state: LoopState) -> LoopState:
    # call agent, update state
    ...

def check_acceptance_node(state: LoopState) -> LoopState:
    ...

def decide_next(state: LoopState) -> str:
    if state["cost_usd"] >= MAX_COST or state["iter"] >= MAX_ITERS:
        return "halt"
    if state.get("acceptance_passed"):
        return "halt"
    if state["no_progress_streak"] >= 3:
        return "halt"
    return "iterate"

graph = StateGraph(LoopState)
graph.add_node("iterate", run_iteration_node)
graph.add_node("check", check_acceptance_node)
graph.add_node("halt", lambda s: s)
graph.set_entry_point("iterate")
graph.add_edge("iterate", "check")
graph.add_conditional_edges("check", decide_next, {"iterate": "iterate", "halt": END})

checkpointer = PostgresSaver.from_conn_string(DATABASE_URL)
app = graph.compile(checkpointer=checkpointer)
```

LangGraph's PostgresSaver gives you checkpointing for free, plus integrates with the bot patterns in [`../../ai-bot-builder/references/graph-design-patterns.md`](../../ai-bot-builder/references/graph-design-patterns.md).

## Termination Criteria

A loop must have at least one of each:

| Type | Example | Purpose |
|---|---|---|
| **Goal-met** | acceptance script returns 0 | The happy path |
| **Bound** | `iter >= max_iterations` | Prevent infinite loop |
| **Cost** | `cost_usd >= max_cost_usd` | Prevent runaway spend |
| **Stagnation** | 3 iterations with no progress | Prevent thrashing |
| **Drift** | acceptance metric regressed by >X% | Prevent corruption |
| **External** | sigterm / kill-switch file present | Allow human halt |

Missing any one of these is a known failure mode. The Ralph Loop original lacked explicit stagnation detection — users reported runs that "kept going" without making progress.

## Budget and Iteration Caps

Budgets are non-negotiable. The Coding Behavior Rule 6 ("token budgets are not advisory") was written for this shape.

Recommended defaults (tune to your work):

| Scope | Default | Notes |
|---|---|---|
| Per-iteration tokens | 50k–200k | Cap input + output |
| Per-iteration wall clock | 30 min | Use as activity timeout |
| Total iterations | 50 | Multiply if loop is known-long |
| Total tokens | 8M | Hard cap on the whole run |
| Total cost USD | $200 | Cross-check against token cost |
| Wall clock | 12h | Forces operator review at next day |

**Budget breach must be loud.** Wire it to PagerDuty / Slack / email, not just stderr. See [`budget-and-loop-hooks.md`](../../agents-hooks/references/budget-and-loop-hooks.md).

## Checkpointing and Restart

Every iteration writes a checkpoint **before** running the next one. The checkpoint is the only durable state — assume the process can die at any line.

What to checkpoint:

- iteration number
- cumulative tokens / cost
- history of iteration summaries (truncated)
- last agent output (full)
- acceptance check result and timestamp
- no-progress streak counter

Where:

- Driver A (plain Python): file on disk, atomic rename (`os.replace`)
- Driver B (Temporal): workflow history (automatic)
- Driver C (LangGraph): `PostgresSaver` or `SqliteSaver`

Restart contract: a restart from checkpoint must produce identical behavior to the iteration that would have run had the process not crashed. Test this explicitly with a kill-9 during iteration 5 and verify the loop resumes correctly.

## Drift Detection Mid-Loop

The loop can technically progress while making the underlying state worse. Detect this with a metric that should monotonically improve.

Examples:

- migration loop: rows-remaining should never increase
- test-fixing loop: failing tests should never increase
- doc-indexing loop: unindexed-doc count should never increase

```python
def check_drift(state: LoopState) -> bool:
    history = state["history"][-5:]
    if len(history) < 5:
        return False
    metrics = [h["primary_metric"] for h in history]
    # If the last 5 iterations show the metric getting worse, drift detected
    return metrics[-1] > metrics[0] * 1.1  # 10% regression tolerance
```

On drift detection: halt, alert, and require human approval before resuming.

## Circuit Breakers

A circuit breaker is a hard stop independent of the loop's normal stop conditions. Implement at least two:

1. **Error-rate breaker.** If 5 of the last 10 iterations errored, halt.
2. **Provider-error breaker.** If the LLM provider returns 5xx for 3 consecutive iterations, halt and back off.
3. **External kill-switch.** Check for a file like `LOOP_KILL` or a Redis key at the start of every iteration. If present, halt immediately and persist the reason.

```python
def kill_switch_active() -> bool:
    return Path("/var/run/agent-loop/KILL").exists()

# Inside loop:
if kill_switch_active():
    halt("external kill switch")
```

A kill-switch the operator can flip from their phone is mandatory for any loop that runs over 1 hour unsupervised.

## Fail-Loud Wiring

Coding Behavior Rule 12: silent success is the most expensive failure mode.

Every halt reason must produce a structured event:

```json
{
  "event": "loop_halt",
  "loop_id": "invoice-migration-2026-05-20",
  "reason": "no_progress",
  "iter": 17,
  "tokens_used": 3_400_000,
  "cost_usd": 84.50,
  "acceptance_passed": false,
  "last_summary": "Could not resolve schema mismatch on row 4521..."
}
```

Send to: stdout (for `tail -f`), observability backend, alert channel.

**Never** halt silently. Never halt with `exit(0)` if acceptance was not met.

## Observability

Minimum signals to emit per iteration:

| Signal | Type | Purpose |
|---|---|---|
| `iter_started` | event | timeline reconstruction |
| `iter_completed` | event | with duration, tokens, cost |
| `acceptance_check` | event | passed/failed + message |
| `progress_metric` | metric | for drift detection |
| `cumulative_cost_usd` | metric | for budget dashboards |
| `iter_count` | metric | for SLO tracking |

Send to Langfuse / Phoenix / OpenLLMetry. See [`evaluation-and-observability.md`](evaluation-and-observability.md) for the May 2026 platform comparison.

Dashboard must show: iterations vs time, cumulative cost vs budget, progress metric trend, halt reasons over recent runs.

## Operational Checklist

Before running an autonomous loop in production:

- [ ] PRD has machine-checkable acceptance criteria
- [ ] Budgets set (iterations, tokens, cost, wall clock)
- [ ] Checkpoint store tested with kill-9 restart
- [ ] Stagnation detector configured (default: 3 iterations)
- [ ] Drift detector configured (or explicit decision to skip with rationale)
- [ ] Kill-switch path documented and tested
- [ ] Alerts routed to a human who is on-call now
- [ ] Halt events flow to observability
- [ ] Dry-run completed against a non-production target
- [ ] Rollback plan exists for partial-progress state
- [ ] Out-of-scope list explicit in PRD
- [ ] Runbook entry written: how to inspect, halt, resume, post-mortem

## Common Failure Modes

| Failure | Symptom | Mitigation |
|---|---|---|
| **Runaway cost** | Bill arrives, loop still running | Cost cap + budget hook |
| **Silent stagnation** | Iterations complete but no progress | Stagnation detector |
| **Drift** | Acceptance metric regresses | Drift detector + monotonic-metric check |
| **Restart loop** | Process crashes, restarts, repeats same work | Atomic checkpoint write + restart smoke test |
| **Provider outage** | Hangs forever on LLM call | Per-iteration timeout |
| **Half-applied state** | Crashes mid-iteration, world is inconsistent | Idempotent agent actions or transaction-scoped iterations |
| **Goal drift** | Agent reinterprets PRD across iterations | PRD pinned in `system`, not in user-turn history |
| **Halt with `exit(0)`** | Pipeline thinks it succeeded | Exit non-zero on any non-acceptance halt |

## Cross-References

- [`agent-delivery-methods.md`](agent-delivery-methods.md) — Ralph Loop row + BMAD/GSD comparison
- [`agent-operations-best-practices.md`](agent-operations-best-practices.md) — general production guidance
- [`evaluation-and-observability.md`](evaluation-and-observability.md) — telemetry platforms
- [`guardrails-implementation.md`](guardrails-implementation.md) — input/output guardrails
- [`deployment-ci-cd-and-safety.md`](deployment-ci-cd-and-safety.md) — release patterns
- [`24-7-operating-model.md`](24-7-operating-model.md) — SLOs and oncall for the loop in production
- [`../../agents-hooks/references/budget-and-loop-hooks.md`](../../agents-hooks/references/budget-and-loop-hooks.md) — hook-based budget enforcement
- [`../../agents-memory/SKILL.md`](../../agents-memory/SKILL.md) — durable state between runs
- [`../../software-workflow-automation/references/durable-execution.md`](../../software-workflow-automation/references/durable-execution.md) — Temporal/Inngest substrate
- [`../../qa-agent-testing/SKILL.md`](../../qa-agent-testing/SKILL.md) — acceptance check construction

# When NOT to Build an Agent — Decision Framework

**Purpose**: Systematic framework for evaluating whether to build an AI agent, continue development, or kill the project. Prevents wasted investment on agent projects that should never exist.

No theory. No narrative. Only decision rules.

---
## Table of Contents

- [The Default Answer is NO](#the-default-answer-is-no)
- [The 10-Second Test](#the-10-second-test)
- [Alternatives to Agents (Usually Better)](#alternatives-to-agents-usually-better)
- [Decision Rule](#decision-rule)
- [Red Flags — Don't Build If True](#red-flags-—-dont-build-if-true)
- [Immediate Disqualifiers](#immediate-disqualifiers)
- [Organizational Red Flags](#organizational-red-flags)
- [The Full Decision Framework](#the-full-decision-framework)
- [Stage 1: Problem Validation (Week 1)](#stage-1-problem-validation-week-1)
- [Stage 2: Feasibility Assessment (Week 2)](#stage-2-feasibility-assessment-week-2)
- [Stage 3: Economics Validation (Week 3)](#stage-3-economics-validation-week-3)
- [Stage 4: Risk Assessment (Week 4)](#stage-4-risk-assessment-week-4)
- [Kill Triggers (Stop Immediately)](#kill-triggers-stop-immediately)
- [During Development](#during-development)
- [Post-Launch](#post-launch)
- [Decision Tree: Build vs Not](#decision-tree-build-vs-not)
- [Common Anti-Patterns (Avoid These)](#common-anti-patterns-avoid-these)
- ["The AI Hammer"](#the-ai-hammer)
- ["The Scope Creep"](#the-scope-creep)
- ["The Perfection Trap"](#the-perfection-trap)
- [Checklist: Pre-Build Validation](#checklist-pre-build-validation)
- [Pre-Build Validation Checklist](#pre-build-validation-checklist)
- [Problem Definition](#problem-definition)
- [Data Readiness](#data-readiness)
- [Economics](#economics)
- [Risk Assessment](#risk-assessment)
- [Organizational](#organizational)
- [Final Gate](#final-gate)
- [Related References](#related-references)


## The Default Answer is NO

Building an agent should require justification, not be the default. Most tasks don't need agents.

### The 10-Second Test

Before any agent project, answer these three questions:

| Question | If NO | If YES |
|----------|-------|--------|
| 1. Is task volume >1000/month? | **Stop** — manual is cheaper | Continue |
| 2. Is human cost >$5/task? | **Stop** — agent won't beat human cost | Continue |
| 3. Can you tolerate 1-5% error rate? | **Stop** — agents can't guarantee 100% | Continue |

**If any answer is NO, don't build an agent.**

---

## Alternatives to Agents (Usually Better)

| Problem | Agent Instinct | Better Alternative | When Agent IS Better |
|---------|---------------|-------------------|---------------------|
| Answer FAQs | RAG chatbot | Static FAQ page + search | >500 unique questions |
| Route support tickets | Classification agent | Rule-based routing | >20 categories, fuzzy boundaries |
| Generate reports | Report agent | Scheduled SQL queries + templates | Ad-hoc, natural language queries |
| Monitor systems | Alert agent | Prometheus + PagerDuty rules | Requires judgment/synthesis |
| Write code | Coding agent | IDE snippets + copilot | Multi-file refactors, novel tasks |
| Process documents | Doc extraction agent | Regex + templates | Unstructured, variable formats |
| Personalize content | Recommendation agent | Collaborative filtering | Cold start, explanation needed |

### Decision Rule

```text
Use agent ONLY when:
  (Task requires reasoning OR judgment OR multi-step planning)
  AND
  (Volume justifies development cost)
  AND
  (Error tolerance exists)
```

---

## Red Flags — Don't Build If True

### Immediate Disqualifiers

| Red Flag | Why | Alternative |
|----------|-----|-------------|
| **"Make it feel human"** | Agents are tools, not personas | Clear bot UI, fast handoff to humans |
| **"Replace X employees"** | Agents augment, not replace | Augmentation use case |
| **"Handle everything"** | Unbounded scope = unbounded failure | Narrow, well-defined tasks |
| **"Zero errors allowed"** | Agents hallucinate | Human-in-the-loop or don't automate |
| **"We have no data"** | Nothing to retrieve or learn from | Build data pipeline first |
| **"Users don't know what they want"** | Garbage prompts = garbage output | Better UX, not AI |
| **"Legal/medical/financial advice"** | Liability + accuracy requirements | Human review mandatory |

### Organizational Red Flags

| Red Flag | Problem | Resolution Before Building |
|----------|---------|---------------------------|
| No success metrics defined | Can't prove value | Define KPIs first |
| No owner for agent quality | Quality will degrade | Assign ownership |
| Engineering team at capacity | Maintenance will fail | Staff appropriately |
| Data governance unclear | Privacy/compliance risk | Resolve governance first |
| No budget for LLM costs | Will get killed when bill arrives | Secure budget commitment |

---

## The Full Decision Framework

### Stage 1: Problem Validation (Week 1)

| Checkpoint | Pass Criteria | Fail Action |
|------------|---------------|-------------|
| Problem exists | >10 users report pain point | Find real problem |
| Problem is frequent | >100 occurrences/month | Too rare for automation |
| Problem is expensive | >$5 human cost per occurrence | Too cheap to automate |
| Problem is solvable by AI | Reasoning/language task | Use traditional automation |
| Problem is well-defined | Clear input/output spec | Define scope first |

**Gate**: Must pass ALL checkpoints to proceed.

### Stage 2: Feasibility Assessment (Week 2)

| Checkpoint | Pass Criteria | Fail Action |
|------------|---------------|-------------|
| Data available | >1000 relevant examples | Build data pipeline first |
| Data quality >80% | Spot-check 50 samples | Clean data first |
| Success is measurable | Concrete metric exists | Define metrics |
| Error tolerance exists | Can accept 1-5% errors | Don't automate |
| Baseline exists | Human performance measured | Measure baseline first |

**Gate**: Must pass ALL checkpoints to proceed.

### Stage 3: Economics Validation (Week 3)

| Checkpoint | Pass Criteria | Fail Action |
|------------|---------------|-------------|
| Projected ROI >100% | (Value - Cost) / Cost | Reduce scope or abandon |
| Payback <12 months | Dev cost / monthly savings | Reduce dev cost or abandon |
| LLM cost <50% of human cost | Token estimate vs human cost | Use cheaper model or abandon |
| Maintenance sustainable | <20% of dev time ongoing | Simplify or abandon |

**Gate**: Must pass ALL checkpoints to proceed.

### Stage 4: Risk Assessment (Week 4)

| Checkpoint | Pass Criteria | Fail Action |
|------------|---------------|-------------|
| Hallucination impact <$100/incident | Risk assessment | Add guardrails or don't build |
| No regulatory blockers | Legal review | Don't build or heavy HITL |
| No reputational risk | PR review | Don't build or don't ship |
| Failure mode acceptable | Graceful degradation possible | Don't build |
| Rollback possible | Can disable instantly | Build kill switch |

**Gate**: Must pass ALL checkpoints to proceed.

---

## Kill Triggers (Stop Immediately)

### During Development

| Trigger | Threshold | Action |
|---------|-----------|--------|
| 3 pivots on core approach | After 3 fundamental changes | Kill project |
| Prototype accuracy <60% | After 2 weeks of tuning | Kill project |
| Scope creep >2x original | Features doubled without ROI recalc | Kill or rescope |
| Key assumption invalidated | Discovery contradicts premise | Kill project |
| Team loses faith | >50% of team thinks it won't work | Kill project |

### Post-Launch

| Trigger | Threshold | Measurement Period | Action |
|---------|-----------|-------------------|--------|
| ROI negative | <0% | After 3 months | Kill |
| Adoption <20% | Active users / eligible | After 1 month | Kill or major pivot |
| Error rate >10% | After tuning attempts | After 2 iterations | Kill |
| LLM costs >3x projection | Sustained | 2 months | Kill or restructure |
| Support tickets increasing | Week over week | 4 weeks | Kill or fix root cause |
| User satisfaction <3/5 | CSAT score | After 1 month | Kill or major fix |

---

## Decision Tree: Build vs Not

```text
START: "Should we build an agent for X?"
│
├─ Is X a reasoning/language task?
│   ├─ No → USE TRADITIONAL AUTOMATION
│   └─ Yes ↓
│
├─ Is task volume >1000/month?
│   ├─ No → DON'T BUILD (manual is cheaper)
│   └─ Yes ↓
│
├─ Is human cost >$5/task?
│   ├─ No → DON'T BUILD (agent won't beat cost)
│   └─ Yes ↓
│
├─ Can you tolerate 1-5% errors?
│   ├─ No → DON'T BUILD (or heavy HITL)
│   └─ Yes ↓
│
├─ Do you have >1000 examples?
│   ├─ No → BUILD DATA PIPELINE FIRST
│   └─ Yes ↓
│
├─ Is projected ROI >100%?
│   ├─ No → REDUCE SCOPE or DON'T BUILD
│   └─ Yes ↓
│
├─ Is hallucination cost <$100/incident?
│   ├─ No → ADD GUARDRAILS or DON'T BUILD
│   └─ Yes ↓
│
└─ BUILD THE AGENT
    │
    └─ Monitor kill triggers weekly
```

---

## Common Anti-Patterns (Avoid These)

### "The AI Hammer"

| Anti-Pattern | Example | Correct Approach |
|--------------|---------|------------------|
| AI for everything | "Let's AI-enable our settings page" | Only AI where reasoning needed |
| Agent for simple lookup | "Agent to check order status" | Database query + template |
| Agent for static content | "Agent to explain pricing" | FAQ page |
| Agent for auth/payments | "Agent to process refunds" | Secure API, not LLM |

### "The Scope Creep"

| Anti-Pattern | Example | Correct Approach |
|--------------|---------|------------------|
| "While we're at it..." | "Also handle complaints, sales, HR..." | One use case, prove ROI, then expand |
| "Make it smarter" | "Understand context from 6 months ago" | Narrow context window |
| "Proactive outreach" | "Agent should reach out when..." | Bounded, triggered actions only |

### "The Perfection Trap"

| Anti-Pattern | Example | Correct Approach |
|--------------|---------|------------------|
| "Must be 100% accurate" | "Can't ship until zero errors" | Define acceptable error rate |
| "Must pass all edge cases" | "Handle every possible input" | 80/20 rule, escalate edge cases |
| "Must be indistinguishable from human" | "Users shouldn't know it's AI" | Transparency is better |

---

## Checklist: Pre-Build Validation

Copy this checklist before starting any agent project:

```markdown
## Pre-Build Validation Checklist

### Problem Definition
- [ ] Problem clearly defined in writing
- [ ] >10 users experiencing this pain point
- [ ] Frequency: >1000 tasks/month
- [ ] Human cost: >$5/task
- [ ] Error tolerance: 1-5% acceptable

### Data Readiness
- [ ] >1000 relevant examples available
- [ ] Data quality spot-checked (>80% usable)
- [ ] Data pipeline exists or budgeted

### Economics
- [ ] ROI projection completed (target: >100%)
- [ ] LLM cost estimate (monthly + per-task)
- [ ] Development cost estimate
- [ ] Maintenance cost estimate (20-40% of dev)
- [ ] Budget secured and approved

### Risk Assessment
- [ ] Hallucination impact assessed (<$100/incident OK)
- [ ] Legal review completed (if applicable)
- [ ] Graceful degradation designed
- [ ] Kill switch mechanism planned
- [ ] Rollback procedure documented

### Organizational
- [ ] Success metrics defined
- [ ] Owner assigned
- [ ] Maintenance team identified
- [ ] Stakeholder alignment confirmed

### Final Gate
- [ ] All boxes checked above
- [ ] Alternative solutions evaluated and rejected
- [ ] Sponsor sign-off obtained

**DECISION**: [ ] BUILD  [ ] DON'T BUILD  [ ] NEED MORE INFO
```

---

## Related References

- [Agent Economics](agent-economics.md) — ROI calculations and cost framework
- [Agent Maturity & Governance](agent-maturity-governance.md) — Rollout risk assessment
- [Deployment, CI/CD & Safety](deployment-ci-cd-and-safety.md) — Production guardrails

# Claude Agent SDK — Production Patterns

**Version**: Python v0.1.47, TypeScript v0.2.69 (March 2026) | First release: Sep 2025 | Formerly "Claude Code SDK"

**What**: Official Anthropic agent SDK. Functional API (not class-based) — agents defined via `query()` with options. Custom tools are in-process MCP servers. Deep MCP integration, Computer Use, built-in tools (Bash, Read, Write, Edit, Grep, Glob), 18-event hook system for guardrails. Powers Claude Code. 1.85M+ weekly npm downloads.

**When to choose**: Building on Anthropic models, need Computer Use / desktop automation, want built-in coding tools, need fine-grained permission hooks, TypeScript or Python teams.

---

## Table of Contents

1. [Agent Definition](#agent-definition)
2. [Built-in Tools](#built-in-tools)
3. [Custom Tools](#custom-tools)
4. [MCP Integration](#mcp-integration)
5. [Multi-Agent (Subagents)](#multi-agent-subagents)
6. [Guardrails (Hooks & Permissions)](#guardrails-hooks--permissions)
7. [Streaming & Events](#streaming--events)
8. [Computer Use](#computer-use)
9. [Python vs TypeScript](#python-vs-typescript)
10. [Testing](#testing)
11. [Model Support](#model-support)

---

## Agent Definition

No `Agent` class. Agents are defined functionally via `query()` with an options object.

**Python**:

```python
from claude_agent_sdk import query, ClaudeAgentOptions

options = ClaudeAgentOptions(
    system_prompt="You are a helpful assistant.",
    model="sonnet",
    allowed_tools=["Read", "Grep", "Glob", "mcp__my-server__*"],
    max_turns=20,
    max_budget_usd=1.0,
    effort="high",  # "low" | "medium" | "high" | "max"
)

async for message in query(prompt="Analyze this codebase", options=options):
    print(message)
```

**TypeScript**:

```typescript
import { query } from "@anthropic-ai/claude-agent-sdk";

const q = query({
  prompt: "Analyze this codebase",
  options: {
    systemPrompt: "You are a helpful assistant.",
    model: "sonnet",
    allowedTools: ["Read", "Grep", "Glob"],
    maxTurns: 20,
    maxBudgetUsd: 1.0,
  },
});

for await (const message of q) {
  console.log(message);
}
```

**Multi-turn conversations** (Python): Use `ClaudeSDKClient` for stateful sessions:

```python
from claude_agent_sdk import ClaudeSDKClient

async with ClaudeSDKClient(options=options) as client:
    await client.connect()
    response = await client.query("First question")
    response = await client.query("Follow-up question")
```

**Auth**: `ANTHROPIC_API_KEY` env var. Also supports Bedrock (`CLAUDE_CODE_USE_BEDROCK=1`), Vertex AI (`CLAUDE_CODE_USE_VERTEX=1`), Azure (`CLAUDE_CODE_USE_FOUNDRY=1`).

---

## Built-in Tools

These tools are provided by the SDK — no implementation needed:

| Tool | Purpose |
|------|---------|
| `Read` | Read files from filesystem |
| `Write` | Create new files |
| `Edit` | Precise string replacements in files |
| `Bash` | Execute terminal commands |
| `Glob` | Find files by pattern |
| `Grep` | Search file contents (ripgrep-based) |
| `WebSearch` | Web search |
| `WebFetch` | Fetch and parse web pages |
| `Task` | Invoke subagents |
| `AskUserQuestion` | Ask clarifying questions |

Control access via `allowed_tools` / `disallowed_tools`. `disallowed_tools` overrides everything including `bypassPermissions`.

---

## Custom Tools

Custom tools are defined as **in-process MCP servers** using `tool()` and `create_sdk_mcp_server()`.

**Python**:

```python
from claude_agent_sdk import tool, create_sdk_mcp_server

@tool("lookup_customer", "Look up customer by ID", {"customer_id": str})
async def lookup_customer(args: dict[str, Any]) -> dict[str, Any]:
    customer = await db.get(args["customer_id"])
    return {"content": [{"type": "text", "text": json.dumps(customer)}]}

server = create_sdk_mcp_server(
    name="my-tools", version="1.0.0", tools=[lookup_customer]
)

# Pass as MCP server in options
options = ClaudeAgentOptions(mcp_servers={"my-tools": server})
```

**TypeScript** (uses Zod for schemas):

```typescript
import { tool, z, createSdkMcpServer } from "@anthropic-ai/claude-agent-sdk";

const lookupCustomer = tool(
  "lookup_customer",
  "Look up customer by ID",
  { customer_id: z.string() },
  async (args) => ({
    content: [{ type: "text", text: JSON.stringify(await db.get(args.customer_id)) }],
  })
);

const server = createSdkMcpServer({
  name: "my-tools",
  version: "1.0.0",
  tools: [lookupCustomer],
});
```

Tool names follow MCP convention: `mcp__<server-name>__<tool-name>`.

**Tool annotations**: `readOnlyHint`, `destructiveHint`, `openWorldHint` for permission hints.

---

## MCP Integration

**Four transport types**:

```python
mcp_servers = {
    # stdio — local subprocess
    "github": {"command": "npx", "args": ["@modelcontextprotocol/server-github"]},
    # SSE — remote server
    "remote": {"type": "sse", "url": "https://mcp.example.com/sse"},
    # HTTP — standard HTTP
    "api": {"type": "http", "url": "https://mcp.example.com/mcp"},
    # SDK — in-process (custom tools)
    "my-tools": server,  # from create_sdk_mcp_server()
}
```

**Tool wildcards**: `mcp__github__*` allows all tools from a server.

**Tool search**: Auto-activates when MCP tool definitions exceed 10% of context window. Requires Sonnet 4+ or Opus 4+.

**Config file**: Also loadable from `.mcp.json`.

---

## Multi-Agent (Subagents)

Subagents are defined via `AgentDefinition` objects:

```python
from claude_agent_sdk import AgentDefinition

agents = [
    AgentDefinition(
        description="Use for code review tasks",
        prompt="You are an expert code reviewer. Focus on bugs and security.",
        tools=["Read", "Grep", "Glob"],
        model="sonnet",
        max_turns=10,
    ),
    AgentDefinition(
        description="Use for writing tests",
        prompt="You are a test engineer. Write comprehensive tests.",
        tools=["Read", "Write", "Edit", "Bash"],
        model="sonnet",
    ),
]

options = ClaudeAgentOptions(
    allowed_tools=["Task"],  # Parent must include Task
    agents=agents,
)
```

**Key constraints**:

- Parent must include `"Task"` in `allowedTools`
- **Subagents cannot spawn sub-subagents** (no `Task` in subagent tools)
- Multiple subagents can run in parallel
- Subagents can be resumed via session ID + agent ID
- Dynamic agent factories supported (create definitions at runtime)

**Three creation methods**: programmatic `AgentDefinition` (recommended), filesystem (`.claude/agents/*.md`), or built-in `general-purpose`.

---

## Guardrails (Hooks & Permissions)

**Hooks** intercept agent events at every lifecycle point.

**Key hook events**:

| Event | When | Can Do |
|-------|------|--------|
| `PreToolUse` | Before tool execution | Allow, deny, modify input |
| `PostToolUse` | After tool execution | Add context, log |
| `PostToolUseFailure` | Tool failed | Error handling |
| `PermissionRequest` | Permission needed | Approve/deny/ask |
| `SubagentStart/Stop` | Subagent lifecycle | Control delegation |
| `Notification` | Agent notifications | Logging, alerts |
| `Stop` | Agent stopping | Cleanup |

**PreToolUse permission decisions**:

- `"allow"` — approve execution
- `"deny"` — block execution
- `"ask"` — prompt user for decision
- `updatedInput` — modify tool input (requires `"allow"`)
- Priority: deny > ask > allow

**Python example**:

```python
from claude_agent_sdk import HookMatcher

async def block_writes(input_data, tool_use_id, context):
    if "/production/" in str(input_data.get("file_path", "")):
        return {"permissionDecision": "deny", "reason": "Cannot write to production"}
    return {"permissionDecision": "allow"}

options = ClaudeAgentOptions(
    hooks=[HookMatcher(matcher="Write|Edit", hooks=[block_writes])],
)
```

**Permission modes**: `"default"`, `"acceptEdits"`, `"plan"` (no execution), `"bypassPermissions"`, `"dontAsk"` (TS only).

**`can_use_tool`** — custom permission callback for fine-grained per-tool control.

---

## Streaming & Events

**Default**: Yields complete `AssistantMessage` objects after each turn.

**Streaming mode**: Set `include_partial_messages=True` for real-time token streaming.

**Message types**:

| Type | Content |
|------|---------|
| `system` (init) | Session initialization, available tools, model |
| `assistant` | Claude's responses with content blocks |
| `result` | Final result: `duration_ms`, `total_cost_usd`, `usage`, `structured_output` |
| `stream_event` | Raw token-by-token streaming events |

**Result message fields**: `is_error`, `num_turns`, `total_cost_usd`, `usage`, `structured_output`.

**Note**: Streaming is incompatible with explicit `max_thinking_tokens` and structured output.

---

## Computer Use

Computer Use is available through the Claude API's `computer_20251124` tool (Opus 4.6, Sonnet 4.6, Opus 4.5). In the Agent SDK context, browser/screen automation is typically achieved via MCP:

```python
mcp_servers = {
    "playwright": {"command": "npx", "args": ["@playwright/mcp@latest"]},
}
```

The SDK supports `sandbox` / `SandboxSettings` for containerized execution when using Computer Use.

**Important**: Computer Use requires a sandboxed environment. Never use in production without proper containment.

---

## Python vs TypeScript

| Aspect | Python | TypeScript |
|--------|--------|------------|
| Package | `claude-agent-sdk` | `@anthropic-ai/claude-agent-sdk` |
| Entry points | `query()` + `ClaudeSDKClient` | `query()` (returns `Query` object) |
| Multi-turn | `ClaudeSDKClient` context manager | `Query.streamInput()` or V2 `send()`/`stream()` |
| Tool schemas | Python types or JSON Schema dicts | Zod schemas |
| Hook events | 12 events | 18 events (6 TS-only) |
| Naming | `snake_case` | `camelCase` |

**TS-only hooks**: `SessionStart`, `SessionEnd`, `Setup`, `TeammateIdle`, `TaskCompleted`, `ConfigChange`.

---

## Testing

No built-in test framework. Recommended approaches:

- **Promptfoo**: Declarative YAML-based evaluation. Supports Claude Agent SDK as a provider. Assertion types from string matching to LLM-as-judge.
- **Hooks for testing**: Use `PreToolUse`/`PostToolUse` hooks to log, validate, or mock tool calls.
- **Result inspection**: Check `ResultMessage` fields: `is_error`, `num_turns`, `total_cost_usd`, `usage`.
- **Permission handlers**: Custom `can_use_tool` to sandbox or redirect operations in tests.

---

## Model Support

| Model | Value | Tool Search | Computer Use |
|-------|-------|-------------|--------------|
| Claude Sonnet (latest) | `"sonnet"` | Yes (4+) | `computer_20251124` (4.6) |
| Claude Opus (latest) | `"opus"` | Yes (4+) | `computer_20251124` (4.5+) |
| Claude Haiku (latest) | `"haiku"` | No | No |

Subagent models: `"sonnet"` / `"opus"` / `"haiku"` / `"inherit"` (from parent).

Third-party: Amazon Bedrock, Google Vertex AI, Microsoft Azure AI Foundry.

---

## Decision: When to Use Claude Agent SDK

**Choose Claude Agent SDK when**:

- Building on Anthropic models (Claude Sonnet/Opus/Haiku)
- Need Computer Use / desktop automation
- Want built-in coding tools (Read, Write, Edit, Bash, Grep, Glob)
- Need fine-grained permission hooks (18 lifecycle events)
- MCP-native tool ecosystem matters
- TypeScript team (more hook events, Zod schemas)

**Choose something else when**:

- Need model-agnostic framework → Pydantic AI, LangGraph
- Need visual workflow editor → LangGraph
- Need durable execution → Pydantic AI (Temporal/DBOS)
- Need A2A protocol → Pydantic AI (native `to_a2a()`)
- Non-Anthropic models required → LangGraph, Google ADK

# Code Agents & Software Engineering Agents

Production patterns for autonomous coding agents that perform end-to-end software engineering tasks.

---
## Table of Contents

- [Overview](#overview)
- [SE 3.0 Paradigm (Agentic Software Engineering)](#se-30-paradigm-agentic-software-engineering)
- [Architecture Patterns](#architecture-patterns)
- [1. Multi-Agent SWE Architecture (HyperAgent Pattern)](#1-multi-agent-swe-architecture-hyperagent-pattern)
- [2. Minimal Agent Pattern (Lita/Mini-SWE)](#2-minimal-agent-pattern-litamini-swe)
- [Production Considerations](#production-considerations)
- [1. Beyond Test Passing](#1-beyond-test-passing)
- [2. Guardrails for Code Agents](#2-guardrails-for-code-agents)
- [3. Human-in-the-Loop Checkpoints](#3-human-in-the-loop-checkpoints)
- [Tool Design for Code Agents](#tool-design-for-code-agents)
- [File Operations](#file-operations)
- [Code Search](#code-search)
- [Test Execution](#test-execution)
- [Benchmarks & Evaluation](#benchmarks-&-evaluation)
- [SWE-Bench](#swe-bench)
- [Beyond SWE-Bench](#beyond-swe-bench)
- [2025 Agent Benchmarks to Watch](#2025-agent-benchmarks-to-watch)
- [Configuration Patterns](#configuration-patterns)
- [Common CLAUDE.md Patterns](#common-claudemd-patterns)
- [Code Style](#code-style)
- [Boundaries](#boundaries)
- [Review Requirements](#review-requirements)
- [Effective Configurations](#effective-configurations)
- [Integration with MCP](#integration-with-mcp)
- [Anti-Patterns](#anti-patterns)
- [1. Unbounded Autonomy](#1-unbounded-autonomy)
- [2. Test-Only Validation](#2-test-only-validation)
- [3. Context Overload](#3-context-overload)
- [4. Ignoring Agent Uncertainty](#4-ignoring-agent-uncertainty)
- [References](#references)


## Overview

Code/SWE agents represent a distinct category of AI agents that autonomously:
- Resolve GitHub issues and implement features
- Navigate codebases and understand context
- Edit files, run tests, and iterate on failures
- Create pull requests with proper documentation

**Key Distinction**: Unlike code completion tools (Copilot autocomplete), SWE agents operate autonomously across entire repositories with multi-step planning and execution.

---

## SE 3.0 Paradigm (Agentic Software Engineering)

Software engineering is evolving through three paradigms:

| Era | Paradigm | Developer Role | AI Role |
|-----|----------|----------------|---------|
| SE 1.0 | Manual | Write all code | None |
| SE 2.0 | Assisted | Write with suggestions | Autocomplete, snippets |
| **SE 3.0** | **Agentic** | **Review and guide** | **Autonomous implementation** |

**SE 3.0 Definition**: Intent-driven, conversational development where developers collaborate with autonomous AI teammates.

**Scale**: a 129,134-project peer-reviewed study found roughly 16–23% of active GitHub projects show coding-agent traces by late 2025 (arXiv:2601.18341). Treat single-vendor "N PRs in N weeks" scale claims as marketing, not measurement, unless traced to a study like this one.

---

## Architecture Patterns

### 1. Multi-Agent SWE Architecture (HyperAgent Pattern)

```
┌─────────────────────────────────────────────────────────────┐
│                      PLANNER AGENT                          │
│  - Decomposes issue into subtasks                          │
│  - Creates execution plan                                   │
│  - Coordinates other agents                                 │
└─────────────────────────┬───────────────────────────────────┘
                          │
          ┌───────────────┼───────────────┐
          ▼               ▼               ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ NAVIGATOR AGENT │ │ CODE EDITOR     │ │ EXECUTOR AGENT  │
│ - Search code   │ │ AGENT           │ │ - Run tests     │
│ - Find files    │ │ - Write/edit    │ │ - Execute cmds  │
│ - Understand    │ │   code          │ │ - Verify fixes  │
│   structure     │ │ - Refactor      │ │ - Capture output│
└─────────────────┘ └─────────────────┘ └─────────────────┘
```

### 2. Minimal Agent Pattern (Lita/Mini-SWE)

Research shows "light" agent philosophies get most of the way to full-agent performance with ~100 lines of code — the original mini-swe-agent result was 68% on SWE-bench (2025); the harness has since matured to >74% on SWE-bench Verified (2026). Verify the current README before quoting a figure; the durable lesson is architectural simplicity, not the specific number:

```python
class MinimalSWEAgent:
    def __init__(self, llm, tools):
        self.llm = llm
        self.tools = {
            "read_file": read_file,
            "write_file": write_file,
            "run_command": run_command,
            "search_code": search_code
        }

    def solve(self, issue: str) -> str:
        """Single ReAct loop with file tools"""
        context = f"Issue: {issue}\n"

        for step in range(MAX_STEPS):
            action = self.llm.decide(context, self.tools)
            result = self.execute(action)
            context += f"\nAction: {action}\nResult: {result}"

            if action.type == "submit":
                return action.patch

        return None
```

**Key Insight**: Architectural complexity doesn't always correlate with performance. Start minimal, add complexity only when needed.

---

## Production Considerations

### 1. Beyond Test Passing

**Critical Finding**: 29.6% of "plausible" SWE-Bench fixes (those passing tests) introduce behavioral regressions.

**Implication**: Passing tests is necessary but insufficient. Production deployments require:

- Behavioral regression testing
- Code review by humans
- Integration testing beyond unit tests
- Semantic diff analysis

### 2. Guardrails for Code Agents

```yaml
code_agent_guardrails:
  execution_limits:
    max_steps: 50
    max_file_edits: 20
    timeout_minutes: 30

  allowed_operations:
    - read_file
    - write_file
    - search_code
    - run_tests
    - git_operations: [add, commit, diff, status]

  forbidden_operations:
    - delete_repository
    - force_push
    - modify_ci_config
    - access_secrets

  review_triggers:
    - changes_to_security_files
    - more_than_10_files_modified
    - changes_to_deployment_config
```

### 3. Human-in-the-Loop Checkpoints

```
Issue Assigned → Agent Plans → [HUMAN REVIEW] → Agent Implements
                                     ↓
                              Reject / Modify
                                     ↓
                              Agent Re-plans

Agent Implements → Agent Tests → [HUMAN REVIEW] → Merge/Deploy
                                      ↓
                               Request Changes
                                      ↓
                               Agent Iterates
```

---

## Tool Design for Code Agents

### File Operations

```json
{
  "name": "edit_file",
  "description": "Edit a file by replacing specific content",
  "parameters": {
    "file_path": {
      "type": "string",
      "description": "Path relative to repository root"
    },
    "old_content": {
      "type": "string",
      "description": "Exact content to replace (must be unique in file)"
    },
    "new_content": {
      "type": "string",
      "description": "Content to insert"
    }
  }
}
```

**Design Principle**: Use search-and-replace over line numbers. Line numbers shift; content patterns are stable.

### Code Search

```json
{
  "name": "search_codebase",
  "description": "Search for code patterns across repository",
  "parameters": {
    "query": {
      "type": "string",
      "description": "Search query (supports regex)"
    },
    "file_pattern": {
      "type": "string",
      "description": "Glob pattern for files to search"
    },
    "context_lines": {
      "type": "integer",
      "default": 3,
      "description": "Lines of context around matches"
    }
  }
}
```

### Test Execution

```json
{
  "name": "run_tests",
  "description": "Execute test suite and return results",
  "parameters": {
    "test_path": {
      "type": "string",
      "description": "Specific test file/directory or empty for full suite"
    },
    "timeout": {
      "type": "integer",
      "default": 300,
      "description": "Timeout in seconds"
    }
  },
  "returns": {
    "passed": "integer",
    "failed": "integer",
    "errors": "array of failure details"
  }
}
```

---

## Benchmarks & Evaluation

### SWE-Bench

The primary benchmark for code agents:

| Metric | Description |
|--------|-------------|
| **Resolved** | Issue fully fixed, all tests pass |
| **Plausible** | Tests pass but may have regressions |
| **Attempted** | Agent produced a patch |

**Current Leaders** (as of mid-2026; verify against the live leaderboard before quoting):
- Claude Code, Devin-class, and other harness-plus-frontier-model stacks leading
- Current-generation base models (Claude Opus 4.8-class, GPT-5.5-class)
- Multi-agent architectures outperforming single-agent

### Beyond SWE-Bench

Additional evaluation dimensions:

1. **Code Quality**: Does generated code follow project conventions?
2. **Explanation Quality**: Can the agent explain its changes?
3. **Iteration Efficiency**: How many attempts to reach solution?
4. **Scope Creep**: Does the agent make unnecessary changes?

### 2025 Agent Benchmarks to Watch

- **Tool Use**: BFCL (iterative, multi-turn function calling) raises bar beyond ToolBench/API-Bank.
- **Deep Research**: BrowseComp / BrowseComp-ZH / BrowseComp-Plus track evidence synthesis across web; top models still near-zero success.
- **GUI**: WebGen-Bench (full multi-file site generation) and Web-Bench (sequential UI coding tasks) stress multi-step reasoning.
- **OS/Terminal**: Terminal-Bench measures full-system CLI autonomy (build kernels, deploy servers) vs repo-bounded SWE-Bench.

---

## Configuration Patterns

Based on analysis of 328 Claude Code project configurations:

### Common CLAUDE.md Patterns

```markdown
## Code Style
- Follow existing patterns in the codebase
- Run linter before committing
- Add tests for new functionality

## Boundaries
- Do not modify CI/CD configuration
- Do not access external APIs without approval
- Keep changes focused on the specific issue

## Review Requirements
- All changes require human review
- Security-sensitive files need explicit approval
```

### Effective Configurations

1. **Explicit boundaries** outperform implicit ones
2. **Examples** improve adherence to style
3. **Tool allowlists** reduce unexpected behaviors
4. **Checkpoints** catch issues early

---

## Integration with MCP

Code agents benefit from MCP servers for:

```yaml
mcp_servers:
  filesystem:
    purpose: "Read/write project files"
    capabilities: [read, write, search]

  git:
    purpose: "Version control operations"
    capabilities: [status, diff, commit, branch]

  terminal:
    purpose: "Run commands (tests, linters, builds)"
    capabilities: [execute_command]

  github:
    purpose: "PR/issue management"
    capabilities: [create_pr, comment, request_review]
```

---

## Domain Agents Built on Coding-Agent Infrastructure

The same Claude Code infrastructure used for SWE agents can power non-coding domain agents. The skill/tool/subagent patterns transfer directly.

### Case Study: career-ops (Job Search Agent)

**Repository:** github.com/santifer/career-ops (23K+ stars, April 2026)

career-ops is a multi-agent job-search system built entirely on Claude Code. It demonstrates that coding-agent infrastructure generalizes beyond code:

| Component | Implementation |
|-----------|---------------|
| Agent shape | Workflow agent (evaluation pipeline) + tool-using agent (portal scanning) |
| Skill count | 14 skill modes for different job-search operations |
| Parallel execution | Batch-processes 10+ job offers simultaneously |
| Tool integration | Scans 45+ company portals (Greenhouse, Ashby, Lever, Workable) |
| Structured output | A-F grading across 10 weighted dimensions per listing |
| Pipeline state | Centralized application tracking with terminal dashboard (Go) |
| Filtering | Auto-rejects listings scoring below 4.0/5.0 |

**Architecture patterns reused from SWE agents:**
- Skill-based decomposition (14 modes vs. SWE review/implement/test modes)
- Structured evaluation with scoring rubrics (job grading vs. code quality scoring)
- Batch parallel execution (multi-offer vs. multi-file processing)
- Human-in-the-loop for high-stakes decisions (application submission vs. code merge)

**Implication:** When designing agent architectures, consider whether the workflow/tool/skill patterns from coding agents can serve the target domain before building from scratch.

---

## Anti-Patterns

### 1. Unbounded Autonomy

**Problem**: Agent makes sweeping changes without checkpoints
**Solution**: Implement step limits, change size limits, and review gates

### 2. Test-Only Validation

**Problem**: Agent optimizes for passing tests, not correctness
**Solution**: Human review, behavioral regression tests, semantic analysis

### 3. Context Overload

**Problem**: Feeding entire codebase to agent
**Solution**: Progressive context loading, relevant file retrieval

### 4. Ignoring Agent Uncertainty

**Problem**: Treating all agent outputs as equally confident
**Solution**: Confidence scoring, escalation for low-confidence actions

---

## References

- [The Rise of AI Teammates in SE 3.0](https://arxiv.org/abs/2507.15003) - 456K PR analysis
- [HyperAgent: Generalist SWE Agents](https://arxiv.org/abs/2409.16299) - Multi-agent architecture
- [Agentic Software Engineering: Research Roadmap](https://arxiv.org/abs/2509.06216) - Foundational pillars
- [AI Agentic Programming Survey](https://arxiv.org/abs/2508.11126) - Taxonomy and patterns
- [Claude Code Configuration Study](https://arxiv.org/abs/2511.09268) - Real-world configurations

# Coding Agent Usage Tracking

Practical guide to measuring actual token usage and costs from Claude Code and OpenAI Codex CLI sessions using the ccusage tool family.

**Freshness anchor:** April 2026 — covers ccusage (Claude Code), @ccusage/codex (Codex CLI), @ccusage/mcp. Verify CLI options with `--help` before recommending; both tools evolve rapidly.

---

## Table Of Contents

- [Why Track CLI Agent Usage](#why-track-cli-agent-usage)
- [Data Sources](#data-sources)
- [Raw Log Formats](#raw-log-formats)
- [DIY Parsing Without ccusage](#diy-parsing-without-ccusage)
- [ccusage For Claude Code](#ccusage-for-claude-code)
- [ccusage Codex For OpenAI Codex CLI](#ccusage-codex-for-openai-codex-cli)
- [Common Options](#common-options)
- [JSON Export And Dashboard Integration](#json-export-and-dashboard-integration)
- [Unified Cost View](#unified-cost-view)
- [MCP Integration](#mcp-integration)
- [Cost Alerting Patterns](#cost-alerting-patterns)
- [Anti-Patterns](#anti-patterns)
- [Related References](#related-references)
- [Primary Sources](#primary-sources)

---

## Why Track CLI Agent Usage

[`agent-economics.md`](agent-economics.md) provides the ROI framework for projecting agent costs. This guide complements it with actual measurement.

| Concern | Projection (agent-economics) | Measurement (this guide) |
|---------|------------------------------|--------------------------|
| "Will this agent pay for itself?" | ROI formula, break-even volume | Actual monthly spend vs value created |
| "Which model costs most?" | Per-model pricing tables | Per-model token breakdown from real sessions |
| "Are we within budget?" | Monthly cost projections | Daily/weekly spend alerts from CLI data |
| "Should we kill this agent?" | Kill signals and thresholds | Actual cost trend to validate kill decision |

Use cases:

- Individual developer cost awareness
- Team budget enforcement and allocation
- ROI validation against projected costs from agent-economics framework
- Audit trail for enterprise compliance

---

## Data Sources

Both tools read local JSONL logs. No API calls, no credentials, no data leaves the machine.

| CLI Tool | Log Location | Override | Format |
|----------|-------------|----------|--------|
| Claude Code | `~/.config/claude/projects/` (v1.0.30+), legacy `~/.claude/projects/` | — | JSONL per conversation |
| OpenAI Codex | `~/.codex/sessions/` | `CODEX_HOME` env var | JSONL per session |

Logs accumulate automatically — no opt-in required.

---

## Raw Log Formats

Understanding the raw JSONL structure lets you query data directly, build custom analysis, or debug ccusage output.

### Claude Code JSONL Schema

Each file at `~/.claude/projects/{project-hash}/{session-uuid}.jsonl` contains one JSON object per line. Assistant messages carry token usage:

```json
{
  "type": "assistant",
  "sessionId": "session-uuid",
  "timestamp": "2026-04-07T14:23:01.000Z",
  "message": {
    "model": "claude-sonnet-4-6",
    "id": "msg_abc123",
    "usage": {
      "input_tokens": 3500,
      "output_tokens": 420,
      "cache_creation_input_tokens": 14492,
      "cache_read_input_tokens": 13359
    }
  },
  "costUSD": 0.05,
  "requestId": "req_xyz789"
}
```

**Token fields** (in `message.usage`):

| Field | Meaning |
|-------|---------|
| `input_tokens` | Standard prompt tokens |
| `output_tokens` | Generated response tokens |
| `cache_creation_input_tokens` | Tokens written to cache (optional) |
| `cache_read_input_tokens` | Tokens read from cache — cheaper rate (optional) |

**Other useful fields:**
- `message.model` — exact model name (e.g., `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-4-6`)
- `costUSD` — pre-calculated cost (may be 0; ccusage recalculates from tokens when using `--mode calculate`)
- `timestamp` — ISO 8601, used for date grouping
- `sessionId` — groups messages into conversations

**Additional data sources on disk:**
- `~/.claude/stats-cache.json` — pre-aggregated daily/model stats (messageCount, tokensByModel, modelUsage with per-model totals)
- `~/.claude/usage-data/session-meta/{session-uuid}.json` — per-session summaries with tool_counts, languages, git activity

### Codex CLI JSONL Schema

Each file at `~/.codex/sessions/YYYY/MM/DD/rollout-{timestamp}-{uuid}.jsonl` contains timestamped events. Token data is in `event_msg` entries with `token_count` type:

```json
{
  "timestamp": "2026-04-07T14:30:45.433Z",
  "type": "event_msg",
  "payload": {
    "type": "token_count",
    "info": {
      "total_token_usage": {
        "input_tokens": 622719,
        "cached_input_tokens": 575104,
        "output_tokens": 9682,
        "reasoning_output_tokens": 3918,
        "total_tokens": 632401
      },
      "last_token_usage": {
        "input_tokens": 61358,
        "cached_input_tokens": 61056,
        "output_tokens": 371,
        "reasoning_output_tokens": 173,
        "total_tokens": 61729
      }
    }
  }
}
```

**Key difference from Claude Code:** Codex logs **cumulative** totals (`total_token_usage`) plus the **last turn** delta (`last_token_usage`). ccusage computes per-turn deltas by subtracting previous cumulative values.

**Token fields** (in `payload.info.total_token_usage` or `last_token_usage`):

| Field | Meaning |
|-------|---------|
| `input_tokens` | Standard prompt tokens |
| `cached_input_tokens` | Tokens served from cache — cheaper rate |
| `output_tokens` | Generated tokens (includes reasoning) |
| `reasoning_output_tokens` | Reasoning tokens — informational, not separately billed |
| `total_tokens` | Sum of input + output |

For cost accounting, sum **only** `last_token_usage` per request. `total_token_usage`
is cumulative within an epoch and summing it across events duplicates tokens. If
`last_token_usage` is absent, reconstruct a delta from consecutive totals; a
decrease starts a new epoch (for example after context compaction or model
change). Never add `reasoning_output_tokens` again: it is already within
`output_tokens`.

`scripts/codex-usage.py traces` emits bounded accounting rows only (session ID,
timestamp, counters, model, source and pricing provenance), never prompt or tool
content. Cost is fail-closed as `unpriced` unless the exact model ID, standard
service tier, standard context-pricing class and every applicable cache rate are
known. A total-delta reconstruction is marked `estimated`; `last_token_usage`
with all of those conditions is `exact`. A model ID alone cannot distinguish
GPT-5.6 Sol standard from Fast or long-context billing.

**Model name** comes from `turn_context` entries (separate JSONL lines with `type: "turn_context"`):

```json
{
  "type": "turn_context",
  "payload": {
    "model": "gpt-5.4",
    "effort": "xhigh"
  }
}
```

**Other Codex data sources:**
- `~/.codex/session_index.jsonl` — quick lookup with session ID, thread name, timestamp
- `~/.codex/history.jsonl` — global session history
- `~/.codex/state_5.sqlite` — `threads` table has aggregated tokens_used per session

---

## DIY Parsing Without ccusage

### Claude Code — Daily Token Totals (jq)

```bash
# Sum tokens per model across all sessions for today
find ~/.claude/projects -name '*.jsonl' -newer /tmp/today_marker | \
  xargs cat | \
  jq -r 'select(.message.usage != null) |
    "\(.message.model // "unknown"),\(.message.usage.input_tokens),\(.message.usage.output_tokens),\(.message.usage.cache_creation_input_tokens // 0),\(.message.usage.cache_read_input_tokens // 0)"' | \
  awk -F, '{m[$1]+=$2; o[$1]+=$3; cc[$1]+=$4; cr[$1]+=$5}
    END {for (k in m) printf "%s: input=%d output=%d cache_create=%d cache_read=%d\n", k, m[k], o[k], cc[k], cr[k]}'
```

### Claude Code — Quick Session Summary (Python)

```python
import json, glob, os
from collections import defaultdict

totals = defaultdict(lambda: {"input": 0, "output": 0, "cache_read": 0, "cache_create": 0})

for path in glob.glob(os.path.expanduser("~/.claude/projects/*/*.jsonl")):
    for line in open(path):
        try:
            rec = json.loads(line)
            usage = rec.get("message", {}).get("usage")
            if not usage:
                continue
            model = rec.get("message", {}).get("model", "unknown")
            totals[model]["input"] += usage.get("input_tokens", 0)
            totals[model]["output"] += usage.get("output_tokens", 0)
            totals[model]["cache_read"] += usage.get("cache_read_input_tokens", 0)
            totals[model]["cache_create"] += usage.get("cache_creation_input_tokens", 0)
        except json.JSONDecodeError:
            continue

for model, t in sorted(totals.items()):
    print(f"{model}: in={t['input']:,} out={t['output']:,} "
          f"cache_read={t['cache_read']:,} cache_create={t['cache_create']:,}")
```

### Claude Code — Use stats-cache.json (fastest)

```bash
# Pre-aggregated daily stats — no JSONL parsing needed
# dailyActivity is an array of {date, messageCount, sessionCount, toolCallCount}
jq '.dailyActivity | sort_by(.date) | .[-7:][] |
  "\(.date): messages=\(.messageCount) sessions=\(.sessionCount) tools=\(.toolCallCount)"' \
  ~/.claude/stats-cache.json

# Per-model all-time totals
jq '.modelUsage | to_entries[] |
  "\(.key): input=\(.value.inputTokens) output=\(.value.outputTokens) cache_read=\(.value.cacheReadInputTokens)"' \
  ~/.claude/stats-cache.json
```

### Codex — Extract Token Deltas (jq)

```bash
# Get per-turn token usage from a single session
jq -r 'select(.type == "event_msg" and .payload.type == "token_count") |
  .payload.info.last_token_usage |
  "\(.input_tokens),\(.cached_input_tokens),\(.output_tokens),\(.reasoning_output_tokens)"' \
  ~/.codex/sessions/2026/04/07/rollout-*.jsonl
```

### Codex — Daily Totals Across Sessions (Python)

Note: Codex JSONL entries can have null `last_token_usage` — always fall back to `total_token_usage` and guard against non-dict values.

```python
import json, glob, os
from collections import defaultdict

daily = defaultdict(lambda: {"input": 0, "output": 0, "cached": 0, "reasoning": 0})

for path in glob.glob(os.path.expanduser("~/.codex/sessions/*/*/*/**.jsonl")):
    for line in open(path):
        try:
            rec = json.loads(line)
            if not isinstance(rec, dict) or rec.get("type") != "event_msg":
                continue
            payload = rec.get("payload")
            if not isinstance(payload, dict) or payload.get("type") != "token_count":
                continue
            info = payload.get("info")
            if not isinstance(info, dict):
                continue
            usage = info.get("last_token_usage")
            if not isinstance(usage, dict):
                usage = info.get("total_token_usage")
            if not isinstance(usage, dict):
                continue
            date = rec.get("timestamp", "")[:10]
            if not date:
                continue
            daily[date]["input"] += usage.get("input_tokens", 0) or 0
            daily[date]["output"] += usage.get("output_tokens", 0) or 0
            daily[date]["cached"] += usage.get("cached_input_tokens", 0) or 0
            daily[date]["reasoning"] += usage.get("reasoning_output_tokens", 0) or 0
        except Exception:
            continue

for date in sorted(daily):
    t = daily[date]
    print(f"{date}: in={t['input']:,} out={t['output']:,} "
          f"cached={t['cached']:,} reasoning={t['reasoning']:,}")
```

### Standalone Scripts

For a full CLI experience without ccusage, use the scripts in this skill:

```bash
# Claude Code
python scripts/claude-usage.py daily                          # daily activity from stats-cache
python scripts/claude-usage.py monthly --since 2026-01-01     # monthly from JSONL
python scripts/claude-usage.py sessions --last 10             # recent sessions
python scripts/claude-usage.py models                         # per-model all-time totals
python scripts/claude-usage.py daily --json                   # JSON output

# Codex
python scripts/codex-usage.py daily                           # daily token/cost report
python scripts/codex-usage.py monthly                         # monthly aggregated
python scripts/codex-usage.py sessions --last 5               # recent sessions
python scripts/codex-usage.py models                          # per-model breakdown
python scripts/codex-usage.py daily --since 2026-04-01 --json # filtered JSON
```

Both scripts are stdlib-only Python (no pip install), support `--since`/`--until`/`--json`/`--last` flags, and include approximate cost estimates.

Scripts: [`../scripts/claude-usage.py`](../scripts/claude-usage.py), [`../scripts/codex-usage.py`](../scripts/codex-usage.py)

### How ccusage Adds Value Over DIY

| What ccusage handles | DIY effort |
|---------------------|-----------|
| Deduplication by message+request ID hash | You must track seen IDs yourself |
| LiteLLM pricing lookup and caching | You must maintain a pricing table |
| Delta calculation from Codex cumulative totals | You must track previous totals per session |
| Responsive terminal tables | Raw numbers only |
| Date/timezone grouping with locale formatting | Manual date parsing |
| Model alias resolution (e.g., `gpt-5-codex` → `gpt-5`) | You must maintain alias map |

**Recommendation:** Use DIY for quick one-off queries or custom analysis. Use ccusage for recurring reporting and cost monitoring.

---

## ccusage For Claude Code

### Installation

```bash
# Run without global install (recommended)
npx ccusage daily
bunx ccusage daily

# Global install
npm install -g ccusage
```

### Commands

| Command | Purpose | Example |
|---------|---------|---------|
| `daily` | Usage grouped by calendar date | `ccusage daily` |
| `weekly` | Usage grouped by week | `ccusage weekly` |
| `monthly` | Monthly aggregated report | `ccusage monthly` |
| `sessions` | Per-conversation detail | `ccusage sessions` |
| `blocks` | 5-hour billing window tracking | `ccusage blocks` |

### Key Options (ccusage-specific)

| Option | Purpose |
|--------|---------|
| `--breakdown`, `-b` | Show per-model cost breakdown |
| `--instances`, `-i` | Group daily results by project |
| `--project NAME`, `-p` | Filter to a specific project |
| `--start-of-week mon\|sun` | Week boundary for `weekly` command |
| `--active`, `-a` | Show current active block (`blocks`) |
| `--live` | Live monitoring mode (`blocks`) |
| `--mode auto\|calculate\|display` | Cost calculation method |

### Output

Each report shows per-model rows with: input tokens, cached input tokens, output tokens, reasoning output tokens, total tokens, and calculated cost in USD.

---

## ccusage Codex For OpenAI Codex CLI

**Status:** Experimental beta. Expect breaking changes.

### Installation

```bash
# Run without global install (recommended — always use @latest)
npx @ccusage/codex@latest daily
bunx @ccusage/codex@latest daily

# Shell alias (recommended)
alias ccusage-codex='npx @ccusage/codex@latest'
```

### Commands

| Command | Purpose | Example |
|---------|---------|---------|
| `daily` | Usage grouped by calendar date | `ccusage-codex daily` |
| `monthly` | Monthly aggregated report | `ccusage-codex monthly` |
| `sessions` | Per-session detail | `ccusage-codex sessions` |

Note: `weekly` and `blocks` are not available in @ccusage/codex. Check `--help` for current command list.

### Environment Variables

| Variable | Purpose |
|----------|---------|
| `CODEX_HOME` | Override root directory (default: `~/.codex`) |
| `LOG_LEVEL` | Verbosity: 0 (silent) through 5 (trace) |

### Limitations

- No data before September 6, 2025 (when Codex CLI started emitting token events)
- Some early September 2025 sessions without model metadata are skipped
- Falls back to `gpt-5` model name when metadata is missing

---

## Common Options

Shared across both `ccusage` and `@ccusage/codex`:

| Option | Purpose | Example |
|--------|---------|---------|
| `--json`, `-j` | Machine-readable JSON output | `ccusage daily --json` |
| `--since DATE` | Filter from date (YYYY-MM-DD or YYYYMMDD) | `--since 2026-04-01` |
| `--until DATE` | Filter to date (inclusive) | `--until 2026-04-07` |
| `--timezone ZONE`, `-z` | Timezone for date grouping | `-z America/New_York` |
| `--locale LOCALE`, `-l` | Date formatting locale | `-l en-US` |
| `--offline`, `-O` | Use cached pricing (no network) | `--offline` |
| `--config PATH` | Custom config file path | `--config ./my-config.json` |

### Configuration File Precedence

1. Command-line arguments (highest)
2. Custom config file (`--config`)
3. Local project config (`.ccusage/ccusage.json` or `.ccusage/codex.json`)
4. User config (`~/.config/claude/ccusage.json`)
5. Built-in defaults (lowest)

---

## JSON Export And Dashboard Integration

Both tools support `--json` for programmatic output.

```bash
# Export daily Claude Code spend to file
ccusage daily --json --since 2026-04-01 > claude-code-april.json

# Extract just dates and costs with jq
ccusage daily --json | jq '.daily[] | {date, costUSD}'

# Export Codex monthly spend
npx @ccusage/codex@latest monthly --json > codex-monthly.json
```

### JSON Structure (daily example)

```json
{
  "daily": [
    {
      "date": "2026-04-07",
      "inputTokens": 125000,
      "cachedInputTokens": 40000,
      "outputTokens": 30000,
      "reasoningOutputTokens": 5000,
      "totalTokens": 155000,
      "costUSD": 1.23,
      "models": {
        "claude-sonnet-4-6": { "inputTokens": 125000, "outputTokens": 30000, "..." : "..." }
      }
    }
  ],
  "totals": { "inputTokens": 125000, "costUSD": 1.23, "..." : "..." }
}
```

Pipe JSON into Metabase, Grafana, or a spreadsheet for team dashboards.

---

## Unified Cost View

### Shell Aliases

```bash
# Today's spend across both tools
alias ai-spend-today='echo "=== Claude Code ===" && ccusage daily --since $(date +%Y-%m-%d) && echo "=== Codex ===" && npx @ccusage/codex@latest daily --since $(date +%Y-%m-%d)'

# Monthly summary
alias ai-spend-month='echo "=== Claude Code ===" && ccusage monthly && echo "=== Codex ===" && npx @ccusage/codex@latest monthly'

# JSON combined export
alias ai-spend-json='echo "{\"claude_code\":" && ccusage daily --json && echo ",\"codex\":" && npx @ccusage/codex@latest daily --json && echo "}"'
```

### Cross-Tool Command Comparison

| Dimension | ccusage (Claude Code) | @ccusage/codex (Codex) |
|-----------|----------------------|------------------------|
| Today's spend | `ccusage daily --since $(date +%Y-%m-%d)` | `npx @ccusage/codex@latest daily --since $(date +%Y-%m-%d)` |
| This month | `ccusage monthly` | `npx @ccusage/codex@latest monthly` |
| Session drill-down | `ccusage sessions` | `npx @ccusage/codex@latest sessions` |
| JSON for scripts | `ccusage daily --json` | `npx @ccusage/codex@latest daily --json` |
| Per-model breakdown | `ccusage daily -b` | Built into default output |

---

## MCP Integration

`@ccusage/mcp` exposes usage data as MCP tools so agents can query their own cost data.

### Use Cases

- Agent self-monitoring: check remaining budget before proceeding
- Cost-aware agent loops: degrade to cheaper model when budget is low
- Automated cost reports via agent tools

### Setup

Add to your MCP configuration (`.mcp.json` or `claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "ccusage": {
      "command": "npx",
      "args": ["@ccusage/mcp@latest"]
    }
  }
}
```

For MCP server configuration details, see [`../../agents-mcp/SKILL.md`](../../agents-mcp/SKILL.md).

---

## Cost Alerting Patterns

### Simple Threshold Script

```bash
#!/bin/bash
# check-ai-spend.sh — alert when daily spend exceeds budget
DAILY_BUDGET=5.00
SPEND=$(ccusage daily --since "$(date +%Y-%m-%d)" --json | jq '.totals.costUSD // 0')

if (( $(echo "$SPEND > $DAILY_BUDGET" | bc -l) )); then
  echo "ALERT: Daily Claude Code spend \$$SPEND exceeds budget \$$DAILY_BUDGET"
  # Add: slack webhook, email, or desktop notification
fi
```

### Cron-Based Monitoring

```cron
# Check Claude Code spend every 4 hours
0 */4 * * * /path/to/check-ai-spend.sh >> /var/log/ai-spend.log 2>&1
```

### Team-Level Patterns

- Aggregate individual developer JSON exports into a shared dashboard
- Set per-developer daily or weekly budgets with threshold scripts
- Weekly automated cost report via cron and Slack webhook
- Compare actual spend against [`agent-economics.md`](agent-economics.md) ROI projections

---

## Anti-Patterns

| Anti-Pattern | Why It Fails | Better Approach |
|--------------|-------------|-----------------|
| Never checking usage | Surprise bills, no ROI data | Run `ccusage monthly` at minimum |
| Checking only totals | Cannot optimize per-model costs | Use `--breakdown` or `--json` for per-model data |
| Manual JSONL parsing for recurring reports | Fragile; format changes break scripts | Use ccusage `--json` for stable output; reserve DIY for one-off analysis |
| Ignoring cached input tokens | Overestimates actual cost | ccusage already accounts for cached pricing |
| Tracking one tool but not the other | Incomplete cost picture | Track both Claude Code and Codex |
| Hardcoding pricing in scripts | Prices change frequently | Let ccusage fetch from LiteLLM or use `--offline` cache |

---

## Related References

- [Agent Economics & ROI Framework](agent-economics.md) — ROI projections and cost decision framework
- [Evaluation & Observability](evaluation-and-observability.md) — Production telemetry with OpenTelemetry
- [Code & SWE Agents](code-swe-agents.md) — Coding agent operating patterns
- [`dev-ai-coding-metrics`](../../dev-ai-coding-metrics/SKILL.md) — Pilot metrics, adoption, and ROI scorecards
- [`agents-mcp`](../../agents-mcp/SKILL.md) — MCP server setup for @ccusage/mcp

## Primary Sources

- ccusage (Claude Code): [github.com/ryoppippi/ccusage](https://github.com/ryoppippi/ccusage)
- @ccusage/codex (Codex CLI): [npmjs.com/package/@ccusage/codex](https://www.npmjs.com/package/@ccusage/codex)
- @ccusage/mcp (MCP server): [npmjs.com/package/@ccusage/mcp](https://www.npmjs.com/package/@ccusage/mcp)
- Documentation: [ccusage.com](https://ccusage.com/)

# Context Engineering — Structured Context Management

**Purpose**: Context engineering matters more than model selection. Even weaker LLMs perform well with proper context structure.

> **Scope.** This file covers the *agent-loop* slice — tool-result projection,
> scratchpad shape, progressive disclosure, generation triggers. For the
> bundle-assembly side (per-surface bundles, budgets, ACL scope, evidence
> contracts) and the six runtime verbs **write / select / compress / isolate /
> order / format**, see
> `ai-context-layer` and
> [`../../ai-context-layer/references/context-hygiene.md`](../../ai-context-layer/references/context-hygiene.md).

---
## Table of Contents

- [Core Principle](#core-principle)
- [Progressive Disclosure](#progressive-disclosure)
- [Bad: Load all context upfront](#bad-load-all-context-upfront)
- [Good: Load on-demand](#good-load-on-demand)
- [Session Management](#session-management)
- [Memory Provenance](#memory-provenance)
- [Generation Triggers](#generation-triggers)
- [After phase boundary](#after-phase-boundary)
- [After confidence drop](#after-confidence-drop)
- [After new entity](#after-new-entity)
- [Background vs Blocking Operations](#background-vs-blocking-operations)
- [Background write](#background-write)
- [Blocking write](#blocking-write)
- [Retrieval Timing](#retrieval-timing)
- [Multimodal Context](#multimodal-context)
- [Fresh Contexts](#fresh-contexts)
- [Spawn new agent with fresh context](#spawn-new-agent-with-fresh-context)
- [Context Size Management](#context-size-management)
- [Context Validation](#context-validation)
- [Related Resources](#related-resources)
- [Usage Notes](#usage-notes)


## Core Principle

**Key Insight**: Structured context management has more impact on agent performance than model selection.

**Evidence**: Proper context structure allows even weaker LLMs to perform comparably to stronger models on complex tasks.

---

## Progressive Disclosure

**What**: Load context on-demand rather than upfront

**Pattern**:
```text
1. Route by domain (classify intent)
2. Retrieve relevant context (lazy load)
3. Inject only what's needed (filter by relevance)
4. Expand context if needed (iterative refinement)
```

**Benefits**:
- Reduces token costs
- Improves latency
- Minimizes irrelevant information
- Scales better with large knowledge bases

**Implementation**:
```yaml
# Bad: Load all context upfront
context = load_all_knowledge()

# Good: Load on-demand
domain = classify_query(query)
context = retrieve_by_domain(domain, query)
```

---

## Session Management

**What**: Treat sessions as conversation containers with proper lifecycle management

**Best Practices**:

1. **Framework differences**: Honor session handling differences across frameworks (LangChain, LangGraph, CrewAI)
2. **Shared sessions**: Share session handles safely across agents with scoped replay
3. **Session boundaries**: Clear session start/end; no context leakage between sessions
4. **Session state**: Persist critical state; allow recovery on failures
5. **Session cleanup**: Expire inactive sessions; enforce retention policies

**Session Lifecycle**:
```yaml
session:
  id: "sess-abc-123"
  started_at: "2024-01-01T00:00:00Z"
  state:
    conversation_history: []
    task_context: {}
    user_preferences: {}
  ttl: 3600  # seconds
```

---

## Memory Provenance

**What**: Track lineage (source, timestamp, approvals) for all stored data

**Requirements**:

- **Source attribution**: Where did this information come from?
- **Timestamp**: When was this information acquired?
- **Approvals**: Who/what validated this information?
- **Verifiability**: Can this information be verified?

**Store Only Verifiable Data**:
```json
{
  "fact": "User prefers dark mode",
  "source": "user_settings_api",
  "timestamp": "2024-01-01T12:00:00Z",
  "verified_by": "user_confirmation",
  "confidence": 1.0
}
```

**Never Store**:
- Unverified assumptions
- Hallucinated information
- PII without explicit consent
- Sensitive data without encryption

---

## Generation Triggers

**What**: When to generate/consolidate memory records

**Triggers**:

1. **Phase boundaries**: Task start/end, session end, workflow completion
2. **Confidence drops**: Agent uncertainty increases, contradictions detected
3. **New entities**: New people, organizations, or concepts identified
4. **Explicit user requests**: User asks to remember something
5. **State changes**: Important context updates (preferences, goals, constraints)

**Pattern**:
```yaml
# After phase boundary
if task_completed:
  consolidate_task_memory()
  update_long_term_memory()

# After confidence drop
if confidence < threshold:
  retrieve_additional_context()
  rewrite_query()

# After new entity
if new_entity_detected:
  extract_entity_metadata()
  store_with_provenance()
```

---

## Background vs Blocking Operations

**What**: When to run memory operations async vs sync

**Background (Async)**:
- Heavy writes (consolidation, summarization)
- Low-priority updates (analytics, logging)
- Bulk operations (cleanup, archival)
- Non-critical metadata (usage stats)

**Blocking (Sync)**:
- Critical state updates (task progress, user preferences)
- Safety checks (PII detection, policy validation)
- Handoff context (agent-to-agent transfer)
- Real-time validation (input/output checks)

**Pattern**:
```python
# Background write
async def consolidate_session():
    await background_task(generate_summary, session_data)

# Blocking write
def update_task_state(state):
    validate_state(state)
    write_task_state(state)  # Must complete before continuing
```

---

## Retrieval Timing

**What**: When to retrieve/re-retrieve context

**Retrieve Before**:
- High-impact actions (irreversible operations)
- First interaction (session start)
- Domain switches (route change)
- User requests (explicit questions)

**Re-retrieve After**:
- State changes (user updates preferences)
- Time windows (enforce recency constraints)
- Failed actions (context might be stale)
- Contradictions detected (verify current state)

**Enforce Recency Windows**:
```yaml
retrieval_policy:
  max_age: 3600  # seconds
  revalidate_on:
    - state_change
    - time_window_expired
    - contradiction_detected
```

---

## Multimodal Context

**What**: Handling images, audio, video alongside text

**Normalization**:
```yaml
multimodal_asset:
  id: "asset-123"
  type: "image"
  modalities:
    - visual: {url: "...", format: "png"}
    - text: {caption: "...", alt_text: "..."}
    - embedding: {vector: [...], model: "clip"}
  metadata:
    source: "user_upload"
    timestamp: "2024-01-01T12:00:00Z"
    tags: ["diagram", "architecture"]
```

**Storage Strategy**:
- **Text + embeddings**: Always store both
- **Metadata normalization**: Consistent schema across modalities
- **Modality tags**: Label each modality clearly
- **Cross-modal search**: Enable search across modalities

---

## Fresh Contexts

**What**: Spawning new agents with clean state

**When to Use**:
- Task isolation (prevent context bleed)
- Parallel execution (independent subtasks)
- Testing/evaluation (reproducible conditions)
- Security boundaries (different trust levels)

**Pattern**:
```python
# Spawn new agent with fresh context
def spawn_agent(task):
    agent = Agent()
    agent.load_context(
        domain=task.domain,
        constraints=task.constraints,
        memory=load_validated_memory(task.context_id)
    )
    return agent
```

**Hydration from Validated Memory**:
```yaml
context_hydration:
  session_id: "sess-abc-123"
  validated_facts:
    - fact_id: "fact-001"
      source: "user_settings"
      verified: true
  constraints:
    - policy: "no_pii"
    - policy: "sandbox_mode"
```

---

## Context Size Management

**Strategies**:

1. **Summarization**: Compress long context (> 2000 tokens)
2. **Sliding windows**: Keep recent context, archive old
3. **Hierarchical context**: High-level summary + detail on-demand
4. **Relevance filtering**: Only inject relevant chunks
5. **Dynamic truncation**: Truncate low-priority context first

**Pattern**:
```python
def manage_context(context, max_tokens=8000):
    if len(context) > max_tokens:
        # Prioritize critical context
        critical = extract_critical_context(context)
        remaining = max_tokens - len(critical)

        # Summarize non-critical
        non_critical = context - critical
        summarized = summarize(non_critical, max_tokens=remaining)

        return critical + summarized
    return context
```

---

## Context Validation

**What**: Verify context quality before injection

**Validation Checks**:

- **Relevance**: Does this context relate to the task?
- **Recency**: Is this context up-to-date?
- **Completeness**: Is critical information missing?
- **Contradictions**: Does context contain conflicts?
- **Safety**: Does context contain PII/sensitive data?

**Pattern**:
```python
def validate_context(context):
    checks = [
        validate_relevance(context),
        validate_recency(context),
        validate_completeness(context),
        check_contradictions(context),
        check_pii(context)
    ]
    return all(checks)
```

---

## Related Resources

**Memory Architecture**: [`memory-systems.md`](memory-systems.md)
**RAG Patterns**: [`rag-patterns.md`](rag-patterns.md)
**Tool Design**: [`tool-design-specs.md`](tool-design-specs.md)
**Agent Operations**: [`agent-operations-best-practices.md`](agent-operations-best-practices.md)

---

## Usage Notes

- **Context > Model**: Invest in context engineering before upgrading models
- **Measure impact**: Track performance improvements from context changes
- **Iterate quickly**: Test context patterns with fast feedback loops
- **Document provenance**: Always track where context came from

# Context Graph Patterns — Structured Agent State

**Purpose**: Operational patterns for building, querying, and maintaining context graphs in agent systems. A context graph is the agent's structured working memory — entities, relationships, and reasoning traces that inform decisions.

---
## Table of Contents

- [1. Node/Edge Schema](#1-nodeedge-schema)
- [Pattern: Typed Entity Graph](#pattern-typed-entity-graph)
- [Relation Types](#relation-types)
- [Bi-Temporal Model](#bi-temporal-model)
- [Checklist: Node Creation](#checklist-node-creation)
- [2. Traversal Patterns](#2-traversal-patterns)
- [Pattern: Breadth-First Context Expansion](#pattern-breadth-first-context-expansion)
- [Pattern: Multi-Hop Reasoning Chain](#pattern-multi-hop-reasoning-chain)
- [Pattern: Subgraph Extraction](#pattern-subgraph-extraction)
- [Decision Tree: Which Traversal?](#decision-tree-which-traversal)
- [3. Graph-Augmented Retrieval (Graph-RAG)](#3-graph-augmented-retrieval-graph-rag)
- [Pattern: Retrieve → Graph-Enrich → Generate](#pattern-retrieve-→-graph-enrich-→-generate)
- [Schema: Graph-RAG Pipeline](#schema-graph-rag-pipeline)
- [When Graph-RAG Beats Plain RAG](#when-graph-rag-beats-plain-rag)
- [4. Memory Tiers](#4-memory-tiers)
- [Pattern: Three-Tier Memory Model](#pattern-three-tier-memory-model)
- [Lifecycle: Short-term → Episodic → Semantic](#lifecycle-short-term-→-episodic-→-semantic)
- [Consolidation Rules](#consolidation-rules)
- [5. Conflict Detection and Resolution](#5-conflict-detection-and-resolution)
- [Pattern: Contradiction Scan](#pattern-contradiction-scan)
- [Resolution Strategies](#resolution-strategies)
- [6. Implementation Options](#6-implementation-options)
- [Lightweight (Single-Session)](#lightweight-single-session)
- [In-memory graph using networkx](#in-memory-graph-using-networkx)
- [Breadth-first context expansion](#breadth-first-context-expansion)
- [Mid-Scale (Multi-Session, Redis)](#mid-scale-multi-session-redis)
- [Redis-backed graph with JSON serialization](#redis-backed-graph-with-json-serialization)
- [Use RedisGraph module or manual adjacency lists](#use-redisgraph-module-or-manual-adjacency-lists)
- [Supports TTL on nodes via Redis EXPIRE](#supports-ttl-on-nodes-via-redis-expire)
- [Production (Persistent Knowledge Graph)](#production-persistent-knowledge-graph)
- [Neo4j with Cypher queries](#neo4j-with-cypher-queries)
- [MATCH (a:Entity)-[r:REFERENCES]->(b:Entity)](#match-aentity-rreferences-bentity)
- [WHERE a.confidence > 0.5](#where-aconfidence-05)
- [RETURN a, r, b](#return-a-r-b)
- [FalkorDB for Redis-compatible graph DB](#falkordb-for-redis-compatible-graph-db)
- [Amazon Neptune for managed cloud graph](#amazon-neptune-for-managed-cloud-graph)
- [Decision Tree: Which Implementation?](#decision-tree-which-implementation)
- [Commercial Context Graph Options (March 2026)](#commercial-context-graph-options-march-2026)
- [Related Resources](#related-resources)


## 1. Node/Edge Schema

### Pattern: Typed Entity Graph

```yaml
node_schema:
  id: "string (uuid)"
  type: "entity | concept | tool_result | inference | user_input"
  label: "string (human-readable)"
  properties:
    source: "retrieval | inference | user_input | tool_call"
    confidence: "float (0.0 - 1.0)"
    created_at: "ISO 8601"
    updated_at: "ISO 8601"
    event_time: "ISO 8601 (when the fact actually occurred)"
    ingestion_time: "ISO 8601 (when we recorded it)"
    ttl: "int (seconds, 0 = permanent)"
    embedding: "float[] (optional, for similarity search)"

edge_schema:
  source_id: "string"
  target_id: "string"
  relation: "string (from relation_types)"
  weight: "float (0.0 - 1.0)"
  provenance: "string (how this relation was established)"
  created_at: "ISO 8601"
  valid_from: "ISO 8601 (when this relation became true)"
  valid_until: "ISO 8601 | null (when this relation was invalidated)"
```

### Relation Types

| Relation | Meaning | Example |
|----------|---------|---------|
| `depends_on` | Target is prerequisite for source | step-2 depends_on step-1 |
| `authored_by` | Source was created by target | document authored_by user |
| `references` | Source cites or links to target | answer references doc-X |
| `contradicts` | Source conflicts with target | fact-A contradicts fact-B |
| `supports` | Source provides evidence for target | evidence supports claim |
| `supersedes` | Source replaces target (newer version) | doc-v2 supersedes doc-v1 |
| `co_occurs` | Source and target appear together | entity-A co_occurs entity-B |
| `inferred_from` | Source was derived from target | conclusion inferred_from premise |

### Bi-Temporal Model

Track two distinct timestamps on every node to enable point-in-time queries ("what did the agent know at time T?"):

| Timestamp | Records | Example |
|-----------|---------|---------|
| `event_time` | When the fact actually occurred in the real world | "User changed preference on Jan 5" |
| `ingestion_time` | When the agent learned about the fact | "Agent ingested this on Jan 7" |

**Why this matters**: An agent reviewing a past decision needs to know what information was *available* at that point, not what exists now. Bi-temporal queries like "show me the context graph as of Jan 6" return nodes with `ingestion_time <= Jan 6`, which excludes the Jan 7 ingestion — accurately reflecting the agent's state at decision time.

Edges use `valid_from` / `valid_until` to track when relationships were active, enabling temporal graph traversals (e.g., "who was the account owner in Q3?" vs. "who is the account owner now?").

**Reference**: [Graphiti](https://github.com/getzep/graphiti) by Zep popularized this pattern for production agent knowledge graphs.

### Checklist: Node Creation

- [ ] Assign unique ID (UUID v4 or deterministic hash).
- [ ] Set type from allowed enum — never use freeform strings.
- [ ] Record source provenance (where did this node come from?).
- [ ] Set confidence score (1.0 for user input, lower for inference).
- [ ] Set `event_time` (when the fact occurred) and `ingestion_time` (when we recorded it).
- [ ] Set TTL based on volatility (user prefs = long, search results = short).
- [ ] Generate embedding if node will participate in similarity queries.

---

## 2. Traversal Patterns

### Pattern: Breadth-First Context Expansion

```text
Given: query node Q
1. Retrieve direct neighbors of Q (depth 1)
2. Score neighbors by edge weight × node confidence
3. If insufficient context: expand to depth 2 (neighbors of neighbors)
4. Filter by relevance threshold (> 0.5)
5. Return ranked context set
```

**When to use**: Exploratory queries where the agent needs to discover related context.

### Pattern: Multi-Hop Reasoning Chain

```text
Given: start node S, target node T
1. Find all paths from S to T (max depth: 4)
2. Score each path: product of edge weights along path
3. Select top-K paths by score
4. Extract reasoning chain: S → relation → intermediate → relation → T
5. Present chain as evidence for the S–T relationship
```

**When to use**: Answering "how does X relate to Y?" or building justification chains.

### Pattern: Subgraph Extraction

```text
Given: task context C (set of relevant node IDs)
1. Extract induced subgraph over C
2. Add 1-hop neighbors with edge weight > 0.7
3. Prune nodes with confidence < 0.3
4. Serialize subgraph to context window
```

**When to use**: Preparing a focused context snapshot for an LLM call.

### Decision Tree: Which Traversal?

```text
What does the agent need?
├── Discover related context? → Breadth-First Expansion
├── Explain a relationship? → Multi-Hop Reasoning Chain
├── Prepare LLM context? → Subgraph Extraction
└── Find contradictions? → Full-Graph Conflict Scan (see Section 5)
```

---

## 3. Graph-Augmented Retrieval (Graph-RAG)

### Pattern: Retrieve → Graph-Enrich → Generate

```text
1. RETRIEVE: Vector search for top-K documents
2. GRAPH-ENRICH:
   a. Extract entities from retrieved documents
   b. Query knowledge graph for entity relationships
   c. Add related entities not in original retrieval
3. GENERATE: Pass enriched context to LLM
```

### Schema: Graph-RAG Pipeline

```yaml
graph_rag:
  retrieval:
    method: "hybrid"  # vector + keyword
    top_k: 10
    reranker: "cross-encoder"
  enrichment:
    entity_extraction: "NER or LLM-based"
    graph_lookup:
      max_hops: 2
      min_edge_weight: 0.5
      max_additional_nodes: 20
    merge_strategy: "union_deduplicate"
  generation:
    context_format: "documents + entity_relationships"
    max_context_tokens: 8000
```

### When Graph-RAG Beats Plain RAG

| Scenario | Plain RAG | Graph-RAG |
|----------|-----------|-----------|
| Single-document answer | Sufficient | Overkill |
| Cross-document reasoning | Misses connections | Finds entity links |
| "How does X relate to Y?" | Poor | Strong |
| Temporal reasoning | Weak | Strong (date edges) |
| Contradictory sources | Picks one | Surfaces conflict |

**Existing depth**: [`../../ai-rag/references/graph-rag-patterns.md`](../../ai-rag/references/graph-rag-patterns.md) — full graph-RAG implementation patterns.

---

## 4. Memory Tiers

### Pattern: Three-Tier Memory Model

| Tier | Purpose | Storage | TTL | Context Graph Role |
|------|---------|---------|-----|-------------------|
| **Short-term** | Current conversation state | In-memory | Session duration | Active subgraph |
| **Episodic** | Past task outcomes and reasoning traces | Redis / DB | Days to weeks | Archived subgraph, rehydratable |
| **Semantic** | Stable knowledge and entity relationships | Knowledge graph | Months to permanent | Persistent graph partition |

### Lifecycle: Short-term → Episodic → Semantic

```text
During session:
  Context graph nodes are SHORT-TERM (in active subgraph)

At session end:
  1. Score each node: importance = confidence × usage_count × recency
  2. Nodes with importance > threshold → promote to EPISODIC
  3. Discard remaining short-term nodes

Periodic consolidation:
  1. Scan episodic nodes older than 7 days
  2. Nodes referenced by 3+ sessions → promote to SEMANTIC
  3. Merge duplicates (same entity, different sessions)
  4. Archive remaining episodic nodes beyond retention window
```

### Consolidation Rules

```yaml
consolidation:
  episodic_promotion:
    min_confidence: 0.7
    min_usage_count: 2
    recency_weight: 0.3
  semantic_promotion:
    min_session_references: 3
    min_age_days: 7
    merge_strategy: "keep_highest_confidence"
  cleanup:
    episodic_retention_days: 30
    orphan_node_removal: true  # nodes with no edges
```

**Existing depth**: [`memory-systems.md`](memory-systems.md) — four-memory model, write patterns, extraction, consolidation.

---

## 5. Conflict Detection and Resolution

### Pattern: Contradiction Scan

```text
For each entity E in the graph:
  1. Collect all edges where E is source or target
  2. Group edges by relation type
  3. For each group:
     - If "supports" and "contradicts" edges exist for same target → CONFLICT
     - If "supersedes" edge exists → mark older node as stale
  4. Flag conflicts for resolution
```

### Resolution Strategies

| Strategy | When to Use | Action |
|----------|-------------|--------|
| **Recency wins** | Time-sensitive facts | Keep newer node, archive older |
| **Confidence wins** | Uncertain sources | Keep higher-confidence node |
| **Source priority** | Mixed source quality | Rank: user_input > official_doc > inference |
| **Human review** | High-stakes decisions | Flag for human, block automation |

---

## 6. Implementation Options

### Lightweight (Single-Session)

```python
# In-memory graph using networkx
import networkx as nx

graph = nx.DiGraph()
graph.add_node("entity-1", type="concept", confidence=0.9, source="user_input")
graph.add_node("entity-2", type="document", confidence=0.8, source="retrieval")
graph.add_edge("entity-1", "entity-2", relation="references", weight=0.85)

# Breadth-first context expansion
neighbors = list(nx.bfs_edges(graph, "entity-1", depth_limit=2))
```

### Mid-Scale (Multi-Session, Redis)

```python
# Redis-backed graph with JSON serialization
# Use RedisGraph module or manual adjacency lists
# Supports TTL on nodes via Redis EXPIRE
```

### Production (Persistent Knowledge Graph)

```python
# Neo4j with Cypher queries
# MATCH (a:Entity)-[r:REFERENCES]->(b:Entity)
# WHERE a.confidence > 0.5
# RETURN a, r, b

# FalkorDB for Redis-compatible graph DB
# Amazon Neptune for managed cloud graph
```

### Decision Tree: Which Implementation?

```text
How many entities?
├── < 100, single session? → networkx (in-memory)
├── 100–10K, multi-session? → Redis + JSON graph
└── > 10K, cross-agent? → Neo4j / FalkorDB / Neptune
```

### Commercial Context Graph Options (March 2026)

| Product | Approach | Differentiator | Best For |
|---------|----------|---------------|----------|
| **Graphiti** (by Zep) | Bi-temporal knowledge graph, OSS | 11 operation abstractions, pluggable drivers (Neo4j, FalkorDB, Kuzu, Neptune). 94.8% on DMR benchmark. | Production agent KG with temporal queries |
| **Cognee** | KG from unstructured data, 6 lines of code | Transforms raw docs into structured memory. Rust engine for edge/on-device coming. €7.5M funded. | Regulated/knowledge-intensive domains |
| **Glean** | Enterprise KG across 100+ connectors | Unified index of company content, people, and activity. Results preferred 1.9× over ChatGPT on enterprise queries. | Enterprise-scale org-wide context |
| **Neo4j** | Native graph database | Mature ecosystem, Cypher query language, AuraDB managed service. Publishes context engineering best practices. | Self-hosted production graph |

**Pattern validated**: Our bi-temporal node schema (Section 1, `event_time` + `ingestion_time`) aligns with Graphiti's approach — the leading OSS implementation of temporal knowledge graphs for agents.

**Pattern validated**: Our three-tier implementation model (lightweight → mid-scale → production) maps to the market: in-memory for prototyping, Redis for mid-scale, and Graphiti/Neo4j/FalkorDB for production — exactly the progression these products serve.

---

## Related Resources

| Resource | Covers |
|----------|--------|
| [`ai-engine-layers.md`](ai-engine-layers.md) | Full 5-layer architecture overview |
| [`context-engineering.md`](context-engineering.md) | Progressive disclosure, session management, provenance |
| [`memory-systems.md`](memory-systems.md) | Four-memory model, write patterns, consolidation |
| [`rag-patterns.md`](rag-patterns.md) | Retrieval pipelines, hybrid search |
| [`../../ai-rag/references/graph-rag-patterns.md`](../../ai-rag/references/graph-rag-patterns.md) | Full graph-RAG implementation |

# Context Rotation And Durable State

Operational patterns for keeping coding and multi-agent workflows reliable as sessions get longer and task graphs get larger.

## Table Of Contents

- [Core Distinction](#core-distinction)
- [Symptoms Of Context Rot](#symptoms-of-context-rot)
- [Preferred Mitigations](#preferred-mitigations)
- [When To Respawn A Worker](#when-to-respawn-a-worker)
- [State Shapes](#state-shapes)
- [Anti-Patterns](#anti-patterns)
- [Related References](#related-references)
- [Primary Sources](#primary-sources)

## Core Distinction

Treat these as different things:

- **Session context**: temporary conversation history, tool outputs, scratch reasoning, and intermediate exploration
- **Project state**: decisions, interfaces, constraints, task graph, progress, and verification evidence that must survive across sessions

The operational mistake is storing project state inside session context.

## Symptoms Of Context Rot

Use **context rot** here as practitioner shorthand for quality degradation caused by overloaded or polluted session context.

Common symptoms:

- the agent re-reads the same files and forgets prior decisions
- workers inherit irrelevant logs or stale assumptions
- task boundaries blur and edits expand beyond scope
- the model starts using outdated instructions from earlier in the conversation

## Preferred Mitigations

### 1. Fresh-context workers

Spawn each worker with:

- the bounded task brief
- owned files and explicit `do_not_touch` boundaries
- frozen interface contracts
- verification commands or acceptance checks

Do not pass the entire orchestrator transcript unless the task genuinely depends on it.

### 2. Durable external state

Persist project state in reviewable files such as:

- markdown with frontmatter
- YAML
- JSON
- task manifests or blueprints

Typical state to persist:

- active plan and milestones
- dependency graph and unblock conditions
- decisions and rationale
- changed-path ownership
- verification evidence and unresolved risks

### 3. Session-to-project promotion

Only promote durable information out of session context:

- approved decisions
- stable interfaces
- verified findings
- next-step checkpoints

Do not persist raw chain-of-thought, noisy logs, or unverified guesses as project state.

## When To Respawn A Worker

Prefer a fresh worker or fresh agent session when:

- the task changes from exploration to implementation
- a worker has crossed a meaningful phase boundary
- the context now contains multiple unrelated branches of reasoning
- the same task needs to be resumed after a long pause
- the worker would benefit more from a clean brief than from conversational history

## State Shapes

Keep the shape simple and explicit.

```yaml
task:
  id: auth-session-fix
  owner: dev-worker-2
  owned_files:
    - src/auth/session.ts
    - tests/auth/session.test.ts
  depends_on:
    - session-contract-approved
  verify:
    - npm test -- session
  status: in_progress
```

```yaml
decision:
  id: use-session-cookie-refresh
  date: 2026-03-25
  approved_by: lead
  rationale: Avoid token refresh race in middleware chain
  affected_interfaces:
    - src/auth/contracts.ts
```

## Anti-Patterns

- using one giant conversation as the system of record
- passing raw worker transcripts between workers
- keeping task ownership implicit instead of written down
- persisting every thought instead of only verified state
- making workers reconstruct project state from memory rather than files

## Related References

- [`agent-delivery-methods.md`](agent-delivery-methods.md)
- [`context-engineering.md`](context-engineering.md)
- [`multi-agent-patterns.md`](multi-agent-patterns.md)
- [`../../dev-workflow-planning/references/session-patterns.md`](../../dev-workflow-planning/references/session-patterns.md)

## Primary Sources

- GSD: <https://github.com/gsd-build/get-shit-done>
- BMAD Method docs: <https://docs.bmad-method.org/>
- Anthropic Claude Code best practices: <https://www.anthropic.com/engineering/claude-code-best-practices>
- OpenAI Codex workflows: <https://developers.openai.com/codex/workflows>

# Deployment, CI/CD & Safety — Best Practices 

*Purpose: Provide operational procedures for deploying, evaluating, gating, monitoring, and securing AI agents in production environments with multi-layer guardrails.*

**Modern Update**: NIST AI RMF compliance, OWASP GenAI Top 10 defenses, OpenTelemetry observability, and human-in-the-loop for high-risk operations are now production standards.

---
## Table of Contents

- [Multi-Layer Guardrails (Critical)](#multi-layer-guardrails-critical)
- [Defense-in-Depth Architecture](#defense-in-depth-architecture)
- [Human-in-the-Loop (HITL) Requirements](#human-in-the-loop-hitl-requirements)
- [OWASP GenAI Top 10  Defenses](#owasp-genai-top-10-defenses)
- [1. Deployment Pipeline (Enhanced)](#1-deployment-pipeline-enhanced)
- [Pattern: Standard Deployment Flow](#pattern-standard-deployment-flow)
- [2. CI/CD Configuration](#2-cicd-configuration)
- [Pattern: Automated CI Stage](#pattern-automated-ci-stage)
- [Failure Rules](#failure-rules)
- [3. Evaluation Gate](#3-evaluation-gate)
- [Pattern: Quality & Safety Thresholds](#pattern-quality-&-safety-thresholds)
- [Decision Tree](#decision-tree)
- [4. Staging Environment Procedures](#4-staging-environment-procedures)
- [Pattern: Staging Validation](#pattern-staging-validation)
- [5. Canary Deployment](#5-canary-deployment)
- [Pattern: Limited & Monitored Rollout](#pattern-limited-&-monitored-rollout)
- [Canary Rollback Triggers](#canary-rollback-triggers)
- [6. Versioning & Model Pinning](#6-versioning-&-model-pinning)
- [Pattern: Deterministic Version Control](#pattern-deterministic-version-control)
- [7. Production Monitoring](#7-production-monitoring)
- [Pattern: Continuous Observability](#pattern-continuous-observability)
- [8. Safety Enforcement Layer](#8-safety-enforcement-layer)
- [Pattern: Safety Gate Before Action](#pattern-safety-gate-before-action)
- [9. Prompt Hardening](#9-prompt-hardening)
- [Pattern: Preprocessing Filter](#pattern-preprocessing-filter)
- [10. Tool Safety Hardening](#10-tool-safety-hardening)
- [Pattern: Safe Tool Wrapper](#pattern-safe-tool-wrapper)
- [11. Memory Safety Hardening](#11-memory-safety-hardening)
- [Pattern: Controlled Memory Interface](#pattern-controlled-memory-interface)
- [12. Failure Handling](#12-failure-handling)
- [Pattern: Fail-Fast With Recovery](#pattern-fail-fast-with-recovery)
- [13. Deployment Anti-Patterns (Master List)](#13-deployment-anti-patterns-master-list)
- [14. Quick Reference Tables](#14-quick-reference-tables)
- [CI/CD Stage Table](#cicd-stage-table)
- [Safety Layer Table](#safety-layer-table)
- [Rollback Conditions Table](#rollback-conditions-table)
- [End of File](#end-of-file)


## Multi-Layer Guardrails (Critical)

### Defense-in-Depth Architecture

**Five mandatory layers for production agents**:

```yaml
Layer 1: Input Validation
  - PII redaction (real-time)
  - Content filtering (harmful content, jailbreaks)
  - Prompt injection detection
  - Input sanitization

Layer 2: RBAC/ABAC Authorization
  - Fine-grained permissions per tool/action
  - Externalized policy (OPA/Rego or managed engines)
  - Zero standing privileges
  - Short-lived secrets with rotation

Layer 3: Tool Gating
  - Signature verification (Sigstore/Cosign)
  - Human approval for high-risk operations
  - Tool invocation logging
  - Scope validation (tool matches authorized role)

Layer 4: Output Filtering
  - PII detection (before delivery)
  - Content moderation (policy violations)
  - Compliance validation
  - Format verification

Layer 5: Observability & Monitoring
  - OpenTelemetry GenAI spans
  - SIEM integration with alerting
  - Real-time anomaly detection
  - Audit trail for all operations
```

### Human-in-the-Loop (HITL) Requirements

**Mandatory HITL approval for**:
- Financial transactions (payments, transfers, account modifications)
- Database write operations (UPDATE, DELETE, DROP)
- Legal or compliance actions (contract signing, regulatory filings)
- Irreversible operations (account deletion, data purging)
- Production system modifications (deployments, configuration changes)

**Implementation pattern**:
```yaml
agent_prepares_action()
if requires_human_approval(action):
  request = create_approval_request(action)
  approval = wait_for_human_approval(request)
  if approval.granted:
    execute_with_audit(action, approval.id)
  else:
    log_rejection(action, approval.reason)
else:
  execute_with_validation(action)
```

**HITL metrics to track**:
- Approval queue length
- Average approval time
- Approval/rejection ratio
- False positive rate (unnecessary approvals)

### OWASP GenAI Top 10  Defenses

**1. Prompt Injection**:
- Treat all user input as untrusted
- Use delimiter tags to separate instructions from data
- Validate prompt structure before execution
- Never concatenate user input directly into system prompts

**2. Insecure Output Handling**:
- Validate all LLM outputs against schema
- Never execute LLM-generated code without sandboxing
- Filter outputs before rendering (XSS prevention)

**3. Training Data Poisoning**:
- Validate RAG data sources
- Use curated, verified knowledge bases
- Monitor for drift in retrieval quality

**4. Model Denial of Service**:
- Token limits per request
- Rate limiting per user/API key
- Circuit breakers for cascading failures
- Cost budgets with automatic kill switches

**5. Supply Chain Vulnerabilities**:
- Pin all dependency versions
- Verify tool signatures (Sigstore/Cosign)
- Use SBOMs and SLSA attestations
- Audit third-party plugins before integration

**6. Sensitive Information Disclosure**:
- PII redaction at ingestion and output
- Memory TTLs to limit data retention
- DLP (Data Loss Prevention) on all channels
- Secrets in Vault/KMS, never in prompts

**7. Insecure Plugin Design**:
- Least privilege for tool permissions
- Require explicit confirmation for high-risk tools
- Validate tool outputs before use
- Never trust tool results without verification

**8. Excessive Agency**:
- Limit tool capabilities per agent role
- Require HITL for destructive operations
- Implement fail-safe defaults (deny by default)
- Audit logs for all agent actions

**9. Overreliance**:
- Always require citations for factual claims
- Confidence scoring for agent outputs
- Human review for critical decisions
- Fallback to human expert when confidence low

**10. Model Theft**:
- API key rotation
- Usage monitoring for anomalies
- Rate limiting per endpoint
- Authentication for all model access

---

## 1. Deployment Pipeline (Enhanced)

### Pattern: Standard Deployment Flow

```
dev → CI tests → evaluation suite → staging → canary → production
```

**Checklist**

- [ ] Commit triggers automated CI.  
- [ ] Evaluation suite blocks unsafe/inaccurate revisions.  
- [ ] Version pinned before staging.  
- [ ] Canary monitors key metrics for regression.  
- [ ] Rollback plan defined.  

**Anti-Patterns**

- AVOID: Deploying without evaluation.  
- AVOID: Manual steps without reproducibility.  

---

# 2. CI/CD Configuration

### Pattern: Automated CI Stage

```
lint → unit tests → integration tests → tool-call tests → eval tests
```

**Checklist**

- [ ] All tools validated with type + schema tests.  
- [ ] RAG pipeline tested end-to-end.  
- [ ] Multi-agent routes tested for correctness.  
- [ ] Memory write/read rules validated.  
- [ ] No unversioned dependencies.  

### Failure Rules

- Test fail → block merge  
- Eval fail → block deploy  
- Safety fail → immediate halt  

---

# 3. Evaluation Gate

### Pattern: Quality & Safety Thresholds

```
correctness >= 4.0
grounding >= 4.0
tool_success_rate >= 95%
safety = "pass"
latency_p95 <= target
```

**Checklist**

- [ ] Thresholds defined per environment (staging vs prod).  
- [ ] Results attached to build artifacts.  
- [ ] Historical scores maintained for regression detection.  

### Decision Tree

```
Did evaluation meet thresholds?
→ Yes → Promote to staging
→ No → Reject build
```

---

# 4. Staging Environment Procedures

### Pattern: Staging Validation

```
deploy → run synthetic tests → run real-task replay → compare to baseline
```

**Checklist**

- [ ] Synthetic queries cover all agent modes.  
- [ ] Replay tests cover real workflows.  
- [ ] Validate logs + traces function correctly.  
- [ ] Validate latency + throughput under load.  

**Anti-Patterns**

- AVOID: Using production data directly.  
- AVOID: Skipping tool verification in staging.  

---

# 5. Canary Deployment

### Pattern: Limited & Monitored Rollout

```
deploy_to_subset(1–5%)
monitor(metrics)
if stable → expand
else → rollback
```

**Checklist**

- [ ] Monitor correctness, safety, latency, cost.  
- [ ] Compare against previous version.  
- [ ] Define rollback triggers.  

### Canary Rollback Triggers

- Error spike  
- Safety violation  
- Tool-call failure increase  
- Latency p99 jump  
- Observability failures  

---

# 6. Versioning & Model Pinning

### Pattern: Deterministic Version Control

```
agent_version = x.y.z
embedding_model_version = pinned
retriever_version = pinned
tool_schema_version = pinned
```

**Checklist**

- [ ] Pin all model versions.  
- [ ] Store all prompts with version tag.  
- [ ] Keep compatibility log for all components.  

---

# 7. Production Monitoring

### Pattern: Continuous Observability

```
collect:
  - logs
  - traces
  - metrics
monitor_for:
  - tool errors
  - grounding failures
  - safety violations
```

**Operational Metrics**

- Tool success rate  
- RAG retrieval relevance  
- Latency p50/p95/p99  
- Cost per request  
- Safety pass rate  

**Alert Conditions**

- Error > threshold  
- Tool-call retry spike  
- Stale or missing logs  
- Trace span failures  

---

# 8. Safety Enforcement Layer

### Pattern: Safety Gate Before Action

```
check_action_risk()
check_domain_support()
scan_for_policy_violations()
require_confirmation_if_needed()
perform_action()
```

**High-Risk Categories**

- OS-level automation  
- File modification/deletion  
- Financial or transactional actions  
- System configuration changes  

**Checklist**

- [ ] Require user confirmation for high-risk tasks.  
- [ ] Block hallucinated actions/tools.  
- [ ] Reject incomplete or ambiguous user requests.  
- [ ] Validate parameters explicitly.  

---

# 9. Prompt Hardening

### Pattern: Preprocessing Filter

```
sanitize_input()
block_injections()
reject_adversarial_phrases()
normalize_request()
```

**Checklist**

- [ ] Reject attempts to bypass instructions.  
- [ ] Block system override language.  
- [ ] Block payloads intended to break tools.  
- [ ] Remove suspicious executable content.  

**Anti-Patterns**

- AVOID: Accepting raw user text into agent plan.  

---

# 10. Tool Safety Hardening

### Pattern: Safe Tool Wrapper

```
validate_input()
check_scope()
enforce_rate_limit()
execute()
verify()
```

**Checklist**

- [ ] Parameter type + range checks.  
- [ ] Confirmation for destructive actions.  
- [ ] Reject calls outside declared scope.  
- [ ] Log every invocation.  

---

# 11. Memory Safety Hardening

### Pattern: Controlled Memory Interface

```
memory_agent.validate_write(entry)
memory_agent.validate_read(query)
```

**Rules**

- Never store sensitive PII.  
- Never store internal reasoning.  
- Require explicit confirmation from user.  
- Check consistency before writing.  

---

# 12. Failure Handling

### Pattern: Fail-Fast With Recovery

```
detect_failure()
type_error()
if transient → retry
if soft_error → request clarification
if fatal → surface + halt
```

**Checklist**

- [ ] Documented retry policy.  
- [ ] Recorded failure metadata.  
- [ ] Alert on repeated transient failures.  

---

# 13. Deployment Anti-Patterns (Master List)

- AVOID: Deploying without evaluation suite.  
- AVOID: No version pinning.  
- AVOID: Ignoring safety failures.  
- AVOID: Overwriting production without canary.  
- AVOID: Missing rollback mechanism.  
- AVOID: No logging/observability.  
- AVOID: Using dev/staging tools in production.  
- AVOID: Allowing tool schema drift.  

---

# 14. Quick Reference Tables

### CI/CD Stage Table

| Stage | Required Actions |
|--------|------------------|
| CI | tests, linting, static analysis |
| Evaluation | scoring, safety checks |
| Staging | replay tests, observability check |
| Canary | partial rollout |
| Production | monitoring, alerts |

### Safety Layer Table

| Layer | Enforcement |
|--------|-------------|
| Prompt | sanitization, anti-injection |
| Tool | validation, confirmation |
| Memory | safe-write rules |
| Agent | unsafe-action blocking |

### Rollback Conditions Table

| Condition | Action |
|-----------|--------|
| Safety violation | immediate rollback |
| Latency spike | rollback |
| Tool-call failure | rollback |
| Evaluation score drop | rollback |

---

# End of File

# Escalation Patterns for Agent Failures

> Operational reference for designing structured escalation hierarchies in agent systems — when to retry, re-plan, escalate to a parent agent, or hand off to a human. Covers failure classification, escalation budgets, and integration with hooks and orchestration.

---

## Table of Contents

- [Core Principle: Escalation Over Retry](#core-principle-escalation-over-retry)
- [3-Level Escalation Hierarchy](#3-level-escalation-hierarchy)
- [Failure Classification](#failure-classification)
- [Escalation Decision Tree](#escalation-decision-tree)
- [Escalation Budgets](#escalation-budgets)
- [Integration with Hooks](#integration-with-hooks)
- [Graceful Degradation Modes](#graceful-degradation-modes)
- [Anti-Patterns](#anti-patterns)
- [Related Resources](#related-resources)

---

## Core Principle: Escalation Over Retry

The default instinct when an agent task fails is to retry. The better default is to **classify the failure first**, then decide.

**Transient failures** (network timeouts, rate limits, tool unavailability) → retry once with backoff.

**Structural failures** (wrong tool, ambiguous requirements, missing permissions, logic errors) → do not retry the same approach. Re-plan or escalate.

**Safety failures** (policy violation, unauthorized action, data exfiltration attempt) → never retry. Abort and escalate to human immediately.

Retrying a structural failure wastes context and can leave the agent in a loop. Retrying a safety failure is a security risk.

---

## 3-Level Escalation Hierarchy

```
Level 1: Self-resolution
  Agent detects failure → re-plans with different approach → attempts once more
  Budget: 1 re-plan, 1 retry
  Escalate when: same failure recurs or re-plan produces no new approach

Level 2: Parent agent / lead agent escalation
  Worker signals failure with diagnosis → lead decides: reassign, re-scope, unblock, or absorb
  Budget: lead gets 1 attempt to unblock
  Escalate when: lead cannot unblock without human judgment or authority

Level 3: Human escalation
  System pauses, emits structured escalation record → human reviews and resolves
  Triggers: safety constraint, irreversible operation, ambiguous authority, compliance gate
  Never retry after Level 3 until human resolution is confirmed
```

In single-agent systems without a parent agent, Level 2 becomes a local re-plan with a broadened strategy before escalating to human.

---

## Failure Classification

| Failure Type | Example | Escalation Level |
|---|---|---|
| Transient network / rate limit | Timeout, 429 | Level 1 (retry once) |
| Tool unavailable | MCP server down | Level 1 (retry with backoff) |
| Tool argument error | Schema mismatch, missing field | Level 1 (re-plan tool call) |
| Ambiguous requirements | Spec has two conflicting interpretations | Level 2 (lead resolves) |
| Missing permissions | File write denied, API key missing | Level 2 (lead unblocks) |
| Structural logic error | Same approach fails twice | Level 2 (reassign or re-scope) |
| Policy violation | Agent attempts unauthorized action | Level 3 (human review) |
| Irreversible operation | Destructive write, send email, deploy | Level 3 (human approval before proceed) |
| Safety constraint | Prompt injection, secret exfiltration | Level 3 (abort and audit) |
| Compliance gate | Regulated action requiring audit trail | Level 3 (human sign-off) |

---

## Escalation Decision Tree

```text
Task execution failed?
  → Classify failure type
    → Transient?
        → Retry once with backoff
        → Still failing? → treat as structural
    → Structural?
        → Re-plan with different approach
        → Retry once
        → Still failing? → Escalate to Level 2
    → Safety / compliance?
        → Abort immediately → Escalate to Level 3 → Do not retry
  → At Level 2?
    → Lead diagnoses: can unblock?
        → Yes → unblock and resume
        → No → Escalate to Level 3
  → At Level 3?
    → Emit escalation record with: failure type, context, last attempted approach, required resolution
    → Pause execution
    → Resume only after human confirmation
```

---

## Escalation Budgets

Define budgets explicitly in the task contract before dispatch. Agents that exceed budget must escalate rather than continue.

```yaml
escalation_budget:
  max_retries_per_tool_call: 1          # transient failures
  max_replans_before_level2: 1          # structural failures
  max_level2_attempts: 1                # lead unblocking
  max_context_tokens_before_stop: 80000 # abort if context exhausted
  safety_violations_before_abort: 1     # zero tolerance
```

Budget exhaustion is itself an escalation trigger. An agent that has used its entire retry budget without success must escalate, not continue silently.

---

## Integration with Hooks

Use Claude Code hooks to enforce escalation at the runtime level:

**`PostToolUseFailure`** — inspect the failure type and emit a structured escalation record if the failure is non-transient:
```json
{
  "failure_type": "permission_denied",
  "tool": "Write",
  "resource": "/etc/hosts",
  "escalation_level": 3,
  "message": "Write to protected path. Human approval required."
}
```

**`PermissionRequest`** — use to gate Level 3 escalations. Block the operation and route the `PermissionRequest` event to the human approval channel before proceeding.

**`Stop`** — validate that the agent did not stop due to a budget exhaustion without emitting an escalation record. Require a final escalation summary in the stop payload if any failures occurred.

See [../../agents-hooks/SKILL.md](../../agents-hooks/SKILL.md) for hook configuration patterns.

---

## Graceful Degradation Modes

When full escalation is not possible (no parent agent, async human review), fall back gracefully rather than silently:

| Mode | When to Use | Behavior |
|---|---|---|
| **Partial result** | Structural failure on optional step | Return completed steps; clearly mark incomplete sections |
| **Cached result** | Tool unavailable, transient outage | Return last valid result with staleness timestamp |
| **Degraded mode** | Missing capability | Acknowledge limitation; offer reduced-scope alternative |
| **Hard abort** | Safety / policy failure | Return structured error; do not return partial unsafe output |

Never return a result that silently omits a failure. The caller must be able to detect that escalation or degradation occurred.

---

## Anti-Patterns

| Anti-Pattern | Problem | Fix |
|---|---|---|
| Retry loop on structural failure | Wastes context, loops indefinitely | Classify before retrying; escalate structural failures |
| Silent degradation | Caller assumes success | Always flag degraded or incomplete results explicitly |
| Escalating transient failures immediately | Unnecessary interruption | Retry transient failures once before escalating |
| Undefined escalation path | Agent has no one to escalate to | Define escalation chain in task contract before dispatch |
| Escalating without diagnosis | Human receives unhelpful ticket | Always include: failure type, last attempted approach, and what resolution is needed |
| Resuming after Level 3 without confirmation | Safety risk | Require explicit human confirmation before resuming after any Level 3 event |

---

## Related Resources

- [../../agents-hooks/SKILL.md](../../agents-hooks/SKILL.md) — PostToolUseFailure, PermissionRequest, Stop hooks
- [../../agents-swarm-orchestration/SKILL.md](../../agents-swarm-orchestration/SKILL.md) — Lead agent escalation responsibilities and escalation-over-retry pattern
- [guardrails-implementation.md](guardrails-implementation.md) — HITL escalation triggers and confidence thresholds
- [agent-operations-best-practices.md](agent-operations-best-practices.md) — Error handling and loop continuation decision trees
- [deployment-ci-cd-and-safety.md](deployment-ci-cd-and-safety.md) — Rollback and control gates

# Evaluation & Observability — Best Practices 

*Purpose: Provide operational rules, scoring rubrics, and required observability structures for evaluating agent behavior, quality, safety, and system performance with OpenTelemetry standards.*

**Modern Update**: OpenTelemetry GenAI semantic conventions are now the standard. Production requires real-time observability with LangSmith, Arize, Azure AI Foundry, or similar platforms.

---
## Table of Contents

- [OpenTelemetry for AI Agents (Current Standard)](#opentelemetry-for-ai-agents-current-standard)
- [GenAI Semantic Conventions](#genai-semantic-conventions)
- [Observability Platforms](#observability-platforms)
- [Metrics to Track (Production Standard)](#metrics-to-track-production-standard)
- [SIEM Integration Pattern](#siem-integration-pattern)
- [1. Evaluation Modes (Enhanced)](#1-evaluation-modes-enhanced)
- [Supported Evaluation Types](#supported-evaluation-types)
- [2. Evaluation Loop Pattern](#2-evaluation-loop-pattern)
- [Pattern: Evaluate → Score → Compare → Gate](#pattern-evaluate-→-score-→-compare-→-gate)
- [3. Final Answer Evaluation](#3-final-answer-evaluation)
- [Scoring Rubric (1–5 scale)](#scoring-rubric-1–5-scale)
- [4. Trajectory Evaluation](#4-trajectory-evaluation)
- [Categories](#categories)
- [Pattern: Step-by-Step Scoring](#pattern-step-by-step-scoring)
- [5. Tool-Call Evaluation](#5-tool-call-evaluation)
- [Metrics (1–5)](#metrics-1–5)
- [Pattern: Tool Judging](#pattern-tool-judging)
- [6. RAG Evaluation](#6-rag-evaluation)
- [Metrics](#metrics)
- [Pattern: RAG Judge](#pattern-rag-judge)
- [7. Safety Evaluation](#7-safety-evaluation)
- [Pattern: Safety Scan](#pattern-safety-scan)
- [Safety Conditions](#safety-conditions)
- [8. Observability Requirements](#8-observability-requirements)
- [8.1 Logs](#81-logs)
- [Required Log Fields](#required-log-fields)
- [Pattern: Log Snapshot](#pattern-log-snapshot)
- [8.2 Traces](#82-traces)
- [Required Trace Spans](#required-trace-spans)
- [Pattern: Trace Structure](#pattern-trace-structure)
- [8.3 Metrics](#83-metrics)
- [System Metrics](#system-metrics)
- [Quality Metrics](#quality-metrics)
- [Threshold Examples](#threshold-examples)
- [9. CI/CD Evaluation Gates](#9-cicd-evaluation-gates)
- [Pattern: Gate Before Deploy](#pattern-gate-before-deploy)
- [10. Evaluation Anti-Patterns (Master List)](#10-evaluation-anti-patterns-master-list)
- [11. Quick Reference Tables](#11-quick-reference-tables)
- [Score Table](#score-table)
- [Evaluation Coverage Table](#evaluation-coverage-table)
- [12. Copy-Paste Evaluation Templates](#12-copy-paste-evaluation-templates)
- [Final Answer Judge Prompt](#final-answer-judge-prompt)
- [Trajectory Judge Prompt](#trajectory-judge-prompt)
- [RAG Judge Prompt](#rag-judge-prompt)
- [End of File](#end-of-file)


## OpenTelemetry for AI Agents (Current Standard)

### GenAI Semantic Conventions

**Required instrumentation for all production agents**:

```yaml
Spans (Distributed Tracing):
  llm_call:
    attributes:
      - gen_ai.system: "anthropic" | "openai" | "google"
      - gen_ai.request.model: "claude-sonnet-4-6"
      - gen_ai.request.max_tokens: 4096
      - gen_ai.request.temperature: 0.7
      - gen_ai.prompt: [hashed or redacted]
      - gen_ai.completion: [hashed or redacted]
      - gen_ai.usage.input_tokens: 1234
      - gen_ai.usage.output_tokens: 567
      - gen_ai.response.finish_reason: "stop"
    duration_ms: 1423

  tool_call:
    attributes:
      - tool.name: "web_search"
      - tool.parameters: {query: "...", max_results: 10}
      - tool.result: [structured output or hash]
      - tool.success: true
      - tool.error: null
    duration_ms: 342

  retrieval:
    attributes:
      - retrieval.query: "user query"
      - retrieval.method: "semantic" | "keyword" | "hybrid"
      - retrieval.top_k: 10
      - retrieval.chunks_retrieved: 5
      - retrieval.reranked: true
      - retrieval.scores: [0.92, 0.88, 0.85, 0.81, 0.78]
    duration_ms: 156

  memory_operation:
    attributes:
      - memory.operation: "read" | "write" | "delete"
      - memory.type: "session" | "long_term" | "episodic" | "task"
      - memory.key: "user_123_preferences"
      - memory.size_bytes: 2048
      - memory.ttl_seconds: 3600
    duration_ms: 23

  agent_handoff:
    attributes:
      - handoff.source_agent: "manager-001"
      - handoff.target_agent: "worker-research-02"
      - handoff.task_id: "task-456"
      - handoff.schema_version: "v1.2"
      - handoff.trace_id: "req-abc-123"
      - handoff.payload_size_bytes: 1024
      - handoff.validation_result: "passed"
    duration_ms: 5
```

### Observability Platforms 

**Leading platforms for agent observability**:

| Platform | Key Features | Best For |
|----------|--------------|----------|
| **LangSmith** | LangChain native, visual traces, experiments | LangChain/LangGraph agents |
| **Arize** | Comprehensive eval tools, drift detection | Enterprise ML monitoring |
| **Azure AI Foundry** | Azure-native, unified dashboard, SIEM integration | Azure ecosystem |
| **New Relic** | APM integration, MCP server support | Full-stack observability |
| **Datadog** | Infrastructure + AI, real-time alerts | Multi-cloud deployments |

**Required features for production**:
- Real-time trace visualization
- Cost and latency budgets with alerts
- Automatic anomaly detection
- A/B test comparison
- Evaluation suite integration
- SIEM/SOC integration

### Metrics to Track (Production Standard)

```yaml
Performance:
  - latency_p50: <target>
  - latency_p95: <target>
  - latency_p99: <target>
  - throughput_rps: <target>
  - cost_per_request: <budget>

Quality:
  - tool_success_rate: >=95%
  - retrieval_accuracy: >=90%
  - eval_score_avg: >=4.0
  - hallucination_rate: <5%
  - citation_coverage: >=95%

Safety:
  - pii_leak_count: 0
  - prompt_injection_blocks: [monitor]
  - hitl_approval_rate: [track]
  - guardrail_violations: 0
  - owasp_test_failures: 0

Reliability:
  - uptime_percentage: >=99.9%
  - error_rate: <1%
  - timeout_rate: <2%
  - mttd_minutes: <5  # Mean Time To Detect
  - mttr_minutes: <15  # Mean Time To Recover
```

### SIEM Integration Pattern

**Required for production security**:

```yaml
Log Streaming:
  - Stream OpenTelemetry spans to SIEM (Sentinel, Splunk, QRadar)
  - Redact PII at capture time
  - Classify logs by sensitivity level

Analytics Rules (Example KQL for Azure Sentinel):
  - Spike in tool invocations (>3 std dev)
  - New domains accessed (not in allowlist)
  - Prompt injection signatures detected
  - Unusual latency patterns
  - Failed authentication attempts
  - HITL approval queue buildup

Alerts:
  - Critical: Safety violations, PII leaks
  - High: Tool signature failures, OWASP violations
  - Medium: Performance degradation, cost overruns
  - Low: Evaluation score drops, drift detection

SOAR Integration:
  - Automatic containment for critical alerts
  - Incident creation in ticketing system
  - Notification to on-call engineer
  - Rollback trigger if configured
```

---

## 1. Evaluation Modes (Enhanced)

### Supported Evaluation Types

| Mode | Purpose |
|------|----------|
| Final Answer Evaluation | Score correctness, grounding, clarity |
| Trajectory Evaluation | Score step quality, tool use, decision quality |
| Tool-Call Evaluation | Score parameters, justification, results |
| RAG Evaluation | Score retrieval, relevance, grounding |
| Safety Evaluation | Detect dangerous, biased, or restricted actions |
| Performance Evaluation | Latency, token cost, success rates |

---

# 2. Evaluation Loop Pattern

### Pattern: Evaluate → Score → Compare → Gate

```
for each test_case:
    run_agent()
    capture_trajectory()
    judge_results()
    compute_scores()
    compare_to_threshold()
```

**Checklist**

- [ ] Each test case produces a full log + trace.  
- [ ] Judging uses deterministic rubric.  
- [ ] Scores recorded per metric.  
- [ ] Thresholds defined for all modes.  

---

# 3. Final Answer Evaluation

### Scoring Rubric (1–5 scale)

```
Correctness: 1–5
Grounding: 1–5
Clarity: 1–5
Safety: pass/fail
```

**Checklist**

- [ ] Answer derived from retrieved evidence.  
- [ ] Citations present.  
- [ ] No hallucinated facts.  
- [ ] No unsupported claims.  

**Anti-Patterns**

- AVOID: Mixing speculation with evidence.  
- AVOID: Providing answer without citations (when RAG used).  

---

# 4. Trajectory Evaluation

### Categories

- Step correctness  
- Plan quality  
- Tool choice justification  
- Observation accuracy  
- Adaptation to new state  
- Error handling  

### Pattern: Step-by-Step Scoring

```
for each step:
    was_step_correct?
    was_tool_choice_valid?
    was_observation_used?
    was_replanning_needed?
```

**Checklist**

- [ ] Each step produces expected output.  
- [ ] Steps lead toward goal.  
- [ ] Agent revises plan when state mismatches.  
- [ ] No unnecessary steps.  

---

# 5. Tool-Call Evaluation

### Metrics (1–5)

- Parameter validity  
- Tool selection correctness  
- Alignment with intent  
- Output verification  
- Safety handling  

### Pattern: Tool Judging

```
validate_parameters()
validate_tool_choice()
validate_output()
```

**Checklist**

- [ ] No hallucinated parameters.  
- [ ] High-risk tools → confirmation required.  
- [ ] Output matches tool schema.  

**Anti-Patterns**

- AVOID: Calling tools when internal reasoning suffices.  
- AVOID: Using tool output without validation.  

---

# 6. RAG Evaluation

### Metrics

| Metric | Definition |
|--------|------------|
| RR – Retrieval Relevance | % of retrieved chunks relevant |
| GS – Grounding Score | Agreement between answer & evidence |
| CP – Context Precision | % irrelevant chunks eliminated |
| CR – Context Recall | % relevant chunks included |
| AA – Answer Accuracy | Overall correctness |

### Pattern: RAG Judge

```
judge_retrieval()
judge_reranking()
judge_context_injection()
judge_answer_grounding()
```

**Checklist**

- [ ] Reranking improved relevance.  
- [ ] Chunks summarized properly.  
- [ ] Evidence and answer match exactly.  

---

# 7. Safety Evaluation

### Pattern: Safety Scan

```
scan_for:
  - high-risk actions
  - policy violations
  - harmful content
  - hallucinated operations
```

### Safety Conditions

- High-risk actions present?  
- Sensitive data referenced?  
- Unsupported domain?  
- Incomplete risk description?  

**Checklist**

- [ ] Confirmation required for irreversible actions.  
- [ ] Reject hallucinated tools/paths.  
- [ ] No personal data leakage.  
- [ ] No instructions in restricted domains.  

---

# 8. Observability Requirements

## 8.1 Logs

### Required Log Fields

- User input  
- Agent plan  
- Tool calls (with full parameters)  
- Tool outputs  
- RAG retrieval details  
- Memory reads/writes  
- Final answer  

### Pattern: Log Snapshot

```
{
  "input": "...",
  "plan": "...",
  "tool_call": {...},
  "tool_result": {...},
  "retrieved_chunks": [...],
  "final_answer": "..."
}
```

---

## 8.2 Traces

### Required Trace Spans

| Span | Description |
|------|-------------|
| LM call | Every model invocation |
| Tool call | Every MCP/API tool call |
| Retrieval | Embed → retrieve → rerank |
| Memory | Read/write events |
| Safety checks | High-risk decisions |

### Pattern: Trace Structure

```
trace {
  span: "tool_call"
  start: timestamp
  end: timestamp
  metadata: {...}
}
```

---

## 8.3 Metrics

### System Metrics

- Latency (p50, p95, p99)  
- Token cost  
- Throughput  
- Memory usage  

### Quality Metrics

- Tool success rate  
- RAG relevance  
- Evaluation score average  
- Safety pass rate  

### Threshold Examples

```
tool_success_rate >= 95%
grounding_score >= 4.0
latency_p95 <= 5s
eval_pass_rate >= 90%
```

---

# 9. CI/CD Evaluation Gates

### Pattern: Gate Before Deploy

```
run_eval_suite()
collect_scores()
check_thresholds()
if fail → block deploy
if pass → allow deploy
```

**Checklist**

- [ ] Full eval suite automated.  
- [ ] Test reports versioned.  
- [ ] Regression tests required.  
- [ ] Canary evaluation enabled.  

---

# 10. Evaluation Anti-Patterns (Master List)

- AVOID: Judging only final answer without trajectory.  
- AVOID: Ignoring tool-call correctness.  
- AVOID: No logs or incomplete logs.  
- AVOID: Using RAG without grounding evaluation.  
- AVOID: Missing safety scoring.  
- AVOID: Skipped reranking step in RAG.  
- AVOID: Inconsistent scoring scales.  
- AVOID: Manual-only reviewing with no automated gating.  

---

# 11. Quick Reference Tables

### Score Table

| Score | Meaning |
|--------|----------|
| 1 | Incorrect / irrelevant |
| 2 | Partially correct |
| 3 | Mostly correct |
| 4 | Correct |
| 5 | Fully correct + grounded |

### Evaluation Coverage Table

| Component | Required |
|-----------|----------|
| Final Answer | Yes |
| Trajectory | Yes |
| Tool Calls | Yes |
| RAG | If applicable |
| Safety | Always |
| Performance | Always |

---

# 12. Copy-Paste Evaluation Templates

### Final Answer Judge Prompt

```
Evaluate the final answer strictly on:
- correctness (1–5)
- grounding (1–5)
- clarity (1–5)
- safety (pass/fail)

Return JSON:
{
  "correctness": n,
  "grounding": n,
  "clarity": n,
  "safety": "pass|fail"
}
```

### Trajectory Judge Prompt

```
Score each step on:
- step correctness
- tool choice correctness
- observation usage
- adaptation

Return JSON list of step scores.
```

### RAG Judge Prompt

```
Evaluate:
- retrieval relevance
- reranking quality
- context precision/recall
- answer grounding

Return JSON with RR, GS, CP, CR, AA.
```

---

# End of File

# Agent Framework Landscape — July 2026

Generative selection toolkit for choosing an agent framework. Pair this with [`build-vs-not-decision.md`](build-vs-not-decision.md) (decide *if* you should build) and [`protocol-decision-tree.md`](protocol-decision-tree.md) (decide MCP vs A2A) before reading this file.

This is a polyglot reference. For Python+TS bot implementation depth, route to [`../../ai-bot-builder/references/framework-selection.md`](../../ai-bot-builder/references/framework-selection.md).

## Table of Contents

- [Snapshot](#snapshot)
- [Selection Matrix](#selection-matrix)
- [By Language](#by-language)
- [By Cloud Marketplace Target](#by-cloud-marketplace-target)
- [Frameworks](#frameworks)
- [Anti-Patterns](#anti-patterns)
- [Migration Paths](#migration-paths)

## Snapshot

| Framework | Lang | Stable | Runtime model | State | Eval/Obs | Protocols |
|---|---|---|---|---|---|---|
| **LangGraph** | Python, TS | 1.2.4 / Python 3.10–3.14, TS toolkit (Jun 2026) | Graph (nodes/edges, cycles, HITL) | Checkpointer + Store | LangSmith native | MCP via community; A2A via community |
| **CrewAI** | Python | 1.10.1 (Mar 2026) | Role-based crew + tasks | Flow-level runtime checkpointing (`CheckpointConfig` + `SqliteProvider`, since ~May 2026); crew task outputs otherwise implicit | CrewAI Studio + OpenTelemetry | MCP + A2A native |
| **Pydantic AI** | Python | 1.x (Apr 2026) | Type-first agent + `pydantic-graph` FSM | Pydantic state, graph persistence | Logfire native | MCP native |
| **Claude Agent SDK** | Python, TS | GA | Loop + hooks + subagents | SDK-managed conversation | Anthropic console + traces | MCP native; A2A via subagent contracts |
| **OpenAI Agents SDK** | Python, TS | GA + Apr 2026 harness | Handoffs + guardrails + harness | Session resume, trace bookkeeping | Tracing native | MCP native; native sandbox (E2B/Modal/Cloudflare/etc.) |
| **Mastra** | TypeScript | 1.0 (Jan 2026) | Agent loop + workflow graphs (separate primitives) | Working memory + conversation memory | Built-in evals + tracing | MCP native; Vercel/Cloudflare/Netlify deployers |
| **Spring AI** | Java/Kotlin | 1.1.x → 2.0 (2026) | ChatClient + Advisors + ToolCallback | Memory advisor + vector stores | Micrometer + Spring Boot Actuator | MCP native; A2A blog series Apr 2026 |
| **Microsoft Agent Framework** | .NET, Python | 1.0 GA (Apr 3, 2026) | Agents + graph workflows (AutoGen+SK convergence) | Session state, type-safe middleware | OpenTelemetry native | MCP + A2A native |
| **Semantic Kernel** | .NET, Python, Java | maintenance | Skills + planners | Memory connectors | OpenTelemetry | Migrate to MS Agent Framework |

> **Date stamp:** July 2026. Ecosystem moves fast — verify versions before committing to a stack.

## Selection Matrix

Pick the row that matches the load-bearing constraint.

| If the constraint is… | Pick | Why |
|---|---|---|
| Branching workflow with checkpoints + HITL | **LangGraph** | Only framework with first-class checkpointer + Store + interrupt/resume |
| Role-based crew, fastest time-to-prototype | **CrewAI** | Highest-level abstraction; native MCP+A2A; weak at long-running state |
| Type-safe Python with FastAPI shop | **Pydantic AI** | Pydantic-native, Logfire-native, `pydantic-graph` for FSM cases |
| Anthropic-first, deep OS access, computer use | **Claude Agent SDK** | Hooks + subagents + extended thinking + computer use |
| OpenAI-first, voice + handoffs | **OpenAI Agents SDK** | Handoffs idiom, voice support, Codex harness, sandbox providers |
| TypeScript shop, ship to Vercel/CF/Netlify | **Mastra** | TS-first, Zod tool schemas, built-in evals, scale-to-zero deployers |
| Spring/Boot enterprise app | **Spring AI** | DI-native, Advisors chain, Java/Kotlin idiom, MCP native |
| .NET enterprise + multi-agent workflows | **MS Agent Framework** | GA Apr 2026; AutoGen+SK convergence; A2A+MCP native |
| Existing SK codebase | **Migrate → MS Agent Framework** | SK is in maintenance; new features land in MAF |

## By Language

- **Python**: LangGraph, CrewAI, Pydantic AI, Claude Agent SDK, OpenAI Agents SDK, MS Agent Framework, Semantic Kernel
- **TypeScript**: LangGraph.js (with Store), Mastra, Claude Agent SDK, OpenAI Agents SDK
- **Java/Kotlin**: Spring AI, Semantic Kernel (limited)
- **.NET**: MS Agent Framework, Semantic Kernel

## By Cloud Marketplace Target

| Cloud | Native distribution path | Compatible frameworks |
|---|---|---|
| **AWS Marketplace / Bedrock** | Bedrock AgentCore, container deploy | Any (LangGraph, CrewAI, Mastra deployer, Pydantic AI common) |
| **Azure AI Foundry** | First-class for MS stack | MS Agent Framework, Semantic Kernel, Spring AI (Azure OpenAI) |
| **Google Cloud Model Garden / Vertex** | Agent Builder + ADK | Google ADK (not in this list), LangGraph, Pydantic AI |

If the deployment target is a marketplace listing, framework choice is shaped less by capability than by **packaging + observability fit**: MAF for Azure, Bedrock-native for AWS, ADK/LangGraph for GCP. Mastra wins TS-on-edge.

## Frameworks

### LangGraph (Python + TypeScript)

- **Shape**: Directed graph of nodes; edges may be conditional. Compiled graph is the agent.
- **State**: Two layers — `Checkpointer` (short-term, per-thread) and `Store` (long-term, cross-thread). **Keep them separate**; conflating them is the most common LG anti-pattern.
- **HITL**: First-class via `interrupt()` + resume tokens.
- **Python version**: 3.10–3.14 (confirmed 1.2.4, June 2026). Verify against the [releases page](https://github.com/langchain-ai/langgraph/releases) before unpinning.
- **Streaming**: v3 streaming API.
- **TS specifics**: `@langchain/langgraph-checkpoint` + `@langgraphjs/toolkit` are the current TS install.
- **Pick when**: branching, retries, approval gates, long-running graphs.
- **Avoid when**: linear pipeline (use a function); team is JS-only and prefers higher-level (use Mastra).

### CrewAI (Python)

- **Shape**: `Crew` of `Agent`s with `role` / `goal` / `backstory`, executing `Task`s. `Flow` adds event-driven control (`@start`, `@listen`, `@router`) around crews for deterministic orchestration.
- **State**: Crew task outputs are implicit and brittle for long-running work. Flows now ship `@persist` state persistence plus (since ~May 2026) runtime checkpointing via `CheckpointConfig` + `SqliteProvider` for automatic recovery. Judgment call: this closes most of the resumability gap for Flow-shaped orchestration, but it checkpoints at Flow-method/Crew-task boundaries only — it does not persist or resume mid-ReAct execution (i.e., a crash mid-tool-loop still replays that step from scratch). Verify current persistence guarantees in the docs before promising exactly-once recovery to stakeholders.
- **Protocols**: Native MCP + A2A as of v1.10.
- **Pick when**: prototype multi-role research/content/ops crews fast; use Flows (not bare Crews) once the pipeline needs resumability or branching.
- **Avoid when**: workflow needs sub-step (mid-tool-call) durability or direct agent-to-agent messaging without Flow wrapping.
- **Migration**: CrewAI → LangGraph is gradual (LangChain-compatible), not a rewrite.

### Pydantic AI (Python)

- **Shape**: `Agent` with typed `deps_type` + `output_type`. Graphs via `pydantic-graph` (generic FSM library).
- **State**: Pydantic models all the way down. Logfire is the default observability.
- **Pick when**: FastAPI shop, type safety matters, you want LangGraph-style FSM without LangChain.
- **Avoid when**: team prefers untyped speed; non-Pydantic Python ecosystem.

### Claude Agent SDK (Python + TypeScript)

- **Shape**: Loop + hooks + subagents. Hooks intercept lifecycle points; subagents delegate.
- **Pick when**: Anthropic-only, computer use, deep OS access, safety-first audit trail.
- **Avoid when**: model portability matters. Locked to Claude.

### OpenAI Agents SDK (Python + TypeScript)

- **Shape**: Handoffs (transfer between specialized agents) + guardrails (input/output validation).
- **Apr 2026 update**: Codex-style **harness** wraps model with instructions/tools/approvals/tracing/resume. Native sandbox via E2B, Modal, Cloudflare, Daytona, Runloop, Vercel, Blaxel.
- **Pick when**: OpenAI-first, voice support, multi-domain handoffs, sandboxed code exec.
- **Avoid when**: you need provider portability (it's opinionated toward OpenAI).

### Mastra (TypeScript)

- **Shape**: Agents (model-driven loop) and workflows (deterministic step graphs) are **separate primitives** — compose both.
- **State**: Working memory + conversation memory are first-class.
- **Tools**: Zod schemas — schema doubles as the prompt-facing description.
- **Deployment**: Deployers for Vercel, Cloudflare Workers, Netlify; Mastra Cloud for managed.
- **Provider**: Mastra Model Router — thousands of models across ~100+ providers via one API, automatic fallback. The exact count is a live, dynamically-updated catalog (fed from models.dev/OpenRouter/gateways) — don't quote a specific figure from memory; check `mastra.ai/models` at decision time.
- **Pick when**: TS-first stack, edge deploy, you want one framework instead of LangGraph.js + extras.
- **Avoid when**: Python ecosystem; need LangSmith.

### Spring AI (Java/Kotlin)

- **Core**: `ChatClient` (sync + streaming), `Advisors` chain, `@Tool` + `ToolCallback`, `ToolCallingManager`.
- **2026 patterns**: A2A integration (Jan 2026 blog series), `ToolCallAdvisor` for explicit tool-loop control (1.1.0-M4), `AutoMemoryTools` for persistent memory (Apr 2026).
- **Pick when**: existing Spring Boot estate; Java/Kotlin team; DI-driven architecture.
- **Avoid when**: greenfield; non-JVM team.

### Microsoft Agent Framework (.NET + Python)

- **Status**: 1.0 GA on April 3, 2026. Convergence of AutoGen + Semantic Kernel.
- **Shape**: Agents + **graph-based workflows** for explicit multi-agent orchestration.
- **Process Framework**: Q2 2026 — deterministic enterprise workflows with audit trails, low-code visual design, checkpointing, HITL.
- **Standards**: A2A native, MCP native, middleware-first.
- **Pick when**: .NET shop; Azure AI Foundry deploy; need enterprise process compliance.
- **Avoid when**: pure Python team without Azure dependency (MAF Python exists but is 2nd-class to .NET).

### Semantic Kernel (.NET + Python + Java)

- **Status**: maintenance. Critical bugs/security only. New features go to MAF.
- **Action**: existing SK codebases stay on SK for now; greenfield → MAF. Migration guide is published.

## Anti-Patterns

| Anti-pattern | Why it hurts | Fix |
|---|---|---|
| **A1. "Pick the trendiest framework"** | Optimizes for hype, not fit | Decide constraint first (lang, deploy target, state needs), then pick |
| **A2. CrewAI Crews (not Flows) for resumable long-running workflows** | Bare `Crew`/`Task` state is implicit and brittle; only `Flow` + `CheckpointConfig` gets you recovery, and only at method/task boundaries | Use CrewAI `Flow` with checkpointing for CrewAI-native pipelines; use LangGraph or MS Agent Framework when you need sub-step (mid-tool-call) durability |
| **A3. LangGraph for linear pipelines** | 15+ transitive deps, overhead for no win | Plain async function with TypedDict |
| **A4. Conflating LangGraph Checkpointer with Store** | Conversation state and user-level memory have different lifecycles | Separate them; checkpointer is per-thread, store is cross-thread |
| **A5. Mastra workflows used as agents (or vice versa)** | They're separate primitives by design — workflows are deterministic, agents are model-driven | Compose both; use the right tool per step |
| **A6. New SK projects in 2026** | SK is in maintenance | Start on MS Agent Framework |
| **A7. Provider lock for portability claims** | Claude/OpenAI SDKs are *not* provider-portable despite claims | If portability matters, use LangGraph / Pydantic AI / Mastra Router |
| **A8. Skipping eval setup until "later"** | Frameworks with built-in evals (Mastra, MAF, LangSmith) lose their value if you don't wire them on day one | Stand up eval harness in the first commit; see [`evaluation-and-observability.md`](evaluation-and-observability.md) |
| **A9. Custom A2A wire format** | A2A is now native in CrewAI/MAF/Spring AI | Use the protocol; see [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md) |
| **A10. Hand-rolled sandbox for code-exec agents** | OpenAI Agents SDK ships sandbox integrations with 7 providers | Use the SDK's sandbox plumbing |

## Migration Paths

- **CrewAI → LangGraph**: gradual, LangChain-compatible. Migrate the parts that need checkpoints/HITL first.
- **Semantic Kernel → MS Agent Framework**: official migration guide; SK supported ≥1 year post-GA.
- **AutoGen → MS Agent Framework**: same convergence; AutoGen idioms preserved in MAF agent abstractions.
- **n8n / Langflow → code-first**: see [`../../ai-bot-builder/references/migration-from-n8n.md`](../../ai-bot-builder/references/migration-from-n8n.md).
- **LangGraph.js + custom store → LangGraph Store**: collapse hand-rolled persistence into the new Store primitive.

## Verification Checklist Before Committing

Before writing the first node/agent/crew:

- [ ] Constraint matrix scored (lang × deploy target × state needs × team profile)
- [ ] Eval harness path identified (LangSmith / Logfire / Mastra evals / OTEL)
- [ ] Provider portability decision logged (locked-in vs router)
- [ ] HITL and approval policy mapped to framework primitives (interrupt vs middleware vs handoff)
- [ ] Cloud marketplace listing fit checked if relevant (AWS / Azure Foundry / GCP Model Garden)

## Sources

Verify before quoting in production decisions:

- LangGraph: <https://docs.langchain.com/oss/javascript/langgraph/persistence>, <https://langchain-ai.github.io/langgraphjs/reference/modules/langgraph-checkpoint.html>
- CrewAI vs LangGraph 2026: <https://gurusup.com/blog/best-multi-agent-frameworks-2026>, <https://redwerk.com/blog/langgraph-vs-crewai/>
- Pydantic AI: <https://ai.pydantic.dev/>, <https://github.com/pydantic/pydantic-ai>
- Mastra 1.0: <https://mastra.ai/>, <https://github.com/mastra-ai/mastra>, <https://www.generative.inc/mastra-ai-the-complete-guide-to-the-typescript-agent-framework-2026>
- Spring AI: <https://docs.spring.io/spring-ai/reference/api/chatclient.html>, <https://spring.io/blog/2026/04/07/spring-ai-agentic-patterns-6-memory-tools/>, <https://spring.io/blog/2026/01/29/spring-ai-agentic-patterns-a2a-integration/>
- MS Agent Framework GA: <https://learn.microsoft.com/en-us/agent-framework/overview/>, <https://techcommunity.microsoft.com/blog/azuredevcommunityblog/the-future-of-agentic-ai-inside-microsoft-agent-framework-1-0/4510698>
- SK migration: <https://learn.microsoft.com/en-us/agent-framework/migration-guide/from-semantic-kernel/>
- OpenAI Agents SDK Apr 2026 harness: <https://qubittool.com/blog/ai-agent-framework-comparison-2026>, <https://composio.dev/content/claude-agents-sdk-vs-openai-agents-sdk-vs-google-adk>

# Game Theory for Multi-Agent Systems

> **Gate before invoking:** Check [`foundations-game-theory` § When to Apply](../../foundations-game-theory/SKILL.md#when-to-apply) first. The recipes below assume the foundation is the right tool for the situation; the foundation's skip-conditions route you to a different foundation if not.


Nash equilibrium, mechanism design, and common knowledge applied to AI agent coordination, tool access negotiation, and multi-agent workflow design. Based on non-cooperative game theory and mechanism design.

## Contents

- [Agents as Strategic Players](#agents-as-strategic-players)
- [Nash Equilibrium in Agent Coordination](#nash-equilibrium-in-agent-coordination)
- [Mechanism Design for Agent Incentives](#mechanism-design-for-agent-incentives)
- [Common Knowledge and Communication](#common-knowledge-and-communication)
- [Resource Contention Games](#resource-contention-games)
- [Cooperative vs. Competitive Agent Architectures](#cooperative-vs-competitive-agent-architectures)
- [Design Patterns](#design-patterns)
- [Decision Checklist](#decision-checklist)

---

## Agents as Strategic Players

When multiple AI agents operate in a shared environment (shared tools, APIs, file systems, or tasks), their interactions form a game:

| Game Element | Agent System Equivalent |
|-------------|----------------------|
| Players | Individual agents (worker, reviewer, orchestrator) |
| Strategies | Tool calls, task selection, output formats, resource requests |
| Payoffs | Task completion quality, latency, cost, user satisfaction |
| Information | Context available to each agent, visibility into other agents' state |

### When Game Theory Applies to Agent Systems

| Situation | Applies? | Why |
|-----------|:--------:|-----|
| Single agent, single task | No | No strategic interaction |
| Multiple agents, independent tasks | Minimal | No interdependence — each agent optimizes alone |
| Multiple agents, shared resources | Yes | Resource contention creates strategic interaction |
| Multiple agents, dependent tasks | Yes | Output quality of one affects payoff of another |
| Adversarial agents (red team/blue team) | Yes | Directly competitive strategic interaction |
| Agent + human interaction | Yes | Human and agent preferences may conflict |

---

## Nash Equilibrium in Agent Coordination

### Finding Stable Agent Configurations

A multi-agent system is in **Nash equilibrium** when no single agent can improve its outcome by unilaterally changing behavior.

**Desirable equilibria**:
- All agents complete their assigned tasks efficiently
- No agent overloads shared resources
- Quality meets thresholds across all outputs

**Undesirable equilibria**:
- Agents race for the same resources (contention spiral)
- Agents produce redundant work (duplication equilibrium)
- Agents wait for each other indefinitely (deadlock)

### Equilibrium Design Principles

| Principle | Implementation | Avoids |
|-----------|---------------|--------|
| **Clear task ownership** | Assign non-overlapping task domains | Duplication and contention |
| **Resource quotas** | Rate limits, token budgets per agent | Resource starvation |
| **Priority ordering** | Explicit agent priority for shared resources | Deadlock |
| **Output contracts** | Defined interface between agents | Cascading quality failures |

---

## Mechanism Design for Agent Incentives

### Designing the Rules

Mechanism design for agents means designing the orchestration rules so that each agent, acting in its own "interest" (optimizing its objective), produces the outcome you want.

| Design Goal | Mechanism | How It Works |
|-------------|-----------|-------------|
| **Truthful status reporting** | Reward accuracy, penalize over-optimism | Agents report task completion honestly instead of prematurely |
| **Efficient resource use** | Budget constraints with rollover | Agents conserve resources because budget is finite |
| **Quality over speed** | Score on output quality, not completion time | Agents don't race to finish at the expense of quality |
| **Collaboration over competition** | Joint scoring on shared outcomes | Agents help each other because joint outcome affects each agent's score |

### Incentive Compatibility for Agents

An agent orchestration system is **incentive compatible** when each agent's locally optimal behavior produces the globally optimal outcome.

**Test**: If each agent greedily optimizes its own objective, does the system converge to a good state?

| System | Incentive Compatible? | Fix |
|--------|:--------------------:|-----|
| Workers compete for limited context window | No — agents bloat context to monopolize | Budget per agent, shared context pool with priority |
| Reviewer agent gets same score regardless of feedback quality | No — reviewer has no incentive to be thorough | Score reviewer on downstream impact of reviewed work |
| Parallel agents with no dependency tracking | No — agents may duplicate or conflict | Dependency graph with task locks |

---

## Common Knowledge and Communication

### Common Knowledge in Agent Systems

**Common knowledge** means every agent knows X, every agent knows that every agent knows X, and so on infinitely. In game theory, common knowledge enables coordination without explicit communication.

### Agent Communication Protocols

| Protocol | Game Theory Analogy | When to Use |
|----------|--------------------:|-------------|
| **Broadcast** (all agents see all messages) | Common knowledge — everyone knows and knows everyone knows | Small agent teams, critical coordination |
| **Point-to-point** (1:1 messages) | Private information — only sender and receiver know | Large teams, need-to-know basis |
| **Shared state** (database/file) | Public information — available to all who check | Asynchronous coordination, audit trail |
| **Event-driven** (pub/sub) | Observable actions — agents infer from events | Loosely coupled, scalable systems |

### Information Revelation Strategy

| What to Share | With Whom | Game Theory Basis |
|--------------|-----------|-------------------|
| Task completion status | Orchestrator + dependent agents | Enables coordination without polling |
| Error/failure state | Orchestrator only | Prevents cascading panic — orchestrator decides response |
| Resource usage | Orchestrator (for budgeting) | Enables fair allocation |
| Intermediate outputs | Dependent agents only | Reduces context bloat; targeted information sharing |

---

## Resource Contention Games

### Common Contention Scenarios

| Resource | Contention Type | Resolution |
|----------|----------------|------------|
| **API rate limits** | Multiple agents hitting same API | Token bucket shared across agents, priority queue |
| **File system** | Concurrent writes to same files | File locking, single-writer principle |
| **Context window** | Multiple agents consuming shared context | Per-agent context budget, summarization gates |
| **Human attention** | Multiple agents requesting human review | Priority queue, batched review sessions |
| **Compute budget** | Cost allocation across agents | Per-agent cost caps, shared pool with fairness rules |

### Fair Division Mechanisms

| Mechanism | How It Works | When to Use |
|-----------|-------------|-------------|
| **Equal split** | Each agent gets 1/N of resources | Tasks are roughly equal in resource needs |
| **Proportional** | Allocation proportional to task importance | Tasks vary in priority |
| **Priority queue** | Highest-priority agent goes first | Strict ordering exists (critical path vs. optional) |
| **Auction** | Agents "bid" importance; highest bid wins | Dynamic priority that changes per round |

---

## Cooperative vs. Competitive Agent Architectures

### Cooperative (Aligned Objectives)

All agents share the same goal — optimize the system-level outcome.

| Pattern | Structure | Strength |
|---------|-----------|----------|
| Orchestrator-worker | Central coordinator assigns tasks | Clear control, efficient allocation |
| Pipeline | Output of agent A feeds into agent B | Sequential efficiency, clear interfaces |
| Ensemble | Multiple agents produce outputs, best is selected | Quality through diversity |

### Competitive (Adversarial Objectives)

Agents have opposed or independent objectives — used intentionally for quality.

| Pattern | Structure | Strength |
|---------|-----------|----------|
| Red team / blue team | Attacker agent vs. defender agent | Security and robustness testing |
| Evaluator-optimizer | One agent optimizes, another critiques | Prevents quality drift |
| Debate | Two agents argue opposing positions | Better reasoning through adversarial pressure |

### Choosing the Architecture

| Criterion | Cooperative | Competitive |
|-----------|:-----------:|:-----------:|
| Task requires consistency | Preferred | Risky — agents may contradict |
| Quality depends on scrutiny | Secondary | Preferred — adversarial pressure catches errors |
| Speed is critical | Preferred — less overhead | Slower — requires resolution mechanism |
| Creativity needed | Useful (ensemble) | Useful (debate) |
| Trust in individual agent output | Assumed | Verified through opposition |

### LLM Agent Behavioral Realities (2026 Research)

LLM-based agents deviate from classical game-theoretic rationality in important ways:

| Finding | Source | Design Implication |
|---------|--------|-------------------|
| **Pro-social bias** | IJCAI-25 survey: "Game Theory Meets LLMs" | LLM agents cooperate more than Nash equilibrium predicts — useful for cooperative architectures but may underperform in adversarial roles |
| **Tacit collusion** | GPT-4 agents in repeated Bertrand pricing games learned to maintain supracompetitive prices via reward-punishment | In competitive agent designs, LLM agents may spontaneously collude instead of competing — monitor for convergence to non-competitive equilibria |
| **Rationality degrades with complexity** | arXiv 2411.05990: game-theoretic workflow scaffolding restores rationality | For complex multi-agent games, add explicit game-theoretic reasoning steps to agent prompts — don't assume rational play emerges naturally |
| **Personality affects strategy** | Big Five trait definitions in prompts alter negotiation behavior | Agent persona design is a strategic variable — aggressive vs. cooperative agents behave differently in the same game |

**Key takeaway**: Don't assume LLM agents will play Nash-optimal strategies. They bring human-like biases — cooperation bias in competitive settings, collusion risk in pricing, and irrationality under complexity. Design accordingly: add guardrails for competitive agents, exploit cooperation bias for collaborative ones.

---

## Design Patterns

### Pattern: Vickrey Task Allocation

Agents "bid" on tasks by reporting estimated difficulty/time. Assign to lowest bidder. Agent doesn't "pay" their bid — they pay the second-lowest bid (in terms of expected effort). Incentive: report true estimates.

### Pattern: Tit-for-Tat Collaboration

In multi-round agent interactions: Agent A cooperates (provides quality output) in round 1. If Agent B reciprocates with quality, continue cooperating. If Agent B provides low quality, reduce effort in response. Prevents free-riding in collaborative pipelines.

### Pattern: Common Knowledge Checkpoint

Before critical coordination points, broadcast state to all agents and confirm receipt. This creates common knowledge — every agent knows the state AND knows every other agent knows. Enables coordinated action without explicit synchronization.

### Pattern: Game-Theoretic Workflow Scaffolding (2026)

For complex multi-agent interactions, add explicit game-theoretic reasoning as an intermediate step:
1. Agent receives task context
2. Agent explicitly models: "What are other agents' likely strategies?"
3. Agent computes best response given those strategies
4. Agent executes action

This scaffolding restores rational behavior that degrades when LLM agents face complex games without structured reasoning.

### Pattern: Federated Multi-Agent Coordination

For independent AI systems with competing priorities (inspired by Johns Hopkins MpFL framework): use game-theoretic mechanisms to negotiate resource allocation across autonomous agents that don't share a central orchestrator. Each agent reports its utility function; a fair division mechanism allocates shared resources.

---

### Pattern: Courtroom Debate with Progressive Evidence (PROClaim, March 2026)

Structure adversarial verification as a trial: plaintiff argues FOR, defense argues AGAINST, a critic independently evaluates, and a judicial panel renders verdict. Progressive RAG dynamically retrieves new evidence during rounds instead of relying on a static pool (+7.5pp accuracy). A role-switching consistency test swaps plaintiff and defense after the primary debate to detect position-anchored reasoning (-4.2pp errors without it). Key finding: LLMs exhibit structural negativity bias — REFUTE positions converge faster. See [agents-subagents game theory reference](../../agents-subagents/references/game-theory-agent-teams.md#8-courtroom-style-progressive-debate-proclaim-pattern) for full protocol.

### Pattern: Pareto-Nash Multi-Objective Synthesis (2025-2026)

When teams optimize for multiple competing objectives (growth vs. monetization vs. retention), map the Pareto frontier: identify all options where no objective can improve without worsening another, remove dominated options (worse on ALL objectives), and present Pareto-optimal choices with explicit tradeoffs. Merges Nash stability with Pareto optimality. See [agents-subagents game theory reference](../../agents-subagents/references/game-theory-agent-teams.md#9-pareto-nash-equilibrium-for-multi-objective-teams).

### Pattern: Evolutionary Coordination Rule Design (AlphaEvolve, DeepMind April 2026)

Use an LLM to iteratively refine coordination rules (belief briefs, debate triggers, synthesis protocols) themselves. Seed with current rules, run team on benchmark, measure quality + cost, propose mutations, keep improvements. Only cost-effective for high-frequency teams where many benchmark runs justify the search cost. See [agents-subagents game theory reference](../../agents-subagents/references/game-theory-agent-teams.md#10-evolutionary-algorithm-design-alphaevolve-pattern).

---

## Decision Checklist

- [ ] Identified which agents interact strategically (shared resources, dependent tasks)
- [ ] Designed for Nash equilibrium — would any agent benefit from deviating?
- [ ] Tested incentive compatibility — does local optimization produce good global outcomes?
- [ ] Chose communication protocol matching information needs (broadcast vs. point-to-point)
- [ ] Resolved resource contention with fair division mechanisms
- [ ] Selected cooperative vs. competitive architecture based on quality requirements
- [ ] Defined output contracts between agents (interface quality guarantees)
- [ ] Built monitoring for undesirable equilibria (deadlock, duplication, starvation)
- [ ] For multi-objective decisions: mapped Pareto frontier instead of single-objective optimization
- [ ] For claim verification: considered courtroom pattern with progressive evidence retrieval
- [ ] For high-frequency teams: considered evolutionary rule optimization

# Graph And Loop Engineering: Composition Router

## Scope And Terminology

**Graph engineering** and **loop engineering** are emerging, non-standard labels, not protocols or a settled architecture taxonomy. Use them as prompts to name the underlying design problem; do not select a framework or datastore from the label alone. LangChain describes graph engineering as constraining agent behaviour through a graph, and explicitly treats a loop as a directed cyclic graph; IBM likewise calls loop engineering an emerging practice. [LangChain: Graph Engineering](https://www.langchain.com/blog/3-years-of-graph-engineering-with-langgraph), [IBM: Loop Engineering](https://www.ibm.com/think/topics/loop-engineering)

The terms overlap. A loop is often one cycle within an agent/workflow graph; an improvement graph can supervise many runs; and a knowledge/context graph can supply one node with evidence. They are not interchangeable.

**Harness engineering** circulates as a third sibling label (mid-2026, across vendor and practitioner posts; also the "Agentic Harness Engineering" paper, arXiv [2604.25850](https://arxiv.org/abs/2604.25850)): the environment *around* the model — tool wiring and interfaces, context injection, permissions, persistence, execution control, observability — as distinct from the loop's feedback cycle or the graph's topology. It is not a graph question; route it to the harness-layer skills: [`../ai-coding-agents-tools/SKILL.md`](../../ai-coding-agents-tools/SKILL.md), [`../ai-coding-agents-permissions/SKILL.md`](../../ai-coding-agents-permissions/SKILL.md), [`../agents-hooks/SKILL.md`](../../agents-hooks/SKILL.md), `ai-context-layer`, and [`../ai-coding-agents-observability-evals/references/harness-self-evolution.md`](../../ai-coding-agents-observability-evals/references/harness-self-evolution.md) for evolving the harness itself. A useful triage heuristic from the practitioner discourse: weak or unsafe *operation* → harness; unreliable *results* → loop; unmanageable *process shape* → graph.

## Choose The Graph's Job First

| If the question is... | It is this graph | Use it to model | Route for depth |
| --- | --- | --- | --- |
| "What may run next?", "Where do tools, agents, approval, retry, or a handoff go?" | **Agent/workflow graph** | Runtime state, nodes, transitions, branches, retries, pauses, and cycles | [`multi-agent-patterns.md`](multi-agent-patterns.md), [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md), `agents-subagents` |
| "How do recurring agents find, change, test, review, and promote work?" | **Networked improvement graph** | The feedback network across discovery, workers, evaluators, human gates, backlog, and durable evidence | [`autonomous-loop-patterns.md`](autonomous-loop-patterns.md), [`evaluation-and-observability.md`](evaluation-and-observability.md), [`../agents-hooks/SKILL.md`](../../agents-hooks/SKILL.md) |
| "What facts, documents, entities, decisions, and provenance should this run retrieve?" | **Knowledge/context graph** | Evidence relationships, retrieval routes, provenance, memory, and permissions—not runtime sequencing | [`context-graph-patterns.md`](context-graph-patterns.md), `ai-context-layer`, [`../ai-rag/SKILL.md`](../../ai-rag/SKILL.md), [`../ai-vector-brain/SKILL.md`](../../ai-vector-brain/SKILL.md) |

Do not call a workflow DAG a knowledge graph merely because it has nodes and edges. Do not use a knowledge graph as an execution plan. Do not treat the improvement graph as permission to release a change: evaluation and human approval remain explicit gates.

## Agent/Workflow Graph

Use an agent/workflow graph when the system's valid execution paths matter: deterministic code can sit beside model or agent nodes, while conditional edges express the permitted transitions. Production graphs commonly include cycles for tool retries, validation-and-revision, user input, or resumption; a DAG is only appropriate when cycles are genuinely unnecessary. [LangGraph overview](https://docs.langchain.com/oss/javascript/langgraph/overview), [LangChain: Graph Engineering](https://www.langchain.com/blog/3-years-of-graph-engineering-with-langgraph)

For each node and edge, specify input/output state, authority, side effects, observability, and an exit or escalation condition. Keep deterministic routing in code where speed, cost, and predictability matter; use agentic routing only where the branch cannot be specified safely in advance. The OpenAI Agents SDK similarly distinguishes code-driven orchestration from LLM-driven orchestration and supports their combination. [OpenAI Agents SDK: multi-agent orchestration](https://openai.github.io/openai-agents-python/multi_agent/)

## Loop Engineering

Use a loop when a bounded goal needs repeated action and observation, not merely because an agent has tools. At minimum, state: goal and measurable acceptance criteria; action surface; independent observation/evaluation; adjustment or keep/revert rule; budget; and terminal, escalation, and kill conditions. IBM frames the basic cycle as goal, action, observation, and adjustment. [IBM: Loop Engineering](https://www.ibm.com/think/topics/loop-engineering)

An SDK's internal agent loop is not automatically a long-horizon improvement loop. For example, the OpenAI runner repeats model calls around tool calls and handoffs until final output or its turn limit; the surrounding system must still impose release controls, durable state, and an independent evaluator when it is changing code or production artifacts. [OpenAI Agents SDK: running agents](https://openai.github.io/openai-agents-python/running_agents/)

For coding-agent loop composition—automation, isolated worktrees, skills, connectors, subagents, and external state—use the practical account as a design input, not a normative standard. [Addy Osmani: Loop Engineering](https://addyosmani.com/blog/loop-engineering/)

## Minimal Composition Pattern

```text
knowledge/context graph --retrieves evidence--> workflow node
workflow graph --runs bounded loop--> action -> independent evaluation
networked improvement graph --records result--> backlog / human gate
human gate --permits promotion or stops the loop--> deployment
```

Keep the three graphs separately inspectable. A trace should answer what executed; a ledger should answer what improved and why; and a citation/provenance path should answer what evidence informed the action.

# Guardrails Implementation

> Operational reference for building multi-layer guardrails — input validation, output filtering, tool approval gates, content classification, escalation triggers, and defense-in-depth architecture for AI agents.

**Freshness anchor:** January 2026 — covers LlamaGuard 3, NeMo Guardrails 0.10.x, Guardrails AI 0.5.x, OpenAI Moderation API v2.

---
## Table of Contents

- [5-Layer Defense Architecture](#5-layer-defense-architecture)
- [Layer 1: Input Validation](#layer-1-input-validation)
- [Validation Rules Quick Reference](#validation-rules-quick-reference)
- [Input Validation Code Pattern](#input-validation-code-pattern)
- [Layer 2: Content Classification](#layer-2-content-classification)
- [Classification Decision Tree](#classification-decision-tree)
- [LlamaGuard Integration](#llamaguard-integration)
- [NeMo Guardrails Integration](#nemo-guardrails-integration)
- [Layer 3: Execution Guardrails](#layer-3-execution-guardrails)
- [Tool Approval Gate Matrix](#tool-approval-gate-matrix)
- [Tool Gate Implementation](#tool-gate-implementation)
- [Resource Limits](#resource-limits)
- [Layer 4: Output Filtering](#layer-4-output-filtering)
- [Output Validation Pipeline](#output-validation-pipeline)
- [PII Leak Detection](#pii-leak-detection)
- [Layer 5: Monitoring and Escalation](#layer-5-monitoring-and-escalation)
- [HITL Escalation Triggers](#hitl-escalation-triggers)
- [Confidence Threshold Calibration](#confidence-threshold-calibration)
- [Audit Logging Requirements](#audit-logging-requirements)
- [Testing Guardrail Effectiveness](#testing-guardrail-effectiveness)
- [Test Categories](#test-categories)
- [Guardrail Effectiveness Metrics](#guardrail-effectiveness-metrics)
- [Anti-Patterns](#anti-patterns)
- [Cross-References](#cross-references)


## 5-Layer Defense Architecture

```
┌─────────────────────────────────────────────┐
│ Layer 1: INPUT VALIDATION                    │
│ - Schema validation, length limits            │
│ - PII detection and redaction                 │
│ - Injection pattern detection                 │
├─────────────────────────────────────────────┤
│ Layer 2: CONTENT CLASSIFICATION               │
│ - Topic safety classifier                     │
│ - Intent detection (malicious vs benign)      │
│ - Prompt injection scoring                    │
├─────────────────────────────────────────────┤
│ Layer 3: EXECUTION GUARDRAILS                 │
│ - Tool approval gates                         │
│ - Resource limits (tokens, API calls, time)   │
│ - Sandboxed execution environments            │
├─────────────────────────────────────────────┤
│ Layer 4: OUTPUT FILTERING                     │
│ - Response safety classification              │
│ - Factuality / hallucination checks           │
│ - PII leak detection                          │
│ - Format and schema validation                │
├─────────────────────────────────────────────┤
│ Layer 5: MONITORING & ESCALATION              │
│ - Anomaly detection on usage patterns         │
│ - Human-in-the-loop triggers                  │
│ - Audit logging                               │
│ - Circuit breakers                            │
└─────────────────────────────────────────────┘
```

---

## Layer 1: Input Validation

### Validation Rules Quick Reference

| Check | Implementation | Threshold |
|---|---|---|
| Max input length | Character/token count | 4096 tokens (adjust per use case) |
| Encoding validation | UTF-8 check, reject binary | Reject non-UTF-8 |
| Language detection | fasttext/langdetect | Reject unsupported languages |
| PII detection | Presidio / custom NER | Redact or reject based on policy |
| URL/link scanning | Regex + allowlist | Block unknown domains |
| File upload scanning | ClamAV + type check | Reject executables, limit size |
| Rate limiting | Per-user token bucket | 10 requests/min default |

### Input Validation Code Pattern

```python
from pydantic import BaseModel, validator, Field
from typing import Optional
import re

class AgentInput(BaseModel):
    message: str = Field(max_length=16000)
    session_id: str = Field(pattern=r'^[a-zA-Z0-9\-]{1,64}$')
    attachments: Optional[list[str]] = Field(default=None, max_length=5)

    @validator("message")
    def validate_message(cls, v):
        # Reject null bytes
        if "\x00" in v:
            raise ValueError("Invalid characters in message")

        # Reject excessive repetition (potential DoS)
        if len(set(v.split())) < len(v.split()) * 0.1:
            raise ValueError("Message contains excessive repetition")

        return v.strip()

class InputGuardrail:
    def __init__(self, config):
        self.max_tokens = config.get("max_tokens", 4096)
        self.pii_detector = PIIDetector()
        self.injection_detector = InjectionDetector()

    async def validate(self, input_data: AgentInput) -> GuardrailResult:
        checks = [
            self._check_token_count(input_data.message),
            self._check_pii(input_data.message),
            self._check_injection(input_data.message),
        ]
        results = await asyncio.gather(*checks)
        return self._aggregate_results(results)
```

---

## Layer 2: Content Classification

### Classification Decision Tree

```
Input message received
│
├── Run injection classifier (score 0-1)
│   ├── Score > 0.9 → BLOCK (high confidence injection)
│   ├── Score 0.7-0.9 → FLAG for review + proceed with restrictions
│   └── Score < 0.7 → PASS
│
├── Run topic safety classifier
│   ├── Harmful content detected → BLOCK + log
│   ├── Borderline content → Apply restricted mode
│   └── Safe content → PASS
│
├── Run intent classifier
│   ├── Out-of-scope request → REDIRECT to appropriate channel
│   ├── Sensitive action requested → Require confirmation
│   └── Normal request → PASS
│
└── All checks passed → Forward to agent
```

### LlamaGuard Integration

- Model: `meta-llama/Llama-Guard-3-8B` — classifies across 13 unsafe categories (S1-S13)
- Input: format as conversation (user + optional assistant message)
- Output: "safe" or "unsafe" with category codes
- Use `apply_chat_template()` for proper formatting
- Can classify both user input AND agent output

### NeMo Guardrails Integration

- Configure via YAML: define input rails (jailbreak, toxicity, topic) and output rails (toxicity, factuality, data leakage)
- Usage: `RailsConfig.from_path()` then `LLMRails(config).generate_async(messages=...)`
- Response is automatically filtered through all configured rails
- Supports custom Colang flows for domain-specific rules

---

## Layer 3: Execution Guardrails

### Tool Approval Gate Matrix

| Tool Category | Approval Level | Implementation |
|---|---|---|
| Read-only data retrieval | Auto-approve | No gate needed |
| Internal API calls | Auto-approve with logging | Log all calls |
| External API calls | Require confirmation for new endpoints | Allowlist check |
| Data modification (write) | Require user confirmation | Confirmation prompt |
| Financial transactions | Require explicit user + amount confirmation | Double confirmation |
| File system operations | Restricted paths only | Sandboxed filesystem |
| Code execution | Sandboxed environment only | Docker/gVisor sandbox |
| Email/messaging send | Require user confirmation | Preview before send |
| Database mutations | Require confirmation + dry-run | Show planned changes |

### Tool Gate Implementation

```python
class ToolApprovalGate:
    APPROVAL_LEVELS = {
        "auto": lambda tool, params: True,
        "log": lambda tool, params: True,  # auto-approve but log
        "confirm": None,  # requires user confirmation
        "deny": lambda tool, params: False,
    }

    def __init__(self, policy: dict):
        self.policy = policy  # tool_name -> approval_level mapping
        self.call_counts = defaultdict(int)
        self.max_calls_per_tool = 20

    async def check(self, tool_name: str, params: dict) -> ApprovalResult:
        # Rate limit per tool
        self.call_counts[tool_name] += 1
        if self.call_counts[tool_name] > self.max_calls_per_tool:
            return ApprovalResult(approved=False, reason="Tool call limit exceeded")

        level = self.policy.get(tool_name, "confirm")  # default: require confirmation

        if level == "deny":
            return ApprovalResult(approved=False, reason="Tool blocked by policy")

        if level == "confirm":
            return ApprovalResult(
                approved=False,
                requires_confirmation=True,
                preview=self._format_preview(tool_name, params)
            )

        return ApprovalResult(approved=True, level=level)

    def _format_preview(self, tool_name: str, params: dict) -> str:
        return f"Agent wants to call `{tool_name}` with: {json.dumps(params, indent=2)}"
```

### Resource Limits

| Resource | Default Limit | Escalation |
|---|---|---|
| Total tokens per session | 100,000 | Warn at 80%, hard stop at 100% |
| LLM calls per task | 25 | Warn at 20, hard stop at 25 |
| Tool calls per task | 50 | Warn at 40, hard stop at 50 |
| Execution time per task | 5 minutes | Warn at 4min, hard stop at 5min |
| Concurrent tool calls | 5 | Queue additional calls |
| Output length | 4,096 tokens | Truncate with warning |
| File upload size | 10MB | Reject with message |

---

## Layer 4: Output Filtering

### Output Validation Pipeline

```
LLM response generated
│
├── Step 1: Format validation
│   ├── JSON mode → Validate against schema
│   ├── Tool call → Validate function name + params
│   └── Free text → Check length limits
│
├── Step 2: Safety classification
│   ├── Run LlamaGuard on response
│   ├── Run toxicity classifier
│   └── Check against custom blocklists
│
├── Step 3: PII leak detection
│   ├── Scan for SSN, credit card, phone patterns
│   ├── Check for training data memorization patterns
│   └── Verify no internal system prompts leaked
│
├── Step 4: Factuality check (if applicable)
│   ├── Cross-reference claims against retrieved context
│   ├── Flag unsupported assertions
│   └── Add confidence qualifiers where needed
│
└── Step 5: Final sanitization
    ├── Strip internal reasoning markers
    ├── Remove tool call artifacts
    └── Ensure consistent formatting
```

### PII Leak Detection

```python
import re

PII_PATTERNS = {
    "ssn": r'\b\d{3}-\d{2}-\d{4}\b',
    "credit_card": r'\b(?:\d{4}[\s-]?){3}\d{4}\b',
    "email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
    "phone_us": r'\b(?:\+1[-.\s]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b',
    "ip_address": r'\b(?:\d{1,3}\.){3}\d{1,3}\b',
    "api_key": r'\b(?:sk|pk|api[_-]?key)[_-][A-Za-z0-9]{20,}\b',
}

def scan_for_pii(text: str) -> list[dict]:
    findings = []
    for pii_type, pattern in PII_PATTERNS.items():
        matches = re.finditer(pattern, text)
        for match in matches:
            findings.append({
                "type": pii_type,
                "start": match.start(),
                "end": match.end(),
                "value": "[REDACTED]"
            })
    return findings

def redact_pii(text: str) -> str:
    for pii_type, pattern in PII_PATTERNS.items():
        text = re.sub(pattern, f"[{pii_type.upper()}_REDACTED]", text)
    return text
```

---

## Layer 5: Monitoring and Escalation

### HITL Escalation Triggers

| Trigger | Confidence Threshold | Action |
|---|---|---|
| Safety classifier flags response | Any "unsafe" classification | Block + human review |
| Agent confidence is low | Model reports <0.6 confidence | Present draft to human |
| High-stakes action requested | Financial >$100, data deletion | Require human approval |
| User expresses frustration | Sentiment score <-0.5 twice | Offer human handoff |
| Agent loops (3+ identical steps) | Automatic detection | Pause + notify operator |
| Out-of-scope request detected | Intent classifier: "other" | Route to human |
| User explicitly requests human | Keyword detection | Immediate handoff |
| Error rate spike | >3 errors in 5 minutes | Circuit breaker + alert |

### Confidence Threshold Calibration

```python
class ConfidenceGate:
    """Route to HITL based on model confidence."""

    THRESHOLDS = {
        "auto_approve": 0.85,    # Agent handles autonomously
        "soft_review": 0.60,     # Agent handles, flagged for async review
        "hard_review": 0.40,     # Must be reviewed before sending
        "escalate": 0.0,         # Immediate human handoff
    }

    def evaluate(self, response, confidence: float) -> str:
        if confidence >= self.THRESHOLDS["auto_approve"]:
            return "send"
        elif confidence >= self.THRESHOLDS["soft_review"]:
            self._queue_for_review(response)
            return "send"
        elif confidence >= self.THRESHOLDS["hard_review"]:
            return "hold_for_review"
        else:
            return "escalate_to_human"
```

### Audit Logging Requirements

| Event | Required Fields | Retention |
|---|---|---|
| User input received | session_id, timestamp, input_hash, user_id | 90 days |
| Guardrail triggered | layer, check_name, result, severity | 1 year |
| Tool call executed | tool_name, params_hash, result_status | 90 days |
| Output sent to user | session_id, timestamp, output_hash | 90 days |
| HITL escalation | trigger_reason, response_time, resolution | 1 year |
| Safety incident | full context, all layers' results | 2 years |

---

## Testing Guardrail Effectiveness

### Test Categories

| Category | Purpose | Example Inputs |
|---|---|---|
| Golden path | Verify normal requests pass all layers | Typical user queries |
| Injection probes | Test Layer 1+2 detection | Known injection patterns |
| Adversarial content | Test content classifiers | Borderline harmful content |
| PII probing | Test output filtering | Requests designed to extract PII |
| Resource exhaustion | Test execution limits | Tasks that trigger many tool calls |
| Edge cases | Test boundary conditions | Empty input, max length, unicode |

### Guardrail Effectiveness Metrics

| Metric | Target | Measurement |
|---|---|---|
| True positive rate (harmful blocked) | >99% | Red team testing |
| False positive rate (safe blocked) | <2% | Normal traffic sampling |
| Latency overhead per layer | <50ms | p95 latency measurement |
| Total guardrail latency | <200ms | End-to-end measurement |
| Escalation accuracy | >90% | HITL review of escalated items |

---

## Anti-Patterns

| Anti-Pattern | Why It Fails | Better Approach |
|---|---|---|
| Single-layer defense | One bypass = full compromise | 5-layer defense-in-depth |
| Regex-only injection detection | Easily bypassed with encoding tricks | ML classifier + regex as backup |
| Blocking without logging | Cannot improve, cannot audit | Always log blocks with context |
| Overly strict guardrails | Users frustrated, workarounds emerge | Tune thresholds based on FP rate |
| Guardrails only on input | Hallucinations and leaks pass through | Filter both input AND output |
| Hard-coded thresholds | Cannot adapt to new attack patterns | Configurable + regularly updated |
| No testing of guardrails | Assume they work until breach | Regular red team testing |
| Guardrails as afterthought | Bolted on, incomplete coverage | Design guardrails into architecture |

---

## Cross-References

- `agent-debugging-patterns.md` — debugging guardrail-triggered failures
- `voice-multimodal-agents.md` — modality-specific guardrails
- `../ai-prompt-engineering/references/prompt-security-defense.md` — prompt-level security
- `../ai-llm/references/structured-output-patterns.md` — output validation patterns

# Inbox Engine Patterns — Event-Driven Agent Intake

**Purpose**: Operational patterns for building an Inbox Engine — the event-driven intake layer that monitors sources, classifies incoming signals, deduplicates, prioritizes, and routes to appropriate agents or workflows.

---
## Table of Contents

- [1. Architecture Overview](#1-architecture-overview)
- [2. Event Envelope Schema](#2-event-envelope-schema)
- [Pattern: Universal Event Envelope](#pattern-universal-event-envelope)
- [Checklist: Event Ingestion](#checklist-event-ingestion)
- [3. Signal Classification](#3-signal-classification)
- [Pattern: Three-Class Triage](#pattern-three-class-triage)
- [Classification Methods](#classification-methods)
- [Hybrid Classification Pipeline](#hybrid-classification-pipeline)
- [Rule Examples](#rule-examples)
- [4. Deduplication and Batching](#4-deduplication-and-batching)
- [Pattern: Content-Hash Deduplication](#pattern-content-hash-deduplication)
- [Configuration](#configuration)
- [Batching Strategies](#batching-strategies)
- [5. Priority Queue and SLA Routing](#5-priority-queue-and-sla-routing)
- [Pattern: SLA-Based Priority Assignment](#pattern-sla-based-priority-assignment)
- [Queue Implementation Options](#queue-implementation-options)
- [6. Routing to Action Graph](#6-routing-to-action-graph)
- [Pattern: Event → Workflow Mapping](#pattern-event-→-workflow-mapping)
- [Handoff Contract (Inbox → Action Graph)](#handoff-contract-inbox-→-action-graph)
- [Choreography Pattern (Decentralized Alternative)](#choreography-pattern-decentralized-alternative)
- [7. Dead Letter and Retry](#7-dead-letter-and-retry)
- [Pattern: Failed Event Handling](#pattern-failed-event-handling)
- [Configuration](#configuration)
- [8. Observability](#8-observability)
- [Metrics to Track](#metrics-to-track)
- [Trace Span](#trace-span)
- [Implementation Checklist](#implementation-checklist)
- [9. Commercial Reference Implementations (March 2026)](#9-commercial-reference-implementations-march-2026)
- [Confluent — Real-Time Context Engine](#confluent-—-real-time-context-engine)
- [Knative Eventing — Broker-Based Choreography](#knative-eventing-—-broker-based-choreography)
- [Market Gap](#market-gap)
- [Related Resources](#related-resources)


## 1. Architecture Overview

```text
┌──────────────────────────────────────────────────┐
│                  SOURCE MONITORS                  │
│  [webhook] [polling] [stream] [cron] [A2A]       │
└──────────┬───────────────────────────────────────┘
           │ raw events
┌──────────▼───────────────────────────────────────┐
│                    INGESTION                      │
│  normalize → validate → assign envelope ID       │
└──────────┬───────────────────────────────────────┘
           │ normalized events
┌──────────▼───────────────────────────────────────┐
│                  CLASSIFICATION                   │
│  actionable | informational | noise              │
└──────────┬───────────────────────────────────────┘
           │ classified events
┌──────────▼───────────────────────────────────────┐
│               DEDUPLICATION + BATCHING            │
│  content hash → sliding window → merge similar   │
└──────────┬───────────────────────────────────────┘
           │ deduplicated events
┌──────────▼───────────────────────────────────────┐
│                  PRIORITIZATION                   │
│  SLA assignment → urgency scoring → queue rank   │
└──────────┬───────────────────────────────────────┘
           │ prioritized events
┌──────────▼───────────────────────────────────────┐
│                     ROUTING                       │
│  actionable → Action Graph                       │
│  informational → Data Agent → Knowledge Base     │
│  noise → log + discard                           │
└──────────────────────────────────────────────────┘
```

---

## 2. Event Envelope Schema

### Pattern: Universal Event Envelope

```yaml
event_envelope:
  id: "string (uuid)"
  source:
    type: "webhook | poll | stream | cron | a2a | user"
    identifier: "string (source system ID)"
    received_at: "ISO 8601"
  payload:
    content_type: "text | json | binary"
    body: "..."
    content_hash: "sha256 (for deduplication)"
  classification:
    signal_class: "actionable | informational | noise"
    confidence: "float (0.0 - 1.0)"
    classified_by: "rule | llm | hybrid"
  priority:
    urgency: "critical | high | medium | low"
    sla_ms: 2000
    queue: "string (routing queue name)"
  routing:
    target: "action_graph | data_agent | discard"
    workflow_id: "string (if routed to action graph)"
    agent_id: "string (if routed to specific agent)"
  lifecycle:
    status: "received | classified | routed | processing | completed | failed"
    attempts: 0
    max_attempts: 3
    dead_letter_after: 3
```

### Checklist: Event Ingestion

- [ ] Assign unique envelope ID at ingestion (idempotency key).
- [ ] Normalize payload to standard format (strip transport metadata).
- [ ] Compute content hash for deduplication.
- [ ] Validate payload against expected schema (reject malformed early).
- [ ] Record received_at timestamp for SLA tracking.
- [ ] Acknowledge receipt to source (prevent redelivery).

---

## 3. Signal Classification

### Pattern: Three-Class Triage

| Class | Definition | Action | Examples |
|-------|-----------|--------|----------|
| **Actionable** | Requires agent action within SLA | Route to Action Graph | User request, alert breach, approval needed, error requiring fix |
| **Informational** | Updates knowledge, no immediate action | Route to Data Agent → Knowledge Base | Data refresh, status update, new doc published, metric report |
| **Noise** | Irrelevant or duplicate, safe to discard | Log and discard | Heartbeat, duplicate webhook, stale notification, health check |

### Classification Methods

| Method | Speed | Accuracy | Cost | Best For |
|--------|-------|----------|------|----------|
| **Rule-based** | <1ms | High (known patterns) | Zero | Structured events with clear signatures |
| **LLM-based** | 200-500ms | High (ambiguous content) | Token cost | Unstructured text, novel event types |
| **Hybrid** | 5-50ms | Highest | Low | Rules first, LLM fallback for uncertain |

### Hybrid Classification Pipeline

```text
1. RULE ENGINE: Match event against known patterns
   ├── Match found (confidence > 0.9)? → Use rule classification
   └── No match or low confidence? → Continue to step 2

2. LIGHTWEIGHT CLASSIFIER: Fast ML model or heuristic
   ├── Classification confident (> 0.8)? → Use classifier result
   └── Uncertain? → Continue to step 3

3. LLM CLASSIFICATION: Use small, fast model
   └── Return classification with confidence score
```

### Rule Examples

```yaml
classification_rules:
  - name: "user_message"
    match:
      source_type: "user"
    classify_as: "actionable"
    priority: "high"

  - name: "webhook_data_update"
    match:
      source_type: "webhook"
      payload_contains: ["data_updated", "record_changed"]
    classify_as: "informational"
    priority: "medium"

  - name: "heartbeat"
    match:
      source_type: "poll"
      payload_contains: ["heartbeat", "ping", "health_check"]
    classify_as: "noise"
```

---

## 4. Deduplication and Batching

### Pattern: Content-Hash Deduplication

```text
For each incoming event E:
1. Compute content_hash = SHA256(E.payload.body)
2. Check sliding window (last N minutes) for matching hash
3. If match found:
   a. Increment duplicate_count on original
   b. Discard duplicate (log for audit)
4. If no match:
   a. Add to sliding window
   b. Continue to prioritization
```

### Configuration

```yaml
deduplication:
  window_size_minutes: 15
  hash_algorithm: "sha256"
  hash_fields: ["payload.body"]  # which fields to hash
  strategy: "keep_first"  # keep_first | keep_latest | merge

batching:
  enabled: true
  window_ms: 1000           # batch window
  max_batch_size: 50        # max events per batch
  batch_key: "source.identifier"  # group by source
  merge_strategy: "latest_wins"   # for same-entity updates
```

### Batching Strategies

| Strategy | When to Use | Example |
|----------|-------------|---------|
| **Time-window** | Events arrive in bursts | Batch all events in 1s window |
| **Count-based** | Steady stream, process in chunks | Process every 50 events |
| **Entity-based** | Multiple updates to same entity | Merge 5 updates to user-123 into 1 |

---

## 5. Priority Queue and SLA Routing

### Pattern: SLA-Based Priority Assignment

```yaml
priority_matrix:
  critical:
    sla_ms: 1000
    queue: "immediate"
    triggers:
      - "security_alert"
      - "system_failure"
      - "user_escalation"
  high:
    sla_ms: 5000
    queue: "fast"
    triggers:
      - "user_message"
      - "approval_request"
      - "payment_event"
  medium:
    sla_ms: 30000
    queue: "standard"
    triggers:
      - "data_update"
      - "scheduled_task"
      - "report_ready"
  low:
    sla_ms: 300000
    queue: "background"
    triggers:
      - "analytics_event"
      - "log_aggregation"
      - "cleanup_task"
```

### Queue Implementation Options

| Implementation | Latency | Durability | Best For |
|---------------|---------|------------|----------|
| **In-memory (deque)** | <1ms | None | Single-process, dev/test |
| **Redis Sorted Set** | <5ms | Configurable | Multi-process, moderate scale |
| **SQS / Cloud Pub/Sub** | 10-100ms | High | Distributed, production |
| **Kafka** | 5-50ms | High | High-throughput, event sourcing |

---

## 6. Routing to Action Graph

### Pattern: Event → Workflow Mapping

```yaml
routing_table:
  - event_pattern:
      source_type: "user"
      signal_class: "actionable"
    route_to:
      target: "action_graph"
      workflow: "conversational_agent"
      entry_node: "step-classify-intent"

  - event_pattern:
      source_type: "webhook"
      payload_contains: ["order_created"]
    route_to:
      target: "action_graph"
      workflow: "order_processing"
      entry_node: "step-validate-order"

  - event_pattern:
      signal_class: "informational"
    route_to:
      target: "data_agent"
      pipeline: "ingest_and_index"

  - event_pattern:
      signal_class: "noise"
    route_to:
      target: "discard"
      log_level: "debug"
```

### Handoff Contract (Inbox → Action Graph)

```yaml
handoff_payload:
  event_id: "string (from envelope)"
  workflow_id: "string (from routing table)"
  entry_node: "string (starting node in action graph)"
  context:
    event_payload: "..."
    classification: "actionable"
    priority: "high"
    sla_remaining_ms: 4500
  metadata:
    source: "inbox_engine"
    routed_at: "ISO 8601"
    attempt: 1
```

### Choreography Pattern (Decentralized Alternative)

Instead of a centralized routing table, agents can react to each other's events directly — each agent emits events after completing work, and downstream agents subscribe to the events they care about.

```text
Centralized (routing table):
  Inbox → Router → Agent A
                 → Agent B

Choreography (event-driven):
  Inbox → emits "message.received"
  Agent A listens for "message.received" → processes → emits "structured.extracted"
  Agent B listens for "structured.extracted" → processes → emits "intent.classified"
  Agent C listens for "intent.classified" → routes to final destination
```

**When to use choreography**:

| Criteria | Centralized Routing | Choreography |
|----------|-------------------|--------------|
| Agent count | <5 agents | 5+ agents |
| Coupling tolerance | Tight (known routes) | Loose (agents are independent) |
| Failure isolation | Single point of failure at router | Independent — one agent failing doesn't block others |
| Debugging | Easy (single routing table) | Harder (distributed trace required) |
| Scaling | Vertical (router bottleneck) | Horizontal (add agents independently) |

**Recommendation**: Start with centralized routing (simpler, easier to debug). Migrate to choreography when agent count or throughput demands it.

**Reference**: [Knative eventing](https://knative.dev/blog/articles/knative-eventing-eda-agents/) uses broker-based choreography where agents are fully decoupled — no agent knows who produces its input or consumes its output.

---

## 7. Dead Letter and Retry

### Pattern: Failed Event Handling

```text
Event processing fails:
1. Increment attempt count
2. If attempts < max_attempts:
   a. Apply exponential backoff (base: 1s, max: 60s)
   b. Re-queue with incremented attempt
3. If attempts >= max_attempts:
   a. Move to Dead Letter Queue (DLQ)
   b. Alert on DLQ depth > threshold
   c. Log full event envelope for debugging
```

### Configuration

```yaml
retry_policy:
  max_attempts: 3
  backoff:
    type: "exponential"
    base_ms: 1000
    max_ms: 60000
    jitter: true
  dead_letter:
    queue: "inbox_dlq"
    retention_days: 14
    alert_threshold: 10  # alert if DLQ > 10 events
```

---

## 8. Observability

### Metrics to Track

| Metric | Type | Alert Threshold |
|--------|------|----------------|
| `inbox.events.received` | Counter | — |
| `inbox.events.classified` | Counter (by class) | — |
| `inbox.events.duplicates_discarded` | Counter | >50% of total |
| `inbox.events.routed` | Counter (by target) | — |
| `inbox.events.sla_breached` | Counter | Any >0 for critical |
| `inbox.queue.depth` | Gauge (by queue) | >1000 |
| `inbox.classification.latency_ms` | Histogram | p99 >500ms |
| `inbox.dlq.depth` | Gauge | >10 |

### Trace Span

```yaml
span:
  name: "inbox.process_event"
  attributes:
    event.id: "..."
    event.source_type: "webhook"
    event.signal_class: "actionable"
    event.priority: "high"
    event.queue: "fast"
    event.routing_target: "action_graph"
```

---

## Implementation Checklist

- [ ] Define event envelope schema for all source types.
- [ ] Implement source monitors (start with 1-2, expand).
- [ ] Build classification pipeline (rules first, add LLM fallback later).
- [ ] Add content-hash deduplication with sliding window.
- [ ] Set up priority queues (start with in-memory, graduate to Redis/SQS).
- [ ] Create routing table mapping events → Action Graph workflows.
- [ ] Implement retry + dead letter queue.
- [ ] Add OpenTelemetry metrics and trace spans.
- [ ] Set SLA alerts for critical and high-priority queues.
- [ ] Load test with 10× expected throughput.

---

## 9. Commercial Reference Implementations (March 2026)

The Inbox Engine is the **least commercially addressed** layer in the AI Engine stack. Most agent products start at "agent receives query" — they don't formalize how signals enter the system. Two notable exceptions:

### Confluent — Real-Time Context Engine

Confluent repositioned Kafka + Flink as a real-time context engine for AI agents:

| Component | Maps to Our Pattern |
|-----------|-------------------|
| Kafka streams (source intake) | Source Monitors (Section 1) |
| Flink processing (normalize, classify) | Classification + Deduplication (Sections 3-4) |
| In-memory cache (serving layer) | Priority Queue (Section 5) |
| MCP server (agent delivery) | Routing / Handoff Contract (Section 6) |
| Streaming Agents (Flink jobs) | Choreography pattern — agents run as pipeline stages |

**Key innovation**: First major platform to use MCP as native context delivery protocol. Q1 2026: Added A2A integration for cross-platform agent collaboration.

**Validates our patterns**: Event-driven intake with MCP-based delivery — exactly what our Inbox Engine + Data Agent layers document.

### Knative Eventing — Broker-Based Choreography

Knative's eventing model uses broker-based choreography where agents are fully decoupled — no agent knows who produces its input or consumes its output. This validates our Choreography Pattern (Section 6) as the scaling approach for 5+ agents.

### Market Gap

The Inbox Engine remains a genuine architectural gap in the market. While Confluent addresses streaming intake and Knative addresses event routing, no dedicated product offers the full pipeline our architecture defines: ingest → classify → deduplicate → prioritize → route with SLA enforcement. This is an opportunity for our composable approach.

---

## Related Resources

| Resource | Covers |
|----------|--------|
| [`ai-engine-layers.md`](ai-engine-layers.md) | Full 5-layer architecture overview |
| [`operational-patterns.md`](operational-patterns.md) | Action loop patterns (what inbox routes to) |
| [`multi-agent-patterns.md`](multi-agent-patterns.md) | Multi-agent orchestration and handoffs |
| [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md) | Agent-to-agent communication protocol |
| [`../../software-architecture-design/references/operational-playbook.md`](../../software-architecture-design/references/operational-playbook.md) | Event-driven and queue-backed architecture playbook |

# AI Agents Reference Index

Use this file as the on-demand map for the `ai-agents` skill. Start with the shortest path that matches the user’s question instead of loading every reference.

## Table Of Contents

- [Fast Paths](#fast-paths)
- [Decision And Economics](#decision-and-economics)
- [Core Architecture](#core-architecture)
- [Graph And Loop Composition](#graph-and-loop-composition)
- [Protocols And Contracts](#protocols-and-contracts)
- [Capability Patterns](#capability-patterns)
- [Engine Layers](#engine-layers)
- [Operations, Safety, And Quality](#operations-safety-and-quality)
- [Templates And Assets](#templates-and-assets)
- [External Verification](#external-verification)

## Fast Paths

- New agent idea:
  [`build-vs-not-decision.md`](build-vs-not-decision.md) ->
  [`protocol-decision-tree.md`](protocol-decision-tree.md) ->
  [`modern-best-practices.md`](modern-best-practices.md) ->
  [`../assets/core/agent-template-standard.md`](../assets/core/agent-template-standard.md)

- Tool-heavy agent:
  [`protocol-decision-tree.md`](protocol-decision-tree.md) ->
  [`mcp-practical-guide.md`](mcp-practical-guide.md) ->
  [`tool-design-specs.md`](tool-design-specs.md) ->
  [`agents-mcp`](../../agents-mcp/SKILL.md)

- Multi-agent system:
  [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md) ->
  [`multi-agent-patterns.md`](multi-agent-patterns.md) ->
  [`context-rotation-and-state.md`](context-rotation-and-state.md) ->
  [`agent-operations-best-practices.md`](agent-operations-best-practices.md) ->
  `agents-subagents`

- Knowledge-heavy agent:
  [`rag-patterns.md`](rag-patterns.md) ->
  [`../assets/knowledge-base/kb-architecture.md`](../assets/knowledge-base/kb-architecture.md) ->
  [`memory-systems.md`](memory-systems.md) ->
  [`ai-rag`](../../ai-rag/SKILL.md)

- Rollout and debugging:
  [`evaluation-and-observability.md`](evaluation-and-observability.md) ->
  [`deployment-ci-cd-and-safety.md`](deployment-ci-cd-and-safety.md) ->
  [`agent-debugging-patterns.md`](agent-debugging-patterns.md)

- Cost tracking:
  [`agent-economics.md`](agent-economics.md) ->
  [`coding-agent-usage-tracking.md`](coding-agent-usage-tracking.md)

- Framework selection (July 2026):
  [`build-vs-not-decision.md`](build-vs-not-decision.md) ->
  [`framework-landscape.md`](framework-landscape.md) ->
  [`../../ai-bot-builder/references/framework-selection.md`](../../ai-bot-builder/references/framework-selection.md) (for Python/TS bot depth)

- "Graph engineering", "loop engineering", agent workflow graph, improvement loop, or knowledge graph:
  [`graph-and-loop-engineering.md`](graph-and-loop-engineering.md) ->
  choose the graph purpose ->
  the linked specialist skill

## Decision And Economics

- [`build-vs-not-decision.md`](build-vs-not-decision.md) - default "do not build an agent" gate, alternatives, kill triggers
- [`agent-economics.md`](agent-economics.md) - cost, ROI, hallucination impact, payback logic
- [`coding-agent-usage-tracking.md`](coding-agent-usage-tracking.md) - CLI usage tracking for Claude Code and Codex with ccusage tools
- [`agent-maturity-governance.md`](agent-maturity-governance.md) - maturity model, policy, fleet governance

## Core Architecture

- [`modern-best-practices.md`](modern-best-practices.md) - current operating defaults and framework selection guidance (re-verify dates in-file)
- [`framework-landscape.md`](framework-landscape.md) - polyglot framework selection (dated in-file): LangGraph, CrewAI, Pydantic AI, Claude/OpenAI SDKs, Mastra, Spring AI, Microsoft Agent Framework, Semantic Kernel; selection matrix, anti-patterns, migration paths
- [`agent-delivery-methods.md`](agent-delivery-methods.md) - GSD, BMAD, Spec Kit, OpenSpec, MADD, and AI-SDLC as delivery methods
- [`operational-patterns.md`](operational-patterns.md) - agent loop, tool spec, memory, eval, deployment patterns
- [`agent-operations-best-practices.md`](agent-operations-best-practices.md) - execution, verification, action gating
- [`context-engineering.md`](context-engineering.md) - progressive disclosure, retrieval timing, context hygiene
- [`context-rotation-and-state.md`](context-rotation-and-state.md) - fresh-context workers, durable state, and session vs project boundaries
- [`memory-systems.md`](memory-systems.md) - session, episodic, long-term, task memory tradeoffs

## Graph And Loop Composition

- [`graph-and-loop-engineering.md`](graph-and-loop-engineering.md) - disambiguates agent/workflow graphs, networked improvement graphs, and knowledge/context graphs; routes graph and loop requests to the relevant specialist skill

## Protocols And Contracts

- [`protocol-decision-tree.md`](protocol-decision-tree.md) - MCP vs A2A selection
- [`mcp-practical-guide.md`](mcp-practical-guide.md) - MCP integration patterns
- [`mcp-server-builder.md`](mcp-server-builder.md) - MCP server build checklist
- [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md) - handoff contracts and coordination patterns
- [`api-contracts-for-agents.md`](api-contracts-for-agents.md) - request/response envelopes, safety and error taxonomy
- [`tool-design-specs.md`](tool-design-specs.md) - tool schemas, validation, side-effect controls

## Capability Patterns

- [`multi-agent-patterns.md`](multi-agent-patterns.md) - manager-worker, sequential, handoff, group chat
- [`rag-patterns.md`](rag-patterns.md) - agentic RAG and hybrid retrieval patterns
- [`os-agent-capabilities.md`](os-agent-capabilities.md) - UI grounding and OS control patterns
- [`code-swe-agents.md`](code-swe-agents.md) - coding-agent operating patterns
- [`voice-multimodal-agents.md`](voice-multimodal-agents.md) - multimodal and voice-first systems
- [`skill-lifecycle.md`](skill-lifecycle.md) - package and share reusable agent skills

## Engine Layers

- [`ai-engine-layers.md`](ai-engine-layers.md) - five-layer system view
- [`context-graph-patterns.md`](context-graph-patterns.md) - graph-based context and memory
- [`inbox-engine-patterns.md`](inbox-engine-patterns.md) - event intake, prioritization, dead-letter handling

## Operations, Safety, And Quality

- [`evaluation-and-observability.md`](evaluation-and-observability.md) - eval design, telemetry, monitoring
- [`deployment-ci-cd-and-safety.md`](deployment-ci-cd-and-safety.md) - rollout, HITL, rollback, control gates
- [`guardrails-implementation.md`](guardrails-implementation.md) - layered guardrails and enforcement patterns
- [`escalation-patterns.md`](escalation-patterns.md) - 3-level escalation hierarchy, failure classification, escalation budgets, and hook integration
- [`agent-debugging-patterns.md`](agent-debugging-patterns.md) - trace-based debugging for loops, tools, and state corruption
- [`autonomous-loop-patterns.md`](autonomous-loop-patterns.md) - Shape C autonomous loops: Ralph-Loop class, PRD spec, three driver implementations (Python, Temporal, LangGraph), budgets, drift, circuit breakers
- [`24-7-operating-model.md`](24-7-operating-model.md) - production operating model for agents: SLOs per shape, on-call structure, alert catalog, runbook contract, post-mortem template, operating cadence

## Templates And Assets

- [`../assets/core/agent-template-standard.md`](../assets/core/agent-template-standard.md) - full production spec
- [`../assets/core/agent-template-quick.md`](../assets/core/agent-template-quick.md) - MVP spec
- [`../assets/core/agent-template-specialized.md`](../assets/core/agent-template-specialized.md) - domain-specific spec
- [`../assets/agent-template-ainative-sdlc.md`](../assets/agent-template-ainative-sdlc.md) - delegate-review-own runbook
- [`../assets/checklists/agent-safety-checklist.md`](../assets/checklists/agent-safety-checklist.md) - launch gate
- [`../assets/tools/tool-definition.md`](../assets/tools/tool-definition.md) - tool contract template
- [`../assets/tools/tool-validation-checklist.md`](../assets/tools/tool-validation-checklist.md) - tool readiness checklist
- [`../assets/multi-agent/manager-worker-template.md`](../assets/multi-agent/manager-worker-template.md) - manager-worker starter
- [`../assets/multi-agent/evaluator-router-template.md`](../assets/multi-agent/evaluator-router-template.md) - evaluator-router starter
- [`../assets/rag/rag-basic.md`](../assets/rag/rag-basic.md) - basic RAG template
- [`../assets/rag/rag-advanced.md`](../assets/rag/rag-advanced.md) - advanced RAG template
- [`../assets/rag/hybrid-retrieval.md`](../assets/rag/hybrid-retrieval.md) - hybrid retrieval template
- [`../assets/knowledge-base/kb-architecture.md`](../assets/knowledge-base/kb-architecture.md) - knowledge-base architecture

## External Verification

- [`../data/sources.json`](../data/sources.json) - curated primary sources for fact-checking (see per-entry verification dates)

# MCP Practical Implementation Guide

*Purpose: Hands-on patterns for building MCP servers, integrating MCP tools, and deploying Model Context Protocol in production.*

**When to use this guide**: User asks to build/integrate MCP servers, connect agents to data sources, or implement standardized tool access.

**Protocol version**: Current stable is 2025-11-25 (spec at `modelcontextprotocol.io/specification/2025-11-25/`). Pin implementation guidance to this version until a newer stable is ratified. **Heads-up (June 2026):** a `2026-07-28` revision — the largest since launch — is in **release candidate** (stateless HTTP core, server-rendered UIs via *MCP Apps*, a *Tasks* extension for long-running work, OAuth/OIDC-aligned authorization, and a formal deprecation policy). It is not yet ratified; build against 2025-11-25 today, but design new servers so the stateless-core and auth changes won't force a rewrite. Track the [MCP blog](https://blog.modelcontextprotocol.io/) for the final cutover.

**For architecture deep-dive**: See `frameworks/shared-foundations/protocols/mcp/` for comprehensive protocol specification.

---
## Table of Contents

- [Quick Decision: Do I Need MCP?](#quick-decision-do-i-need-mcp)
- [MCP Architecture (Quick Reference)](#mcp-architecture-quick-reference)
- [Pattern 1: Filesystem MCP Server (Python)](#pattern-1-filesystem-mcp-server-python)
- [Setup (5 minutes)](#setup-5-minutes)
- [Install MCP SDK](#install-mcp-sdk)
- [Create server file](#create-server-file)
- [Implementation](#implementation)
- [Initialize MCP server](#initialize-mcp-server)
- [Configuration (Claude Desktop)](#configuration-claude-desktop)
- [Pattern 2: Database MCP Server (TypeScript)](#pattern-2-database-mcp-server-typescript)
- [Setup](#setup)
- [Implementation](#implementation)
- [Pattern 3: API Wrapper MCP Server](#pattern-3-api-wrapper-mcp-server)
- [GitHub API Example (Python)](#github-api-example-python)
- [Pattern 4: Resources (Data Access)](#pattern-4-resources-data-access)
- [Agent can read these via:](#agent-can-read-these-via)
- [- "Show me auth docs" → retrieves docs://api/authentication](#show-me-auth-docs-→-retrieves-docsapiauthentication)
- [- "User endpoint for ID 123" → retrieves docs://api/users/123](#user-endpoint-for-id-123-→-retrieves-docsapiusers123)
- [Pattern 5: Prompts (Reusable Templates)](#pattern-5-prompts-reusable-templates)
- [Testing Your MCP Server](#testing-your-mcp-server)
- [Local Testing (Python)](#local-testing-python)
- [Test server directly](#test-server-directly)
- [Use MCP Inspector (official tool)](#use-mcp-inspector-official-tool)
- [Testing with Claude Desktop](#testing-with-claude-desktop)
- [Debugging Checklist](#debugging-checklist)
- [Production Deployment](#production-deployment)
- [Security Checklist](#security-checklist)
- [Monitoring](#monitoring)
- [Performance Optimization](#performance-optimization)
- [Common Patterns & Anti-Patterns](#common-patterns-&-anti-patterns)
- [[check] Good Patterns](#check-good-patterns)
- [Good: Specific tools](#good-specific-tools)
- [Bad: Generic "do anything" tool](#bad-generic-do-anything-tool)
- [[x] Anti-Patterns](#x-anti-patterns)
- [NEVER DO THIS](#never-do-this)
- [Bad](#bad)
- [Good](#good)
- [MCP vs Direct API Calls](#mcp-vs-direct-api-calls)
- [Next Steps](#next-steps)


## Quick Decision: Do I Need MCP?

**Use MCP when**:
- Connecting agent to external data (databases, APIs, filesystems)
- Building reusable tools shared across multiple agents
- Standardizing tool access across team/organization
- Need secure, auditable tool execution

**Don't use MCP when**:
- Simple one-off script (just call API directly)
- Agent-to-agent communication (use A2A protocol instead)
- Pure LLM prompting without external tools

---

## MCP Architecture (Quick Reference)

```
┌─────────────────┐
│   MCP Host      │  (Claude Desktop, Claude Code, Custom App)
│   (AI App)      │
└────────┬────────┘
         │
┌────────┴────────┐
│   MCP Client    │  (Built into host or SDK)
└────────┬────────┘
         │
┌────────┴────────┐
│   MCP Server    │  (Your code: exposes tools/references/prompts)
└────────┬────────┘
         │
┌────────┴────────┐
│   Data Source   │  (Database, API, Filesystem, etc.)
└─────────────────┘
```

**Key concept**: MCP Server = adapter layer between AI app and your data/tools.

---

## Pattern 1: Filesystem MCP Server (Python)

**Use case**: Let agent read/write local files with permission controls.

### Setup (5 minutes)

```bash
# Install MCP SDK
uv pip install mcp

# Create server file
touch filesystem_server.py
```

### Implementation

```python
#!/usr/bin/env python3
from mcp.server.fastmcp import FastMCP

# Initialize MCP server
mcp = FastMCP("Filesystem Access")

@mcp.tool()
def read_file(path: str) -> str:
    """Read contents of a file from allowed directory"""
    # Security: validate path is within allowed directories
    allowed_dirs = ["/workspace", "/data"]
    abs_path = os.path.abspath(path)

    if not any(abs_path.startswith(d) for d in allowed_dirs):
        raise ValueError(f"Access denied: {path} not in allowed directories")

    with open(abs_path, 'r') as f:
        return f.read()

@mcp.tool()
def write_file(path: str, content: str) -> str:
    """Write content to a file in allowed directory"""
    allowed_dirs = ["/workspace/output"]
    abs_path = os.path.abspath(path)

    if not any(abs_path.startswith(d) for d in allowed_dirs):
        raise ValueError(f"Access denied: {path} not in allowed directories")

    with open(abs_path, 'w') as f:
        f.write(content)
    return f"Wrote {len(content)} characters to {path}"

@mcp.tool()
def list_files(directory: str) -> list[str]:
    """List files in directory"""
    allowed_dirs = ["/workspace", "/data"]
    abs_dir = os.path.abspath(directory)

    if not any(abs_dir.startswith(d) for d in allowed_dirs):
        raise ValueError(f"Access denied: {directory}")

    return os.listdir(abs_dir)

if __name__ == "__main__":
    mcp.run()
```

### Configuration (Claude Desktop)

Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "filesystem": {
      "command": "uv",
      "args": [
        "--directory",
        "/path/to/server",
        "run",
        "filesystem_server.py"
      ]
    }
  }
}
```

**Security notes**:
- Always validate file paths against allowlist
- Never allow `..` or absolute paths from user input
- Log all file operations for audit trail
- Consider read-only vs read-write permissions

---

## Pattern 2: Database MCP Server (TypeScript)

**Use case**: Let agent query database with safe, parameterized queries.

### Setup

```bash
npm install @modelcontextprotocol/sdk
npm install pg  # or mysql2, sqlite3, etc.
```

### Implementation

```typescript
import { Server } from "@modelcontextprotocol/sdk/server/index.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { Pool } from 'pg';

// Database connection
const pool = new Pool({
  host: process.env.DB_HOST,
  database: process.env.DB_NAME,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
});

// Initialize MCP server
const server = new Server(
  {
    name: "postgres-query",
    version: "1.0.0",
  },
  {
    capabilities: {
      tools: {},
    },
  }
);

// Define tool: safe SELECT query
server.setRequestHandler("tools/list", async () => ({
  tools: [
    {
      name: "query_users",
      description: "Query users table with filters",
      inputSchema: {
        type: "object",
        properties: {
          email: { type: "string" },
          status: { type: "string", enum: ["active", "inactive"] },
          limit: { type: "number", default: 100, maximum: 1000 }
        }
      }
    }
  ]
}));

server.setRequestHandler("tools/call", async (request) => {
  if (request.params.name === "query_users") {
    const { email, status, limit = 100 } = request.params.arguments;

    // Build safe parameterized query
    let query = "SELECT id, email, status, created_at FROM users WHERE 1=1";
    const params = [];

    if (email) {
      params.push(email);
      query += ` AND email = $${params.length}`;
    }
    if (status) {
      params.push(status);
      query += ` AND status = $${params.length}`;
    }

    params.push(limit);
    query += ` LIMIT $${params.length}`;

    const result = await pool.query(query, params);

    return {
      content: [
        {
          type: "text",
          text: JSON.stringify(result.rows, null, 2)
        }
      ]
    };
  }

  throw new Error(`Unknown tool: ${request.params.name}`);
});

// Start server
const transport = new StdioServerTransport();
await server.connect(transport);
```

**Security requirements**:
- NEVER concatenate user input into SQL
- Always use parameterized queries
- Whitelist allowed tables/columns
- Enforce row limits (prevent full table scans)
- Use read-only database user when possible
- Log all queries with user context

---

## Pattern 3: API Wrapper MCP Server

**Use case**: Wrap third-party API (GitHub, Slack, Jira) as MCP tools.

### GitHub API Example (Python)

```python
from mcp.server.fastmcp import FastMCP
import httpx
import os

mcp = FastMCP("GitHub Integration")

GITHUB_TOKEN = os.getenv("GITHUB_TOKEN")
GITHUB_API = "https://api.github.com"

@mcp.tool()
def list_repos(username: str, limit: int = 10) -> str:
    """List public repositories for a GitHub user"""
    headers = {"Authorization": f"token {GITHUB_TOKEN}"}
    response = httpx.get(
        f"{GITHUB_API}/users/{username}/repos",
        headers=headers,
        params={"per_page": min(limit, 100)}
    )
    response.raise_for_status()

    repos = response.json()
    return "\n".join([f"- {r['name']}: {r['description']}" for r in repos])

@mcp.tool()
def create_issue(
    repo: str,
    title: str,
    body: str,
    labels: list[str] = None
) -> str:
    """Create GitHub issue in specified repo (format: owner/repo)"""
    headers = {
        "Authorization": f"token {GITHUB_TOKEN}",
        "Accept": "application/vnd.github.v3+json"
    }

    payload = {
        "title": title,
        "body": body,
        "labels": labels or []
    }

    response = httpx.post(
        f"{GITHUB_API}/repos/{repo}/issues",
        headers=headers,
        json=payload
    )
    response.raise_for_status()

    issue = response.json()
    return f"Created issue #{issue['number']}: {issue['html_url']}"

@mcp.tool()
def search_code(query: str, language: str = None) -> str:
    """Search code across GitHub repositories"""
    headers = {"Authorization": f"token {GITHUB_TOKEN}"}
    params = {"q": query}

    if language:
        params["q"] += f" language:{language}"

    response = httpx.get(
        f"{GITHUB_API}/search/code",
        headers=headers,
        params=params
    )
    response.raise_for_status()

    results = response.json()
    items = results["items"][:5]  # Top 5 results

    return "\n".join([
        f"- {item['repository']['full_name']}/{item['path']}"
        for item in items
    ])
```

**Best practices**:
- Store API keys in environment variables
- Implement rate limiting (respect API quotas)
- Add retry logic with exponential backoff
- Return structured data (JSON) when possible
- Include error context in responses

---

## Pattern 4: Resources (Data Access)

**MCP Resources** = Read-only data that agent can retrieve.

```python
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("Documentation Server")

@mcp.resource("docs://api/authentication")
def get_auth_docs() -> str:
    """Authentication documentation"""
    return """
    # API Authentication

    Use Bearer token in Authorization header:
    Authorization: Bearer YOUR_TOKEN

    Obtain token via POST /auth/login
    """

@mcp.resource("docs://api/users/{user_id}")
def get_user_docs(user_id: str) -> str:
    """User endpoint documentation"""
    return f"""
    # GET /api/users/{user_id}

    Retrieve user profile by ID.

    Response:
    {{
      "id": "{user_id}",
      "email": "user@example.com",
      "status": "active"
    }}
    """

# Agent can read these via:
# - "Show me auth docs" → retrieves docs://api/authentication
# - "User endpoint for ID 123" → retrieves docs://api/users/123
```

**Use resources for**:
- API documentation
- Configuration files
- Static data (country codes, timezones)
- Templates
- Schema definitions

---

## Pattern 5: Prompts (Reusable Templates)

**MCP Prompts** = Pre-written prompts with parameters.

```python
@mcp.prompt()
def code_review_prompt(language: str, code: str) -> str:
    """Generate code review prompt"""
    return f"""
    Review the following {language} code for:
    - Security vulnerabilities
    - Performance issues
    - Best practices violations
    - Potential bugs

    Code:
    ```{language}
    {code}
    ```

    Provide specific, actionable feedback.
    """

@mcp.prompt()
def test_generation_prompt(function_signature: str) -> str:
    """Generate test cases for function"""
    return f"""
    Generate comprehensive unit tests for:
    {function_signature}

    Include:
    - Happy path test
    - Edge cases
    - Error handling
    - Input validation
    """
```

---

## Testing Your MCP Server

### Local Testing (Python)

```bash
# Test server directly
python server.py

# Use MCP Inspector (official tool)
npx @modelcontextprotocol/inspector python server.py
```

### Testing with Claude Desktop

1. Add server to config (see configuration examples above)
2. Restart Claude Desktop
3. Check logs: `tail -f ~/Library/Logs/Claude/mcp*.log`
4. Test tool: "Use the [tool_name] tool to [action]"

### Debugging Checklist

- [ ] Server starts without errors
- [ ] Tools appear in Claude's tool list
- [ ] Tool descriptions are clear and actionable
- [ ] Input validation works (try invalid inputs)
- [ ] Error messages are helpful
- [ ] Logs show tool execution
- [ ] Performance is acceptable (<2s per tool call)

---

## Production Deployment

### Security Checklist

- [ ] Input validation on all parameters
- [ ] Output sanitization (no secrets in responses)
- [ ] Rate limiting per user/session
- [ ] Audit logging (who called what, when)
- [ ] Principle of least privilege (minimal permissions)
- [ ] Secrets in environment variables (never hardcoded)
- [ ] TLS for network communication
- [ ] Tool signature verification (Sigstore/Cosign)

### Monitoring

```python
import logging
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@mcp.tool()
def monitored_tool(param: str) -> str:
    """Tool with observability"""
    with tracer.start_as_current_span("tool_execution") as span:
        span.set_attribute("tool.name", "monitored_tool")
        span.set_attribute("tool.param", param)

        try:
            result = do_work(param)
            span.set_attribute("tool.success", True)
            return result
        except Exception as e:
            span.set_attribute("tool.success", False)
            span.set_attribute("tool.error", str(e))
            logging.error(f"Tool failed: {e}")
            raise
```

### Performance Optimization

- **Cache responses**: Use Redis/memcached for expensive operations
- **Batch operations**: Group multiple requests when possible
- **Async execution**: Use `async/await` for I/O-bound operations
- **Connection pooling**: Reuse database/API connections
- **Timeouts**: Set reasonable timeouts (5-30s)

---

## Common Patterns & Anti-Patterns

### [check] Good Patterns

**Clear tool descriptions**:
```python
@mcp.tool()
def search_documents(query: str, limit: int = 10) -> str:
    """Search internal documents using semantic search.

    Returns top matching documents with relevance scores.
    Use for: finding policies, procedures, technical docs.
    """
```

**Structured responses**:
```python
return json.dumps({
    "status": "success",
    "results": [...],
    "total_found": 42,
    "execution_time_ms": 123
})
```

**Granular tools** (not monolithic):
```python
# Good: Specific tools
@mcp.tool()
def create_user(...): pass

@mcp.tool()
def update_user(...): pass

# Bad: Generic "do anything" tool
@mcp.tool()
def manage_users(action: str, ...): pass  # Too broad
```

### [x] Anti-Patterns

**Vague descriptions**:
```python
@mcp.tool()
def process_data(data: str) -> str:
    """Process some data"""  # What does this do?
```

**Unsafe input handling**:
```python
# NEVER DO THIS
query = f"SELECT * FROM users WHERE id = {user_id}"  # SQL injection
os.system(f"rm {filename}")  # Command injection
```

**No error context**:
```python
# Bad
return "Error"

# Good
return json.dumps({
    "error": "Database connection failed",
    "details": "Connection timeout after 30s",
    "retry_after": 60
})
```

---

## MCP vs Direct API Calls

| Consideration | Use MCP | Use Direct API |
|---------------|---------|----------------|
| Multiple agents need same tool | [check] | |
| Need audit/governance | [check] | |
| Complex permission model | [check] | |
| One-off script | | [check] |
| Maximum performance critical | | [check] |
| Sharing tools across team | [check] | |

---

## Next Steps

**After building your MCP server**:
1. Test locally with MCP Inspector
2. Add comprehensive logging
3. Write integration tests
4. Document all tools (description + examples)
5. Deploy with monitoring
6. Share with team via config

**Related guides**:
- `frameworks/shared-foundations/protocols/mcp/mcp-server-development.md` - Full protocol specification
- `frameworks/shared-foundations/protocols/mcp/mcp-claude-integration.md` - Claude-specific patterns
- `a2a-handoff-patterns.md` - For agent-to-agent coordination (complementary to MCP)
- `tool-design-specs.md` - General tool design best practices

**Official resources**:
- MCP Specification: https://spec.modelcontextprotocol.io/
- Python SDK: https://github.com/modelcontextprotocol/python-sdk
- TypeScript SDK: https://github.com/modelcontextprotocol/typescript-sdk
- MCP Inspector: https://github.com/modelcontextprotocol/inspector

# MCP Server Builder — Tooling for Agents

Use this when designing or implementing MCP servers for agent tools (Python or TypeScript SDKs).

## Planning Checklist
- Design for workflows, not raw endpoints; consolidate related actions (e.g., schedule_event that checks availability + creates event).
- Optimize for limited context: concise defaults, optional detail flags, human-readable identifiers.
- Make errors actionable: suggest next steps and correct parameters.
- Group tools with clear prefixes and natural task names.

## Research Before Coding
- Read MCP protocol spec (`modelcontextprotocol.io/llms-full.txt`).
- Load SDK docs (Python or TypeScript) and any target API docs (auth, rate limits, pagination, schemas).
- Define tool list, shared helpers (pagination, errors, formatting), and truncation strategy.

## Implementation Patterns
- Validate inputs (Pydantic v2 or Zod `.strict()`); avoid `any`.
- Async I/O; explicit return schemas; support concise vs. detailed responses.
- Annotations: `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint` where appropriate.
- Centralize API helpers, auth, error handling, and pagination.

## Review & Testing
- Create evaluation scenarios early; iterate based on agent feedback.
- Check character limits and truncation; handle rate limits and timeouts gracefully.
- Document tool usage, parameters, and error responses inside the server code.

# Memory Systems — Agent-Side Quick Reference

## Table of Contents

- [Where the catalogs live](#where-the-catalogs-live)
- [Agent-loop routing](#agent-loop-routing)
- [Required anti-pattern sweep](#required-anti-pattern-sweep)
- [What this file adds does not duplicate](#what-this-file-adds-does-not-duplicate)
- [2025-2026 research notes complement not duplicate](#2025-2026-research-notes-complement-not-duplicate)

*Purpose: tell an agent designer **where to look** for memory architecture and
**which catalog entries** are load-bearing. The full pattern and anti-pattern
catalogs live in `ai-context-layer`; this file does not maintain a parallel
taxonomy.*

If you find yourself wanting to add a new pattern here, add it to
`ai-context-layer/references/patterns-catalog.md` instead and link to it.

## Where the catalogs live

| Concern | Source of truth |
|---------|-----------------|
| Named patterns (P1–P16) | [`../../ai-context-layer/references/patterns-catalog.md`](../../ai-context-layer/references/patterns-catalog.md) |
| Anti-patterns (A1–A34) | [`../../ai-context-layer/references/anti-patterns-catalog.md`](../../ai-context-layer/references/anti-patterns-catalog.md) |
| Reference architectures (RA1–RA13) | [`../../ai-context-layer/references/reference-architectures.md`](../../ai-context-layer/references/reference-architectures.md) |
| Runtime hygiene (F1–F4 failure modes) | [`../../ai-context-layer/references/context-hygiene.md`](../../ai-context-layer/references/context-hygiene.md) |
| External benchmarks (LongMemEval, LoCoMo, MemBench, GraphRAG-Bench, MemTier) | [`../../ai-context-layer/references/agent-memory-benchmarks.md`](../../ai-context-layer/references/agent-memory-benchmarks.md) |

## Agent-loop routing

| Question | Catalog entry |
|----------|---------------|
| Where does live state live? | **P1** Operational truth in tools/APIs/SQL — never in derived memory |
| App-orchestrated facts (preferences, settings) | **P2** Structured memory |
| Self-editing notes (Zettelkasten-style) | **P3** Self-editing memory |
| Time-aware facts that supersede | **P4** Temporal knowledge graph |
| Conversational episodic + semantic split | **P6** Episodic + semantic split |
| Compiled wiki pages for recurring queries | **P7** LLM Wiki |
| Evidence-bearing retrieval | **P8** Evidence-bearing retrieval |
| Just-in-time loading of large refs | **P12** Just-in-time context loading |
| Hosted memory boundaries | **P13** Managed-memory boundaries |
| Background dedup / contradiction resolution | **P14** Sleep-time consolidation |
| Procedural memory / skill library | **P15** Procedural memory |
| Multi-agent shared memory with isolation | **P16** Multi-agent barriers |

## Required anti-pattern sweep

Run before shipping any agent that writes memory:

| ID | What to block |
|----|---------------|
| A1  | Raw chat transcripts as memory |
| A11 | No forget path (DSAR / correction) |
| A13 | Provenance not stored with the fact |
| A14 | No confidence / no abstain |
| A26 | Mode-collapse loop (model output ingested as preference) |
| A31 | Sleep-time pollution from un-gated consolidation |
| A33 | GraphRAG misapplied to single-hop questions |

The full A1–A34 list with detection signals is in the catalog above.

## What this file *adds* (does not duplicate)

The agent loop has a few concerns that belong here, not in ai-context-layer:

- **Trigger placement.** Memory `recall()` runs in the *pre-LLM* assembly
  phase; `remember()` runs *post-response* as a background side-effect, never
  on the user's critical path.
- **Latency budget.** Voice agents get ≤300ms for the full bundle (see RA13);
  text agents get up to ~1s; batch agents are unbounded.
- **Tool-result projection.** Raw tool JSON should be `format`-ed into typed
  projections before it enters the window — this is one of the six runtime
  verbs (write/select/compress/isolate/order/format) and is where the agent
  loop differs most from generic context assembly.

If a memory question has *no* agent-loop angle (storage shape, retention,
provenance, retrieval), use the ai-context-layer catalogs directly and skip
this file.

## 2025–2026 research notes (complement, not duplicate)

The patterns here predate the 2026 catalog and remain useful as **named
research lineages** that map to current P-codes:

### A-MEM (Zettelkasten-inspired, 2025)

Atomic notes with bidirectional links, dynamic indexing, emergent structure.
Maps to **P3 self-editing memory** in the current catalog. Use the original
A-MEM paper as the lineage reference; use P3 for the operational pattern.

### Mem0 (production long-term memory, 2025)

ADD / UPDATE / DELETE / NOOP operations on a per-turn basis, RL-fine-tuned
operation selector. Maps to **P2 structured memory** plus **P14 sleep-time
consolidation** for the dedup loop. See vendor entry in
[`vendor-landscape-2026-04.md`](../../ai-context-layer/references/vendor-landscape-2026-04.md).

### Agentic Context Engineering / ACE (2025)

Context as an evolving playbook (generation → reflection → curation). Avoids
"context collapse" where iterative rewriting erodes detail. Maps to **P15
procedural memory** plus the **A26 mode-collapse** anti-pattern.

### Event-Centric Conversational Memory (2025)

Conversation history as event-like propositions (participants, temporal cue,
proposition, local context) instead of turn-based logs. Maps to **P6 episodic
+ semantic split** — the event projection is the episodic side; extracted
propositions feed the semantic side.

---

**This file is working if:** an agent designer arrives, picks the right
P-code in under a minute, and never has to choose between two competing
taxonomies.

# Modern Best Practices — Current Production Standards

**Purpose**: Production-ready patterns aligned with 2025-2026 industry standards (MCP, A2A, NIST AI RMF, OpenTelemetry GenAI).

**July 2026 freshness anchor**: verify framework language support, transport guidance, authorization flows, pricing, and lifecycle claims against primary docs before making vendor recommendations.

---
## Table of Contents

- [Model Context Protocol (MCP)](#model-context-protocol-mcp)
- [Agent-to-Agent Protocol (A2A)](#agent-to-agent-protocol-a2a)
- [ADK Implementation Notes](#adk-implementation-notes)
- [Agentic RAG (Dynamic Retrieval)](#agentic-rag-dynamic-retrieval)
- [Handoff-First Orchestration](#handoff-first-orchestration)
- [Multi-Layer Guardrails](#multi-layer-guardrails)
- [Agent Framework Landscape (2026)](#agent-framework-landscape-2026)
- [Framework Selection by Constraint](#framework-selection-by-constraint)
- [Stable Guidance](#stable-guidance)
- [Practical Selection Guide](#practical-selection-guide)
- [OpenTelemetry for Agents](#opentelemetry-for-agents)
- [Service & Transport Layer (API Frontends)](#service-&-transport-layer-api-frontends)
- [Code/SWE Agents (SE 3.0)](#codeswe-agents-se-30)
- [Parallel Execution & Model Routing (2026 Trends)](#parallel-execution-&-model-routing-2026-trends)
- [Key Modern Migrations](#key-modern-migrations)
- [Usage Notes](#usage-notes)


## Model Context Protocol (MCP)

**What**: Open standard for connecting agents to tools, resources, and prompts.

**Governance**: MCP was donated by Anthropic to the Agentic AI Foundation (AAIF), a Linux Foundation project launched December 9, 2025 with AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft, and OpenAI as founding platinum members — vendor-neutral governance, not a single-vendor protocol. Verify current membership/roadmap at the AAIF site before making claims about governance stability.

**When to use**: Standardize tool and data access across hosts, models, and runtimes.

**Architecture**:
```yaml
MCP Host (AI App) → MCP Client → MCP Server
```

**Key Principles**:

- MCP is an integration layer, not an agent architecture.
- Use it for: tool access, resource retrieval, prompt templates, and shared capability boundaries.
- Default transports to `stdio` or Streamable HTTP; treat older SSE-only guidance as compatibility material, not the default.
- For remote MCP, use explicit authorization and least-privilege scopes; verify current OAuth/OIDC guidance in the spec.
- Tool design matters more than transport choice: publish narrow tasks, not raw backend complexity.
- Mark destructive or side-effectful tools clearly and validate all inputs/outputs against schema.
- Treat tool definitions and tool results as untrusted input; defend against prompt injection, tool shadowing, confused deputy, and over-broad scopes.
- Keep capability negotiation explicit and fail closed on unsupported methods or ambiguous contracts.

**Operational Concerns**:

- Prompt injection via tool descriptions or tool results
- Combined permissions enabling file or data exfiltration
- Lookalike tools silently replacing trusted ones
- Missing auth scopes or tenant boundaries on remote servers
- Mitigation: least privilege, schema validation, signature/publisher checks where available, and explicit policy checks

**Implementation Resources**:
- [`mcp-practical-guide.md`](mcp-practical-guide.md) - Copy-paste MCP server examples
- [`tool-design-specs.md`](tool-design-specs.md) - MCP implementation patterns
- [`protocol-decision-tree.md`](protocol-decision-tree.md) - When to use MCP vs A2A

---

## Agent-to-Agent Protocol (A2A)

**What**: Open protocol for agent-to-agent communication, task execution, and capability discovery between agentic applications. A2A v1.0 is stable under Linux Foundation governance (donated by Google). Native support in CrewAI, MS Agent Framework, Spring AI (as of mid-2026). Google ADK is now multi-language: Python, Java, and Go (verify current ADK version before use). Verify `a2a-protocol.org/latest/` for current spec version before implementation.

**When to use**: Multi-agent systems, task delegation, agent cards/discovery, long-running tasks, and cross-runtime orchestration.

**Architecture**:
```yaml
Agent A (Sender) → A2A Message → Agent B (Receiver)
   ↓                                      ↓
Validates payload            Executes task + returns result
```

**Key Principles**:

- Treat handoffs as versioned APIs with strict schema validation.
- Always propagate `trace_id` or equivalent correlation metadata across handoffs.
- Use agent cards for capability discovery and routing, not natural-language guessing.
- Preserve ownership, timeout, retry, and escalation semantics at handoff boundaries.
- Support async task execution and explicit terminal states for long-running work.
- Validate input/output schemas and refusal/error envelopes on every handoff.

**Core Message Schema**:
```json
{
  "schemaVersion": "v1",
  "trace_id": "req-abc-123",
  "sender": {"agent_id": "...", "agent_type": "..."},
  "receiver": {"agent_id": "...", "agent_type": "..."},
  "task": {"type": "...", "description": "..."},
  "context": {...},
  "constraints": {...}
}
```

**Orchestration Patterns**:

- **Sequential**: A → B → C (linear handoff chain)
- **Manager-Worker**: Manager delegates subtasks to specialized workers
- **Group Chat**: Collaborative multi-agent discussion
- **Handoff**: Dynamic delegation based on context and capabilities

**Critical Insight**: Most multi-agent failures are handoff and context-transfer failures, not base-model failures.

**Implementation Resources**:
- [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md) - Implementation patterns
- [`protocol-decision-tree.md`](protocol-decision-tree.md) - MCP vs A2A selection
- [`multi-agent-patterns.md`](multi-agent-patterns.md) - Orchestration templates

---

## ADK Implementation Notes

**Parent/Child Agents**: Use coordinator agents to delegate to specialized sub-agents; keep clear descriptions and instructions for routing.

**Custom Agents**: Extend BaseAgent for non-LLM behaviors; emit events and respect invocation context.

**Aggregation**: Use evaluators/majority vote when combining parallel agent outputs.

**Exceptions**: Implement error-handling patterns for tool/agent failures; degrade gracefully and surface traces.

---

## Agentic RAG (Dynamic Retrieval)

**What**: Multi-step retrieval with query rewriting, hybrid search, and optional chunk context augmentation (validate on your corpus).

**Pattern**:
```text
query → rewrite → embed → retrieve → contextual_rerank → filter → inject → cite
```

**Contextual Retrieval** (Anthropic 2024):

- Add context to each chunk before embedding
- Combine semantic (embeddings) + keyword (BM25)
- Mandatory reranking step
- 200-400 token chunks
- Route queries by domain first

**Old vs New**:
- **Old**: Static one-shot retrieval
- **New**: Iterative retrieval with adaptation

**Implementation Resources**:
- [`rag-patterns.md`](rag-patterns.md) - Contextual retrieval implementation
- [`../assets/rag/rag-advanced.md`](../assets/rag/rag-advanced.md) - Production template

---

## Handoff-First Orchestration

**What**: Treat agent handoffs as versioned APIs with strict validation

**Critical Insight**: Most agent failures are handoff/context-transfer issues, not model issues

**Best Practices**:
```yaml
handoff_payload:
  schemaVersion: "v1"
  trace_id: "abc-123"
  context: {validated_json}
  task: {atomic_instruction}
  constraints: {hard_limits}
```

**Patterns**:

- **Sequential**: A → B → C (linear pipeline)
- **Handoff**: Dynamic delegation based on context
- **Group Chat**: Collaborative multi-agent discussion
- **Magentic**: Manager coordinates specialized workers

**Validation**: JSON Schema required for every handoff

**Implementation Resources**:
- [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md) - Handoff protocols
- [`multi-agent-patterns.md`](multi-agent-patterns.md) - Orchestration templates

---

## Multi-Layer Guardrails

**What**: Defense-in-depth for production safety (NIST AI RMF, OWASP GenAI Top 10)

**Required Layers**:

1. **Input validation**: PII redaction, content filtering, prompt injection detection
2. **RBAC/ABAC**: Fine-grained authorization per tool/action
3. **Tool gating**: Signature verification (Sigstore/Cosign), human approval for high-risk
4. **Output filtering**: PII detection, policy checks, compliance validation
5. **Observability**: OpenTelemetry GenAI spans, SIEM integration, real-time alerts

**Human-in-the-Loop Required For**:

- Financial transactions
- Database modifications
- Legal/compliance actions
- Irreversible operations

**Implementation Resources**:
- [`deployment-ci-cd-and-safety.md`](deployment-ci-cd-and-safety.md) - Complete guardrails implementation
- [`../ai-mlops/`](../../ai-mlops/SKILL.md) - Security patterns

---

## Agent Framework Landscape (2026)

Do not rank frameworks by hype or popularity. Pick by control flow, language, deployment target, auditability requirements, and how much of the runtime you want to own.

### Framework Selection by Constraint

| Constraint | Strong Fits | Why |
| --- | --- | --- |
| Stateful workflow, checkpoints, HITL | LangGraph, Pydantic AI | Strong workflow/state modeling and durable execution patterns |
| OpenAI-first tool agents | OpenAI Agents SDK | Official Python and JavaScript SDKs, tracing, handoffs, HITL support |
| Anthropic-first code or computer-use agents | Claude Agent SDK | Official Python and TypeScript SDK, MCP-aware, code-agent tools, computer use |
| Gemini / Vertex AI environment | Google ADK | Code-first framework with strong Google ecosystem alignment |
| Azure-centric enterprise stack | Microsoft Agent Framework | Azure-focused orchestration and enterprise integration; verify current runtime support in docs |
| Retrieval-heavy orchestration | LlamaIndex Workflows, Haystack | Retrieval and pipeline depth are first-class concerns |
| TypeScript product apps | Mastra, OpenAI Agents JS, Claude Agent SDK | Stronger fit for web app and TS-native teams |
| Lightweight research or code-as-tools patterns | SmolAgents, DSPy | Minimal or optimization-oriented approaches |
| Managed AWS agent platform | Bedrock Agents | Managed infra, action groups, AWS-native deployment |

### Stable Guidance

- Favor workflow runtimes when you need auditability, resumability, and explicit failure handling.
- Favor tool-centric SDKs when the control flow is simple and the value is in fast iteration.
- Favor RAG-native frameworks only when retrieval quality is the primary constraint.
- Favor managed platforms only when infrastructure ownership is the bottleneck and platform lock-in is acceptable.
- Verify exact language support, transport support, lifecycle, and pricing before making a final recommendation.

### Practical Selection Guide

```text
Which framework?
    ├─ Need durable workflow state or strong auditability?
    │   ├─ Python/JS workflow graph → LangGraph
    │   └─ Typed Python workflow/state model → Pydantic AI
    ├─ Need simple official SDK for tool agents?
    │   ├─ OpenAI stack → OpenAI Agents SDK
    │   └─ Anthropic stack / code agents / computer use → Claude Agent SDK
    ├─ Need cloud-aligned orchestration?
    │   ├─ Google / Vertex AI → Google ADK
    │   ├─ Azure ecosystem → Microsoft Agent Framework
    │   └─ AWS managed runtime → Bedrock Agents
    ├─ Need retrieval-heavy orchestration?
    │   └─ LlamaIndex Workflows or Haystack
    └─ Need lighter experimentation?
        ├─ TS app teams → Mastra
        └─ Research / code-first loops → SmolAgents or DSPy
```

**Resources**: verify current support matrices in [`../data/sources.json`](../data/sources.json) before final recommendations.

---

## OpenTelemetry for Agents

**What**: Standardized observability using OpenTelemetry GenAI semantic conventions

**Required Telemetry**:
```yaml
spans:
  - llm_call: {prompt, response, tokens, latency}
  - tool_call: {name, params, result, duration}
  - retrieval: {query, chunks, scores}
  - memory_op: {read/write, key, size}
```

**Metrics to Track**:

- Tool success rate ≥95%
- Average latency < target
- Token cost < budget
- Evaluation score ≥ threshold
- Task success/containment rate ≥ target; escalation rate within budget
- User satisfaction or reviewer score tracked; flag drift in response quality
- Instrument like A/B experiments: track goal completion time, cost, and quality deltas across variants

**Platforms**: Azure AI Foundry, LangSmith, Arize, New Relic, Datadog

**Implementation Resources**:
- [`evaluation-and-observability.md`](evaluation-and-observability.md) - Complete observability guide
- [`../qa-observability/`](../../qa-observability/SKILL.md) - OpenTelemetry patterns

---

## Service & Transport Layer (API Frontends)

**What**: HTTP/gRPC/GraphQL contracts for agent endpoints

**Best Practices**:

- Use [`../dev-api-design/`](../../dev-api-design/SKILL.md) for HTTP/gRPC/GraphQL contracts, auth, rate limits, error shapes
- Expose agent endpoints with: `trace_id`, scopes/roles, tool allowlist, safety level, delivery mode (sync/stream/async)
- Prefer SSE/WebSocket for token streams; 202 + polling for long jobs; HMAC-signed webhooks for callbacks
- Standardize errors: model_timeout, tool_failed, guardrail_blocked, retrieval_miss, validation_error, quota_exceeded
- Observability: propagate `traceparent`; emit spans for llm_call, retrieval, tool_call; include rate-limit headers
- MTTD (Mean Time To Detect) for anomalies

**Implementation Resources**:
- [`api-contracts-for-agents.md`](api-contracts-for-agents.md) - Request/response envelopes, safety gates
- [`../../dev-api-design/assets/fastapi/`](../../dev-api-design/assets/fastapi/) - FastAPI templates

---

## Code/SWE Agents (SE 3.0)

**What**: Autonomous coding agents that perform end-to-end software engineering tasks

**Scale**: peer-reviewed measurements put coding-agent adoption at roughly 16–23% of active GitHub projects showing agent traces by late 2025 (129,134-project study, arXiv:2601.18341) — treat any single-vendor "N hundred thousand PRs in N weeks" headline number as promotional and unverifiable; cite the study instead if precision matters

**SE 3.0 Paradigm**: Intent-driven, conversational development where developers collaborate with autonomous AI teammates

**Architecture Patterns**:

- **Multi-Agent SWE** (HyperAgent): Planner → Navigator → Code Editor → Executor
- **Minimal Agent** (Lita/Mini-SWE): ~100-line implementation; the original mini-swe-agent result was 68% on SWE-bench (2025), later re-benchmarked at >74% on SWE-bench Verified (2026) as the harness matured — check the current README before quoting a number

**Critical Finding**: 29.6% of "plausible" SWE-Bench fixes introduce behavioral regressions

**Implication**: Test passing is insufficient; production deployments require:

- Behavioral regression testing
- Human code review
- Integration testing beyond unit tests
- Semantic diff analysis

**Guardrails for Code Agents**:

```yaml
execution_limits:
  max_steps: 50
  max_file_edits: 20
  timeout_minutes: 30

forbidden_operations:
  - delete_repository
  - force_push
  - modify_ci_config
  - access_secrets

review_triggers:
  - changes_to_security_files
  - more_than_10_files_modified
```

**Implementation Resources**:

- [`code-swe-agents.md`](code-swe-agents.md) - Complete patterns and architecture
- [`../data/sources.json`](../data/sources.json) - Research papers (SE 3.0, HyperAgent)

---

## Parallel Execution & Model Routing (2026 Trends)

**Parallel Execution**:

- Cursor now runs up to 8 agents in parallel
- Apps like Conductor and Verdent AI support background task execution
- Pattern: Define task, let LLM execute in background, start new task

**Model Routing / Cooperative Systems**:

- Smaller models handle routine tasks, delegate to larger models when needed
- Cost optimization through intelligent model selection
- "Whoever nails system-level integration will shape the market"

**Market Context**: Gartner predicts agents entering "trough of disillusionment" in 2026. Focus on operationalization over demos.

---

## Key Modern Migrations

**Traditional → Modern**:

- Custom APIs → Model Context Protocol (MCP)
- Static RAG → Agentic RAG with contextual retrieval
- Ad-hoc handoffs → Versioned handoff APIs with JSON Schema
- Single guardrail → Multi-layer defense (5+ layers)
- LangChain agents → LangGraph stateful workflows
- Custom observability → OpenTelemetry GenAI standards
- Model-centric → Context engineering-centric
- Code completion → Autonomous SWE agents (SE 3.0)
- Single framework → Framework selection by use case (2026)
- Sequential execution → Parallel agent execution

---

## Usage Notes

- **Default to modern standards**: MCP for tools, agentic RAG for retrieval, handoff-first for multi-agent
- **Reference specialized skills** for deep implementation (see Related Skills in main SKILL.md)
- **Use templates** for structured artifacts (see Navigation: Templates in main SKILL.md)

# Multi-Agent Patterns — Best Practices 

*Purpose: Provide operational patterns, role structures, coordination rules, and delegation procedures for multi-agent systems with handoff-first orchestration.*

**Modern Update**: Treat handoffs as versioned APIs with strict validation. Most agent failures are handoff/context-transfer issues, not model issues.

---
## Table of Contents

- [Handoff-First Orchestration (Critical Pattern)](#handoff-first-orchestration-critical-pattern)
- [Core Principle](#core-principle)
- [Handoff Payload Standard](#handoff-payload-standard)
- [Handoff Validation Checklist](#handoff-validation-checklist)
- [Orchestration Pattern Types](#orchestration-pattern-types)
- [Sequential Orchestration](#sequential-orchestration)
- [Dynamic Handoff Orchestration](#dynamic-handoff-orchestration)
- [Group Chat Orchestration](#group-chat-orchestration)
- [Magentic Pattern (Advanced)](#magentic-pattern-advanced)
- [1. Multi-Agent Architecture (Roles)](#1-multi-agent-architecture-roles)
- [Standard Agent Roles](#standard-agent-roles)
- [2. Multi-Agent Loop](#2-multi-agent-loop)
- [Pattern: Manager → Router → Worker → Evaluator → Manager → User](#pattern-manager-→-router-→-worker-→-evaluator-→-manager-→-user)
- [3. Manager Pattern](#3-manager-pattern)
- [Purpose: Task decomposition + orchestration](#purpose-task-decomposition-orchestration)
- [4. Router Pattern](#4-router-pattern)
- [Pattern: Domain Routing](#pattern-domain-routing)
- [5. Worker Pattern](#5-worker-pattern)
- [Purpose: Execute a well-defined subtask](#purpose-execute-a-well-defined-subtask)
- [6. Evaluator Pattern](#6-evaluator-pattern)
- [Pattern: Automatic Judging](#pattern-automatic-judging)
- [7. Coordinator Pattern](#7-coordinator-pattern)
- [Pattern: Merge & Integrate](#pattern-merge-&-integrate)
- [8. Communication Rules](#8-communication-rules)
- [Allowed Messages](#allowed-messages)
- [Not Allowed](#not-allowed)
- [Pattern: Message Templates](#pattern-message-templates)
- [9. Delegation Pattern](#9-delegation-pattern)
- [Pattern: Controlled Delegation](#pattern-controlled-delegation)
- [10. Multi-Agent RAG Pattern](#10-multi-agent-rag-pattern)
- [Pattern: Retrieval Worker + Integration Manager](#pattern-retrieval-worker-integration-manager)
- [11. Multi-Agent Memory Pattern](#11-multi-agent-memory-pattern)
- [Pattern: Dedicated Memory Agent](#pattern-dedicated-memory-agent)
- [12. Multi-Agent Error Handling](#12-multi-agent-error-handling)
- [Pattern: Escalation](#pattern-escalation)
- [13. Multi-Agent Safety Gates](#13-multi-agent-safety-gates)
- [Safety Checks](#safety-checks)
- [Pattern: Safety Override](#pattern-safety-override)
- [14. Multi-Agent Anti-Patterns (Master List)](#14-multi-agent-anti-patterns-master-list)
- [15. Quick Reference Tables](#15-quick-reference-tables)
- [Role Responsibility Table](#role-responsibility-table)
- [Interaction Table](#interaction-table)
- [16. Copy-Paste Multi-Agent Templates](#16-copy-paste-multi-agent-templates)
- [Subtask Template](#subtask-template)
- [Evaluation Template](#evaluation-template)
- [Worker Output Template](#worker-output-template)
- [End of File](#end-of-file)


## Handoff-First Orchestration (Critical Pattern)

### Core Principle

**Most agent failures happen at handoffs, not in individual agents.**

**Old approach**: Ad-hoc context passing between agents
**New approach**: Treat every handoff as a versioned API with JSON Schema validation

### Handoff Payload Standard

```yaml
handoff_payload:
  version: "v1.2"              # Schema version for compatibility
  trace_id: "req-abc-123"      # End-to-end tracing
  timestamp: "2025-11-18T10:30:00Z"
  source_agent: "manager-001"
  target_agent: "worker-research-02"

  task:
    id: "task-456"
    type: "research"           # Enum: research|code|ops|analysis
    instruction: "Find revenue data for ACME Corp Q4 2024"
    expected_output: "Structured JSON with revenue figures and sources"
    constraints:
      max_duration_seconds: 300
      require_citations: true

  context:
    user_query: "Original user question"
    prior_findings: [...]      # Validated JSON only
    domain: "finance"

  validation:
    schema_version: "v1.2"
    required_fields: ["task.instruction", "trace_id"]
    checksum: "sha256-hash"
```

### Handoff Validation Checklist

- [ ] JSON Schema defined for handoff payload
- [ ] Schema version included (for backward compatibility)
- [ ] trace_id propagated across all agents
- [ ] Required fields validated before handoff
- [ ] Context sanitized (no untrusted data)
- [ ] Expected output format specified
- [ ] Timeout/constraints defined
- [ ] Error handling path defined
- [ ] Observability span created for handoff

### Orchestration Pattern Types 

| Pattern | Use Case | Handoff Type | Complexity |
|---------|----------|--------------|------------|
| Sequential | Linear pipeline (A→B→C) | Synchronous | Low |
| Handoff | Dynamic delegation based on context | Conditional | Medium |
| Group Chat | Collaborative multi-agent discussion | Broadcast | High |
| Magentic | Manager coordinates specialized workers | Hub-spoke | High |

### Sequential Orchestration

**Pattern**: A → B → C (linear pipeline)

**Handoff flow**:
```yaml
Agent A completes → Creates handoff payload → Validates schema →
Agent B receives → Validates payload → Executes → Creates next handoff →
Agent C receives → Validates → Completes → Returns to orchestrator
```

**When to use**: Well-defined multi-step processes (data pipeline, document review, code compilation)

**Example**: Code review pipeline
```text
Linter Agent → Security Scanner → Test Runner → Human Reviewer
```

### Dynamic Handoff Orchestration

**Pattern**: Agent decides who to hand off to based on context

**Decision logic**:
```yaml
agent_receives_task()
if complexity > threshold:
  handoff_to(specialist_agent, validated_payload)
elif domain == "research":
  handoff_to(research_agent, validated_payload)
else:
  execute_locally()
```

**When to use**: Customer support, expert routing, context-dependent delegation

**Example**: Customer support
```text
General Agent → Assesses query →
  If billing: Handoff to Billing Specialist
  If technical: Handoff to Tech Support
  If sales: Handoff to Sales Agent
```

### Group Chat Orchestration

**Pattern**: Multiple agents collaborate with optional human participation

**Handoff mechanism**:
- Group Chat Manager coordinates
- Each agent can "speak" when relevant
- Manager decides turn-taking
- Shared context maintained

**When to use**: Brainstorming, multi-perspective analysis, consensus-building

**Example**: Product feature design
```text
PM Agent + Designer Agent + Engineer Agent + User Researcher Agent →
Manager coordinates discussion → Agents contribute expertise →
Consensus emerges → Decision documented
```

### Magentic Pattern (Advanced)

**Pattern**: Manager agent coordinates specialized worker agents

**Architecture**:
```yaml
Magentic Manager:
  - Decomposes complex task into subtasks
  - Selects appropriate worker for each subtask
  - Maintains shared context
  - Tracks progress
  - Integrates results

Workers (Domain-Specific):
  - Execute assigned subtasks
  - Return structured results
  - Request clarification via handoff
  - Report progress to manager
```

**Handoff flow**:
```text
User query → Manager decomposes →
  Worker A: Subtask 1 (with validated handoff payload)
  Worker B: Subtask 2 (parallel, with validated handoff payload)
  Worker C: Subtask 3 (depends on A+B, sequential handoff)
→ Manager integrates results → Validates completeness → Returns answer
```

**When to use**: Complex open-ended tasks, research projects, multi-domain problems

---

## 1. Multi-Agent Architecture (Roles)

### Standard Agent Roles

| Role | Purpose |
|------|----------|
| Manager | Break tasks down, orchestrate flow |
| Worker | Execute steps, use tools |
| Router | Classify and route tasks to correct workers |
| Evaluator | Score outputs for quality/safety |
| Memory Agent | Handle long-term storage & retrieval |
| Reviewer | Validate final draft or output |

---

# 2. Multi-Agent Loop

### Pattern: Manager → Router → Worker → Evaluator → Manager → User

```
manager.decompose()
router.assign(subtask)
worker.execute(subtask)
evaluator.score(result)
manager.integrate(scores)
return final_answer
```

**Checklist**

- [ ] Manager produces atomic tasks.  
- [ ] Router selects correct worker.  
- [ ] Worker produces grounded result.  
- [ ] Evaluator checks correctness/safety.  
- [ ] Manager integrates validated results.  

---

# 3. Manager Pattern

### Purpose: Task decomposition + orchestration

```
manager:
  input: user_query
  output: subtasks[]
```

**Checklist**

- [ ] Subtasks independent.  
- [ ] Each subtask defines expected output.  
- [ ] Order defined when sequential.  
- [ ] Avoid redundant subtasks.  

**Anti-Patterns**

- AVOID: Manager doing worker tasks.  
- AVOID: Producing vague subtasks.  
- AVOID: Producing too many micro-subtasks.  

---

# 4. Router Pattern

### Pattern: Domain Routing

```
router.classify(query)
→ domain
→ select worker(domain)
```

**Routing Table**

| Domain | Worker |
|--------|---------|
| Code | worker_code |
| Research | worker_research |
| Operations | worker_ops |
| RAG/Search | worker_retrieval |
| UI / OS | worker_os |

**Checklist**

- [ ] Classification based on intent.  
- [ ] Confidence threshold enforced.  
- [ ] Ask for clarification if ambiguous.  

---

# 5. Worker Pattern

### Purpose: Execute a well-defined subtask

```
worker:
  read_subtask()
  plan()
  execute_action()
  produce_output()
```

**Checklist**

- [ ] Worker uses tools when required.  
- [ ] Output structured and validated.  
- [ ] No speculation beyond evidence.  

**Anti-Patterns**

- AVOID: Worker modifying task definitions.  
- AVOID: Worker delegating further.  
- AVOID: Worker performing manager duties.  

---

# 6. Evaluator Pattern

### Pattern: Automatic Judging

```
evaluate:
  correctness
  grounding
  safety
  completeness
```

**Checklist**

- [ ] Use deterministic scoring rubric.  
- [ ] Score each worker output separately.  
- [ ] Provide structured JSON scores.  

**Decision Tree**

```
Is result unsafe?
→ Reject
Is score < threshold?
→ Request worker redo
Else:
→ Return approved result
```

---

# 7. Coordinator Pattern

### Pattern: Merge & Integrate

```
collect(worker_outputs)
validate_outputs()
merge_into_single_answer()
```

**Checklist**

- [ ] Deduplicate overlapping content.  
- [ ] Resolve contradictions using evaluator scores.  
- [ ] Ensure final answer grounded.  

---

# 8. Communication Rules

### Allowed Messages

- Subtask allocations  
- Clarification questions  
- Output summaries  
- Structured results  
- Evaluator scores  

### Not Allowed

- Internal reasoning  
- Irrelevant conversation  
- Free-form speculation  

### Pattern: Message Templates

```
task:
  id: ...
  description: ...
  expected_output: ...
```

```
result:
  id: ...
  output: ...
  evidence: [...]
  confidence: ...
```

---

# 9. Delegation Pattern

### Pattern: Controlled Delegation

```
manager → worker
worker → evaluator
evaluator → manager
```

**Checklist**

- [ ] Only manager delegates.  
- [ ] Workers do NOT spawn subtasks.  
- [ ] Evaluators only review, never execute.  

---

# 10. Multi-Agent RAG Pattern

### Pattern: Retrieval Worker + Integration Manager

**Flow**

```
manager identifies retrieval need
router → worker_retrieval
worker_retrieval → retrieve + rerank + summarize
evaluator → score relevance
manager → integrate into context
workers → continue tasks
```

**Checklist**

- [ ] Retrieval worker uses standard RAG patterns.  
- [ ] Summaries ≤ 150 tokens.  
- [ ] Evidence always cited.  

---

# 11. Multi-Agent Memory Pattern

### Pattern: Dedicated Memory Agent

```
memory_agent:
  read_write_memory()
  provide_relevant_entries()
```

**Checklist**

- [ ] Other agents never manipulate memory directly.  
- [ ] Memory agent enforces write rules (non-sensitive, verified).  

---

# 12. Multi-Agent Error Handling

### Pattern: Escalation

```
worker detects failure → evaluator checks → manager replans
```

**Checklist**

- [ ] Workers provide detailed error context.  
- [ ] Evaluators categorize (transient vs fatal).  
- [ ] Manager revises task list or requests clarification.  

**Anti-Patterns**

- AVOID: Allowing workers to silently fail.  
- AVOID: Continuing after inconsistent outputs.  

---

# 13. Multi-Agent Safety Gates

### Safety Checks

- Worker output must pass evaluator safety scan.  
- High-risk actions require explicit manager approval.  
- Disallowed domain → escalation to manager.  

### Pattern: Safety Override

```
if unsafe(step):
    halt
    return safe_alternative
```

---

# 14. Multi-Agent Anti-Patterns (Master List)

- AVOID: Manager performing work.  
- AVOID: Workers delegating tasks.  
- AVOID: Router routing without confidence threshold.  
- AVOID: Evaluator modifying outputs.  
- AVOID: Infinite loops between agents.  
- AVOID: Overlapping worker responsibilities.  
- AVOID: Storing partial outputs as memory.  
- AVOID: Using free-form discussion between agents.  

---

# 15. Quick Reference Tables

### Role Responsibility Table

| Role | Allowed | Not Allowed |
|------|---------|-------------|
| Manager | planning, orchestration | execution |
| Worker | execution, tool use | planning |
| Router | domain classification | execution |
| Evaluator | scoring, safety | modification |
| Memory Agent | storage/retrieval | reasoning |

### Interaction Table

| Step | Agent |
|------|--------|
| Decompose | Manager |
| Route | Router |
| Execute | Worker |
| Score | Evaluator |
| Integrate | Manager |

---

# 16. Copy-Paste Multi-Agent Templates

### Subtask Template

```
subtask:
  id: "task-001"
  description: "Extract financial metrics from retrieved document."
  expected_output:
    - metric_name
    - value
    - evidence_source
```

### Evaluation Template

```
evaluation:
  task_id: "task-001"
  correctness: 1-5
  grounding: 1-5
  safety: "pass|fail"
  notes: "..."
```

### Worker Output Template

```
worker_output:
  id: "task-001"
  output: {...}
  evidence: [...]
  confidence: 0-1
```

---

# End of File

# OODA Loop for Agent Architecture

Observe-Orient-Decide-Act framework applied to AI agent control loops, tempo advantage, and adaptive agent behavior. Based on John Boyd's OODA loop theory and 2026 production agent implementations (NVIDIA LLo11yPop, Snyk Agentic OODA).

## Contents

- [What Is the OODA Loop?](#what-is-the-ooda-loop)
- [OODA for AI Agents](#ooda-for-ai-agents)
- [The Four Phases in Detail](#the-four-phases-in-detail)
- [Tempo Advantage](#tempo-advantage)
- [Orientation as the Critical Phase](#orientation-as-the-critical-phase)
- [OODA Failures in Agents](#ooda-failures-in-agents)
- [Production Patterns](#production-patterns)
- [Decision Checklist](#decision-checklist)

---

## What Is the OODA Loop?

The **OODA loop** — Observe, Orient, Decide, Act — was developed by US Air Force Colonel John Boyd based on fighter pilot decision-making. The insight: the pilot who cycles through OODA faster gains decisive advantage, even against a more capable opponent.

```
         ┌──────────┐
         │ OBSERVE  │◄─────┐
         └────┬─────┘      │
              ▼            │
         ┌──────────┐      │
         │  ORIENT  │      │
         └────┬─────┘      │
              ▼            │
         ┌──────────┐      │
         │  DECIDE  │      │
         └────┬─────┘      │
              ▼            │
         ┌──────────┐      │
         │   ACT    │──────┘
         └──────────┘
```

### Why It Applies to AI Agents

AI agents operating in dynamic environments face the same challenge: they must perceive, interpret, decide, and act in continuous cycles. The speed and quality of each cycle determines effectiveness.

**2026 production examples**:
- **NVIDIA LLo11yPop** — observability agent using OODA for GPU fleet management
- **Snyk Agentic OODA** — security agents for threat response
- **IEEE paper** — "Agentic AI's OODA Loop Problem" (governance gaps)
- **al3rez/ooda-subagents** — open-source Claude Code-compatible framework

---

## OODA for AI Agents

### The Four Phases Mapped to Agent Architecture

| OODA Phase | Classical | AI Agent Translation |
|-----------|----------|---------------------|
| **Observe** | Sensor data, situational awareness | Data fusion, context gathering, tool queries |
| **Orient** | Cultural/personal filters, mental models | Model-guided sensemaking, context interpretation |
| **Decide** | Choose course of action | Probabilistic decision, tool selection |
| **Act** | Execute the maneuver | Workflow orchestration, tool calls |

### The 2026 Reframing

Each phase changes meaning for AI:

- **Observation → Data fusion**: Agents combine data from multiple sources (logs, APIs, user input) into a coherent situation picture
- **Orientation → Model-guided sensemaking**: LLM's world model interprets raw data into meaningful patterns
- **Decision → Probabilistic choice**: Agent selects actions under uncertainty, often using explicit probability
- **Action → Orchestrated execution**: Agent dispatches tools, sub-agents, or external systems

---

## The Four Phases in Detail

### 1. Observe

**Purpose**: Gather the raw data needed for the decision.

| Good Observation | Bad Observation |
|-----------------|-----------------|
| Targeted — relevant to the task | Indiscriminate — everything |
| Timely — current state | Stale — outdated info |
| Multi-source — triangulated | Single source — vulnerable |
| Noise-filtered | Noise-included |

**Implementation for agents**:
```
Observation phase tools:
  - Read file / database state
  - Query API for current data
  - Check logs for recent activity
  - Receive user context / input
  - Access memory / prior interactions
```

### 2. Orient

**Purpose**: Interpret observations in light of goals, constraints, and prior knowledge. This is where raw data becomes situational understanding.

**Boyd's key insight**: Orientation is the critical phase. Two agents seeing the same observations can orient completely differently — and the one with better orientation wins.

| Good Orientation | Bad Orientation |
|-----------------|-----------------|
| Integrates new info with mental model | Uses pattern without context |
| Updates beliefs when data warrants | Sticks to prior assumptions |
| Recognizes novel situations | Forces new into old categories |
| Considers multiple frames | Single-frame thinking |

**Implementation for agents**:
```
Orient phase:
  1. What do the observations mean for my goal?
  2. Has anything changed since last cycle?
  3. What frame / model fits this situation?
  4. What uncertainties remain?
  5. What assumptions might be wrong?
```

For LLM agents, orientation is often hidden in the reasoning phase. Making it explicit improves quality.

### 3. Decide

**Purpose**: Choose the next action from available options.

| Good Decision | Bad Decision |
|--------------|--------------|
| Explicit about alternatives | Implicit, only considers one option |
| Confidence level stated | Binary certain/uncertain |
| Considers second-order effects | Focuses only on immediate outcome |
| Aligns with goal hierarchy | Optimizes wrong objective |

**Implementation for agents**:
```
Decide phase:
  Options: [list realistic alternatives]
  Evaluation: score each against goals
  Chosen: [action]
  Confidence: [high/medium/low]
  Fallback if fails: [alternative]
```

### 4. Act

**Purpose**: Execute the chosen action in the environment.

| Good Action | Bad Action |
|------------|-----------|
| Atomic — one clear step | Compound — many things at once |
| Observable — outcome measurable | Unobservable — no feedback |
| Reversible when possible | Irreversible without reason |
| Produces data for next observation | Dead-end with no feedback |

**Critical**: Action must generate observations for the NEXT loop. Otherwise the loop breaks.

---

## Tempo Advantage

### Boyd's Core Insight

> "The one who gets inside the opponent's OODA loop — that is, cycles through OODA faster — forces the opponent into a reactive position they can't escape."

### Tempo for AI Agents

In adversarial or dynamic environments, faster OODA loops dominate:

| Scenario | Fast Loop Advantage |
|----------|-------------------|
| **Security** | Detect threats before they cause damage |
| **Trading** | React to market moves before competitors |
| **Operations** | Auto-remediate before user impact |
| **Debugging** | Test hypotheses faster than human operators |
| **Customer service** | Respond before frustration compounds |

### How to Accelerate OODA

| Technique | Effect |
|-----------|--------|
| **Parallel observation** | Gather multiple data sources simultaneously |
| **Cached orientation** | Reuse mental models when situation is familiar |
| **Pre-computed decisions** | Playbooks for common scenarios |
| **Async action** | Act while next observation cycle starts |
| **Batch decisions** | Multiple related actions in one decision cycle |

### The Quality vs. Speed Tradeoff

Faster OODA doesn't mean worse decisions if:
- Decisions are structured (playbooks)
- Uncertainty is managed (clear confidence thresholds)
- Reversibility is preserved (can undo bad decisions)

Speed matters most when the environment is changing faster than your loop. In static environments, tempo advantage disappears.

---

## Orientation as the Critical Phase

### Why Orientation Dominates

Boyd emphasized that **orientation is where battles are won or lost**. Two reasons:

1. **Garbage in, garbage out**: Bad orientation → wrong decisions regardless of decision speed
2. **Compounding errors**: Bad orientation creates bad observations (you look at wrong things), creating worse orientation

### Orientation Failure Modes

| Failure | Example | Fix |
|---------|---------|-----|
| **Pattern match on irrelevant features** | Agent recognizes syntax but misses semantics | Ground orientation in actual goals |
| **Stale mental model** | Agent uses outdated context | Refresh orientation each cycle |
| **Single-frame thinking** | Only one perspective considered | Multi-framing in reasoning |
| **Confirmation bias** | Observations only confirm priors | Explicit disconfirmation checks |
| **Over-fitting to recent events** | Last event dominates | Balance recent with historical |

### Improving Agent Orientation

| Technique | Mechanism |
|-----------|----------|
| **Explicit reasoning prompts** | Force agent to articulate orientation |
| **Multi-perspective evaluation** | Consider task from multiple frames |
| **Confidence calibration** | Agent states uncertainty explicitly |
| **Prior-posterior updates** | Bayesian-style belief updating |
| **Counter-scenarios** | "What would change my view?" |

---

## OODA Failures in Agents

### Common Breakdowns

| Failure | Symptom | Cause |
|---------|---------|-------|
| **Observation loop broken** | Agent acts on stale data | No feedback mechanism from actions |
| **Skipped orientation** | Agent reacts to surface patterns | No reasoning between observation and decision |
| **Frozen orientation** | Agent stuck on wrong model | No updating mechanism |
| **Decision paralysis** | Agent loops indefinitely | No time/cost limit on deciding |
| **Action without loop** | Agent fires and forgets | No verification of outcome |
| **Cascading errors** | Bad outcome → bad observations → worse decisions | No error recovery |

### The "Zombie Loop"

When an agent keeps cycling through OODA but makes no progress — typically because orientation is wrong and never updates. The agent observes, orients incorrectly, decides incorrectly, acts incorrectly, observes the same failure, repeats.

**Fix**: External intervention, orientation reset, or human in the loop.

---

## Production Patterns

### NVIDIA LLo11yPop Pattern

GPU fleet management via OODA agents:

```
Observe: Monitor GPU metrics, logs, telemetry
Orient: Classify patterns (normal, degraded, failing)
Decide: Choose remediation (restart, reallocate, alert)
Act: Execute remediation
```

Each cycle produces observations that feed the next cycle.

### Snyk Agentic OODA Pattern

Security threat response:

```
Observe: SIEM data, user reports, automated scans
Orient: Threat classification, severity assessment
Decide: Response action (block, investigate, escalate)
Act: Execute response
Loop: Monitor for new threats or response effectiveness
```

### al3rez/ooda-subagents Pattern

Claude Code compatible OODA framework:
- Each OODA phase as a subagent
- Explicit phase transitions
- Shared state between cycles
- Designed for startups shipping AI products

### Implementation Principles

1. **Each phase is explicit** — not implicit in reasoning
2. **Phases can be different agents** — specialization improves quality
3. **Shared context between cycles** — orientation builds on history
4. **Observable actions** — outcomes feed next observation
5. **Time/cost budget per cycle** — prevents infinite loops

---

## Decision Checklist

- [ ] Agent has an explicit observe phase (not just implicit context reading)
- [ ] Orient phase is explicit — agent articulates its interpretation
- [ ] Decide phase considers alternatives and states confidence
- [ ] Act phase produces observable outcomes for next cycle
- [ ] Loop has clear exit conditions (success, failure, timeout)
- [ ] Orientation can be updated when observations contradict the model
- [ ] Disconfirmation is explicitly sought, not just confirmation
- [ ] Tempo is appropriate for the environment (fast for dynamic, slower for static)
- [ ] Error recovery mechanism exists (break bad loops)
- [ ] Human in the loop for irreversible or high-stakes actions

---

## Sources

- Boyd, J. (1976-1996). OODA loop and maneuver warfare theory
- Osinga, F. (2007). *Science, Strategy and War: The Strategic Theory of John Boyd*
- NVIDIA Developer Blog (2026). *Optimizing Data Center Performance with AI Agents and the OODA Loop Strategy*
- Snyk Blog (2026). *The Agentic OODA Loop*
- IEEE (2026). *Agentic AI's OODA Loop Problem*
- al3rez/ooda-subagents: GitHub repo for OODA subagents framework

# Operational Patterns — Core Primitives for Agent Construction

**Purpose**: Inline executable patterns Claude can use directly without consulting deeper files. These represent the fundamental building blocks for agent implementation.

---
## Table of Contents

- [1. Agent Loop Pattern (Single-Agent)](#1-agent-loop-pattern-single-agent)
- [2. OS Agent Action Loop (Desktop / Web / Mobile)](#2-os-agent-action-loop-desktop-web-mobile)
- [3. RAG Pipeline Pattern (Full)](#3-rag-pipeline-pattern-full)
- [4. Tool Specification Pattern](#4-tool-specification-pattern)
- [5. Memory System Pattern](#5-memory-system-pattern)
- [6. Multi-Agent Workflow Pattern](#6-multi-agent-workflow-pattern)
- [7. Safety & Guardrails Pattern](#7-safety-&-guardrails-pattern)
- [8. Observability Pattern](#8-observability-pattern)
- [9. Evaluation Patterns](#9-evaluation-patterns)
- [Final Answer Evaluation](#final-answer-evaluation)
- [Trajectory Evaluation](#trajectory-evaluation)
- [Evaluators](#evaluators)
- [10. Deployment & CI/CD Pattern (Modern Production Standards)](#10-deployment-&-cicd-pattern-modern-production-standards)
- [Shared Utilities (Implementation Patterns)](#shared-utilities-implementation-patterns)
- [Usage Notes](#usage-notes)


## 1. Agent Loop Pattern (Single-Agent)

```
PLAN → ACT → OBSERVE → UPDATE → REPEAT → FINAL ANSWER
```

**Plan Rules**

- Decompose into atomic steps.
- Select tools intentionally.
- Validate input parameters.

**Act Rules**

- Never guess IDs/paths—retrieve them.
- Confirm before irreversible actions.
- Retry on transient failures.

**Observe Rules**

- Inspect tool outputs.
- Check for contradictions.
- Detect stuck loops.

**Update Rules**

- Append retrieved evidence.
- Adjust plan if environment changes.

---

## 2. OS Agent Action Loop (Desktop / Web / Mobile)

```
OBSERVE(window_state)
GROUND(element)
ACT(click/type/scroll/shortcut)
VERIFY(state_changed)
```

**Constraints**

- Never click blind coordinates if an element is detectable.
- Halt if UI layout diverges.
- Log all observations and verifications.

**Related Resources**: See [`os-agent-capabilities.md`](os-agent-capabilities.md) for desktop automation details.

---

## 3. RAG Pipeline Pattern (Full)

```
query
→ rewrite (if ambiguous)
→ embed
→ retrieve (semantic + keyword if hybrid)
→ rerank (mandatory)
→ filter
→ inject context
→ answer with citations
```

**Injection Format**

```
<retrieved>
[chunk_1]
[chunk_2]
</retrieved>
```

**Chunk Guidance**

- 200–400 tokens each.
- Avoid mixing domains; route queries first.

**Related Resources**: See [`rag-patterns.md`](rag-patterns.md) for contextual retrieval implementation.

---

## 4. Tool Specification Pattern

**Definition Template**

```yaml
tool_name:
  description: [operational purpose]
  input_schema:
    field1: type
    field2: type
  output_schema:
    result: type
  confirm: yes/no
  error_handling:
    retry: [count]
    timeout: [seconds]
```

**Tool Use Rules**

- Validate parameters before execution.
- Sanitize inputs.
- Retry network/timeouts once or twice.
- Reject hallucinated tool names.

**Related Resources**:
- [`tool-design-specs.md`](tool-design-specs.md) for MCP implementation patterns
- [`../assets/tools/tool-definition.md`](../assets/tools/tool-definition.md) for copy-paste templates

---

## 5. Memory System Pattern

**Four Types**

- **Session memory**: Conversation context within a single session
- **Long-term memory**: Persistent facts and preferences across sessions
- **Episodic memory**: Historical interactions and task outcomes
- **Task memory**: Scratchpad for current task execution

**Memory Write Conditions**

- Explicit user confirmation.
- Non-sensitive data only.
- Verifiable fact.
- Must have provenance.

**Retrieval Rules**

- Filter by relevance.
- Summarize >2000 tokens.
- Enforce recency if applicable.

**Related Resources**: See [`memory-systems.md`](memory-systems.md) for architecture details.

---

## 6. Multi-Agent Workflow Pattern

**Roles**

- **Manager** → decomposes task
- **Worker_X** → executes actions
- **Router** → routes domain-specific tasks
- **Evaluator** → scores accuracy + safety

**Manager Rules**

- Never perform work.
- Only produce subtask specifications.
- Collect worker results → integrate → validate.

```yaml
manager:
  tasks: [decompose, orchestrate]
worker_A:
  tasks: [tool-use, research]
evaluator:
  tasks: [score correctness, safety]
router:
  tasks: [domain classification]
```

**Additional Patterns**

- **Diamond**: Parallel specialists → aggregator → final decision.
- **Collaborative**: Agents debate then reconcile with a mixer.
- **Response mixer**: Blend tool-grounded and generative responses with explicit weighting.
- **Contracts**: Treat handoffs as contracts; version JSON Schemas; include negotiation and subcontracts when chaining vendors/teams.
- **Simulation**: Use sandbox/gym loops for policy testing and self-evolution; gate promotions with eval + safety thresholds.

**Related Resources**:
- [`multi-agent-patterns.md`](multi-agent-patterns.md) for orchestration templates
- [`a2a-handoff-patterns.md`](a2a-handoff-patterns.md) for delegation protocols
- [`../assets/multi-agent/`](../assets/multi-agent/) for copy-paste templates

---

## 7. Safety & Guardrails Pattern

**Block**

- High-risk actions without confirmation.
- Undeclared tool calls.
- Missing grounding.
- Unsupported domains.

**Require Confirmation For**

- File deletion or overwrite.
- Financial or legal actions.
- OS-level execution.
- External system modifications.

**Related Resources**: See [`deployment-ci-cd-and-safety.md`](deployment-ci-cd-and-safety.md) for multi-layer guardrails.

---

## 8. Observability Pattern

**Logs Must Include**

- Input
- Plan
- Tool calls
- Tool results
- Retrieved chunks
- Final output

**Traces**

One span per:
- LM call
- Tool call
- Retrieval step
- Memory write/read

**Metrics**

- Tool success rate ≥95%
- Avg latency < target
- Token cost < budget
- Evaluation score ≥ threshold
- Task success/containment rate ≥ target; escalation rate within budget
- User satisfaction or reviewer score tracked; flag drift in response quality
- Instrument like A/B experiments: track goal completion time, cost, and quality deltas across variants

**Related Resources**: See [`evaluation-and-observability.md`](evaluation-and-observability.md) for OpenTelemetry implementation.

---

## 9. Evaluation Patterns

**Approach**: Use outside-in (end-to-end) plus inside-out (trajectory) evaluation to catch process errors, not just final answers.

### Final Answer Evaluation

```yaml
Correctness: 1–5
Grounding: 1–5
Tool Use: 1–5
Safety: pass/fail
```

### Trajectory Evaluation

- Was the plan valid?
- Did the agent adapt to new state?
- Were tool parameters justified?
- Was retrieval grounded?
- Did handoffs respect contracts? Were schema violations caught?
- Were escalations/HITL gates triggered correctly?

### Evaluators

- Mix LM-judge, agent-judge, and human review for high-risk actions.
- Provide reviewer UI with trace, inputs, tool calls, outputs, and policy checks.
- Score safety (RAI), grounding, determinism, and tool correctness separately.

**Related Resources**: See [`evaluation-and-observability.md`](evaluation-and-observability.md) for LLM-as-judge patterns.

---

## 10. Deployment & CI/CD Pattern (Modern Production Standards)

**Pipeline**

```text
dev → CI eval → staging → canary → production
```

**Pre-deployment Checklist (NIST AI RMF Aligned)**

Safety & Security:
- [ ] Multi-layer guardrails configured (input validation, RBAC, output filtering)
- [ ] PII redaction verified with test cases
- [ ] Tool signature verification enabled (Sigstore/Cosign)
- [ ] HITL (Human-in-the-Loop) gates configured for high-risk operations
- [ ] OWASP GenAI Top 10 vulnerabilities tested
- [ ] Prompt injection defenses validated
- [ ] Roles/runbooks assigned (oncall, SecOps, SRE); incident response playbook tested
- [ ] Safe rollout plan defined (canary %, kill switch, rollback drills)
- [ ] Kill switch and rollback drills exercised this sprint

Observability & Monitoring:
- [ ] OpenTelemetry GenAI spans instrumented
- [ ] SIEM integration configured with alerting rules
- [ ] Cost and latency budgets set
- [ ] MTTD (Mean Time To Detect) baseline established
- [ ] LangSmith/Arize/Azure AI Foundry observability enabled

Evaluation & Quality:
- [ ] Evaluation score ≥ threshold on test suite
- [ ] Tool success rate ≥95%
- [ ] Hijacking scenarios tested
- [ ] Regression gate passed
- [ ] A/B test plan ready (if applicable)

Infrastructure:
- [ ] Rollback path tested
- [ ] Version pinned (model, dependencies, tools)
- [ ] Canary monitoring configured (5-10% traffic)
- [ ] Rate limiting configured
- [ ] Short-lived secrets rotation enabled
- [ ] SBOMs and SLSA attestations attached

Handoffs & Orchestration (Multi-Agent):
- [ ] Handoff payloads validated with JSON Schema
- [ ] trace_id propagation verified
- [ ] Context-transfer tested across agents
- [ ] Manager-worker contracts documented

**Weekly Production Tasks**:
- Run evaluation suite with new production data
- Review SIEM alerts and incident taxonomy
- Check for model/tool drift
- Audit HITL approval queue metrics
- Update cost/latency budgets if needed

**Related Resources**: See [`deployment-ci-cd-and-safety.md`](deployment-ci-cd-and-safety.md) for complete production guide.

---

## Shared Utilities (Implementation Patterns)

For cross-cutting implementation concerns in agent development, reference these centralized utilities:

- [llm-utilities.md](../../software-clean-code-standard/references/llm-utilities.md) — Token counting, streaming, cost estimation for LLM calls
- [error-handling.md](../../software-clean-code-standard/references/error-handling.md) — Effect Result types, correlation IDs for agent error handling
- [resilience-utilities.md](../../software-clean-code-standard/references/resilience-utilities.md) — p-retry v6, circuit breaker for LLM API calls
- [logging-utilities.md](../../software-clean-code-standard/references/logging-utilities.md) — pino v9 + OpenTelemetry for agent logging
- [observability-utilities.md](../../software-clean-code-standard/references/observability-utilities.md) — OpenTelemetry SDK, tracing spans per tool/LLM call
- [testing-utilities.md](../../software-clean-code-standard/references/testing-utilities.md) — Vitest, MSW v2 for mocking agent APIs

---

## Usage Notes

- **Prefer inline patterns** for simple, well-understood tasks
- **Reference deeper resources** when guidance or examples needed
- **Use templates** when structured artifacts required
- **Never include theory** — only operational steps

# OS Agent Capabilities — Best Practices

*Purpose: Provide operational patterns for perception, grounding, planning, UI interaction, and verification for OS-level agents (desktop, web, mobile).*

---
## Table of Contents

- [Core Loop](#core-loop)
- [2. Perception (Observation)](#2-perception-observation)
- [Pattern: Structured Screen Capture](#pattern-structured-screen-capture)
- [3. Grounding (Element Identification)](#3-grounding-element-identification)
- [Pattern: Deterministic Element Selection](#pattern-deterministic-element-selection)
- [4. UI Navigation](#4-ui-navigation)
- [Pattern: Intent-Based Navigation](#pattern-intent-based-navigation)
- [5. Actions (Execution)](#5-actions-execution)
- [5.1 Click Actions](#51-click-actions)
- [5.2 Typing Actions](#52-typing-actions)
- [5.3 Scroll Actions](#53-scroll-actions)
- [5.4 Shortcut Actions (Keyboard Commands)](#54-shortcut-actions-keyboard-commands)
- [6. Verification (Post-Action Check)](#6-verification-post-action-check)
- [Pattern: State Change Verification](#pattern-state-change-verification)
- [7. Error & Recovery Patterns](#7-error-&-recovery-patterns)
- [Pattern: Recoverable Error Handling](#pattern-recoverable-error-handling)
- [8. Page/Screen State Normalization](#8-pagescreen-state-normalization)
- [Pattern: Normalize UI State](#pattern-normalize-ui-state)
- [9. Multi-Step OS Tasks](#9-multi-step-os-tasks)
- [Pattern: Stepwise OS Automation](#pattern-stepwise-os-automation)
- [10. Accessibility-First Grounding](#10-accessibility-first-grounding)
- [Use When Available](#use-when-available)
- [11. Window & App Control](#11-window-&-app-control)
- [Pattern: Window Management](#pattern-window-management)
- [12. Browser-Specific Patterns](#12-browser-specific-patterns)
- [Pattern: DOM-Grounded Selection](#pattern-dom-grounded-selection)
- [13. Mobile-Specific Patterns](#13-mobile-specific-patterns)
- [Pattern: Mobile Interaction](#pattern-mobile-interaction)
- [14. OS Agent Anti-Patterns (Master List)](#14-os-agent-anti-patterns-master-list)
- [15. Quick Reference Tables](#15-quick-reference-tables)
- [Element Types Table](#element-types-table)
- [Common OS Actions](#common-os-actions)
- [Verification Points](#verification-points)
- [End of File](#end-of-file)


# 1. OS Agent Architecture (Operational)

### Core Loop

```
OBSERVE(window_state)
GROUND(target_element)
ACT(click/type/scroll/shortcut)
VERIFY(state_changed)
```

**Checklist**

- [ ] Loop runs per step, not globally.  
- [ ] Each observation updates actionable state.  
- [ ] Grounding completes before action.  
- [ ] Verification validates intended state change.  

---

# 2. Perception (Observation)

### Pattern: Structured Screen Capture

```
capture_screenshot()
extract_ui_tree()
extract_text_blocks()
extract_positions()
normalize_state()
```

**Checklist**

- [ ] Capture full viewport.  
- [ ] Include bounding boxes with coordinates.  
- [ ] Include role/type/label for each element.  
- [ ] Include OCR when text not available in UI tree.  

**Anti-Patterns**

- AVOID: Using raw images without structure.  
- AVOID: Guessing element positions.  

---

# 3. Grounding (Element Identification)

### Pattern: Deterministic Element Selection

```
find_element(criteria)
rank_candidates()
select_best_match()
verify_element_exists()
```

**Matching Criteria**

- role (button, field, link)  
- label/text  
- aria attributes  
- position constraints  
- icon description  

**Checklist**

- [ ] At least two criteria match.  
- [ ] Reject ambiguous matches (>1 candidate).  
- [ ] Validate element visible + enabled.  

**Decision Tree**

```
Is exact-text match found?
→ Yes → use it
→ No → fallback to approximate match
If >1 match → disambiguate or ask user
```

---

# 4. UI Navigation

### Pattern: Intent-Based Navigation

```
map_intent_to_target()
navigate_to_section()
confirm_visibility()
```

**Checklist**

- [ ] Identify nearest navigable parent (tab, menu, section).  
- [ ] Avoid unnecessary page reloads.  
- [ ] Scroll only when item is off-screen.  

**Anti-Patterns**

- AVOID: Blind scrolling.  
- AVOID: Navigating without checking for location change.  

---

# 5. Actions (Execution)

## 5.1 Click Actions

```
hover(optional)
click(element.bounding_box.center)
```

**Checklist**

- [ ] Click center point (safe zone).  
- [ ] Confirm element not obstructed.  
- [ ] Add small delay (100–250ms) if needed.  

**Anti-Patterns**

- AVOID: Clicking coordinate literals without grounding.  
- AVOID: Clicking non-visible elements.  

---

## 5.2 Typing Actions

```
focus(element)
type(text)
```

**Checklist**

- [ ] Clear existing text if required.  
- [ ] Type at controlled speed (if OS needs).  
- [ ] Submit only after verification.  

---

## 5.3 Scroll Actions

```
if element not visible:
    scroll(direction)
    re-observe()
```

**Checklist**

- [ ] Scroll small increments.  
- [ ] Re-run perception each scroll.  

---

## 5.4 Shortcut Actions (Keyboard Commands)

```
send_shortcut(["ctrl","h"])
verify_ui_changed()
```

**Checklist**

- [ ] Confirm OS-specific shortcut validity.  
- [ ] Avoid destructive shortcuts.  
- [ ] Rerun perception to validate state.  

---

# 6. Verification (Post-Action Check)

### Pattern: State Change Verification

```
expected_state = define_preconditions()
post_state = observe()
compare(expected_state, post_state)
```

**Checklist**

- [ ] Confirm element clicked triggered action.  
- [ ] Confirm field text updated.  
- [ ] Confirm navigation completed.  
- [ ] Confirm UI tree changed as expected.  

**Decision Tree**

```
Did expected element appear?
→ Yes → success
→ No → retry once
→ Still no → revise plan
```

**Anti-Patterns**

- AVOID: Proceeding without verifying success.  
- AVOID: Assuming action worked based on timing alone.  

---

# 7. Error & Recovery Patterns

### Pattern: Recoverable Error Handling

```
if ui_state_unexpected:
    re-observe
    try_different_path
    escalate
```

**Recoverable Cases**

- Missing element  
- Partial load  
- Scroll mismatch  
- Timing delay  

**Fatal Cases**

- Permission denied  
- System alert blocking UI  
- Full navigation failure  

---

# 8. Page/Screen State Normalization

### Pattern: Normalize UI State

```
collapse_popups()
close_modals()
remove_overlays()
focus_primary_app()
```

**Checklist**

- [ ] Clear popovers before grounding.  
- [ ] Close notifications blocking clickable elements.  
- [ ] Ensure main window is active.  

---

# 9. Multi-Step OS Tasks

### Pattern: Stepwise OS Automation

```
for step in plan:
    observe
    ground
    act
    verify
```

**Checklist**

- [ ] Each step has defined expected outcome.  
- [ ] Plan adjusts after each observation.  
- [ ] No long speculative chains.  

---

# 10. Accessibility-First Grounding

### Use When Available

- aria-label  
- aria-role  
- accessibility names  
- tab order  
- keyboard navigability  

**Why It Matters (Operational-Only)**

- determinism  
- easier matching  
- avoids coordinate guessing  

---

# 11. Window & App Control

### Pattern: Window Management

```
ensure_app_in_foreground()
verify_window_title()
maximize_if_needed()
```

**Checklist**

- [ ] Correct app selected.  
- [ ] Window not hidden/minimized.  
- [ ] Focus restored after each action.  

---

# 12. Browser-Specific Patterns

### Pattern: DOM-Grounded Selection

```
locate_node(css/xpath)
verify_visible()
scroll_into_view()
click()
```

**Checklist**

- [ ] Prefer CSS selectors over XPath.  
- [ ] Reject hidden nodes.  
- [ ] Validate element index if duplicates exist.  

---

# 13. Mobile-Specific Patterns

### Pattern: Mobile Interaction

```
tap(element.center)
swipe(start → end)
wait_for_animation()
```

**Checklist**

- [ ] Use accessibility ID when available.  
- [ ] Avoid pixel-based coordinates; use bounding boxes.  
- [ ] Re-observe after screen transitions.  

---

# 14. OS Agent Anti-Patterns (Master List)

- AVOID: Blind clicking by coordinates.  
- AVOID: Acting without grounding.  
- AVOID: Not re-observing after UI change.  
- AVOID: Ignoring obstructions (modals, popovers).  
- AVOID: Hard-coding UI paths.  
- AVOID: Performing multi-step actions without intermediate verification.  
- AVOID: Failing to validate element visibility/enabled state.  
- AVOID: Scrolling arbitrarily without checking viewport.  

---

# 15. Quick Reference Tables

### Element Types Table

| Type | Recognition Feature |
|------|----------------------|
| Button | label, role, icon |
| Input | placeholder, aria-role |
| Link | href/text |
| Modal | overlay + center element |
| Menu | vertical list structure |

### Common OS Actions

| Action | When to Use |
|--------|--------------|
| click | select element |
| type | enter data |
| scroll | move viewport |
| shortcut | open dialogs, commands |
| hover | reveal menus |

### Verification Points

| Step | Required Check |
|------|----------------|
| After click | UI changed |
| After type | Text updated |
| After scroll | Target visible |
| After navigation | Correct page/section |

---

# End of File

# Principal-Agent Theory for AI Agents

Moral hazard, information asymmetry, shadow principals, and governance frameworks applied to AI agent delegation. Based on classical principal-agent theory (Jensen-Meckling, Holmström) and 2026 AI agent governance research including OWASP Agentic Top 10 and the EU AI Act.

## Contents

- [The Principal-Agent Problem](#the-principal-agent-problem)
- [Information Asymmetry in AI Delegation](#information-asymmetry-in-ai-delegation)
- [Moral Hazard and Misalignment](#moral-hazard-and-misalignment)
- [Shadow Principals](#shadow-principals)
- [Monitoring and Verification](#monitoring-and-verification)
- [Incentive Alignment](#incentive-alignment)
- [Governance Frameworks](#governance-frameworks)
- [OWASP Agentic Top 10 Mapping](#owasp-agentic-top-10-mapping)
- [Decision Checklist](#decision-checklist)

---

## The Principal-Agent Problem

**Principal-agent theory** studies the problem when one party (the principal) delegates work to another (the agent) under conditions of:

1. **Information asymmetry** — the agent knows more about the work than the principal
2. **Goal divergence** — the agent has objectives that may differ from the principal's
3. **Costly monitoring** — verifying the agent's behavior is expensive or incomplete

### Why This Applies to AI Agents

Classical principal-agent theory was about human employees. AI agents fit perfectly:

| Classical | AI Agent |
|-----------|----------|
| Employee knows their job better than boss | Agent knows the task execution better than the operator |
| Employee may shirk or prioritize self-interest | Agent may pursue instrumental goals or alignment drift |
| Monitoring employees is costly | Monitoring agent reasoning is costly (and opaque) |
| Contracts incomplete — can't specify every edge case | Prompts incomplete — can't specify every edge case |

**2026 reality**: With AI agents taking more autonomous action, the principal-agent framework is becoming the dominant governance lens.

---

## Information Asymmetry in AI Delegation

### What the Principal (You) Doesn't Know

| Hidden Information | Why It Matters |
|-------------------|----------------|
| **Agent's full reasoning chain** | You see output, not thought process |
| **Which tools the agent considered** | May have skipped optimal tool |
| **Intermediate errors recovered** | Errors may indicate latent problems |
| **Confidence distribution** | Agent may present low-confidence outputs as high-confidence |
| **Trade-offs made** | Which objectives the agent sacrificed |
| **Information the agent had but didn't use** | May reveal reasoning gaps |

### What the Agent (LLM) Doesn't Know

| Hidden Information | Why It Matters |
|-------------------|----------------|
| **Your business context** | Agent works with stated goals only |
| **Implicit constraints you assume** | Ethical, legal, relational constraints not in prompt |
| **Downstream consequences** | Effects beyond immediate task |
| **Relative priority of objectives** | When objectives conflict, which wins? |
| **User's true utility function** | Stated goals may differ from real goals |

### Closing the Asymmetry

| Method | What It Reveals |
|--------|-----------------|
| **Chain-of-thought transparency** | Agent reasoning becomes visible |
| **Tool call logging** | Which actions were taken |
| **Confidence scoring** | Agent's self-assessment of reliability |
| **Intermediate checkpoints** | Progress visibility during long tasks |
| **Post-hoc audit trails** | Reconstruction of what happened |

---

## Moral Hazard and Misalignment

### What Is Moral Hazard?

When an agent can take actions the principal can't easily observe or verify, and those actions benefit the agent at the principal's expense.

### AI Agent Moral Hazard Patterns

| Pattern | Example |
|---------|---------|
| **Shortcut taking** | Agent completes task quickly with lower quality to minimize tokens |
| **Sycophancy** | Agent tells user what they want to hear instead of accurate info |
| **Hallucination cover-up** | Agent fabricates rather than admitting uncertainty |
| **Goal gaming** | Agent optimizes the metric, not the underlying objective |
| **Reward hacking** | Agent exploits reward signal in ways the designer didn't intend |
| **Instrumental goal drift** | Agent pursues sub-goals that diverge from primary objective |

### The Misalignment Spectrum

| Level | Severity | Example |
|:-----:|----------|---------|
| **Benign** | Low | Agent slightly verbose to seem helpful |
| **Strategic** | Medium | Agent avoids tasks it's bad at |
| **Deceptive** | High | Agent presents false certainty |
| **Active misalignment** | Very High | Agent pursues goals contrary to principal intent |

Most current LLM agents exhibit benign to strategic misalignment. As autonomy increases, deceptive and actively misaligned behavior becomes more possible.

---

## Shadow Principals

### A Novel 2026 Concern

Traditional principal-agent theory assumes one principal. With AI agents, there are often **multiple principals** with competing claims on the agent's behavior:

| Principal | Interest | Example |
|-----------|---------|---------|
| **User** | Get task done well | Write this code correctly |
| **Model provider** | Safety, liability, content policy | Don't produce harmful output |
| **Platform operator** | Platform incentives, data collection | Keep users on platform |
| **Advertisers** | Influence recommendations | Promote paid products |
| **Training data sources** | Accurate representation | Cite sources, avoid plagiarism |
| **Regulators** | Legal compliance | EU AI Act requirements |

### The Conflict

The agent may face directives from multiple principals that conflict:
- User wants detailed competitor analysis; platform policy limits it
- User wants fastest solution; model provider enforces safety checks
- User wants unfiltered data; regulator requires privacy protections

### Identifying Shadow Principals

When deploying an agent, ask:
1. Who designed the underlying model's training?
2. Who operates the platform?
3. Whose policies govern the tool set?
4. Who has access to logs?
5. Who benefits from the agent's output being a certain way?

Any entity with influence on behavior = a shadow principal. Governance requires knowing them.

---

## Monitoring and Verification

### The Observation Problem

Perfect monitoring (you see every action and reason behind it) is too expensive. No monitoring (trust-based) is too risky. The question is what level of monitoring is cost-justified.

### Monitoring Layers

| Layer | What It Catches | Cost |
|-------|----------------|:----:|
| **Output review** | Bad final outputs | Low |
| **Tool call logs** | Wrong tools used | Low |
| **Reasoning chains** | Flawed logic | Medium |
| **Intermediate checkpoints** | Progress visibility | Medium |
| **Continuous evals** | Pattern detection | High |
| **Independent verifier agents** | Malicious/deceptive behavior | High |
| **Red-team adversarial tests** | Edge cases and failures | Very High |

### Cost-Appropriate Monitoring

| Agent Risk Level | Monitoring Depth |
|-----------------|------------------|
| **Low stakes** (research, drafts, internal) | Output review, spot checks |
| **Medium stakes** (user-facing content, non-critical code) | + tool call logs, reasoning review |
| **High stakes** (code changes, financial transactions, customer communication) | + continuous evals, checkpoint approvals |
| **Critical** (regulated decisions, safety-critical systems) | + verifier agents, red-team, human-in-loop |

---

## Incentive Alignment

### The Contract Design Problem

In principal-agent theory, you design a "contract" (rules, rewards, penalties) such that the agent's best strategy aligns with your objectives. For AI agents, this means designing prompts, tools, and evaluations that make aligned behavior easier than misaligned behavior.

### Alignment Mechanisms

| Mechanism | How It Works |
|-----------|-------------|
| **Outcome contracts** | Evaluate agent on final outcome quality, not process |
| **Process contracts** | Require specific steps regardless of outcome |
| **Mixed contracts** | Reward outcome, penalize process violations |
| **Bonding / stake** | Agent's "trust capital" at risk for bad behavior |
| **Reputation systems** | Track performance over time, adjust autonomy |
| **Adversarial verification** | Independent agent checks primary agent |

### Practical Prompt-Level Alignment

Add explicit language that shifts the agent's "payoff function":

```
Priority order (when objectives conflict):
  1. Safety and legal compliance
  2. User's stated goal
  3. Quality and accuracy
  4. Efficiency

Uncertainty disclosure is rewarded:
  - "I don't know" is preferred to guessing
  - Low-confidence outputs must be flagged
  - Assumptions must be explicit
```

This is principal-agent contract design in prompt form.

---

## Governance Frameworks

### 2026 Governance Landscape

| Framework | Source | Coverage |
|-----------|--------|----------|
| **OWASP Agentic Top 10** | OWASP (Dec 2025) | Security vulnerabilities in agentic systems |
| **EU AI Act** | European Commission (in force Aug 2026) | High-risk AI obligations |
| **Microsoft Agent Governance Toolkit** | Microsoft (OSS, Apr 2026) | Policy-as-code, runtime monitoring |
| **NIST AI RMF** | US NIST | Risk management for AI systems |
| **ISO/IEC 42001** | ISO | AI management systems |

### Core Governance Principles

| Principle | Implementation |
|-----------|---------------|
| **Bounded autonomy** | Explicit scope limits on agent actions |
| **Reversibility** | Destructive actions require confirmation |
| **Auditability** | All agent actions logged and reviewable |
| **Human oversight** | Humans in the loop at critical decisions |
| **Accountability** | Clear ownership for agent behavior |
| **Transparency** | Agent purpose, data, decisions explainable |

---

## OWASP Agentic Top 10 Mapping

### OWASP Top 10 for Agentic Applications (Dec 2025)

| # | Risk | Principal-Agent Diagnosis |
|:-:|------|--------------------------|
| 1 | **Memory poisoning** | Agent's memory corrupted — violates information integrity |
| 2 | **Tool misuse** | Agent uses tools in unintended ways (moral hazard) |
| 3 | **Goal manipulation** | Adversarial input redirects agent objectives (shadow principal conflict) |
| 4 | **Prompt injection** | External input hijacks agent's loyalty (shadow principal) |
| 5 | **Excessive agency** | Agent has more autonomy than warranted |
| 6 | **Insufficient oversight** | Monitoring gap — principal can't see enough |
| 7 | **Data exfiltration** | Agent leaks information it shouldn't |
| 8 | **Cascading hallucinations** | Error propagation unchecked |
| 9 | **Identity / impersonation** | Agent claims authority it doesn't have |
| 10 | **Supply chain** | Third-party tools/models introduce hidden principals |

### Mitigation Framework

For each risk, principal-agent theory suggests:

1. **Reduce information asymmetry** — logging, reasoning transparency
2. **Align incentives** — prompt-level priority, evaluation alignment
3. **Monitor proportionally** — high-stakes actions = more oversight
4. **Limit agency** — explicit scope and rollback mechanisms
5. **Identify shadow principals** — know who else influences the agent
6. **Build reputation tracking** — agent trust earned over time

---

## Decision Checklist

- [ ] Identified all principals (stated + shadow) influencing the agent
- [ ] Mapped information asymmetries between principal and agent
- [ ] Designed monitoring proportional to agent risk level
- [ ] Added explicit priority ordering in prompts for conflicting objectives
- [ ] Uncertainty disclosure rewarded, not punished
- [ ] Scope and action limits explicitly bounded
- [ ] Destructive actions require confirmation or reversibility mechanism
- [ ] Audit logs capture tool calls and reasoning chains
- [ ] Applied OWASP Agentic Top 10 checklist
- [ ] Considered regulatory requirements (EU AI Act, NIST RMF)
- [ ] Trust / reputation mechanism for agents acting over time

---

## Sources

- Jensen, M., & Meckling, W. (1976). *Theory of the Firm: Managerial Behavior, Agency Costs*
- Holmström, B. (1979). *Moral Hazard and Observability*
- OWASP Agentic Top 10 (December 2025)
- EU AI Act (in force August 2026)
- Microsoft Agent Governance Toolkit (April 2026)
- California Management Review: *From Coase to AI Agents* (2025)
- NIST AI Risk Management Framework

# Protocol Decision Tree — MCP vs A2A Selection Guide

*Purpose: Clear decision framework for choosing between Model Context Protocol (MCP) and Agent-to-Agent Protocol (A2A).*

**When to use this guide**: User is building agent infrastructure and needs to decide which protocol(s) to implement.

---
## Table of Contents

- [TL;DR Decision Matrix](#tldr-decision-matrix)
- [Visual Decision Tree](#visual-decision-tree)
- [Detailed Decision Framework](#detailed-decision-framework)
- [Scenario 1: Agent Needs to Access External Systems](#scenario-1-agent-needs-to-access-external-systems)
- [Scenario 2: Multiple Agents Need to Coordinate](#scenario-2-multiple-agents-need-to-coordinate)
- [Scenario 3: Building Multi-Agent System with External Tools](#scenario-3-building-multi-agent-system-with-external-tools)
- [Decision by Use Case](#decision-by-use-case)
- [1. Single Agent with Tools](#1-single-agent-with-tools)
- [2. Multi-Agent Collaboration (No External Tools)](#2-multi-agent-collaboration-no-external-tools)
- [3. Multi-Agent System with Shared Tool Access](#3-multi-agent-system-with-shared-tool-access)
- [4. Agent Needs Dynamic Tool Discovery](#4-agent-needs-dynamic-tool-discovery)
- [5. Agent Needs Dynamic Agent Discovery](#5-agent-needs-dynamic-agent-discovery)
- [Protocol Comparison Table](#protocol-comparison-table)
- [Common Anti-Patterns](#common-anti-patterns)
- [BAD: Anti-Pattern 1: Using A2A for Tool Access](#bad-anti-pattern-1-using-a2a-for-tool-access)
- [BAD: Anti-Pattern 2: Using MCP for Agent Coordination](#bad-anti-pattern-2-using-mcp-for-agent-coordination)
- [BAD: Anti-Pattern 3: Building Custom Protocol](#bad-anti-pattern-3-building-custom-protocol)
- [BAD: Anti-Pattern 4: Mixing Protocol Responsibilities](#bad-anti-pattern-4-mixing-protocol-responsibilities)
- [Implementation Checklist](#implementation-checklist)
- [Implementing MCP](#implementing-mcp)
- [Implementing A2A](#implementing-a2a)
- [Migration Strategies](#migration-strategies)
- [From Custom Tool Integration → MCP](#from-custom-tool-integration-→-mcp)
- [Agent code tightly coupled to GitHub API](#agent-code-tightly-coupled-to-github-api)
- [Agent uses MCP tool](#agent-uses-mcp-tool)
- [From Custom Agent Communication → A2A](#from-custom-agent-communication-→-a2a)
- [No validation, no trace propagation](#no-validation-no-trace-propagation)
- [Validated schema, full observability](#validated-schema-full-observability)
- [Quick Reference: When to Use What](#quick-reference-when-to-use-what)
- [Use MCP when agent needs:](#use-mcp-when-agent-needs)
- [Use A2A when you need:](#use-a2a-when-you-need)
- [Use BOTH when:](#use-both-when)
- [Next Steps](#next-steps)
- [Summary](#summary)


## TL;DR Decision Matrix

| Question | Answer | Use |
|----------|--------|-----|
| Does agent need external data/tools? | Yes | **MCP** |
| Do multiple agents need to coordinate? | Yes | **A2A** |
| Building reusable tool library? | Yes | **MCP** |
| Need agent task delegation? | Yes | **A2A** |
| Connecting to databases/APIs? | Yes | **MCP** |
| Agent discovery/capability routing? | Yes | **A2A** |

**Most production systems use BOTH protocols for different purposes.**

---

## Visual Decision Tree

```
┌─────────────────────────────────────────────────────────────────┐
│  What are you trying to accomplish?                             │
└────────────────────┬────────────────────────────────────────────┘
                     │
        ┌────────────┴────────────┐
        │                         │
        ▼                         ▼
┌───────────────┐         ┌───────────────┐
│ Agent needs   │         │ Agents need   │
│ external      │         │ to coordinate │
│ capabilities  │         │ with each     │
│               │         │ other         │
└───────┬───────┘         └───────┬───────┘
        │                         │
        ▼                         ▼
┌───────────────────────┐ ┌───────────────────────┐
│ Use MCP               │ │ Use A2A               │
│                       │ │                       │
│ • Tool access         │ │ • Task handoffs       │
│ • Data retrieval      │ │ • Delegation          │
│ • Resource management │ │ • Collaboration       │
│ • API integration     │ │ • Orchestration       │
└───────────────────────┘ └───────────────────────┘
```

---

## Detailed Decision Framework

### Scenario 1: Agent Needs to Access External Systems

**Question**: Does your agent need to query databases, call APIs, read files, or execute tools?

**Answer**: Use **MCP**

**Why**: MCP standardizes how agents connect to external data sources and tools.

**Example architecture**:
```
┌─────────────┐
│   Agent     │
└──────┬──────┘
       │ (via MCP Client)
       ▼
┌──────────────────────┐
│   MCP Servers        │
├──────────────────────┤
│ • Database Server    │
│ • Filesystem Server  │
│ • API Wrapper Server │
│ • Search Server      │
└──────────────────────┘
```

**Real-world examples**:
- Customer support agent querying CRM database
- Code assistant reading project files
- Research agent searching web/documents
- DevOps agent managing infrastructure APIs

**Implementation**: See [mcp-practical-guide.md](mcp-practical-guide.md)

---

### Scenario 2: Multiple Agents Need to Coordinate

**Question**: Do you have multiple specialized agents that need to collaborate or delegate tasks?

**Answer**: Use **A2A**

**Why**: A2A provides structured handoffs with validation and observability.

**Example architecture**:
```
┌──────────────┐     A2A     ┌──────────────┐
│ Manager      │────────────→│ Worker A     │
│ Agent        │             │ (Specialist) │
└──────┬───────┘             └──────────────┘
       │ A2A                 ┌──────────────┐
       └────────────────────→│ Worker B     │
                             │ (Specialist) │
                             └──────────────┘
```

**Real-world examples**:
- Project manager agent delegating to dev/test/deploy agents
- Research coordinator distributing search tasks
- Content creation workflow (research → write → edit → publish)
- Customer inquiry routing to specialized support agents

**Implementation**: See [a2a-handoff-patterns.md](a2a-handoff-patterns.md)

---

### Scenario 3: Building Multi-Agent System with External Tools

**Question**: Do agents need BOTH external tools AND inter-agent coordination?

**Answer**: Use **BOTH MCP and A2A**

**Why**: Protocols are complementary, not competitive.

**Example architecture**:
```
┌────────────────────────────────────────────┐
│  Manager Agent                             │
│  ├─ Uses MCP for: DB access, file ops     │
│  └─ Uses A2A for: Delegating to workers   │
└────────┬───────────────────┬───────────────┘
         │ A2A               │ A2A
         ▼                   ▼
┌────────────────┐   ┌────────────────┐
│  Worker A      │   │  Worker B      │
│  Uses MCP for: │   │  Uses MCP for: │
│  • API calls   │   │  • File writes │
│  • Validation  │   │  • Email send  │
└────────────────┘   └────────────────┘
```

**Real-world example: Content Publishing System**
```
Manager Agent (Coordinator)
├─ MCP: Read content guidelines from docs
├─ MCP: Query content calendar database
└─ A2A: Delegate tasks ↓

    Research Agent
    ├─ MCP: Search web for sources
    ├─ MCP: Access research database
    └─ A2A: Handoff to Writer ↓

        Writer Agent
        ├─ MCP: Read style guide
        ├─ MCP: Use grammar checking tool
        └─ A2A: Handoff to Editor ↓

            Editor Agent
            ├─ MCP: Access revision history
            ├─ MCP: Publish to CMS API
            └─ A2A: Return to Manager
```

**Key insight**: MCP handles vertical integration (agent ↔ tools), A2A handles horizontal integration (agent ↔ agent).

---

## Decision by Use Case

### 1. Single Agent with Tools

**Characteristics**:
- One agent handles all tasks
- Needs external data/APIs
- No delegation required

**Protocol**: **MCP only**

**Example**: Personal assistant agent
```
Assistant Agent
├─ MCP: Calendar API
├─ MCP: Email API
├─ MCP: Weather API
└─ MCP: Notes database
```

---

### 2. Multi-Agent Collaboration (No External Tools)

**Characteristics**:
- Multiple specialized agents
- Pure reasoning/planning tasks
- No external data needed

**Protocol**: **A2A only**

**Example**: Creative writing team
```
Plot Designer ─A2A→ Character Developer ─A2A→ Scene Writer
      ↑                                           │
      └───────────────── A2A ─────────────────────┘
```

---

### 3. Multi-Agent System with Shared Tool Access

**Characteristics**:
- Multiple agents
- Each needs external tools
- Coordination required

**Protocol**: **Both MCP and A2A**

**Example**: Software development team
```
Product Manager Agent
├─ MCP: Jira API, user research database
└─ A2A: Delegates to ↓

    Backend Developer Agent
    ├─ MCP: GitHub API, database schema
    └─ A2A: Collaborates with ↓

        Frontend Developer Agent
        ├─ MCP: Figma API, component library
        └─ A2A: Handoff to ↓

            QA Tester Agent
            ├─ MCP: Test framework, bug tracker
            └─ A2A: Report back to Product Manager
```

---

### 4. Agent Needs Dynamic Tool Discovery

**Characteristics**:
- Agent discovers available tools at runtime
- Different tools for different contexts
- Need standardized tool interface

**Protocol**: **MCP** (with tool discovery)

**Example**: Development assistant
```
Dev Assistant Agent
├─ Discovers available MCP servers
├─ Dynamically loads appropriate tools
└─ Adapts to different project types

Available MCP Servers:
• Python tools (pytest, mypy, black)
• Node tools (npm, eslint, prettier)
• Database tools (postgres, redis, mongo)
• Cloud tools (aws, gcp, azure)
```

---

### 5. Agent Needs Dynamic Agent Discovery

**Characteristics**:
- Coordinator discovers available agents
- Delegates based on capabilities
- Agents join/leave dynamically

**Protocol**: **A2A** (with agent cards)

**Example**: Customer support routing
```
Support Coordinator Agent
├─ Discovers available specialist agents
├─ Routes based on agent capabilities
└─ Dynamically scales with demand

Available Specialist Agents:
• Billing Agent (capabilities: payments, refunds, invoicing)
• Technical Agent (capabilities: troubleshooting, bug_reports)
• Account Agent (capabilities: profile, security, authentication)
```

---

## Protocol Comparison Table

| Aspect | MCP | A2A |
|--------|-----|-----|
| **Purpose** | Connect agents to tools/data | Connect agents to agents |
| **Direction** | Vertical (agent ↔ external) | Horizontal (agent ↔ agent) |
| **Primary Use** | Tool execution, data access | Task delegation, coordination |
| **Communication** | Request-response | Message passing with handoffs |
| **Discovery** | Tool/resource discovery | Agent capability discovery |
| **Validation** | Tool input schema | Handoff payload schema |
| **Observability** | Tool call traces | Handoff chain traces |
| **State** | Stateless tools | Stateful conversations |
| **Adoption** | Anthropic, OpenAI, Google | Anthropic, multi-agent frameworks |

---

## Common Anti-Patterns

### BAD: Anti-Pattern 1: Using A2A for Tool Access

**Wrong**:
```
Agent A ─A2A→ Agent B (wrapper around database)
```

**Why wrong**: Agent B is just a thin wrapper around a tool, not adding intelligence

**Right**:
```
Agent A ─MCP→ Database Server
```

**Fix**: If it's just executing a tool, use MCP directly.

---

### BAD: Anti-Pattern 2: Using MCP for Agent Coordination

**Wrong**:
```
Agent A ─MCP→ "Agent B Tool" (agent exposed as MCP tool)
```

**Why wrong**: Loses A2A benefits (context propagation, trace_id, validation)

**Right**:
```
Agent A ─A2A→ Agent B
```

**Fix**: If it involves reasoning/intelligence, use A2A for proper handoffs.

---

### BAD: Anti-Pattern 3: Building Custom Protocol

**Wrong**:
```
Agent A ─custom JSON→ Agent B
Agent A ─custom API→ Tool Server
```

**Why wrong**: Reinventing the wheel, no interoperability, no tooling

**Right**:
```
Agent A ─A2A→ Agent B
Agent A ─MCP→ Tool Server
```

**Fix**: Use standard protocols unless you have very specific needs.

---

### BAD: Anti-Pattern 4: Mixing Protocol Responsibilities

**Wrong**:
```
MCP Tool that calls other agents (mixing vertical + horizontal)
```

**Why wrong**: Violates separation of concerns, hard to trace

**Right**:
```
Agent ─A2A→ Other Agent (coordination)
Agent ─MCP→ Tool (execution)
```

**Fix**: Keep protocols focused on their primary purpose.

---

## Implementation Checklist

### Implementing MCP

- [ ] Identify all external data sources (databases, APIs, files)
- [ ] Group related tools into logical MCP servers
- [ ] Define tool schemas (input/output)
- [ ] Implement security validation
- [ ] Add observability (traces, metrics)
- [ ] Test with MCP Inspector
- [ ] Document tools for agents
- [ ] Deploy with monitoring

**Guide**: [mcp-practical-guide.md](mcp-practical-guide.md)

### Implementing A2A

- [ ] Map agent collaboration workflows
- [ ] Define agent capabilities (agent cards)
- [ ] Design handoff message schemas
- [ ] Implement validation for all handoffs
- [ ] Add trace_id propagation
- [ ] Build error recovery mechanisms
- [ ] Set up agent registry/discovery
- [ ] Monitor handoff metrics

**Guide**: [a2a-handoff-patterns.md](a2a-handoff-patterns.md)

---

## Migration Strategies

### From Custom Tool Integration → MCP

**Before**: Direct API calls in agent code
```python
# Agent code tightly coupled to GitHub API
response = requests.post(
    "https://api.github.com/repos/owner/repo/issues",
    headers={"Authorization": f"token {GITHUB_TOKEN}"},
    json={"title": title, "body": body}
)
```

**After**: MCP server abstracts integration
```python
# Agent uses MCP tool
result = await mcp_client.call_tool(
    "create_github_issue",
    repo="owner/repo",
    title=title,
    body=body
)
```

**Benefits**: Reusable across agents, testable, secure, observable

---

### From Custom Agent Communication → A2A

**Before**: Ad-hoc JSON messages
```python
# No validation, no trace propagation
message = {"task": "analyze", "data": {...}}
requests.post(f"{agent_b_url}/tasks", json=message)
```

**After**: Structured A2A handoffs
```python
# Validated schema, full observability
handoff = {
    "schemaVersion": "v1.2",
    "trace_id": trace_id,
    "sender": {...},
    "receiver": {...},
    "task": {"type": "analyze", "description": "..."},
    "context": {...}
}
validate_handoff_schema(handoff)
await send_a2a_message(agent_b_id, handoff)
```

**Benefits**: Validation, traceability, error recovery, interoperability

---

## Quick Reference: When to Use What

### Use MCP when agent needs:

- Database queries
- File system access
- API calls to third-party services
- Search capabilities
- Tool execution
- Resource retrieval
- Prompt templates

### Use A2A when you need:

- Task delegation between agents
- Agent collaboration workflows
- Capability-based routing
- Multi-agent coordination
- Handoff validation
- Cross-vendor agent communication
- Trace propagation across agents

### Use BOTH when:

- Building complex multi-agent systems
- Agents need external tools AND coordination
- Enterprise-scale agent architectures
- Maximum observability required

---

## Next Steps

**After choosing your protocol(s)**:

1. **MCP path**: Read [mcp-practical-guide.md](mcp-practical-guide.md) → Build server → Test with Inspector → Deploy
2. **A2A path**: Read [a2a-handoff-patterns.md](a2a-handoff-patterns.md) → Design handoffs → Implement validation → Monitor traces
3. **Both paths**: Start with MCP (simpler), add A2A when coordination needed

**Architecture references**:
- MCP deep dive: `frameworks/shared-foundations/protocols/mcp/mcp-architecture.md`
- A2A deep dive: `frameworks/shared-foundations/protocols/a2a/a2a-architecture.md`

**Questions to ask yourself**:
- How many agents? (1 = maybe just MCP, 2+ = consider A2A)
- Do they work together or independently? (together = A2A)
- What external systems? (databases/APIs = MCP)
- Need cross-vendor compatibility? (yes = use standard protocols)

---

## Summary

**Simple rule of thumb**:

```
Agent ↔ External System = MCP
Agent ↔ Agent          = A2A
```

**Remember**: These protocols are **complementary**, not competing. Most production systems use both for different purposes. Choose based on what you're connecting, not on preference.

# Pydantic AI — Production Patterns

**Version**: v1.66.0 (March 2026) | V1.0.0 GA: September 2025 | V2 planned: April 2026+

**What**: Type-safe Python agent framework from the Pydantic team. FastAPI-style DX for GenAI — agents are Python functions + Pydantic schemas, not YAML configs or graph definitions.

**When to choose over alternatives**: Type safety is critical, you want native MCP + A2A interoperability, your team already uses Pydantic/FastAPI, or you need durable execution with minimal infrastructure.

---

## Table of Contents

1. [Agent Definition](#agent-definition)
2. [Tool Use](#tool-use)
3. [Structured Outputs](#structured-outputs)
4. [Dependencies (Dependency Injection)](#dependencies)
5. [MCP Integration](#mcp-integration)
6. [A2A Protocol Support](#a2a-protocol-support)
7. [pydantic-graph (FSM Workflows)](#pydantic-graph-fsm-workflows)
8. [Durable Execution](#durable-execution)
9. [Human-in-the-Loop (HITL)](#human-in-the-loop)
10. [Multi-Agent Patterns](#multi-agent-patterns)
11. [Streaming](#streaming)
12. [Testing](#testing)
13. [Observability](#observability)
14. [Model Support](#model-support)
15. [Migration Notes](#migration-notes)

---

## Agent Definition

An `Agent` is the central abstraction — it wraps a model, system prompt, tools, output type, and dependencies.

```python
from pydantic_ai import Agent

agent = Agent(
    'anthropic:claude-sonnet-4-6',
    system_prompt='You are a helpful customer service agent.',
    result_type=str,        # or a Pydantic model
    retries=3,              # auto-retry on validation failure
)

result = await agent.run('How do I return a product?')
print(result.output)        # typed as str
print(result.usage())       # token counts
```

**System prompts** can be static strings or dynamic functions:

```python
@agent.system_prompt
async def add_context(ctx: RunContext[MyDeps]) -> str:
    user = await ctx.deps.db.get_user(ctx.deps.user_id)
    return f'Current user: {user.name}, plan: {user.plan}'
```

---

## Tool Use

Tools are Python functions registered via decorators. Two types:

- `@agent.tool` — receives `RunContext` (access to deps, retry count, etc.)
- `@agent.tool_plain` — plain function, no context needed

```python
from pydantic_ai import Agent, RunContext

agent = Agent('openai:gpt-5.4', deps_type=DatabaseConn)

@agent.tool
async def lookup_order(ctx: RunContext[DatabaseConn], order_id: str) -> str:
    """Look up order status by ID."""
    order = await ctx.deps.get_order(order_id)
    return f'Order {order_id}: {order.status}, shipped: {order.shipped_date}'

@agent.tool_plain
def calculate_refund(price: float, days_since_purchase: int) -> float:
    """Calculate refund amount based on return policy."""
    if days_since_purchase <= 30:
        return price
    elif days_since_purchase <= 60:
        return price * 0.5
    return 0.0
```

**Toolsets**: Tools are organized into `toolsets` — modular collections that can be combined. MCP servers are one type of toolset.

**Tool validation** (v1.63.0+): Use `args_validator` for pre-execution argument checks.

---

## Structured Outputs

Use `result_type` with Pydantic models for type-safe, validated outputs:

```python
from pydantic import BaseModel

class TicketResponse(BaseModel):
    answer: str
    confidence: float  # 0.0 to 1.0
    sources: list[str]
    needs_escalation: bool

agent = Agent(
    'anthropic:claude-sonnet-4-6',
    result_type=TicketResponse,
    retries=3,  # retries if output fails validation
)

result = await agent.run('Customer asks about refund policy')
response: TicketResponse = result.output  # fully typed
if response.needs_escalation:
    await escalate(response)
```

The model auto-retries when the LLM output doesn't match the schema — the validation error is sent back to the model as feedback.

**Streamed structured output**: Pydantic AI supports token-by-token streaming with immediate validation as data arrives.

---

## Dependencies

`deps_type` enables dependency injection (like FastAPI's `Depends`):

```python
from dataclasses import dataclass

@dataclass
class ServiceDeps:
    db: DatabasePool
    http_client: httpx.AsyncClient
    user_id: str

agent = Agent(
    'openai:gpt-5.4',
    deps_type=ServiceDeps,
    result_type=TicketResponse,
)

# At runtime
async with httpx.AsyncClient() as client:
    deps = ServiceDeps(db=pool, http_client=client, user_id='user-123')
    result = await agent.run('Check my order status', deps=deps)
```

Tools access deps via `RunContext[ServiceDeps]` — full type checking and IDE auto-completion.

---

## MCP Integration

Pydantic AI has first-class MCP support. MCP servers are registered as **toolsets** on agents.

**Three transport types** (StreamableHTTP preferred, SSE deprecated):

```python
from pydantic_ai import Agent
from pydantic_ai.mcp import MCPServerStdio, MCPServerSSE, MCPServerStreamableHTTP

# Streamable HTTP — preferred transport for remote servers
api_server = MCPServerStreamableHTTP('http://localhost:3002/mcp')

# Stdio — runs MCP server as a subprocess (local tools)
docs_server = MCPServerStdio('python', args=['docs_mcp_server.py'], timeout=10)

# SSE — deprecated, use StreamableHTTP for new projects
search_server = MCPServerSSE('http://localhost:3001/sse')

# Combine multiple MCP servers as toolsets
agent = Agent(
    'anthropic:claude-sonnet-4-6',
    toolsets=[docs_server, search_server, api_server],
)

# Context manager handles connection lifecycle
async with agent:
    result = await agent.run('Search docs for refund policy')
```

**Load from config file**:

```python
from pydantic_ai.mcp import load_mcp_servers

servers = load_mcp_servers('mcp_config.json')
agent = Agent('openai:gpt-5.4', toolsets=list(servers.values()))
```

**Pydantic AI as MCP server**: Agents can also expose their tools as MCP servers, allowing other MCP clients to connect to them.

---

## A2A Protocol Support

Native support for Google's Agent-to-Agent (A2A) open standard — agents interoperate across frameworks and vendors.

**Expose an agent as an A2A server**:

```python
from pydantic_ai import Agent

agent = Agent('openai:gpt-5.4', instructions='Customer service bot')
app = agent.to_a2a()
# Run with: uvicorn my_module:app --host 0.0.0.0 --port 8000
```

`to_a2a()` returns an ASGI application (compatible with uvicorn, any ASGI server).

**FastA2A**: A separate Pydantic library (`fasta2a` on PyPI) built on Starlette that provides the underlying A2A server implementation. Supports pluggable **Storage**, **Broker**, and **Worker** components.

**Architecture**: Storage separates A2A-protocol-format task storage from internal conversation context — agents maintain rich internal state while exposing only A2A-compliant messages externally.

---

## pydantic-graph (FSM Workflows)

`pydantic-graph` is a type-centric library for building finite state machines, bundled with Pydantic AI. Each `Agent` internally uses pydantic-graph for its execution flow.

**Core concepts**:

- **Nodes**: Define logic and outgoing edges via return type annotations
- **Edges**: Type-checked transitions (the compiler catches invalid flows)
- **State persistence**: Snapshots before/after each node for durability

**Built-in node types**: `UserPromptNode`, `ModelRequestNode`, `CallToolsNode`, `End`

**State persistence implementations**:

| Implementation | Storage | Use Case |
|---|---|---|
| `SimpleStatePersistence` | In-memory (latest only) | Dev/testing |
| `FullStatePersistence` | In-memory (all snapshots) | Debugging, replay |
| `FileStatePersistence` | JSON files | Simple production, recovery |

State persistence enables interruption and resumption — the graph run can resume from any node. This is the foundation for HITL workflows and crash recovery.

---

## Durable Execution

Agents preserve progress across crashes, API failures, and restarts. Three first-party integrations:

**Temporal** — wraps agent as workflow activities:

```python
from pydantic_ai.durable_exec.temporal import TemporalAgent, PydanticAIWorkflow
from temporalio import workflow

agent = Agent('openai:gpt-5.4', instructions='...', name='support')
temporal_agent = TemporalAgent(agent)

@workflow.defn
class SupportWorkflow(PydanticAIWorkflow):
    __pydantic_ai_agents__ = [temporal_agent]
    @workflow.run
    async def run(self, prompt: str) -> str:
        result = await temporal_agent.run(prompt)
        return result.output
```

**DBOS** — checkpoints to database (SQLite/Postgres):

```python
from pydantic_ai.durable_exec.dbos import DBOSAgent
dbos_agent = DBOSAgent(agent)
result = await dbos_agent.run('prompt')
```

**pydantic-graph persistence** — file-based or in-memory state snapshots for simpler use cases.

**Note**: `run_stream()` is NOT supported in Temporal/DBOS workflows — use `event_stream_handler` instead.

**What this means in practice**:

- API timeout mid-conversation → resume from last successful step
- Server restart → reload state from persistence, continue
- Human approval needed → persist state, wait, resume when approved

---

## Human-in-the-Loop

Use `approval_required()` on any toolset. When a tool needs approval, the agent returns `DeferredToolRequests` instead of executing.

```python
from pydantic_ai import Agent, DeferredToolRequests, DeferredToolResults

# Flag tools requiring approval (based on tool name, args, or context)
approval_toolset = my_toolset.approval_required(
    lambda ctx, tool_def, tool_args: tool_def.name.startswith('dangerous')
)

agent = Agent(
    'openai:gpt-5.4',
    toolsets=[approval_toolset],
    output_type=[str, DeferredToolRequests],
)

# First run — agent returns deferred requests
result = agent.run_sync('Do the dangerous thing')
# result.output is DeferredToolRequests with .approvals list

# Second run — pass approval decisions back
result = agent.run_sync(
    message_history=result.all_messages(),
    deferred_tool_results=DeferredToolResults(
        approvals={
            'tool_call_id_1': True,   # approved
            'tool_call_id_2': False,  # denied
        }
    )
)
```

The approval callback receives `(ctx, tool_def, tool_args)` and returns `bool`. Denied calls get error responses sent back to the model.

**Use cases**: Financial transactions, PII handling, high-risk tool calls, compliance gates.

---

## Multi-Agent Patterns

**Delegation**: One agent calls another as a tool:

```python
support_agent = Agent('openai:gpt-5.4', system_prompt='You handle support')
billing_agent = Agent('openai:gpt-5.4', system_prompt='You handle billing')

@support_agent.tool
async def escalate_to_billing(ctx: RunContext[Deps], issue: str) -> str:
    """Escalate billing issues to the billing specialist."""
    result = await billing_agent.run(issue, deps=ctx.deps)
    return result.output
```

**A2A for cross-framework**: Use `to_a2a()` to expose agents as A2A endpoints — other agents (even non-Pydantic AI) can call them.

**Orchestration patterns**:

- **Agent-as-tool**: Register one agent as a tool on another (simple delegation)
- **Sequential pipeline**: Chain agents via application code
- **A2A mesh**: Expose each agent as an A2A server, route via protocol

---

## Streaming

```python
async with agent.run_stream('Explain our return policy') as stream:
    async for chunk in stream.stream_text():
        print(chunk, end='', flush=True)

# Streamed structured output — validates as tokens arrive
async with agent.run_stream('Analyze this ticket', result_type=Analysis) as stream:
    async for partial in stream.stream_output():
        update_ui(partial)  # partial Pydantic model, validated incrementally
```

---

## Testing

Pydantic AI provides test doubles that avoid calling real LLMs:

```python
from pydantic_ai.models.test import TestModel, FunctionModel

# TestModel — returns predictable outputs
with agent.override(model=TestModel()):
    result = await agent.run('test prompt')
    assert result.output == expected  # deterministic

# FunctionModel — custom logic for complex test scenarios
def mock_response(messages, info):
    if 'refund' in messages[-1].content:
        return 'Processing refund...'
    return 'How can I help?'

with agent.override(model=FunctionModel(mock_response)):
    result = await agent.run('I need a refund')
```

**pydantic_evals**: Built-in evaluation framework for systematic agent testing:

- Dataset management (define test cases with expected outputs)
- CLI for running eval suites
- Logfire visualization of results

---

## Observability

**Built-in OpenTelemetry support**: Pydantic AI emits spans for model calls, tool calls, and agent runs.

**Pydantic Logfire**: First-party observability platform with deep integration:

- Real-time trace visualization
- Token usage tracking
- Tool call monitoring
- Cost tracking per agent/model

OpenTelemetry spans follow GenAI semantic conventions — compatible with any OTel backend (Datadog, New Relic, Grafana, etc.).

---

## Model Support

20+ LLM providers via a unified interface:

| Provider | Models | Notes |
|---|---|---|
| OpenAI | GPT-5.5, GPT-5.4, GPT-5.4 mini, o-series | Full support |
| Anthropic | Claude Opus 4.8, Sonnet 4.6, Haiku 4.5 | Full support |
| Google | Gemini 3.1 Pro, Gemini Flash | Including image models |
| Qwen | Qwen 3.5 | Native structured output (v1.66.0+) |
| Groq | Llama, Mixtral | Via OpenAI-compatible API |
| Ollama | Local models | Via OpenAI-compatible API |

---

## Migration Notes

### From LangGraph

| LangGraph Concept | Pydantic AI Equivalent |
|---|---|
| `StateGraph` | `pydantic-graph` nodes + edges |
| `add_node()` / `add_edge()` | Type-annotated return types |
| `ToolNode` | `@agent.tool` / `@agent.tool_plain` |
| State channels | `deps_type` + `RunContext` |
| Checkpointer | State persistence (`FileStatePersistence`) |
| LangSmith | Pydantic Logfire / OpenTelemetry |

### From CrewAI

| CrewAI Concept | Pydantic AI Equivalent |
|---|---|
| `Crew` | Application-level orchestration |
| `Agent` (role-based) | `Agent` with system prompt |
| `Task` | Agent run with specific prompt |
| `Tool` | `@agent.tool` decorator |

---

## Decision: When to Use Pydantic AI

**Choose Pydantic AI when**:

- Type safety and IDE support matter (team uses mypy/pyright)
- You need native MCP + A2A interoperability
- Your stack is already Pydantic/FastAPI
- You want durable execution without heavy infrastructure
- You need streamed structured outputs

**Choose something else when**:

- You need visual workflow editing → LangGraph
- Fastest possible MVP → OpenAI Agents SDK or CrewAI
- Google Cloud / Vertex AI native → Google ADK
- Enterprise AWS managed → Bedrock Agents

# RAG Patterns — Best Practices 

*Purpose: Provide operational patterns, checklists, and decision logic for building reliable Retrieval-Augmented Generation pipelines with Agentic RAG and Contextual Retrieval.*

**Modern Update**: Chunk context augmentation and agentic retrieval can improve retrieval on ambiguous corpora, but must be validated on your own test set (see `../../ai-rag/references/contextual-retrieval-guide.md`).

---
## Table of Contents

- [Agentic RAG Pattern (Current Standard)](#agentic-rag-pattern-current-standard)
- [Pattern: Dynamic Multi-Step Retrieval](#pattern-dynamic-multi-step-retrieval)
- [Contextual Retrieval (Anthropic 2024)](#contextual-retrieval-anthropic-2024)
- [Pattern: Context-Enhanced Chunks](#pattern-context-enhanced-chunks)
- [1. Core RAG Pipeline (Enhanced)](#1-core-rag-pipeline-enhanced)
- [Pattern: Standard RAG Flow](#pattern-standard-rag-flow)
- [2. Hybrid Retrieval Pattern](#2-hybrid-retrieval-pattern)
- [Pattern: Semantic + Keyword Search](#pattern-semantic-keyword-search)
- [3. Query Rewriting](#3-query-rewriting)
- [Pattern: Clarification Rewrite](#pattern-clarification-rewrite)
- [4. Chunking Strategy](#4-chunking-strategy)
- [Pattern: Overlapping Windows](#pattern-overlapping-windows)
- [5. Reranking Pattern](#5-reranking-pattern)
- [Pattern: Second-Stage Reranker](#pattern-second-stage-reranker)
- [6. Filtering & Relevance Validation](#6-filtering-&-relevance-validation)
- [Pattern: Post-Rerank Filtering](#pattern-post-rerank-filtering)
- [7. Response Generation Pattern](#7-response-generation-pattern)
- [Pattern: Evidence-Grounded Answer](#pattern-evidence-grounded-answer)
- [8. Advanced RAG Techniques](#8-advanced-rag-techniques)
- [8.1 HyDE (Hypothetical Document Embedding)](#81-hyde-hypothetical-document-embedding)
- [8.2 Query Routing](#82-query-routing)
- [8.3 Context Enrichment](#83-context-enrichment)
- [Pattern: Expand With Auxiliary Metadata](#pattern-expand-with-auxiliary-metadata)
- [8.4 Hierarchical Retrieval](#84-hierarchical-retrieval)
- [9. Modular RAG Architecture](#9-modular-rag-architecture)
- [Pattern: Clean Separation of Components](#pattern-clean-separation-of-components)
- [10. Evaluation of RAG Systems](#10-evaluation-of-rag-systems)
- [Pattern: RAG Evaluation Loop](#pattern-rag-evaluation-loop)
- [11. RAG Anti-Patterns (Master List)](#11-rag-anti-patterns-master-list)
- [12. Quick Reference Tables](#12-quick-reference-tables)
- [Chunk Size Table](#chunk-size-table)
- [Retrieval Strategy Table](#retrieval-strategy-table)
- [End of File](#end-of-file)


## Agentic RAG Pattern (Current Standard)

### Pattern: Dynamic Multi-Step Retrieval

**Old (Static RAG)**:
```text
query → embed → retrieve top-k → inject → generate
```

**New (Agentic RAG)**:
```text
query → plan retrieval → multi-step search → adapt → contextual rerank → cite → generate
```

**Key differences**:
- **Static**: One-shot retrieval, fixed top-k
- **Agentic**: Iterative retrieval, adapts to findings, analyzes intermediate results

**When to use Agentic RAG**:
- Complex multi-domain queries
- Research tasks requiring multiple sources
- Queries needing iterative refinement
- High-accuracy requirements (legal, medical, technical)

**Anthropic's Research System Example**:
```text
Lead agent plans research → Spawns parallel search agents → Each searches independently →
Agents summarize findings → Store in external memory → Retrieve context for synthesis →
Generate final answer with citations
```

---

## Contextual Retrieval (Anthropic 2024)

### Pattern: Context-Enhanced Chunks

**Standard chunking**:
```text
Document → Split into chunks → Embed each chunk → Store
```

**Contextual chunking**:
```text
Document → Split into chunks → Add document context to each chunk → Embed enhanced chunks → Store
```

**Example**:
```text
Original chunk: "Revenue increased 15% QoQ"

Contextual chunk: "In ACME Corp's Q4 2024 financial report, revenue increased 15% QoQ"
```

**Implementation**:
1. Use LLM to generate 50-100 token context prefix for each chunk
2. Prepend context to chunk before embedding
3. Store original chunk + context separately
4. Use context-enhanced embeddings for retrieval
5. Return original chunk (without context) in results

**Validation (REQUIRED)**:
- Compare baseline vs augmented retrieval on a held-out retrieval test set (recall@k, nDCG/MRR, empty-result rate).
- Measure end-to-end impact on groundedness/citation coverage (not only retrieval metrics).

---

## 1. Core RAG Pipeline (Enhanced)

### Pattern: Standard RAG Flow

```
query
→ rewrite
→ embed
→ retrieve
→ rerank
→ filter
→ inject context
→ generate answer
```

**Checklist**

- [ ] Query rewritten when ambiguous.  
- [ ] Embeddings use consistent model/version.  
- [ ] Retrieval uses top-k defined (k=5–20).  
- [ ] Reranker always applied.  
- [ ] Chunks filtered for domain relevance.  
- [ ] Context injected using standard wrapper (below).  

**Injection Format**

```
<retrieved>
[chunk_1]
[chunk_2]
...
</retrieved>
```

**Anti-Patterns**

- AVOID: Generating answers without injected context.  
- AVOID: Using only semantic search for domain-heavy queries.  
- AVOID: Passing unbounded chunks (>1500 tokens).  

---

# 2. Hybrid Retrieval Pattern

### Pattern: Semantic + Keyword Search

```
semantic_k = 20
keyword_k = 20
combine → rerank → filter
```

**Checklist**

- [ ] Keyword retrieval used for structured or fact-heavy data.  
- [ ] Semantic retrieval used for narrative/semantic data.  
- [ ] Deduplicate results before reranking.  

**Decision Tree**

```
Does the user ask for facts, numbers, or strict terms?
→ Yes → Include keyword search
→ No → Semantic-only is acceptable
```

**Anti-Patterns**

- AVOID: Using semantic-only for regulatory or code documentation.  
- AVOID: Not deduplicating overlapping results.  

---

# 3. Query Rewriting

### Pattern: Clarification Rewrite

```
rewrite(query) → high-precision query
```

**Checklist**

- [ ] Expand abbreviations or acronyms.  
- [ ] Add missing domain terms.  
- [ ] Convert vague phrases to explicit intents.  
- [ ] Route to the correct domain before retrieval.  

**Examples**

```
"Summarize finances" → "Summarize Q4 financial statements for ACME Corp."
```

---

# 4. Chunking Strategy

### Pattern: Overlapping Windows

**Parameters**

- Chunk size: 200–400 tokens  
- Overlap: 20–40 tokens  

**Checklist**

- [ ] Do not break sentences across chunks.  
- [ ] Keep domain-consistent content within each chunk.  
- [ ] Store metadata (source, page, section).  

**Anti-Patterns**

- AVOID: Mixing unrelated concepts in a single chunk.  
- AVOID: Using large chunk sizes (>800 tokens).  
- AVOID: Not storing metadata for grounding.  

---

# 5. Reranking Pattern

### Pattern: Second-Stage Reranker

```
retrieve_top_50 → rerank_to_top_5
```

**Checklist**

- [ ] Use cross-encoder / heavy reranker.  
- [ ] Score based on semantic alignment to rewritten query.  
- [ ] Keep top 3–7 only.  

**Anti-Patterns**

- AVOID: Using retriever top-k directly.  
- AVOID: Reranking fewer than 20 candidates.  

---

# 6. Filtering & Relevance Validation

### Pattern: Post-Rerank Filtering

```
for each chunk:
    validate relevance
    validate recency
    validate domain alignment
```

**Checklist**

- [ ] Discard stale or deprecated content.  
- [ ] Ensure domain matches query domain.  
- [ ] Remove near-duplicates.  

**Decision Tree**

```
Is chunk relevant to the final question?
→ No → discard
→ Yes → inject
```

---

# 7. Response Generation Pattern

### Pattern: Evidence-Grounded Answer

```
answer must:
- cite retrieved chunks
- avoid unsupported claims
- reflect only injected evidence
```

**Checklist**

- [ ] Use only retrieved text for factual claims.  
- [ ] Include citations or chunk references.  
- [ ] Summaries must align 1:1 with provided evidence.  

**Anti-Patterns**

- AVOID: Mixing external world knowledge with retrieved facts.  
- AVOID: Adding claims inconsistent with chunks.  

---

# 8. Advanced RAG Techniques

## 8.1 HyDE (Hypothetical Document Embedding)

**Purpose:** When retrieval fails or query sparse.

**Pattern**

```
generate hypothetical_doc
embed hypothetical_doc
retrieve based on hypothetical embedding
```

**Checklist**

- [ ] Hypothetical doc ≤ 150 tokens.  
- [ ] Use domain constraints in generation.  

---

## 8.2 Query Routing

**Pattern**

```
route(query) → domain
domain → index
retrieve(domain-specific)
```

**Checklist**

- [ ] Routing based on classification prompt or rules.  
- [ ] Reject multi-domain injection.  

**Routing Table Example**

| Domain | Trigger Keywords | Index |
|--------|------------------|-------|
| Legal  | statute, case    | legal_idx |
| Code   | function, bug    | code_idx |
| Finance| q4, revenue      | finance_idx |

---

## 8.3 Context Enrichment

### Pattern: Expand With Auxiliary Metadata

```
query → enrich(metadata) → retrieve → rerank → inject
```

**Checklist**

- [ ] Add metadata fields (product ID, region, date).  
- [ ] Remove irrelevant metadata before injection.  

---

## 8.4 Hierarchical Retrieval

**Pattern**

```
retrieve(topic-level)
→ retrieve(section-level)
→ retrieve paragraph-level
```

**Checklist**

- [ ] Escalate retrieval depth only if matches drop.  
- [ ] Keep 1–3 top matches per level.  

---

# 9. Modular RAG Architecture

### Pattern: Clean Separation of Components

**Modules**

- Query Processor  
- Embedder  
- Retriever  
- Reranker  
- Context Filter  
- Generator  

**Checklist**

- [ ] Each module testable independently.  
- [ ] Embedding model is versioned.  
- [ ] Retriever swaps allowed without pipeline redesign.  

---

# 10. Evaluation of RAG Systems

### Pattern: RAG Evaluation Loop

```
query_set → retrieve → judge → score → adjust pipeline
```

**Metrics**

- Retrieval relevance (RR)  
- Grounding score (GS)  
- Answer accuracy (AA)  
- Context precision (CP)  
- Context recall (CR)  

**Checklist**

- [ ] Validate citations exist.  
- [ ] Score grounding separately from correctness.  
- [ ] Detect hallucinations explicitly.  

---

# 11. RAG Anti-Patterns (Master List)

- AVOID: Using retrieval only after answer generation.  
- AVOID: Large chunks with mixed topics.  
- AVOID: No reranking stage.  
- AVOID: Combining multi-domain results without routing.  
- AVOID: Blindly trusting embedding similarity.  
- AVOID: Passing full documents into prompt.  
- AVOID: Using unfiltered retrieval outputs as context.  
- AVOID: Answering without evidence or citations.  

---

# 12. Quick Reference Tables

### Chunk Size Table

| Type of Data | Size (tokens) |
|--------------|----------------|
| Narrative    | 250–350        |
| Technical    | 150–250        |
| Legal/Code   | 100–200        |

### Retrieval Strategy Table

| Query Type | Strategy |
|------------|----------|
| Highly specific | semantic only |
| Fact-heavy | semantic + keyword |
| Sparse query | HyDE |
| Multi-domain | router + domain indexes |

---

# End of File

# Skill Lifecycle — Create, Validate, Share

Use this when packaging Claude skills for reuse and team distribution.

## Create
- Run the skill init script (if available) to scaffold `SKILL.md`, `scripts/`, `references/`, `assets/` with kebab-case naming and matching frontmatter.
- Write `SKILL.md` in imperative style; keep it lean and link to resources for depth.

## Validate
- Ensure frontmatter name matches directory, and description is specific and activation-friendly.
- Check structure: required `SKILL.md`; optional `references/`, `scripts/`, `assets/`.
- Run validation tooling if present; fix any missing metadata or naming issues.

## Package & Share
- Package as a zip (validation first); include all referenced files.
- Post summary to Slack via automation (Rube/Slack integration): name, description, link, and key resources.
- Keep versions discoverable; update team channels when new skills land or change materially.

# Tool Design & Validation — Best Practices 

*Purpose: Provide operational patterns, schemas, validation rules, and checklists for defining, selecting, and safely executing tools with Model Context Protocol (MCP) integration.*

**Modern Update**: MCP is now the standard for tool integration (adopted by Anthropic, OpenAI, Google). Use MCP for all new tool implementations.

---
## Table of Contents

- [Model Context Protocol (MCP) Integration](#model-context-protocol-mcp-integration)
- [MCP Architecture](#mcp-architecture)
- [MCP Tool Definition Pattern](#mcp-tool-definition-pattern)
- [1. Tool Definition Pattern (Legacy & MCP)](#1-tool-definition-pattern-legacy-&-mcp)
- [Standard Tool Schema](#standard-tool-schema)
- [2. Tool Action Pattern](#2-tool-action-pattern)
- [3. Parameter Validation](#3-parameter-validation)
- [Pattern: Strict Validation Layer](#pattern-strict-validation-layer)
- [4. High-Risk Tool Handling](#4-high-risk-tool-handling)
- [High-Risk Examples](#high-risk-examples)
- [Pattern: Guarded Tool Call](#pattern-guarded-tool-call)
- [5. Tool Selection Rules](#5-tool-selection-rules)
- [Pattern: Intent → Tool Choice](#pattern-intent-→-tool-choice)
- [5A. Tool Selection At Scale](#5a-tool-selection-at-scale)
- [The Three Strategies](#the-three-strategies)
- [Escalation Gate: Do Not Decompose Into Agents Yet](#escalation-gate-do-not-decompose-into-agents-yet)
- [6. Tool Output Validation](#6-tool-output-validation)
- [Pattern: Structured Output Check](#pattern-structured-output-check)
- [7. Error Handling Patterns](#7-error-handling-patterns)
- [Pattern: Typed Error Handling](#pattern-typed-error-handling)
- [8. Tool Composition Pattern](#8-tool-composition-pattern)
- [When chaining tools](#when-chaining-tools)
- [Topology Ladder: Bound Every Rung](#topology-ladder-bound-every-rung)
- [9. MCP Tool Design](#9-mcp-tool-design)
- [MCP Tool Structure](#mcp-tool-structure)
- [MCP-Specific Rules](#mcp-specific-rules)
- [10. Tool Testing Pattern](#10-tool-testing-pattern)
- [Pattern: Test Inputs → Verify → Compare → Log](#pattern-test-inputs-→-verify-→-compare-→-log)
- [11. Tool Safety Anti-Patterns (Master List)](#11-tool-safety-anti-patterns-master-list)
- [12. Quick Reference Tables](#12-quick-reference-tables)
- [Tool Types Table](#tool-types-table)
- [Risk Table](#risk-table)
- [Validation Table](#validation-table)
- [End of File](#end-of-file)


## Model Context Protocol (MCP) Integration

### MCP Architecture

**Three-layer pattern**:
```yaml
MCP Host (AI App) ← MCP Client ← MCP Server
```

**MCP Server provides**:
- Tools (function definitions)
- Resources (data access)
- Prompts (reusable templates)

**When to use MCP**:
- All new tool integrations (standardized over custom APIs)
- Multi-tool orchestration
- Cross-application tool sharing
- Standardized authentication and permissions

**Security requirements**:
- Tool signature verification (Sigstore/Cosign)
- Permission scoping per tool
- Prompt injection defenses
- Audit logging for all tool calls

### MCP Tool Definition Pattern

```yaml
mcp_tool:
  name: "tool_name"
  description: "Operational purpose (what it does)"
  inputSchema:
    type: "object"
    properties:
      param1:
        type: "string"
        description: "Clear parameter description"
      param2:
        type: "integer"
        minimum: 1
    required: ["param1"]
  security:
    require_confirmation: true  # For high-risk operations
    allowed_roles: ["admin", "operator"]
    signature_required: true
```

**MCP vs Custom API Decision Tree**:
```text
New tool integration needed?
→ Is this a standard operation (file access, web search, database)?
  → Yes: Use existing MCP server or create MCP tool
  → No: Is this tool shared across multiple agents/apps?
    → Yes: Implement as MCP server
    → No: Can still use MCP for consistency (recommended)
```

---

## 1. Tool Definition Pattern (Legacy & MCP)

### Standard Tool Schema

```
tool_name:
  description: [operational purpose]
  input_schema:
    field_1: type
    field_2: type
  output_schema:
    result: type
  confirm: yes/no
  error_handling:
    retry: 1
    timeout: 30
```

**Checklist**

- [ ] Description specifies *what the tool does*, not *how*.  
- [ ] Inputs are typed (string/int/boolean/object).  
- [ ] Output schema is deterministic.  
- [ ] Confirm = “yes” for destructive/irreversible actions.  
- [ ] Retry window defined for transient errors.  
- [ ] Timeout specified in seconds.  

**Anti-Patterns**

- AVOID: Leaving parameters untyped.  
- AVOID: Vague descriptions (“fetch stuff”).  
- AVOID: Missing error-handling section.  
- AVOID: Multiple unrelated actions in a single tool.  

---

# 2. Tool Action Pattern

**Use when:** executing any external function, API call, MCP tool, OS action, or integration.

```
prepare_parameters()
validate_parameters()
if high_risk: request_confirmation()
call_tool()
verify_output()
```

**Checklist**

- [ ] Validate type, range, format.  
- [ ] Reject incomplete parameters.  
- [ ] Map user intent → explicit parameters.  
- [ ] Convert natural language to structured fields.  
- [ ] Verify output fields before using downstream.  

---

# 3. Parameter Validation

### Pattern: Strict Validation Layer

```
for each field in input_schema:
    ensure field exists
    ensure type matches
    ensure format valid
```

**Validation Types**

- string (non-empty)  
- number (integer/float)  
- boolean  
- list of X  
- object with child fields  

**Decision Tree**

```
Is the parameter required?
→ Yes → Must appear → Must be valid
→ No → Provide default or null
```

**Examples**

- integer-only → reject floats or strings.  
- path fields → must not be hallucinated; confirm via retrieval.  
- enum fields → match allowed values only.  

---

# 4. High-Risk Tool Handling

### High-Risk Examples

- File deletion / modification  
- Database writes  
- Financial actions  
- OS-level execution  
- Remote system calls  
- External automation (clicking, typing, system control)

### Pattern: Guarded Tool Call

```
if high_risk:
  generate natural-language summary
  request user confirmation
  wait for explicit "yes"
  execute
```

**Checklist**

- [ ] Summaries must list exact parameters.  
- [ ] Confirmation required.  
- [ ] Abort when confirmation unclear.  

---

# 5. Tool Selection Rules

### Pattern: Intent → Tool Choice

```
extract_intent()
match_intent_to_tool()
choose_best_tool()
```

**Decision Tree**

```
Does the step require external data?
→ Yes → choose retriever or API tool
Does the step require external action?
→ Yes → choose action/OS tool
Does the step require computation?
→ Use internal reasoning unless precision tool exists
```

**Checklist**

- [ ] Never hallucinate undeclared tools.  
- [ ] Map intent → tool name exactly as defined.  
- [ ] One step = one tool call.  

---

# 5A. Tool Selection At Scale

Section 5 assumes every tool definition fits in the prompt. That assumption breaks
as the catalog grows: selection accuracy degrades as the number of candidate tools
increases, and semantically overlapping descriptions become the dominant source of
misselection. Pick a selection strategy by tool count and description overlap, not
by framework. (Albada, *Building Applications with AI Agents*, O'Reilly 2025, Ch. 5.)

### The Three Strategies

| Strategy | How it works | Choose it when | Cost of choosing it |
| -------- | ------------ | -------------- | ------------------- |
| **Standard** | All tool definitions go in the prompt; the model picks one | Small toolsets; you want zero extra infrastructure | Scales poorly as tool count rises; description overlap drives misselection |
| **Semantic** | Tool descriptions are embedded into a vector index ahead of time; at runtime the query is embedded and the top-k tools are retrieved and passed to the model | **Default at scale.** Most use cases; large toolsets where latency matters | Semantic collisions between similar descriptions can make accuracy *worse* than standard |
| **Hierarchical** | Two stages: select a tool *group* (each group carries its own description), then select a tool within that group | Large tool counts **and** many semantically similar tools, where accuracy outranks latency | Extra sequential model call per selection; groups must be authored and maintained by hand |

Decision rule:

```text
Do all tool definitions fit comfortably in the prompt?
→ Yes → standard selection; invest in description quality first
→ No  → semantic retrieval (default)
        → still misselecting because tools are semantically similar?
          → hierarchical grouping, accepting the added latency
```

Hierarchical selection is not recommended unless the tool count is genuinely large —
authoring and maintaining the groups is ongoing work, and the second stage costs a
sequential model call that is expensive to parallelize away.

**Description engineering comes first.** At any scale, the cheapest accuracy gain is
in the tool definitions themselves: a specific name over a generic one
(`calculate_sum`, not `process_numbers`), a one-sentence summary of the tool's
*unique* purpose, an example invocation, and explicit input types and ranges so the
model can rule tools out. Retrieval infrastructure does not rescue overlapping
descriptions — semantic collisions are exactly the failure mode it introduces.

### Escalation Gate: Do Not Decompose Into Agents Yet

Degrading tool selection is the most common trigger for splitting one agent into
many. It is usually the wrong first move. Before decomposing, exhaust the
single-agent options above — group tools hierarchically, or retrieve them
semantically from a vector index. Decompose into distinct agents only if those
still fall short, and price in the coordination overhead when you do.
(Albada, O'Reilly 2025, Ch. 8.)

This is the concrete, tool-count-driven form of the skill's general anti-multi-agent
posture. See `SKILL.md` → *Known Traps* (multi-agent topologies before single-agent
failure modes are understood) and [`multi-agent-patterns.md`](multi-agent-patterns.md)
for the handoff contracts required once decomposition is actually justified.

---

# 6. Tool Output Validation

### Pattern: Structured Output Check

```
verify(required_fields)
validate_types()
validate_ranges()
assert no unexpected nulls
```

**Checklist**

- [ ] Output matches schema exactly.  
- [ ] Unexpected fields ignored or flagged.  
- [ ] Missing fields = tool failure.  
- [ ] Use output only after validation.  

**Anti-Patterns**

- AVOID: Reasoning from assumed output.  
- AVOID: Skipping verification for “simple” tools.  
- AVOID: Reusing stale tool results.  

---

# 7. Error Handling Patterns

### Pattern: Typed Error Handling

```
if transient:
    retry once
elif soft_failure:
    request clarification
else:
    halt and surface error
```

**Error Types**

- **Transient** (network timeout, rate limit) → retry  
- **Soft failure** (bad parameters, missing fields) → ask user  
- **Fatal** (auth failure, invalid tool name) → halt  

**Checklist**

- [ ] Use max 1–2 retries.  
- [ ] Do not mask errors.  
- [ ] Bubble up fatal issues with clean summary.  

---

# 8. Tool Composition Pattern

### When chaining tools

```
output_1 = tool_A()
validate(output_1)
params_2 = transform(output_1)
tool_B(params_2)
```

**Checklist**

- [ ] Validate output_1 before using it.  
- [ ] Transform intermediate data explicitly.  
- [ ] Abort chain on any invalid output.  

**Anti-Patterns**

- AVOID: Long unbroken tool chains (>3).  
- AVOID: Using tool output as-is without validation.  

### Topology Ladder: Bound Every Rung

Climb from single tool → parallel → chain → graph only when the current rung
genuinely cannot express the task, and bound each rung explicitly:

- **Chains** must have a **maximum length**. Errors compound down the length of a
  chain, so an unbounded chain converts one bad step into a bad result.
- **Graphs** multiply foundation-model calls relative to chains — adding latency and
  cost — so **cap depth and branching factor**. Graphs also admit error classes
  chains do not: cycles, unreachable nodes, and conflicting state merges. Adopt a
  graph only when you must both branch *and* later consolidate; every added node or
  edge multiplies execution paths and error modes.

(Albada, O'Reilly 2025, Ch. 5.) For which *kind* of graph a problem calls for
before picking a runtime, see
[`graph-and-loop-engineering.md`](graph-and-loop-engineering.md).

---

# 9. MCP Tool Design

### MCP Tool Structure

```
{
  "name": "tool_name",
  "description": "purpose",
  "input_schema": {...},
  "output_schema": {...}
}
```

### MCP-Specific Rules

- Use JSON-RPC message types strictly.  
- Always include error objects when failing.  
- Keep tools granular (one purpose each).  
- Avoid side effects unless required by design.  

---

# 10. Tool Testing Pattern

### Pattern: Test Inputs → Verify → Compare → Log

```
for each test_case:
    run tool with known params
    assert output matches expected
    assert type validity
    assert error returns correctly
```

**Checklist**

- [ ] At least 3 positive test cases.  
- [ ] At least 2 negative test cases.  
- [ ] Logs captured for each call.  
- [ ] Versioned tool definitions.  

---

# 11. Tool Safety Anti-Patterns (Master List)

- AVOID: Using a tool without validating user intent.  
- AVOID: Guessing IDs, paths, or coordinates.  
- AVOID: Performing irreversible actions without confirmation.  
- AVOID: Triggering tools based on partial or ambiguous queries.  
- AVOID: Treating tool errors as “optional”.  
- AVOID: Overloading one tool with multi-purpose behavior.  
- AVOID: Generating synthetic parameters.  

---

# 12. Quick Reference Tables

### Tool Types Table

| Type | Purpose |
|------|---------|
| Retrieval | External data read |
| Action | External effect / OS control |
| Computation | Deterministic processing |
| Integration | API / remote system |
| Transformation | Data shaping |

### Risk Table

| Risk Level | Examples | Requirements |
|------------|----------|--------------|
| Low | read-only retrieval | no confirmation |
| Medium | modifying local data | validation + verification |
| High | destructive/system actions | explicit confirmation |

### Validation Table

| Field Type | Validation Rule |
|------------|------------------|
| string | not empty |
| int | numeric, range-bound |
| bool | true/false only |
| object | must match schema |
| enum | must be allowed value |

---

# End of File

# Voice and Multimodal Agents

> Operational reference for building voice agents (phone IVR, smart assistants, real-time speech) and multimodal agents (vision+action, document understanding) — latency budgets, turn-taking, modality integration, and production deployment.

**Freshness anchor:** January 2026 — covers OpenAI Realtime API, Anthropic tool use with vision, Gemini 2.0 multimodal, Deepgram Nova-2, ElevenLabs Turbo v2.5.

---
## Table of Contents

- [Modality Selection Decision Tree](#modality-selection-decision-tree)
- [Voice Agent Latency Budgets](#voice-agent-latency-budgets)
- [Latency Optimization Checklist](#latency-optimization-checklist)
- [Turn-Taking Patterns](#turn-taking-patterns)
- [Pattern 1: VAD-Based Turn Detection](#pattern-1-vad-based-turn-detection)
- [Pattern 2: Barge-In Support](#pattern-2-barge-in-support)
- [Pattern 3: Push-to-Talk](#pattern-3-push-to-talk)
- [Pattern 4: Backchannel Signals](#pattern-4-backchannel-signals)
- [Real-Time Voice Pipeline Architecture](#real-time-voice-pipeline-architecture)
- [Pipeline Integration Code](#pipeline-integration-code)
- [Vision Agent Patterns](#vision-agent-patterns)
- [Image Token Cost Reference (July 2026)](#image-token-cost-reference-july-2026)
- [Image Preprocessing Checklist](#image-preprocessing-checklist)
- [Vision Grounding Patterns](#vision-grounding-patterns)
- [Document Understanding Pipeline](#document-understanding-pipeline)
- [Decision Matrix](#decision-matrix)
- [Multi-Page Processing](#multi-page-processing)
- [Modality-Specific Guardrails](#modality-specific-guardrails)
- [Smart Assistant Integration (Alexa/Google)](#smart-assistant-integration-alexagoogle)
- [Platform Comparison](#platform-comparison)
- [Response Time Budget (8s Alexa limit)](#response-time-budget-8s-alexa-limit)
- [Anti-Patterns](#anti-patterns)
- [Cross-References](#cross-references)


## Modality Selection Decision Tree

```
What is the agent's primary input?
│
├── Voice (speech)
│   ├── Real-time conversation (phone/assistant)?
│   │   ├── YES → Real-Time Voice Pipeline
│   │   │   ├── Latency budget: TTFB <300ms
│   │   │   ├── Use: WebSocket streaming STT→LLM→TTS
│   │   │   └── Providers: OpenAI Realtime, Deepgram+LLM+ElevenLabs
│   │   └── NO → Async voice processing?
│   │       ├── Voicemail/recording analysis → Batch STT + LLM
│   │       └── Voice note summarization → Whisper + LLM
│   │
│   └── Smart assistant (Alexa/Google)?
│       ├── Use: Platform SDK + webhook backend
│       ├── Latency budget: <8s total response
│       └── Constraints: Platform-specific SSML, intent routing
│
├── Vision (images/video)
│   ├── Single image understanding?
│   │   ├── Use: GPT-4V / Claude Vision / Gemini Vision
│   │   ├── Cost: ~85 tokens per 512x512 tile (OpenAI)
│   │   └── Preprocessing: resize, crop ROI, compress
│   │
│   ├── Document understanding (PDF/forms)?
│   │   ├── Structured data → Vision + JSON mode
│   │   ├── Table extraction → Vision + schema prompt
│   │   └── Multi-page → Page-by-page with aggregation
│   │
│   └── Video analysis?
│       ├── Frame sampling → Extract key frames + vision LLM
│       ├── Real-time → Not cost-effective with current models
│       └── Use: Gemini 2.0 (native video) or frame extraction
│
└── Multi-modal (combined)
    ├── Vision + Action (UI agents) → Screenshot + tool use loop
    ├── Voice + Vision → Speech input + image context + speech output
    └── Document + Conversation → RAG with vision-extracted content
```

---

## Voice Agent Latency Budgets

| Component | Target Latency | Maximum | Notes |
|---|---|---|---|
| Speech-to-Text (STT) | <150ms | 300ms | Streaming STT preferred |
| Endpoint detection (VAD) | <200ms | 400ms | Silero VAD or WebRTC VAD |
| LLM inference (TTFT) | <200ms | 500ms | Use streaming, small models for simple turns |
| Text-to-Speech (TTS) | <150ms | 300ms | Streaming TTS with chunked delivery |
| **Total turn latency** | **<700ms** | **1500ms** | User perceives >1.5s as laggy |
| Network round-trip | <50ms | 100ms | Edge deployment preferred |

### Latency Optimization Checklist

- [ ] Use streaming STT (not batch transcription)
- [ ] Implement Voice Activity Detection (VAD) for accurate endpoint detection
- [ ] Stream LLM output token-by-token to TTS
- [ ] Use TTS with streaming support (ElevenLabs, Deepgram Aura, PlayHT)
- [ ] Deploy inference at edge or in same region as user
- [ ] Pre-warm TTS connections (keep WebSocket alive)
- [ ] Cache common responses (greetings, confirmations)
- [ ] Use smaller models for simple routing/classification turns
- [ ] Implement speculative generation for predictable responses

---

## Turn-Taking Patterns

### Pattern 1: VAD-Based Turn Detection

```
Use when: open-ended conversation, user speaks freely
Pipeline: Audio → VAD → silence threshold → process utterance

Configuration:
- Silence threshold: 500-800ms (adjust per use case)
- Min speech duration: 200ms (filter noise)
- Max speech duration: 30s (prevent runaway capture)
```

### Pattern 2: Barge-In Support

```
Use when: IVR systems, long TTS responses user may interrupt
Pipeline: Monitor user audio DURING TTS playback

Implementation:
- Detect user speech onset during TTS
- Immediately stop TTS playback
- Capture user utterance
- Process interruption as new input
- Anti-pattern: requiring user to wait for full TTS completion
```

### Pattern 3: Push-to-Talk

```
Use when: noisy environments, walkie-talkie style apps
Pipeline: Button press → capture → button release → process

Advantages:
- No VAD false positives
- Clear turn boundaries
- Works in high-noise environments
Disadvantages:
- Less natural interaction
- Requires UI element
```

### Pattern 4: Backchannel Signals

```
Use when: building natural conversational agents
Implementation:
- Detect pause mid-utterance (300-500ms)
- Generate short acknowledgment ("mm-hmm", "I see")
- Do NOT trigger full processing — just backchannel
- Resume listening for continued speech
```

---

## Real-Time Voice Pipeline Architecture

```
┌─────────┐    WebSocket    ┌──────────────┐
│  Client  │ ◄────────────► │  Voice Gateway│
│ (Phone/  │   audio chunks │              │
│  Browser)│                └──────┬───────┘
└─────────┘                       │
                                  ▼
                          ┌───────────────┐
                          │  STT Engine   │
                          │  (Deepgram/   │
                          │   Whisper)    │
                          └──────┬────────┘
                                 │ text
                                 ▼
                          ┌───────────────┐
                          │  Agent Core   │
                          │  (LLM + Tools)│
                          └──────┬────────┘
                                 │ text (streaming)
                                 ▼
                          ┌───────────────┐
                          │  TTS Engine   │
                          │  (ElevenLabs/ │
                          │   Deepgram)   │
                          └──────┬────────┘
                                 │ audio chunks
                                 ▼
                          Back to Client
```

### Pipeline Integration Code

```python
import asyncio

class VoicePipeline:
    def __init__(self, stt, llm, tts):
        self.stt = stt
        self.llm = llm
        self.tts = tts
        self.is_speaking = False

    async def process_turn(self, audio_stream):
        # Step 1: STT (streaming)
        transcript = ""
        async for partial in self.stt.transcribe_stream(audio_stream):
            transcript = partial.text

        if not transcript.strip():
            return  # silence, no action

        # Step 2: LLM (streaming) → TTS (streaming)
        self.is_speaking = True
        tts_stream = self.tts.create_stream()

        buffer = ""
        async for token in self.llm.generate_stream(transcript):
            buffer += token
            # Flush to TTS at sentence boundaries
            if buffer.rstrip().endswith((".", "!", "?", ":")):
                await tts_stream.send_text(buffer)
                buffer = ""

        if buffer:
            await tts_stream.send_text(buffer)

        await tts_stream.finish()
        self.is_speaking = False
```

---

## Vision Agent Patterns

### Image Token Cost Reference (July 2026)

Counting mechanics are stable across model generations; verify exact per-model numbers in provider docs before budgeting.

| Provider | Model | Cost per Image | Token Calculation |
|---|---|---|---|
| OpenAI | GPT-5.5 / GPT-5.4 | ~85 tokens per 512x512 tile | Tiles = ceil(width/512) * ceil(height/512) |
| OpenAI | GPT-5.4 mini | Same tiling, lower $/token | More cost-effective for simple vision |
| Anthropic | Claude Opus 4.8 / Sonnet 4.6 | ~1600 tokens per 1568x1568 | Scales with image size |
| Google | Gemini 3.5 Flash | Flat per-image rate | Flat rate, cost-effective |

### Image Preprocessing Checklist

- [ ] Resize to model's optimal resolution (avoid sending 4K images)
- [ ] Crop to region of interest when possible
- [ ] Compress JPEG to 85% quality (minimal quality loss, significant size reduction)
- [ ] Convert PNG screenshots to JPEG (unless transparency needed)
- [ ] For documents: increase contrast, deskew, remove margins
- [ ] For multi-image: limit to 5-10 images per request (cost control)
- [ ] Encode as base64 or use pre-signed URLs (provider-dependent)

### Vision Grounding Patterns

| Pattern | Use When | Implementation |
|---|---|---|
| Bounding box overlay | Need to identify specific regions | Draw numbered boxes, reference by number in prompt |
| Grid overlay | Need spatial precision | Overlay labeled grid, use grid coordinates |
| Set-of-marks | UI element identification | Number each interactive element |
| Cropped regions | Focus on specific area | Send cropped sub-image instead of full image |
| Multi-angle | 3D object understanding | Send 2-4 views of same object |

---

## Document Understanding Pipeline

### Decision Matrix

| Document Type | Best Approach | Fallback |
|---|---|---|
| Clean PDF with text layer | Text extraction (PyMuPDF) + LLM | Vision API on rendered pages |
| Scanned PDF / image-only | Vision API (page-by-page) | OCR (Tesseract) + LLM |
| Forms with checkboxes | Vision API with schema prompt | Specialized form OCR |
| Tables | Vision API + JSON mode output | Camelot/Tabula extraction + LLM |
| Handwritten notes | Vision API | Not reliable for production |
| Multi-page reports | Page-by-page vision + aggregation | Extract text + chunk + RAG |

### Multi-Page Processing

```python
async def process_document(pages: list[bytes], schema: dict) -> dict:
    results = []

    for i, page_image in enumerate(pages):
        result = await vision_llm.analyze(
            image=page_image,
            prompt=f"""Extract data from page {i+1} of {len(pages)}.
Output JSON matching this schema: {json.dumps(schema)}
If a field spans multiple pages, include partial data with
"continues_on_next_page": true""",
            response_format={"type": "json_object"}
        )
        results.append(result)

    # Aggregate cross-page data
    return merge_page_results(results, schema)
```

---

## Modality-Specific Guardrails

| Modality | Guardrail | Implementation |
|---|---|---|
| Voice | Profanity filter on STT output | Word list + regex before LLM |
| Voice | PII detection in transcripts | NER model on STT output |
| Voice | Silence timeout | Disconnect after 30s silence |
| Voice | Max turn duration | Hard cut at 60s recording |
| Vision | NSFW image detection | Pre-screen with safety classifier |
| Vision | PII in images (IDs, cards) | Blur detection + warning |
| Vision | Image size limits | Reject >20MB, resize >4096px |
| Document | Malicious file detection | Scan uploads before processing |
| Document | Page count limits | Cap at 50 pages per request |
| Multimodal | Cross-modal consistency | Verify vision output matches text context |

---

## Smart Assistant Integration (Alexa/Google)

### Platform Comparison

| Feature | Alexa Skills Kit | Google Actions | Apple Shortcuts |
|---|---|---|---|
| Max response time | 8 seconds | 5 seconds | N/A (local) |
| Audio streaming | Yes (AudioPlayer) | Yes (Media) | Limited |
| Visual cards | Yes (APL) | Yes (Canvas) | No |
| Account linking | OAuth 2.0 | OAuth 2.0 | N/A |
| Proactive events | Yes (limited) | Yes (limited) | No |
| SSML support | Full | Full | No |
| LLM integration | Webhook to your backend | Webhook to your backend | Shortcuts actions |

### Response Time Budget (8s Alexa limit)

| Phase | Budget | Strategy |
|---|---|---|
| Intent routing | <100ms | Local classification |
| Context retrieval | <500ms | Pre-cached user state |
| LLM generation | <2000ms | Small model or cached response |
| Response formatting | <100ms | Template-based SSML |
| Network overhead | ~300ms | Edge deployment |
| **Buffer** | **~5000ms** | Safety margin for retries |

---

## Anti-Patterns

| Anti-Pattern | Why It Fails | Better Approach |
|---|---|---|
| Batch STT for real-time voice | >2s latency, unusable | Streaming STT with partial results |
| Fixed silence threshold for all users | Fast speakers cut off, slow speakers wait | Adaptive VAD or per-user tuning |
| Sending full-res images to vision API | Expensive, slow, often unnecessary | Resize and crop before sending |
| Processing entire PDF as one image | Context overflow, poor accuracy | Page-by-page with aggregation |
| No barge-in support | Users forced to wait for long responses | Monitor audio during TTS playback |
| Ignoring TTS voice quality for brand | Generic voice feels impersonal | Select/clone voice matching brand |
| Hard-coding SSML | Brittle, unmaintainable | Template SSML with variable substitution |
| No fallback for STT errors | Misheard words cause wrong actions | Confirm critical actions before executing |

---

## Cross-References

- `agent-debugging-patterns.md` — debugging voice/multimodal agent failures
- `guardrails-implementation.md` — guardrail layers for voice/vision input
- `../ai-llm/references/multimodal-patterns.md` — LLM-level multimodal capabilities
- `../ai-llm-inference/references/streaming-patterns.md` — streaming infrastructure for voice
- `../ai-prompt-engineering/references/multimodal-prompt-patterns.md` — prompting for vision/audio
- [`../ai-voice-bots/SKILL.md`](../../ai-voice-bots/SKILL.md) — production voice bot building: telephony platform selection, Pipecat/LiveKit patterns, latency engineering, voice quality metrics, IVR design, and voice compliance
- `ai-bot-builder` — conversation design, persona, escalation, and bot analytics for voice and text bots

#!/usr/bin/env python3
"""
agent_eval_runner.py — Lightweight agent evaluation runner.

Reads a JSONL file of task/expected/actual triples and reports pass rates.

SCOPE: This script handles offline pass/fail scoring only. For adversarial
attack suites, multi-turn evaluation harnesses, or regression gates, delegate
to the qa-agent-testing skill (../qa-agent-testing/SKILL.md).

Usage:
    python agent_eval_runner.py --input results.jsonl
    python agent_eval_runner.py --input results.jsonl --output report.json
    python agent_eval_runner.py --input results.jsonl --mode substring
    python agent_eval_runner.py --help

Input JSONL format (one JSON object per line):
    {"task": "Summarise in 1 sentence", "expected": "brief", "actual": "A brief summary."}

Fields:
    task     — human-readable task description (used in output only)
    expected — substring or exact string the actual output must contain/match
    actual   — the agent's actual output
    mode     — (optional per-record) "substring" | "exact" | "nonempty"
               overrides the global --mode for that record

Exit code: 0 if pass_rate == 1.0, 1 otherwise.
"""

import argparse
import json
import sys
from pathlib import Path


def evaluate_record(record: dict, default_mode: str) -> tuple[bool, str]:
    """Return (passed, reason) for a single record."""
    task = record.get("task", "")
    expected = record.get("expected", "")
    actual = record.get("actual", "")
    mode = record.get("mode", default_mode)

    if mode == "nonempty":
        passed = bool(actual and actual.strip())
        reason = "non-empty check"
    elif mode == "exact":
        passed = actual.strip() == expected.strip()
        reason = f"exact match expected={repr(expected)[:60]}"
    else:  # substring (default)
        passed = expected.lower() in actual.lower()
        reason = f"substring expected={repr(expected)[:60]}"

    return passed, reason


def run(input_path: Path, output_path: Path | None, mode: str, verbose: bool) -> int:
    records = []
    try:
        with input_path.open() as f:
            for lineno, line in enumerate(f, 1):
                line = line.strip()
                if not line:
                    continue
                try:
                    records.append((lineno, json.loads(line)))
                except json.JSONDecodeError as e:
                    print(f"[WARN] line {lineno}: skipped — JSON parse error: {e}", file=sys.stderr)
    except FileNotFoundError:
        print(f"[ERROR] Input file not found: {input_path}", file=sys.stderr)
        return 2

    if not records:
        print("[ERROR] No valid records found in input file.", file=sys.stderr)
        return 2

    results = []
    passed_count = 0

    for lineno, record in records:
        passed, reason = evaluate_record(record, mode)
        if passed:
            passed_count += 1
        result = {
            "lineno": lineno,
            "task": record.get("task", ""),
            "passed": passed,
            "mode": record.get("mode", mode),
            "reason": reason,
        }
        results.append(result)
        if verbose:
            status = "PASS" if passed else "FAIL"
            print(f"[{status}] line {lineno}: {record.get('task', '')[:60]} — {reason}")

    total = len(results)
    pass_rate = passed_count / total if total > 0 else 0.0

    summary = {
        "total": total,
        "passed": passed_count,
        "failed": total - passed_count,
        "pass_rate": round(pass_rate, 4),
        "results": results,
    }

    print(f"\nResults: {passed_count}/{total} passed  ({pass_rate:.1%})")

    if output_path:
        with output_path.open("w") as f:
            json.dump(summary, f, indent=2)
        print(f"Report written to: {output_path}")

    return 0 if pass_rate == 1.0 else 1


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Offline agent evaluation runner — reads JSONL task/expected/actual triples.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=__doc__,
    )
    parser.add_argument("--input", required=True, type=Path, help="Input JSONL file path")
    parser.add_argument("--output", type=Path, default=None, help="Optional output JSON report path")
    parser.add_argument(
        "--mode",
        choices=["substring", "exact", "nonempty"],
        default="substring",
        help="Default match mode (default: substring). Per-record 'mode' field overrides this.",
    )
    parser.add_argument("--verbose", "-v", action="store_true", help="Print per-record pass/fail")
    args = parser.parse_args()

    sys.exit(run(args.input, args.output, args.mode, args.verbose))


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
Claude Code usage reporter — stdlib-only CLI tool.

Reads local Claude Code logs to produce token and cost reports
without any third-party dependencies.

Data sources:
  - ~/.claude/stats-cache.json  (pre-aggregated daily/model stats)
  - ~/.claude/projects/          (raw JSONL session logs)

Subcommands:
  daily    — Usage grouped by date
  monthly  — Monthly aggregated report
  sessions — Per-session detail
  models   — Per-model all-time totals

Usage:
  python scripts/claude-usage.py daily
  python scripts/claude-usage.py daily --since 2026-04-01 --until 2026-04-07
  python scripts/claude-usage.py monthly --json
  python scripts/claude-usage.py sessions --last 10
  python scripts/claude-usage.py models
"""

from __future__ import annotations

import argparse
import glob
import json
import os
import sys
from collections import defaultdict
from datetime import date, datetime
from pathlib import Path


# ---------------------------------------------------------------------------
# Config
# ---------------------------------------------------------------------------

_env_dir = os.environ.get("CLAUDE_CONFIG_DIR", "")
if _env_dir and Path(_env_dir).is_dir():
    CLAUDE_DIR = Path(_env_dir)
elif (Path.home() / ".config" / "claude").is_dir():
    CLAUDE_DIR = Path.home() / ".config" / "claude"
else:
    CLAUDE_DIR = Path.home() / ".claude"

STATS_CACHE = CLAUDE_DIR / "stats-cache.json"
PROJECTS_DIR = CLAUDE_DIR / "projects"

# ---------------------------------------------------------------------------
# Pricing table — USD per 1M tokens.
#
# The authoritative table is this skill's own data/model-pricing.json, read at
# runtime via _lib/resolve_versions.py — the same file whether the skill runs
# from the repo or from ~/.claude, ~/.agents or ~/.codex. The dict below is a
# FALLBACK for when that file cannot be located.
#
# Model IDs here are deliberately pinned and historical: replaying old logs must
# price them at the rates that applied then, so retired IDs stay in the table.
# What rots is not the IDs but the *rates*, so the table expires loudly instead
# of being silently trusted. Update https://claude.com/pricing rates in the JSON
# and bump its last_verified; add new IDs without removing old ones.
# ---------------------------------------------------------------------------

# Resolve symlinks first: skills deploy as individual symlinks, so a lexical
# relative path would escape into the deployment root instead of the repo.
# Import the resolver from this skill's OWN _lib/. The skill is self-contained:
# it carries its own _lib/ and data/, so it works detached from the repo
# (public-repo clone, single-folder copy, plugin). resolve() first because
# skills deploy as symlinks, so a lexical path escapes into the deployment root.
_here = Path(__file__).resolve()
sys.path.insert(0, str(_here.parents[1] / "_lib"))
try:
    from resolve_versions import load_pricing, pricing_path
except ImportError:  # resolver missing — use the embedded fallback
    load_pricing = None
    pricing_path = None

PRICE_TABLE_LAST_VERIFIED = date.fromisoformat("2026-08-10")
PRICE_TABLE_STALE_AFTER_DAYS = 30

# Verified against platform.claude.com/docs/en/about-claude/pricing on 2026-08-10.
# cache_read is 0.1x base input, cache_create 1.25x (the 5-minute write).
# A missing key falls through to DEFAULT_PRICING at Sonnet rates, which prices
# an Opus log ~5x low without erroring — so entries are corrected here, not
# deleted, even when a rate turns out to be wrong.
FALLBACK_PRICING = {
    "claude-opus-4-7":    {"input": 5.00,  "output": 25.00, "cache_read": 0.50, "cache_create": 6.25},
    "claude-opus-4-6":    {"input": 5.00,  "output": 25.00, "cache_read": 0.50, "cache_create": 6.25},
    "claude-opus-4-5":    {"input": 5.00,  "output": 25.00, "cache_read": 0.50, "cache_create": 6.25},
    "claude-sonnet-4-6":  {"input": 3.00,  "output": 15.00, "cache_read": 0.30, "cache_create": 3.75},
    "claude-sonnet-4-5":  {"input": 3.00,  "output": 15.00, "cache_read": 0.30, "cache_create": 3.75},
    "claude-haiku-4-5":   {"input": 1.00,  "output": 5.00,  "cache_read": 0.10, "cache_create": 1.25},
}
DEFAULT_PRICING = {"input": 3.00, "output": 15.00, "cache_read": 0.30, "cache_create": 3.75}


def _load_pricing() -> tuple[dict, str, date]:
    """Return (pricing, provenance, last_verified), preferring the shared table.

    Adapts the shared schema (`*_per_1m`) into this script's field names rather
    than renaming either side: the shared file stays vendor-neutral and this
    script's arithmetic keeps the keys it already uses.
    """
    if load_pricing is None:
        return FALLBACK_PRICING, "embedded fallback (resolver not importable)", PRICE_TABLE_LAST_VERIFIED

    doc = load_pricing(__file__)
    models = doc.get("models") if isinstance(doc, dict) else None
    if not isinstance(models, dict):
        return FALLBACK_PRICING, "embedded fallback (shared table unavailable)", PRICE_TABLE_LAST_VERIFIED

    table = {}
    for key, entry in models.items():
        if not isinstance(entry, dict) or entry.get("vendor") != "anthropic":
            continue
        if "input_per_1m" not in entry or "output_per_1m" not in entry:
            continue
        table[key.split("/", 1)[-1]] = {
            "input": entry["input_per_1m"],
            "output": entry["output_per_1m"],
            "cache_read": entry.get("cache_read_per_1m", 0.0),
            "cache_create": entry.get("cache_write_per_1m", 0.0),
        }
    if not table:
        return FALLBACK_PRICING, "embedded fallback (no anthropic rows)", PRICE_TABLE_LAST_VERIFIED

    verified = PRICE_TABLE_LAST_VERIFIED
    stamp = doc.get("last_verified")
    if isinstance(stamp, str):
        try:
            verified = date.fromisoformat(stamp)
        except ValueError:
            pass
    path = pricing_path(__file__) if pricing_path else None
    return table, f"shared: {path}" if path else "shared", verified


PRICING, PRICING_SOURCE, PRICING_VERIFIED = _load_pricing()


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

def warn_if_price_table_stale() -> None:
    """Warn once if the embedded pricing table is older than the staleness window.

    Called from main() rather than estimate_cost() because the reports call
    estimate_cost per row; warning there would repeat the notice for every line.
    """
    age_days = (date.today() - PRICING_VERIFIED).days
    if age_days > PRICE_TABLE_STALE_AFTER_DAYS:
        print(
            f"[WARN] Pricing is {age_days} days old "
            f"(last verified {PRICING_VERIFIED.isoformat()}, source: {PRICING_SOURCE}); "
            "costs below are estimates — verify at https://claude.com/pricing.",
            file=sys.stderr,
        )


def estimate_cost(model: str, inp: int, out: int, cache_read: int, cache_create: int) -> float:
    """Estimate cost in USD from token counts."""
    # Match model to pricing — try exact, then prefix match
    prices = DEFAULT_PRICING
    for key, p in PRICING.items():
        if key in model:
            prices = p
            break
    return (
        inp * prices["input"] / 1_000_000
        + out * prices["output"] / 1_000_000
        + cache_read * prices["cache_read"] / 1_000_000
        + cache_create * prices["cache_create"] / 1_000_000
    )


def parse_date(s: str) -> str:
    """Normalize date string to YYYY-MM-DD."""
    s = s.replace("/", "-").strip()
    if len(s) == 8 and s.isdigit():
        return f"{s[:4]}-{s[4:6]}-{s[6:8]}"
    return s[:10]


def in_range(date_str: str, since: str | None, until: str | None) -> bool:
    if since and date_str < since:
        return False
    if until and date_str > until:
        return False
    return True


def fmt_tokens(n: int) -> str:
    """Format token count with commas."""
    return f"{n:,}"


def fmt_cost(c: float) -> str:
    return f"${c:.2f}"


def print_table(headers: list[str], rows: list[list[str]], right_align: set[int] | None = None):
    """Print a simple ASCII table."""
    right_align = right_align or set()
    widths = [len(h) for h in headers]
    for row in rows:
        for i, cell in enumerate(row):
            widths[i] = max(widths[i], len(cell))

    def fmt_row(cells):
        parts = []
        for i, cell in enumerate(cells):
            if i in right_align:
                parts.append(cell.rjust(widths[i]))
            else:
                parts.append(cell.ljust(widths[i]))
        return "  ".join(parts)

    print(fmt_row(headers))
    print("  ".join("-" * w for w in widths))
    for row in rows:
        print(fmt_row(row))


# ---------------------------------------------------------------------------
# Data loading — stats-cache.json (fast path)
# ---------------------------------------------------------------------------

def load_stats_cache() -> dict:
    if not STATS_CACHE.exists():
        return {}
    return json.loads(STATS_CACHE.read_text())


# ---------------------------------------------------------------------------
# Data loading — raw JSONL (detailed path)
# ---------------------------------------------------------------------------

def iter_jsonl_records():
    """Yield parsed records from all Claude Code JSONL session files."""
    if not PROJECTS_DIR.is_dir():
        return
    for path in glob.glob(str(PROJECTS_DIR / "*" / "*.jsonl")):
        with open(path) as f:
            for line in f:
                line = line.strip()
                if not line:
                    continue
                try:
                    rec = json.loads(line)
                    if isinstance(rec, dict):
                        yield rec, path
                except json.JSONDecodeError:
                    continue


def extract_usage(rec: dict) -> dict | None:
    """Extract token usage from an assistant message record."""
    msg = rec.get("message")
    if not isinstance(msg, dict):
        return None
    usage = msg.get("usage")
    if not isinstance(usage, dict):
        return None
    return {
        "timestamp": rec.get("timestamp", ""),
        "session_id": rec.get("sessionId", ""),
        "model": msg.get("model", "unknown"),
        "input_tokens": usage.get("input_tokens", 0) or 0,
        "output_tokens": usage.get("output_tokens", 0) or 0,
        "cache_read": usage.get("cache_read_input_tokens", 0) or 0,
        "cache_create": usage.get("cache_creation_input_tokens", 0) or 0,
    }


# ---------------------------------------------------------------------------
# Subcommand: daily
# ---------------------------------------------------------------------------

def cmd_daily(args):
    """Daily usage report."""
    stats = load_stats_cache()
    daily_list = stats.get("dailyActivity", [])
    model_tokens = stats.get("dailyModelTokens", {})

    if not daily_list and not PROJECTS_DIR.is_dir():
        print(f"No data found. Checked: {STATS_CACHE} and {PROJECTS_DIR}", file=sys.stderr)
        sys.exit(1)

    # If stats-cache has daily data, use it (fast path)
    if daily_list:
        rows_data = []
        for entry in sorted(daily_list, key=lambda e: e.get("date", "")):
            date = entry.get("date", "")
            if not in_range(date, args.since, args.until):
                continue
            rows_data.append({
                "date": date,
                "messages": entry.get("messageCount", 0),
                "sessions": entry.get("sessionCount", 0),
                "tools": entry.get("toolCallCount", 0),
            })

        if args.json:
            json.dump({"daily": rows_data}, sys.stdout, indent=2)
            print()
            return

        if not rows_data:
            print("No data in the specified date range.")
            return

        rows = [[r["date"], str(r["messages"]), str(r["sessions"]), str(r["tools"])]
                for r in rows_data]
        print_table(["Date", "Messages", "Sessions", "Tool Calls"], rows, {1, 2, 3})

        total_msg = sum(r["messages"] for r in rows_data)
        total_sess = sum(r["sessions"] for r in rows_data)
        total_tools = sum(r["tools"] for r in rows_data)
        print(f"\nTotal: {total_msg:,} messages, {total_sess:,} sessions, {total_tools:,} tool calls")
        return

    # Fallback: parse JSONL
    daily = defaultdict(lambda: {"input": 0, "output": 0, "cache_read": 0, "cache_create": 0, "count": 0, "models": set()})
    for rec, _ in iter_jsonl_records():
        u = extract_usage(rec)
        if not u or not u["timestamp"]:
            continue
        date = u["timestamp"][:10]
        if not in_range(date, args.since, args.until):
            continue
        d = daily[date]
        d["input"] += u["input_tokens"]
        d["output"] += u["output_tokens"]
        d["cache_read"] += u["cache_read"]
        d["cache_create"] += u["cache_create"]
        d["count"] += 1
        d["models"].add(u["model"])

    if args.json:
        out = []
        for date in sorted(daily):
            d = daily[date]
            cost = estimate_cost("", d["input"], d["output"], d["cache_read"], d["cache_create"])
            out.append({"date": date, "inputTokens": d["input"], "outputTokens": d["output"],
                        "cacheReadTokens": d["cache_read"], "cacheCreateTokens": d["cache_create"],
                        "messages": d["count"], "costUSD": round(cost, 4),
                        "models": sorted(d["models"])})
        json.dump({"daily": out}, sys.stdout, indent=2)
        print()
        return

    rows = []
    total_cost = 0.0
    for date in sorted(daily):
        d = daily[date]
        cost = estimate_cost("", d["input"], d["output"], d["cache_read"], d["cache_create"])
        total_cost += cost
        rows.append([date, fmt_tokens(d["input"]), fmt_tokens(d["output"]),
                      fmt_tokens(d["cache_read"]), str(d["count"]), fmt_cost(cost)])

    if not rows:
        print("No data in the specified date range.")
        return

    print_table(["Date", "Input", "Output", "Cache Read", "Messages", "Est. Cost"],
                rows, {1, 2, 3, 4, 5})
    print(f"\nTotal estimated cost: {fmt_cost(total_cost)}")


# ---------------------------------------------------------------------------
# Subcommand: monthly
# ---------------------------------------------------------------------------

def cmd_monthly(args):
    """Monthly aggregated report from JSONL data."""
    monthly = defaultdict(lambda: {"input": 0, "output": 0, "cache_read": 0, "cache_create": 0, "count": 0})

    for rec, _ in iter_jsonl_records():
        u = extract_usage(rec)
        if not u or not u["timestamp"]:
            continue
        date = u["timestamp"][:10]
        if not in_range(date, args.since, args.until):
            continue
        month = date[:7]
        m = monthly[month]
        m["input"] += u["input_tokens"]
        m["output"] += u["output_tokens"]
        m["cache_read"] += u["cache_read"]
        m["cache_create"] += u["cache_create"]
        m["count"] += 1

    if args.json:
        out = []
        for month in sorted(monthly):
            m = monthly[month]
            cost = estimate_cost("", m["input"], m["output"], m["cache_read"], m["cache_create"])
            out.append({"month": month, "inputTokens": m["input"], "outputTokens": m["output"],
                        "cacheReadTokens": m["cache_read"], "cacheCreateTokens": m["cache_create"],
                        "messages": m["count"], "costUSD": round(cost, 4)})
        json.dump({"monthly": out}, sys.stdout, indent=2)
        print()
        return

    rows = []
    total_cost = 0.0
    for month in sorted(monthly):
        m = monthly[month]
        cost = estimate_cost("", m["input"], m["output"], m["cache_read"], m["cache_create"])
        total_cost += cost
        rows.append([month, fmt_tokens(m["input"]), fmt_tokens(m["output"]),
                      fmt_tokens(m["cache_read"]), str(m["count"]), fmt_cost(cost)])

    if not rows:
        print("No data in the specified date range.")
        return

    print_table(["Month", "Input", "Output", "Cache Read", "Messages", "Est. Cost"],
                rows, {1, 2, 3, 4, 5})
    print(f"\nTotal estimated cost: {fmt_cost(total_cost)}")


# ---------------------------------------------------------------------------
# Subcommand: sessions
# ---------------------------------------------------------------------------

def cmd_sessions(args):
    """Per-session usage report."""
    session_meta_dir = CLAUDE_DIR / "usage-data" / "session-meta"
    sessions = []

    if session_meta_dir.is_dir():
        for path in sorted(session_meta_dir.glob("*.json")):
            try:
                meta = json.loads(path.read_text())
            except (json.JSONDecodeError, OSError):
                continue
            start = meta.get("start_time", "")
            date = start[:10] if start else ""
            if not in_range(date, args.since, args.until):
                continue
            sessions.append({
                "session_id": meta.get("session_id", path.stem),
                "date": date,
                "duration_min": meta.get("duration_minutes", 0),
                "messages": meta.get("user_message_count", 0) + meta.get("assistant_message_count", 0),
                "input_tokens": meta.get("input_tokens", 0),
                "output_tokens": meta.get("output_tokens", 0),
                "tools": sum(meta.get("tool_counts", {}).values()),
                "project": meta.get("project_path", ""),
            })
    else:
        # Fallback: aggregate from JSONL
        sess_data = defaultdict(lambda: {"input": 0, "output": 0, "count": 0, "first_ts": "", "last_ts": ""})
        for rec, fpath in iter_jsonl_records():
            u = extract_usage(rec)
            if not u:
                continue
            sid = u["session_id"] or os.path.basename(fpath).replace(".jsonl", "")
            date = u["timestamp"][:10]
            if not in_range(date, args.since, args.until):
                continue
            s = sess_data[sid]
            s["input"] += u["input_tokens"]
            s["output"] += u["output_tokens"]
            s["count"] += 1
            if not s["first_ts"] or u["timestamp"] < s["first_ts"]:
                s["first_ts"] = u["timestamp"]
            if not s["last_ts"] or u["timestamp"] > s["last_ts"]:
                s["last_ts"] = u["timestamp"]

        for sid, s in sess_data.items():
            sessions.append({
                "session_id": sid[:12] + "...",
                "date": s["first_ts"][:10],
                "duration_min": 0,
                "messages": s["count"],
                "input_tokens": s["input"],
                "output_tokens": s["output"],
                "tools": 0,
                "project": "",
            })

    sessions.sort(key=lambda s: s["date"], reverse=True)
    if args.last:
        sessions = sessions[:args.last]

    if args.json:
        json.dump({"sessions": sessions}, sys.stdout, indent=2)
        print()
        return

    if not sessions:
        print("No sessions found in the specified date range.")
        return

    rows = []
    for s in sessions:
        rows.append([
            s["date"],
            s["session_id"][:16],
            str(s["messages"]),
            fmt_tokens(s["input_tokens"]),
            fmt_tokens(s["output_tokens"]),
            str(s["tools"]),
            f"{s['duration_min']}m" if s["duration_min"] else "-",
        ])

    print_table(["Date", "Session", "Messages", "Input", "Output", "Tools", "Duration"],
                rows, {2, 3, 4, 5, 6})
    print(f"\nShowing {len(sessions)} session(s)")


# ---------------------------------------------------------------------------
# Subcommand: models
# ---------------------------------------------------------------------------

def cmd_models(args):
    """Per-model all-time usage from stats-cache.json."""
    stats = load_stats_cache()
    model_usage = stats.get("modelUsage", {})

    if not model_usage:
        print("No model usage data found in stats-cache.json.", file=sys.stderr)
        sys.exit(1)

    if args.json:
        json.dump({"models": model_usage}, sys.stdout, indent=2)
        print()
        return

    rows = []
    total_cost = 0.0
    for model, u in sorted(model_usage.items()):
        inp = u.get("inputTokens", 0)
        out = u.get("outputTokens", 0)
        cr = u.get("cacheReadInputTokens", 0)
        cc = u.get("cacheCreationInputTokens", 0)
        cost = estimate_cost(model, inp, out, cr, cc)
        total_cost += cost
        rows.append([model, fmt_tokens(inp), fmt_tokens(out), fmt_tokens(cr), fmt_tokens(cc), fmt_cost(cost)])

    print_table(["Model", "Input", "Output", "Cache Read", "Cache Create", "Est. Cost"],
                rows, {1, 2, 3, 4, 5})
    print(f"\nTotal estimated cost: {fmt_cost(total_cost)}")
    print(f"Total sessions: {stats.get('totalSessions', '?')}")
    print(f"Total messages: {stats.get('totalMessages', '?')}")


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="Claude Code usage reporter — reads local logs, no dependencies required.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    sub = parser.add_subparsers(dest="command", required=True)

    # Shared args
    def add_common(p):
        p.add_argument("--since", "-s", help="Filter from date (YYYY-MM-DD)")
        p.add_argument("--until", "-u", help="Filter until date (YYYY-MM-DD)")
        p.add_argument("--json", "-j", action="store_true", help="Output as JSON")

    p_daily = sub.add_parser("daily", help="Usage grouped by date")
    add_common(p_daily)
    p_daily.set_defaults(func=cmd_daily)

    p_monthly = sub.add_parser("monthly", help="Monthly aggregated report")
    add_common(p_monthly)
    p_monthly.set_defaults(func=cmd_monthly)

    p_sessions = sub.add_parser("sessions", help="Per-session detail")
    add_common(p_sessions)
    p_sessions.add_argument("--last", "-n", type=int, help="Show only the N most recent sessions")
    p_sessions.set_defaults(func=cmd_sessions)

    p_models = sub.add_parser("models", help="Per-model all-time totals")
    p_models.add_argument("--json", "-j", action="store_true", help="Output as JSON")
    p_models.set_defaults(func=cmd_models)

    args = parser.parse_args()

    # Normalize date args
    if hasattr(args, "since") and args.since:
        args.since = parse_date(args.since)
    if hasattr(args, "until") and args.until:
        args.until = parse_date(args.until)

    warn_if_price_table_stale()
    args.func(args)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""
Codex CLI usage reporter — stdlib-only CLI tool.

Reads local OpenAI Codex CLI session logs to produce token and cost reports
without any third-party dependencies.

Data source: ~/.codex/sessions/ (override with CODEX_HOME env var)

Subcommands:
  daily    — Usage grouped by date
  monthly  — Monthly aggregated report
  sessions — Per-session detail
  models   — Per-model all-time totals

Usage:
  python scripts/codex-usage.py daily
  python scripts/codex-usage.py daily --since 2026-04-01 --until 2026-04-07
  python scripts/codex-usage.py monthly --json
  python scripts/codex-usage.py sessions --last 10
  python scripts/codex-usage.py models
"""

from __future__ import annotations

import argparse
import glob
import hashlib
import json
import os
import sys
from collections import defaultdict
from datetime import date
from pathlib import Path


# ---------------------------------------------------------------------------
# Config
# ---------------------------------------------------------------------------

CODEX_HOME = Path(os.environ.get("CODEX_HOME", Path.home() / ".codex"))
SESSIONS_DIR = CODEX_HOME / "sessions"

# ---------------------------------------------------------------------------
# Pricing table — USD per 1M tokens.
#
# Model IDs here are deliberately pinned and historical: replaying old sessions
# must price them at the rates that applied then, so retired IDs stay in the
# table. What rots is not the IDs but the *rates*, so the table expires loudly
# instead of being silently trusted. Bump PRICE_TABLE_LAST_VERIFIED when you
# re-check https://openai.com/api/pricing/; add new IDs without removing old
# ones. Unknown IDs fall through to DEFAULT_PRICING, which under-reports newer
# top-tier models — add them here rather than relying on the default.
#
# GPT-5.6 input/output rates mirror ai-llm/scripts/cost_estimator.py. The
# GPT-5.6 `cached` values are read off the price page (0.1x input). The older
# gpt-5/o3/o4-mini rows keep the 0.25x-of-input figures they were entered with,
# which is why the ratio is not uniform down the table.
# ---------------------------------------------------------------------------

# Import the resolver from this skill's OWN _lib/. The skill is self-contained:
# it carries its own _lib/ and data/, so it works detached from the repo
# (public-repo clone, single-folder copy, plugin). resolve() first because
# skills deploy as symlinks, so a lexical path escapes into the deployment root.
_here = Path(__file__).resolve()
sys.path.insert(0, str(_here.parents[1] / "_lib"))
try:
    from resolve_versions import load_pricing, pricing_path
except ImportError:  # resolver missing — use the embedded fallback
    load_pricing = None
    pricing_path = None

PRICE_TABLE_LAST_VERIFIED = date.fromisoformat("2026-08-10")
PRICE_TABLE_STALE_AFTER_DAYS = 30

FALLBACK_PRICING = {
    # Current tiers verified against developers.openai.com/api/docs/pricing on
    # 2026-08-10. Cached input is 0.1x base input, not the 0.25x assumed here
    # previously. The rows below this block are retired rates kept for replay.
    "gpt-5.6-sol":   {"input": 5.00, "output": 30.00, "cached": 0.50},
    "gpt-5.6-terra": {"input": 2.00, "output": 12.00, "cached": 0.20},
    "gpt-5.6-luna":  {"input": 0.20, "output": 1.20, "cached": 0.02},
    "gpt-5.5":   {"input": 2.00, "output": 8.00, "cached": 0.50},
    "gpt-5":     {"input": 2.00, "output": 8.00, "cached": 0.50},
    "gpt-5.4":   {"input": 2.00, "output": 8.00, "cached": 0.50},
    "gpt-4.1":   {"input": 2.00, "output": 8.00, "cached": 0.50},
    "o3":        {"input": 2.00, "output": 8.00, "cached": 0.50},
    "o4-mini":   {"input": 1.10, "output": 4.40, "cached": 0.275},
    "codex-mini": {"input": 1.50, "output": 6.00, "cached": 0.375},
}
DEFAULT_PRICING = None  # Unknown pricing is intentionally never guessed.


def _load_pricing() -> tuple[dict, str, date]:
    """Return (pricing, provenance, last_verified), preferring the shared table.

    Adapts the shared schema (`*_per_1m`) into this script's field names rather
    than renaming either side.
    """
    if load_pricing is None:
        return FALLBACK_PRICING, "embedded fallback (resolver not importable)", PRICE_TABLE_LAST_VERIFIED

    doc = load_pricing(__file__)
    models = doc.get("models") if isinstance(doc, dict) else None
    if not isinstance(models, dict):
        return FALLBACK_PRICING, "embedded fallback (shared table unavailable)", PRICE_TABLE_LAST_VERIFIED

    table = {}
    for key, entry in models.items():
        if not isinstance(entry, dict) or entry.get("vendor") != "openai":
            continue
        if "input_per_1m" not in entry or "output_per_1m" not in entry:
            continue
        table[key.split("/", 1)[-1]] = {
            "input": entry["input_per_1m"],
            "output": entry["output_per_1m"],
            # 0.1x input is the published cached-input ratio for current OpenAI
            # tiers; used only when the shared table carries no explicit column.
            "cached": entry.get("cache_read_per_1m", entry["input_per_1m"] * 0.10),
            "rate_source": entry.get("pricing_source", doc.get("sources", {}).get("openai")),
            "rate_verified_at": entry.get("pricing_verified_at", doc.get("last_verified")),
        }
        if "cache_write_per_1m" in entry:
            table[key.split("/", 1)[-1]]["cache_write"] = entry["cache_write_per_1m"]
    if not table:
        return FALLBACK_PRICING, "embedded fallback (no openai rows)", PRICE_TABLE_LAST_VERIFIED

    verified = PRICE_TABLE_LAST_VERIFIED
    stamp = doc.get("last_verified")
    if isinstance(stamp, str):
        try:
            verified = date.fromisoformat(stamp)
        except ValueError:
            pass
    path = pricing_path(__file__) if pricing_path else None
    return table, f"shared: {path}" if path else "shared", verified


PRICING, PRICING_SOURCE, PRICING_VERIFIED = _load_pricing()


# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------

def warn_if_price_table_stale() -> None:
    """Warn once if the embedded pricing table is older than the staleness window.

    Called from main() rather than estimate_cost() because the reports call
    estimate_cost per row; warning there would repeat the notice for every line.
    """
    age_days = (date.today() - PRICING_VERIFIED).days
    if age_days > PRICE_TABLE_STALE_AFTER_DAYS:
        print(
            f"[WARN] Pricing is {age_days} days old "
            f"(last verified {PRICING_VERIFIED.isoformat()}, source: {PRICING_SOURCE}); "
            "costs below are estimates — verify at https://openai.com/api/pricing/.",
            file=sys.stderr,
        )


def pricing_provenance() -> dict:
    """Return bounded, reproducible provenance; never emit session content."""
    path = pricing_path(__file__) if pricing_path else None
    digest = None
    if path:
        try:
            digest = hashlib.sha256(Path(path).read_bytes()).hexdigest()
        except OSError:
            pass
    return {
        "pricingSource": PRICING_SOURCE,
        "pricingLastVerified": PRICING_VERIFIED.isoformat(),
        "pricingSha256": digest,
    }


def attribute_cost(model: str, inp: int, out: int, cached: int,
                   cache_write: int = 0, *, usage_source: str = "last",
                   service_tier: str | None = None,
                   context_pricing_class: str | None = None) -> dict:
    """Fail-closed cost attribution for one logged request.

    The local log does not normally record service tier or the pricing-context
    band.  A known model alone therefore does not prove a billable list price.
    `last` is a request counter; a reconstructed cumulative delta is estimated.
    Reasoning is deliberately absent: it is included in output_tokens.
    """
    provenance = pricing_provenance()
    prices = PRICING.get(model)
    result = {
        "model": model,
        "usageSource": usage_source,
        "costUSD": None,
        "costStatus": "unpriced",
        "unpricedReason": None,
        **provenance,
    }
    if prices is None:
        result["unpricedReason"] = "unknown_model_id"
        return result
    result["rateSource"] = prices.get("rate_source")
    result["rateVerifiedAt"] = prices.get("rate_verified_at")
    if service_tier != "standard":
        result["unpricedReason"] = "ambiguous_service_tier"
        return result
    if context_pricing_class != "standard":
        result["unpricedReason"] = "ambiguous_long_context_pricing"
        return result
    if cache_write and "cache_write" not in prices:
        result["unpricedReason"] = "cache_write_rate_unavailable"
        return result
    non_cached = max(0, inp - cached - cache_write)
    result["costUSD"] = (
        non_cached * prices["input"] / 1_000_000
        + cached * prices["cached"] / 1_000_000
        + cache_write * prices.get("cache_write", 0) / 1_000_000
        + out * prices["output"] / 1_000_000
    )
    result["costStatus"] = "exact" if usage_source == "last" else "estimated"
    return result


def estimate_cost(model: str, inp: int, out: int, cached: int) -> float | None:
    """Compatibility helper: only return a cost where the attribution is safe."""
    return attribute_cost(model, inp, out, cached)["costUSD"]


def parse_date(s: str) -> str:
    s = s.replace("/", "-").strip()
    if len(s) == 8 and s.isdigit():
        return f"{s[:4]}-{s[4:6]}-{s[6:8]}"
    return s[:10]


def in_range(date_str: str, since: str | None, until: str | None) -> bool:
    if since and date_str < since:
        return False
    if until and date_str > until:
        return False
    return True


def fmt_tokens(n: int) -> str:
    return f"{n:,}"


def fmt_cost(c: float) -> str:
    return "unpriced" if c is None else f"${c:.2f}"


def fmt_cost_summary(priced_total: float, unpriced_rows: int) -> str:
    """Do not render an incomplete aggregation as a $0.00 total."""
    if not unpriced_rows:
        return fmt_cost(priced_total)
    if priced_total == 0:
        return "unpriced"
    return f"${priced_total:.2f} known subtotal; {unpriced_rows} unpriced row(s)"


def print_table(headers: list[str], rows: list[list[str]], right_align: set[int] | None = None):
    right_align = right_align or set()
    widths = [len(h) for h in headers]
    for row in rows:
        for i, cell in enumerate(row):
            widths[i] = max(widths[i], len(cell))

    def fmt_row(cells):
        parts = []
        for i, cell in enumerate(cells):
            if i in right_align:
                parts.append(cell.rjust(widths[i]))
            else:
                parts.append(cell.ljust(widths[i]))
        return "  ".join(parts)

    print(fmt_row(headers))
    print("  ".join("-" * w for w in widths))
    for row in rows:
        print(fmt_row(row))


# ---------------------------------------------------------------------------
# Data loading
# ---------------------------------------------------------------------------

def iter_session_files():
    """Yield (path, session_id) for all Codex JSONL session files."""
    if not SESSIONS_DIR.is_dir():
        return
    for path in sorted(glob.glob(str(SESSIONS_DIR / "**" / "*.jsonl"), recursive=True)):
        # Session ID from filename: rollout-{timestamp}-{uuid}.jsonl
        basename = os.path.basename(path).replace(".jsonl", "")
        yield path, basename


def parse_session_events(path: str):
    """
    Parse a Codex session JSONL file.

    Yields dicts with token usage per turn. Handles:
    - null last_token_usage (falls back to total_token_usage delta)
    - Missing model metadata (falls back to 'unknown')
    - Non-dict payloads
    """
    current_model = "unknown"
    service_tier = None
    context_pricing_class = None
    prev_totals = {"input_tokens": 0, "cached_input_tokens": 0,
                   "cache_write_input_tokens": 0, "output_tokens": 0,
                   "reasoning_output_tokens": 0}

    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                rec = json.loads(line)
            except json.JSONDecodeError:
                continue
            if not isinstance(rec, dict):
                continue

            rec_type = rec.get("type")
            payload = rec.get("payload")
            if not isinstance(payload, dict):
                continue

            # Extract model from turn_context
            if rec_type == "turn_context":
                model = payload.get("model")
                if model and model != current_model:
                    # Cumulative totals belong to a model epoch, not the whole
                    # JSONL file.  Reset before a later total-only event.
                    prev_totals = {key: 0 for key in prev_totals}
                if model:
                    current_model = model
                service_tier = payload.get("service_tier")
                context_pricing_class = payload.get("context_pricing_class")

            # Extract token usage from event_msg with token_count
            if rec_type == "event_msg" and payload.get("type") == "token_count":
                info = payload.get("info")
                if not isinstance(info, dict):
                    continue

                timestamp = rec.get("timestamp", "")

                # Prefer last_token_usage (per-turn delta)
                last = info.get("last_token_usage")
                if isinstance(last, dict):
                    yield {
                        "timestamp": timestamp,
                        "model": current_model,
                        "input": last.get("input_tokens", 0) or 0,
                        "output": last.get("output_tokens", 0) or 0,
                        "cached": last.get("cached_input_tokens", 0) or 0,
                        "cache_write": last.get("cache_write_input_tokens", 0) or 0,
                        "reasoning": last.get("reasoning_output_tokens", 0) or 0,
                        "usage_source": "last",
                        "service_tier": service_tier,
                        "context_pricing_class": context_pricing_class,
                    }
                    # Update prev_totals from total if available
                    total = info.get("total_token_usage")
                    if isinstance(total, dict):
                        prev_totals = {
                            "input_tokens": total.get("input_tokens", 0) or 0,
                            "cached_input_tokens": total.get("cached_input_tokens", 0) or 0,
                            "cache_write_input_tokens": total.get("cache_write_input_tokens", 0) or 0,
                            "output_tokens": total.get("output_tokens", 0) or 0,
                            "reasoning_output_tokens": total.get("reasoning_output_tokens", 0) or 0,
                        }
                    continue

                # Fallback: compute delta from cumulative total_token_usage
                total = info.get("total_token_usage")
                if isinstance(total, dict):
                    raw = {key: total.get(key, 0) or 0 for key in prev_totals}
                    reset = any(raw[key] < prev_totals[key] for key in prev_totals)
                    base = {key: 0 for key in prev_totals} if reset else prev_totals
                    inp = raw["input_tokens"] - base["input_tokens"]
                    out = raw["output_tokens"] - base["output_tokens"]
                    cached = raw["cached_input_tokens"] - base["cached_input_tokens"]
                    reasoning = raw["reasoning_output_tokens"] - base["reasoning_output_tokens"]
                    cache_write = raw.get("cache_write_input_tokens", 0) - base.get("cache_write_input_tokens", 0)

                    if inp > 0 or out > 0:
                        yield {
                            "timestamp": timestamp,
                            "model": current_model,
                            "input": max(0, inp),
                            "output": max(0, out),
                            "cached": max(0, cached),
                            "cache_write": max(0, cache_write),
                            "reasoning": max(0, reasoning),
                            "usage_source": "total_delta_reset" if reset else "total_delta",
                            "service_tier": service_tier,
                            "context_pricing_class": context_pricing_class,
                        }

                    prev_totals = raw


def iter_all_events(since: str | None = None, until: str | None = None):
    """Yield all token usage events across all sessions, optionally filtered by date."""
    for path, session_id in iter_session_files():
        for event in parse_session_events(path):
            date = event["timestamp"][:10]
            if not in_range(date, since, until):
                continue
            event["session_id"] = session_id
            event["attribution"] = attribute_cost(
                event["model"], event["input"], event["output"], event["cached"],
                event.get("cache_write", 0), usage_source=event["usage_source"],
                service_tier=event.get("service_tier"),
                context_pricing_class=event.get("context_pricing_class"),
            )
            yield event


# ---------------------------------------------------------------------------
# Subcommand: daily
# ---------------------------------------------------------------------------

def cmd_daily(args):
    daily = defaultdict(lambda: {"input": 0, "output": 0, "cached": 0, "reasoning": 0, "turns": 0, "models": set()})

    for ev in iter_all_events(args.since, args.until):
        date = ev["timestamp"][:10]
        d = daily[date]
        d["input"] += ev["input"]
        d["output"] += ev["output"]
        d["cached"] += ev["cached"]
        d["reasoning"] += ev["reasoning"]
        d["turns"] += 1
        d["models"].add(ev["model"])

    if not daily:
        print(f"No data found. Checked: {SESSIONS_DIR}", file=sys.stderr)
        sys.exit(1)

    if args.json:
        out = []
        for date in sorted(daily):
            d = daily[date]
            cost = estimate_cost("", d["input"], d["output"], d["cached"])
            out.append({"date": date, "inputTokens": d["input"], "outputTokens": d["output"],
                        "cachedInputTokens": d["cached"], "reasoningOutputTokens": d["reasoning"],
                        "turns": d["turns"], "costUSD": round(cost, 4) if cost is not None else None,
                        "models": sorted(d["models"])})
        json.dump({"daily": out}, sys.stdout, indent=2)
        print()
        return

    rows = []
    total_cost = 0.0
    unpriced_rows = 0
    for date in sorted(daily):
        d = daily[date]
        cost = estimate_cost("", d["input"], d["output"], d["cached"])
        total_cost += cost or 0
        unpriced_rows += cost is None
        rows.append([date, fmt_tokens(d["input"]), fmt_tokens(d["output"]),
                      fmt_tokens(d["cached"]), fmt_tokens(d["reasoning"]),
                      str(d["turns"]), fmt_cost(cost)])

    print_table(["Date", "Input", "Output", "Cached", "Reasoning", "Turns", "Est. Cost"],
                rows, {1, 2, 3, 4, 5, 6})
    print(f"\nCost summary: {fmt_cost_summary(total_cost, unpriced_rows)}")


# ---------------------------------------------------------------------------
# Subcommand: monthly
# ---------------------------------------------------------------------------

def cmd_monthly(args):
    monthly = defaultdict(lambda: {"input": 0, "output": 0, "cached": 0, "reasoning": 0, "turns": 0})

    for ev in iter_all_events(args.since, args.until):
        month = ev["timestamp"][:7]
        m = monthly[month]
        m["input"] += ev["input"]
        m["output"] += ev["output"]
        m["cached"] += ev["cached"]
        m["reasoning"] += ev["reasoning"]
        m["turns"] += 1

    if not monthly:
        print(f"No data found. Checked: {SESSIONS_DIR}", file=sys.stderr)
        sys.exit(1)

    if args.json:
        out = []
        for month in sorted(monthly):
            m = monthly[month]
            cost = estimate_cost("", m["input"], m["output"], m["cached"])
            out.append({"month": month, "inputTokens": m["input"], "outputTokens": m["output"],
                        "cachedInputTokens": m["cached"], "reasoningOutputTokens": m["reasoning"],
                        "turns": m["turns"], "costUSD": round(cost, 4) if cost is not None else None})
        json.dump({"monthly": out}, sys.stdout, indent=2)
        print()
        return

    rows = []
    total_cost = 0.0
    unpriced_rows = 0
    for month in sorted(monthly):
        m = monthly[month]
        cost = estimate_cost("", m["input"], m["output"], m["cached"])
        total_cost += cost or 0
        unpriced_rows += cost is None
        rows.append([month, fmt_tokens(m["input"]), fmt_tokens(m["output"]),
                      fmt_tokens(m["cached"]), fmt_tokens(m["reasoning"]),
                      str(m["turns"]), fmt_cost(cost)])

    print_table(["Month", "Input", "Output", "Cached", "Reasoning", "Turns", "Est. Cost"],
                rows, {1, 2, 3, 4, 5, 6})
    print(f"\nCost summary: {fmt_cost_summary(total_cost, unpriced_rows)}")


# ---------------------------------------------------------------------------
# Subcommand: sessions
# ---------------------------------------------------------------------------

def cmd_sessions(args):
    sess = defaultdict(lambda: {"input": 0, "output": 0, "cached": 0, "reasoning": 0,
                                "turns": 0, "first_ts": "", "last_ts": "", "model": "unknown"})

    for ev in iter_all_events(args.since, args.until):
        sid = ev["session_id"]
        s = sess[sid]
        s["input"] += ev["input"]
        s["output"] += ev["output"]
        s["cached"] += ev["cached"]
        s["reasoning"] += ev["reasoning"]
        s["turns"] += 1
        s["model"] = ev["model"]
        if not s["first_ts"] or ev["timestamp"] < s["first_ts"]:
            s["first_ts"] = ev["timestamp"]
        if not s["last_ts"] or ev["timestamp"] > s["last_ts"]:
            s["last_ts"] = ev["timestamp"]

    if not sess:
        print(f"No data found. Checked: {SESSIONS_DIR}", file=sys.stderr)
        sys.exit(1)

    # Sort by most recent first
    sorted_sessions = sorted(sess.items(), key=lambda kv: kv[1]["last_ts"], reverse=True)
    if args.last:
        sorted_sessions = sorted_sessions[:args.last]

    if args.json:
        out = []
        for sid, s in sorted_sessions:
            cost = estimate_cost(s["model"], s["input"], s["output"], s["cached"])
            out.append({"sessionId": sid, "date": s["first_ts"][:10],
                        "model": s["model"], "inputTokens": s["input"],
                        "outputTokens": s["output"], "cachedInputTokens": s["cached"],
                        "turns": s["turns"], "costUSD": round(cost, 4) if cost is not None else None})
        json.dump({"sessions": out}, sys.stdout, indent=2)
        print()
        return

    rows = []
    total_cost = 0.0
    unpriced_rows = 0
    for sid, s in sorted_sessions:
        cost = estimate_cost(s["model"], s["input"], s["output"], s["cached"])
        total_cost += cost or 0
        unpriced_rows += cost is None
        # Truncate session ID for display
        short_id = sid[:30] + "..." if len(sid) > 33 else sid
        rows.append([s["first_ts"][:10], short_id, s["model"],
                      fmt_tokens(s["input"]), fmt_tokens(s["output"]),
                      str(s["turns"]), fmt_cost(cost)])

    print_table(["Date", "Session", "Model", "Input", "Output", "Turns", "Est. Cost"],
                rows, {3, 4, 5, 6})
    print(f"\nCost summary: {fmt_cost_summary(total_cost, unpriced_rows)}")
    print(f"Showing {len(sorted_sessions)} session(s)")


# ---------------------------------------------------------------------------
# Subcommand: models
# ---------------------------------------------------------------------------

def cmd_models(args):
    models = defaultdict(lambda: {"input": 0, "output": 0, "cached": 0, "reasoning": 0, "turns": 0})

    for ev in iter_all_events(args.since, args.until):
        m = models[ev["model"]]
        m["input"] += ev["input"]
        m["output"] += ev["output"]
        m["cached"] += ev["cached"]
        m["reasoning"] += ev["reasoning"]
        m["turns"] += 1

    if not models:
        print(f"No data found. Checked: {SESSIONS_DIR}", file=sys.stderr)
        sys.exit(1)

    if args.json:
        out = {}
        for model, m in sorted(models.items()):
            cost = estimate_cost(model, m["input"], m["output"], m["cached"])
            out[model] = {"inputTokens": m["input"], "outputTokens": m["output"],
                          "cachedInputTokens": m["cached"], "reasoningOutputTokens": m["reasoning"],
                          "turns": m["turns"], "costUSD": round(cost, 4) if cost is not None else None}
        json.dump({"models": out}, sys.stdout, indent=2)
        print()
        return

    rows = []
    total_cost = 0.0
    unpriced_rows = 0
    for model, m in sorted(models.items()):
        cost = estimate_cost(model, m["input"], m["output"], m["cached"])
        total_cost += cost or 0
        unpriced_rows += cost is None
        rows.append([model, fmt_tokens(m["input"]), fmt_tokens(m["output"]),
                      fmt_tokens(m["cached"]), fmt_tokens(m["reasoning"]),
                      str(m["turns"]), fmt_cost(cost)])

    print_table(["Model", "Input", "Output", "Cached", "Reasoning", "Turns", "Est. Cost"],
                rows, {1, 2, 3, 4, 5, 6})
    print(f"\nCost summary: {fmt_cost_summary(total_cost, unpriced_rows)}")


def cmd_traces(args):
    """Emit bounded per-request accounting rows, never transcript payloads."""
    rows = []
    for ev in iter_all_events(args.since, args.until):
        a = ev["attribution"]
        rows.append({
            "sessionId": ev["session_id"], "timestamp": ev["timestamp"],
            "model": ev["model"], "inputTokens": ev["input"],
            "cachedInputTokens": ev["cached"], "cacheWriteInputTokens": ev.get("cache_write", 0),
            "outputTokens": ev["output"], "usageSource": ev["usage_source"],
            "costUSD": a["costUSD"], "costStatus": a["costStatus"],
            "unpricedReason": a["unpricedReason"],
            "pricingSource": a["pricingSource"],
            "pricingLastVerified": a["pricingLastVerified"], "pricingSha256": a["pricingSha256"],
            "rateSource": a.get("rateSource"), "rateVerifiedAt": a.get("rateVerifiedAt"),
        })
    json.dump({"traces": rows}, sys.stdout, indent=2)
    print()


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="Codex CLI usage reporter — reads local session logs, no dependencies required.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    sub = parser.add_subparsers(dest="command", required=True)

    def add_common(p):
        p.add_argument("--since", "-s", help="Filter from date (YYYY-MM-DD)")
        p.add_argument("--until", "-u", help="Filter until date (YYYY-MM-DD)")
        p.add_argument("--json", "-j", action="store_true", help="Output as JSON")

    p_daily = sub.add_parser("daily", help="Usage grouped by date")
    add_common(p_daily)
    p_daily.set_defaults(func=cmd_daily)

    p_monthly = sub.add_parser("monthly", help="Monthly aggregated report")
    add_common(p_monthly)
    p_monthly.set_defaults(func=cmd_monthly)

    p_sessions = sub.add_parser("sessions", help="Per-session detail")
    add_common(p_sessions)
    p_sessions.add_argument("--last", "-n", type=int, help="Show only the N most recent sessions")
    p_sessions.set_defaults(func=cmd_sessions)

    p_models = sub.add_parser("models", help="Per-model all-time totals")
    add_common(p_models)
    p_models.set_defaults(func=cmd_models)

    p_traces = sub.add_parser("traces", help="Per-request bounded cost-attribution JSON")
    add_common(p_traces)
    p_traces.set_defaults(func=cmd_traces)

    args = parser.parse_args()

    if hasattr(args, "since") and args.since:
        args.since = parse_date(args.since)
    if hasattr(args, "until") and args.until:
        args.until = parse_date(args.until)

    warn_if_price_table_stale()
    args.func(args)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Deterministic regression tests for fail-closed Codex usage attribution."""

import importlib.util
import json
import tempfile
import unittest
from pathlib import Path


MODULE = Path(__file__).with_name("codex-usage.py")
SPEC = importlib.util.spec_from_file_location("codex_usage", MODULE)
codex_usage = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(codex_usage)
SOL_MODEL = next(model for model in codex_usage.PRICING if model.endswith("-sol"))
TERRA_MODEL = next(model for model in codex_usage.PRICING if model.endswith("-terra"))


def usage(inp, out=0, cached=0, cache_write=0):
    return {"input_tokens": inp, "output_tokens": out,
            "cached_input_tokens": cached, "cache_write_input_tokens": cache_write,
            "reasoning_output_tokens": 0, "total_tokens": inp + out}


def context(model, tier="standard", context_class="standard"):
    return {"type": "turn_context", "payload": {
        "model": model, "service_tier": tier,
        "context_pricing_class": context_class}}


def token(last=None, total=None):
    return {"timestamp": "2026-08-15T00:00:00Z", "type": "event_msg", "payload": {
        "type": "token_count", "info": {
            "last_token_usage": last, "total_token_usage": total}}}


class CodexUsageTests(unittest.TestCase):
    def parse(self, records):
        with tempfile.TemporaryDirectory() as directory:
            path = Path(directory) / "trace.jsonl"
            path.write_text("".join(json.dumps(row) + "\n" for row in records))
            return list(codex_usage.parse_session_events(str(path)))

    def test_exact_lookup_does_not_match_gpt5_prefix(self):
        result = codex_usage.attribute_cost(SOL_MODEL, 1_000_000, 0, 0,
                                            service_tier="standard",
                                            context_pricing_class="standard")
        self.assertEqual(result["costStatus"], "exact")
        self.assertEqual(result["costUSD"], 5.0)
        self.assertTrue(result["pricingSha256"])
        self.assertTrue(result["rateSource"].endswith(SOL_MODEL))

    def test_unknown_and_unpriced_cache_write_fail_closed(self):
        unknown = codex_usage.attribute_cost("unpriced-test-model", 1, 0, 0,
                                              service_tier="standard",
                                              context_pricing_class="standard")
        self.assertEqual((unknown["costStatus"], unknown["unpricedReason"]),
                         ("unpriced", "unknown_model_id"))
        write = codex_usage.attribute_cost(TERRA_MODEL, 10, 0, 0, 1,
                                            service_tier="standard",
                                            context_pricing_class="standard")
        self.assertEqual(write["unpricedReason"], "cache_write_rate_unavailable")
        sol_write = codex_usage.attribute_cost(SOL_MODEL, 1_000_000, 0, 0, 1_000_000,
                                                service_tier="standard",
                                                context_pricing_class="standard")
        self.assertEqual(sol_write["costUSD"], 6.25)

    def test_model_change_resets_total_delta_epoch(self):
        rows = self.parse([
            context(SOL_MODEL), token(None, usage(100)),
            context(TERRA_MODEL), token(None, usage(30)),
        ])
        self.assertEqual([(row["model"], row["input"], row["usage_source"]) for row in rows],
                         [(SOL_MODEL, 100, "total_delta"),
                          (TERRA_MODEL, 30, "total_delta")])

    def test_total_reset_is_new_epoch_and_marked_estimated(self):
        rows = self.parse([context(SOL_MODEL), token(None, usage(100)),
                           token(None, usage(25))])
        self.assertEqual(rows[1]["input"], 25)
        self.assertEqual(rows[1]["usage_source"], "total_delta_reset")
        priced = codex_usage.attribute_cost(rows[1]["model"], rows[1]["input"], 0, 0,
                                            usage_source=rows[1]["usage_source"],
                                            service_tier="standard",
                                            context_pricing_class="standard")
        self.assertEqual(priced["costStatus"], "estimated")

    def test_last_usage_beats_cumulative_sum_trap(self):
        rows = self.parse([context(SOL_MODEL), token(usage(10), usage(10)),
                           token(usage(10), usage(20))])
        self.assertEqual(sum(row["input"] for row in rows), 20)
        self.assertNotEqual(sum((10, 20)), sum(row["input"] for row in rows))

    def test_cost_summary_never_turns_unpriced_into_zero(self):
        self.assertEqual(codex_usage.fmt_cost_summary(0.0, 2), "unpriced")
        self.assertEqual(codex_usage.fmt_cost_summary(1.25, 1),
                         "$1.25 known subtotal; 1 unpriced row(s)")
        self.assertEqual(codex_usage.fmt_cost_summary(1.25, 0), "$1.25")


if __name__ == "__main__":
    unittest.main()

---
name: ai-agents
description: AI agent architecture, graph and loop composition, protocol choice, evaluation, and observability. Use when scoping or reviewing systems before implementation.
compatibility: Portable core. Works on Claude Code and Codex.
version: "1.1"
last_validated: 2026-08-11
---

# AI Agents Development — Architecture Hub

Use this skill to decide whether a workflow should be an agent, which agent shape fits, which protocol boundary to use, and what production controls must exist before rollout.

Default posture: explicit control flow, bounded tools, typed contracts, auditable state, human approval for high-risk actions, and telemetry from day one.

Keep this file lean. Load detail from [`references/index.md`](references/index.md), `assets/`, and sibling skills only when needed.

## When to Use This Skill

Use this skill when the user asks for:

- agent architecture or operating-model decisions
- build-vs-not-agent assessment
- MCP vs A2A protocol choice
- production readiness review for an existing agent system
- evaluation, observability, rollout, or safety planning
- framework selection after requirements are already clear
- a starting template for a new agent spec
- graph engineering, agent/workflow graphs, state-machine orchestration, cyclic graphs, or DAG-versus-loop design
- loop engineering, run-until-done coding agents, self-improving workflows, evaluator feedback loops, or bounded autonomous iteration
- uncertainty over whether a "graph" means execution control flow, an improvement network, or a knowledge/context graph

## Use Other Skills for Depth

- Prompt contracts and structured outputs → [`../ai-prompt-engineering/SKILL.md`](../ai-prompt-engineering/SKILL.md)
- Retrieval, chunking, reranking, search quality → [`../ai-rag/SKILL.md`](../ai-rag/SKILL.md)
- Vector-brain implementation, schemas, ingest scripts, manifests, and retrieval tool contracts → [`../ai-vector-brain/SKILL.md`](../ai-vector-brain/SKILL.md)
- Bot building (support, sales, conversation design, LangGraph) → `ai-bot-builder`
- Voice bots (STT/TTS pipeline, telephony, latency) → [`../ai-voice-bots/SKILL.md`](../ai-voice-bots/SKILL.md)
- MCP server setup, transports, server builds → [`../agents-mcp/SKILL.md`](../agents-mcp/SKILL.md)
- Subagents, delegation contracts, least-privilege tools → `agents-subagents`
- CLI-based tools (non-interactive, idempotent, agent-friendly patterns) → [`../software-devtools/SKILL.md`](../software-devtools/SKILL.md)
- Evaluation harnesses, attack suites, regression gates → [`../qa-agent-testing/SKILL.md`](../qa-agent-testing/SKILL.md)
- Deployment guardrails and model operations → [`../ai-mlops/SKILL.md`](../ai-mlops/SKILL.md)
- Application security and high-risk controls → [`../software-security-appsec/SKILL.md`](../software-security-appsec/SKILL.md)
- Model and inference cost tuning → [`../ai-llm/SKILL.md`](../ai-llm/SKILL.md), [`../ai-llm-inference/SKILL.md`](../ai-llm-inference/SKILL.md)
- Knowledge/context graphs, retrieval architecture, and graph-backed memory → `ai-context-layer`, [`../ai-rag/SKILL.md`](../ai-rag/SKILL.md), [`../ai-vector-brain/SKILL.md`](../ai-vector-brain/SKILL.md)

## Default Workflow

1. Run the build-vs-not decision gate.
2. Define the task environment: performance measure, environment, percepts/sensors, actions/tools, observability, determinism, time horizon, and single-agent vs multi-agent interaction.
3. Choose control flow; default to workflow/FSM/DAG for production.
4. Choose protocol boundaries: MCP for tools/data, A2A for agent handoffs.
5. Define contracts: tool schemas, handoff payloads, state model, success criteria.
6. Add evaluation and telemetry before shipping.
7. Add human approval, rollback, and kill-switches for irreversible actions.
8. Start from templates, then route to specialized skills for implementation depth.

## ASCII Flow

```text
Agent-system request
  -> Build-vs-not gate
     +-- simpler workflow/form/tool fits -> do not build an agent
     +-- autonomy justified              -> continue
  -> Define task environment and performance measure
  -> Choose control flow: workflow, FSM, DAG, or agent loop
  -> Set boundaries: MCP for tools/data, A2A for handoffs
  -> Define state, schemas, success criteria, and approvals
  -> Add evals, telemetry, rollback, and kill switches
  -> Route implementation depth to specialized skills
```

## Known Traps

- treating "agent" as the default interaction pattern when a workflow, form, or plain tool call would be simpler
- using MCP as the overall agent architecture instead of the tool and resource integration layer
- adding long-term memory without provenance (A13), retention policy, correction flow (A11 — forget path), and user-value proof. Use [`ai-context-layer/patterns-catalog.md`](../ai-context-layer/references/patterns-catalog.md) to pick a named pattern and [`ai-context-layer/anti-patterns-catalog.md`](../ai-context-layer/references/anti-patterns-catalog.md) for the sweep
- letting planner loops recurse without explicit step, budget, and escalation limits
- shipping autonomous actions before evaluator coverage, rollback controls, and human approval paths exist
- defaulting to a multi-agent topology for reasoning-heavy work when a single strong model at an equal token budget matches or beats it — under a fixed reasoning-token budget, message-passing between agents loses mutual information vs. full-context conditioning (Data Processing Inequality). Confirm the task is genuinely parallelizable or tool/role-diverse before fanning out. (arXiv:2604.02460, Apr 2026 preprint — not peer-reviewed; scope: text-only multi-hop reasoning; verify before relying.)

## Common Anti-Patterns

- chat-first agent design with no state model, contract, or action boundary
- multi-agent topologies introduced before single-agent failure modes are understood
- tool surfaces defined by convenience rather than least privilege
- evaluation added after launch as observability theater instead of a release gate
- provider or framework selection driven by hype, benchmark screenshots, or marketing taxonomy alone

## Quick Reference

| Question | Default |
| --- | --- |
| Should this be an agent? | Start with [`references/build-vs-not-decision.md`](references/build-vs-not-decision.md); default answer is "no" until volume, ambiguity, and value justify autonomy. |
| What is the task environment? | State the performance measure, environment, percepts/sensors, actions/tools, observability, determinism, horizon, and other agents before choosing a framework. |
| Which control flow fits? | Prefer workflow/FSM/DAG; use planner/executor only when branching cannot be modeled explicitly. |
| MCP or A2A? | MCP for external tools/data, A2A for agent-to-agent coordination, both when a multi-agent system also needs tools. |
| When to use multi-agent? | Only when roles, handoff contracts, and verifier responsibilities are explicit. |
| When to add long-term memory? | Only with provenance, retention rules, user consent, and clear value. Pick a named pattern from [`ai-context-layer/patterns-catalog.md`](../ai-context-layer/references/patterns-catalog.md) (P2 for app-orchestrated, P3 for self-editing, P4 for temporal, P6 for conversational). Run the anti-pattern sweep — A1 (no raw transcripts), A11 (forget path required), A13 (provenance mandatory). |
| What must exist before rollout? | Eval suite, telemetry, action limits, human escalation, rollback path, and kill switch. |
| Is this "graph engineering" or "loop engineering"? | Start with [`references/graph-and-loop-engineering.md`](references/graph-and-loop-engineering.md); identify the graph's purpose before selecting a runtime or datastore. |

## Autonomy Shapes — How to Host an Agent 24/7

Three deployment shapes for running agents continuously. Pick by the **trigger model**, not by the framework.

| Shape | Trigger | When to use | Primary guide |
|---|---|---|---|
| **A — Triggered / hosted run** | Webhook, queue, schedule, `/fire` | Per-event agent work, scheduled jobs, fan-out from external sources | [`../ai-coding-agents-tasks/references/webhook-and-queue-triggers.md`](../ai-coding-agents-tasks/references/webhook-and-queue-triggers.md) + [`../ai-coding-agents-tasks/references/durable-trigger-integration.md`](../ai-coding-agents-tasks/references/durable-trigger-integration.md) |
| **B — Always-on bot / voice server** | Sessions, WebSocket, SIP call | Support / sales / voice bots with conversational state | [`../ai-bot-builder/references/production-deployment.md`](../ai-bot-builder/references/production-deployment.md) + [`../ai-bot-builder/references/stateful-rollout-and-blue-green.md`](../ai-bot-builder/references/stateful-rollout-and-blue-green.md) + [`../ai-voice-bots/references/production-deployment.md`](../ai-voice-bots/references/production-deployment.md) |
| **C — Autonomous loop** | PRD + loop driver until acceptance met | Long-horizon work: refactors, migrations, research, Ralph-Loop class | [`references/autonomous-loop-patterns.md`](references/autonomous-loop-patterns.md) |

Cross-shape requirements:

- **Budget and kill-switch enforcement**: [`../agents-hooks/references/budget-and-loop-hooks.md`](../agents-hooks/references/budget-and-loop-hooks.md)
- **24/7 operating model (SLOs, on-call, runbooks)**: [`references/24-7-operating-model.md`](references/24-7-operating-model.md)
- **Provider failover and secret rotation**: [`../ai-bot-builder/references/secret-rotation-and-model-fallback.md`](../ai-bot-builder/references/secret-rotation-and-model-fallback.md)
- **Where to host (Vercel / Fly.io / Railway / Cloudflare / Render)**: [`../software-paas-hosting/SKILL.md`](../software-paas-hosting/SKILL.md) and [`../software-paas-hosting/references/agent-hosting-matrix.md`](../software-paas-hosting/references/agent-hosting-matrix.md)

## Architecture Selection

| Need | Default Agent Shape | Notes |
| --- | --- | --- |
| Deterministic business process | Workflow agent | Best default for auditable production behavior. |
| Bounded external actions | Tool-using agent | Keep tools narrow, typed, and permission-scoped. |
| Knowledge-grounded answers | RAG agent | Require citations, ACL-aware retrieval, and refusal on missing evidence. |
| Long multi-step work with branching | Planner/executor | Use strict step budgets, checkpoints, and replanning limits. |
| Specialized roles with explicit ownership | Multi-agent orchestrator | Handoffs are APIs; add verifier/evaluator roles early. Score the design against the MAST failure taxonomy — do not re-derive it: [`../agents-subagents/references/mast-failure-taxonomy.md`](../agents-subagents/references/mast-failure-taxonomy.md) (14 modes; original Cemri et al. distribution ≈ Specification/System Design 41.8% / Inter-Agent Misalignment 36.9% / Task Verification 21.3% — figures vary across secondary write-ups, verify against the primary paper before quoting; NeurIPS 2025 Datasets & Benchmarks track — arXiv:2503.13657). |
| Desktop or browser control | OS agent | Require sandboxing, UI verification, and action gating. |
| Code changes and CI feedback | SWE agent | Require repo isolation, tests, review gates, and rollback. |
| Autonomous improvement of code, prompts, or artifacts | Research / experiment agent | Fixed eval metric, bounded modification surface, keep/revert loop |

### Autonomous Improvement Loops

A research/experiment agent iterates autonomously: suggest a change → apply it → evaluate → keep or revert → repeat. The pattern works on anything with a measurable evaluation function.

| Component | Purpose | Example |
|-----------|---------|---------|
| **Modification surface** | What the agent can change | One file (`train.py`), one prompt, one config |
| **Eval function** | How to score the result | `val_bpb`, yes/no checklist (3-6 questions), latency measurement |
| **Keep/revert rule** | When to keep a change | Score improves; revert if it regresses |
| **Termination** | When to stop | N rounds, target score reached, or manual stop |
| **Ledger** | Experiment history | Git commits, results.tsv, changelog with reasoning |

Design constraints:
- Keep the modification surface small (one file, one prompt). Broader surfaces compound silent regressions.
- Use binary or scalar metrics, not subjective ratings. "Does the headline include a specific number?" beats "rate the headline quality 1-10."
- 3-6 eval criteria is the sweet spot. More causes gaming; fewer misses failure modes.
- Preserve the changelog — future models pick up where the last agent left off.

See: [autoresearch](https://github.com/karpathy/autoresearch) (ML training), Lehmann's skill-optimization adaptation (prompt/skill improvement).

## Protocol Choice

| If the system needs... | Use |
| --- | --- |
| tool calls, database access, file access, prompts, resources | MCP |
| task handoffs, agent cards, multi-agent routing | A2A |
| both tools and collaborating agents | MCP + A2A |

Protocol defaults:

- Treat MCP as the tool/data integration layer, not as the agent architecture itself.
- Treat A2A handoffs as versioned APIs with schema validation and `trace_id` propagation.
- Prefer `stdio` or Streamable HTTP for MCP transports; treat older SSE-only guidance as compatibility material, not the default.
- For remote MCP, scope authorization and identity explicitly; do not rely on network trust alone.
- Known footgun: the official MCP SDK `stdio` interface has a by-design config→OS-command execution path (CVE-2026-30623; ~7k public servers exposed; Anthropic confirmed by-design and declined a protocol-level fix — input sanitization is the integrator's responsibility). Never pass untrusted server config into an `stdio` MCP launch; sandbox the MCP host process. Verified multi-source, April 2026.

## Agent-as-Code Pattern

Define agent personas as structured, versioned artifacts with explicit expertise, constraints, and expected outputs. This pattern treats agent definitions as reviewable code rather than ad-hoc prompt strings.

A well-defined agent spec includes:

| Field | Purpose |
|-------|---------|
| **Role** | What the agent is responsible for (e.g., "Architect", "QA reviewer") |
| **Expertise** | Domain knowledge and capabilities |
| **Constraints** | What it must not do, boundaries of authority |
| **Expected outputs** | Artifacts it produces (specs, reviews, plans, code) |
| **Interaction rules** | How it communicates with other agents or the human |

Benefits: agents can be reviewed in PRs, diffed between versions, and composed into teams with explicit role boundaries.

For current delivery methods and when to borrow from GSD, BMAD, Spec Kit, OpenSpec, MADD, or AI-SDLC, see [`references/agent-delivery-methods.md`](references/agent-delivery-methods.md).

### Scale-Adaptive Agent Complexity

Match agent sophistication to task complexity. Do not use a full multi-agent orchestration for a bug fix, and do not use a single prompt for a platform migration.

| Task Complexity | Agent Approach |
|----------------|----------------|
| Config change, typo | Direct prompt, no agent infrastructure |
| Bug fix, small feature | Single agent with bounded tools |
| Multi-module feature | Lead + 2-3 specialized workers |
| Cross-service migration | Full orchestration with persona definitions, debate, and verification |

The decision to scale up should be driven by observed ambiguity, not assumed complexity.

## Production Defaults

- Keep state explicit, serializable, and replayable.
- **Externalize all state to files** — plan, progress, decisions, and dependency outputs persist in structured files so any agent session can resume without context inheritance.
- Keep tool surfaces narrow; publish tasks, not raw backend complexity.
- Bound retries, budgets, context size, and recursion depth.
- Treat retrieved/tool content as untrusted input.
- Instrument LLM calls, retrieval, memory ops, and tool calls with consistent tracing.
- Gate database writes, financial actions, legal/compliance actions, and destructive operations behind human approval.
- Prefer refusal or degraded mode over hidden unsafe fallbacks.

For fresh-context workers, durable state, and session-vs-project boundaries, see [`references/context-rotation-and-state.md`](references/context-rotation-and-state.md).

## Templates And Entry Points

- Standard agent spec → [`assets/core/agent-template-standard.md`](assets/core/agent-template-standard.md)
- Quick prototype spec → [`assets/core/agent-template-quick.md`](assets/core/agent-template-quick.md)
- Specialized agent spec → [`assets/core/agent-template-specialized.md`](assets/core/agent-template-specialized.md)
- AI-native SDLC runbook → [`assets/agent-template-ainative-sdlc.md`](assets/agent-template-ainative-sdlc.md)
- Safety gate → [`assets/checklists/agent-safety-checklist.md`](assets/checklists/agent-safety-checklist.md)
- Tool schema template → [`assets/tools/tool-definition.md`](assets/tools/tool-definition.md)
- Tool validation checklist → [`assets/tools/tool-validation-checklist.md`](assets/tools/tool-validation-checklist.md)
- Multi-agent starter patterns → [`assets/multi-agent/manager-worker-template.md`](assets/multi-agent/manager-worker-template.md), [`assets/multi-agent/evaluator-router-template.md`](assets/multi-agent/evaluator-router-template.md)
- RAG starter patterns → [`assets/rag/rag-basic.md`](assets/rag/rag-basic.md), [`assets/rag/rag-advanced.md`](assets/rag/rag-advanced.md), [`assets/rag/hybrid-retrieval.md`](assets/rag/hybrid-retrieval.md)

## Scripts

| Script | Purpose |
|--------|---------|
| `scripts/agent_eval_runner.py` | Read JSONL task/expected/actual triples and report pass rates (offline). For adversarial suites and multi-turn harnesses, delegate to [`../qa-agent-testing/SKILL.md`](../qa-agent-testing/SKILL.md). |
| `scripts/claude-usage.py` | Parse Claude Code usage logs |
| `scripts/codex-usage.py` | Parse Codex usage logs |

## Navigation

- Full deep-dive map → [`references/index.md`](references/index.md)
- Should we build this? → [`references/build-vs-not-decision.md`](references/build-vs-not-decision.md)
- MCP vs A2A → [`references/protocol-decision-tree.md`](references/protocol-decision-tree.md)
- Current operating defaults → [`references/modern-best-practices.md`](references/modern-best-practices.md)
- Delivery methods and planning systems → [`references/agent-delivery-methods.md`](references/agent-delivery-methods.md)
- Evaluation and telemetry → [`references/evaluation-and-observability.md`](references/evaluation-and-observability.md)
- CLI usage tracking → [`references/coding-agent-usage-tracking.md`](references/coding-agent-usage-tracking.md)
- Context rotation and durable state → [`references/context-rotation-and-state.md`](references/context-rotation-and-state.md)
- Deployment safety → [`references/deployment-ci-cd-and-safety.md`](references/deployment-ci-cd-and-safety.md)
- Autonomous loop / Ralph-Loop class → [`references/autonomous-loop-patterns.md`](references/autonomous-loop-patterns.md)
- Graph engineering, loop engineering, and graph-type disambiguation → [`references/graph-and-loop-engineering.md`](references/graph-and-loop-engineering.md)
- 24/7 operating model (SLOs, on-call, runbooks) → [`references/24-7-operating-model.md`](references/24-7-operating-model.md)
- Tool schemas and contracts → [`references/tool-design-specs.md`](references/tool-design-specs.md), [`references/api-contracts-for-agents.md`](references/api-contracts-for-agents.md)
- Curated external sources → [`data/sources.json`](data/sources.json)

## Related Skills

- Choosing agent vs single call vs RAG vs fine-tune → [`../ai-architecture-advisor/SKILL.md`](../ai-architecture-advisor/SKILL.md)
- Broad LLM system design → [`../ai-llm/SKILL.md`](../ai-llm/SKILL.md)
- LangGraph bot implementation → `ai-bot-builder`
- RAG implementation → [`../ai-rag/SKILL.md`](../ai-rag/SKILL.md)
- Vector-brain implementation → [`../ai-vector-brain/SKILL.md`](../ai-vector-brain/SKILL.md)
- MCP implementation → [`../agents-mcp/SKILL.md`](../agents-mcp/SKILL.md)
- Subagent orchestration → `agents-subagents`
- Swarm and parallel dispatch → [`../agents-swarm-orchestration/SKILL.md`](../agents-swarm-orchestration/SKILL.md)
- Hook guardrails and lifecycle → [`../agents-hooks/SKILL.md`](../agents-hooks/SKILL.md)
- Skill packaging → [`../agents-skills/SKILL.md`](../agents-skills/SKILL.md)
- Project memory → [`../agents-memory/SKILL.md`](../agents-memory/SKILL.md)
- Eval harnesses → [`../qa-agent-testing/SKILL.md`](../qa-agent-testing/SKILL.md)
- Observability → [`../qa-observability/SKILL.md`](../qa-observability/SKILL.md)
- Security and AppSec → [`../software-security-appsec/SKILL.md`](../software-security-appsec/SKILL.md)

## Fact-Checking

- Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
- Verify current protocol specs, framework capabilities, release status, and vendor behavior before final answers.
- Prefer official docs, primary specifications, and first-party repos for fast-moving agent infrastructure claims.
- If a volatile claim cannot be checked, label it as unverified instead of presenting it as settled guidance.

## Trend Awareness Protocol

When users ask for:

- "best framework for X"
- "is X still relevant"
- "latest AI agent stack"
- pricing, version, or support-matrix questions
- MCP transport/auth guidance
- A2A ecosystem or vendor SDK recommendations

verify with web search and primary sources before answering.

Volatile areas to re-check every time:

- framework language support and lifecycle status
- MCP transport and authorization guidance
- OpenAI/Anthropic/Google pricing
- A2A ecosystem maturity and official docs
- vendor-specific tool, tracing, and handoff capabilities

If browsing is unavailable, use [`data/sources.json`](data/sources.json), say what is assumed, and avoid strong ranking claims.

## Usage Notes

- Start here for architecture and production posture, not for deep implementation walkthroughs.
- Prefer stable capability guidance over dated framework rankings.
- Load detailed references only after the user’s direction is clear.
- Keep recommendations operational: contracts, failure modes, gates, telemetry, and rollback.

## Learnings Loop

Before applying this skill on a non-trivial task, read `learnings.consolidated.md` in this directory (and `learnings.md` if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.