vasilyu1983/ai-agents-public · GitHub
Designs coding-agent observability and evals. Use when measuring traces, replay, checkpoint lineage, quality trajectories, tool grading, regression, or cost.
프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.
npx skills add vasilyu1983/ai-agents-public --skill ai-coding-agents-observability-evals설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.
agents/openai.yamlinterface: display_name: "AI Coding Agents — Observability And Evals" short_description: "Designs coding-agent observability, traces, evals, lineage, and quality gates" default_prompt: "Use $ai-coding-agents-observability-evals to design traces, replay, regression suites, long-horizon quality trajectories, tool-call grading, latency metrics, or cost accounting for a coding-agent runtime."
assets/templates/golden-task.schema.json{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://skills.ai-coding-agents/observability-evals/golden-task.schema.json",
"title": "GoldenTask",
"description": "A single entry in the golden-task corpus used to drive regression evals for a coding agent. Each entry defines an input, the expected artifact, and the scoring criteria.",
"type": "object",
"required": ["id", "version", "task", "expected", "scoring"],
"additionalProperties": false,
"properties": {
"id": {
"type": "string",
"pattern": "^[a-z0-9][a-z0-9-]{2,79}$",
"description": "Stable, kebab-case identifier. Never reuse or rename — eval history is keyed on this."
},
"version": {
"type": "integer",
"minimum": 1,
"description": "Increment when the expected output or scoring criteria change materially. Old versions should be archived, not overwritten."
},
"tags": {
"type": "array",
"items": { "type": "string" },
"description": "Free-form labels for filtering (e.g. 'regression', 'smoke', 'security', 'refactor')."
},
"task": {
"type": "object",
"required": ["prompt"],
"additionalProperties": false,
"description": "The agent input: the prompt the agent receives, plus optional repo snapshot and tool permissions.",
"properties": {
"prompt": {
"type": "string",
"minLength": 1,
"description": "The exact prompt string sent to the agent. No template variables — must be fully rendered."
},
"repo_snapshot": {
"type": "string",
"description": "Path or URI to a deterministic repo fixture (tarball, git ref, or fixture ID) the agent operates on. Omit for prompt-only tasks."
},
"allowed_tools": {
"type": "array",
"items": { "type": "string" },
"description": "Tool names the agent is permitted to use during this task. Overrides session defaults."
},
"max_turns": {
"type": "integer",
"minimum": 1,
"description": "Hard turn limit for this task. If the agent exceeds this, the run is scored as a timeout."
},
"context_files": {
"type": "array",
"items": { "type": "string" },
"description": "Files to pre-load into the agent's context window before the prompt."
}
}
},
"expected": {
"type": "object",
"required": ["artifact_type"],
"additionalProperties": false,
"description": "The expected output artifact and acceptance criteria.",
"properties": {
"artifact_type": {
"type": "string",
"enum": ["file_diff", "file_create", "command_output", "json_object", "markdown_section", "no_change"],
"description": "Structural type of the expected output."
},
"artifact_path": {
"type": "string",
"description": "File path (relative to repo root) of the expected artifact. Required for 'file_diff' and 'file_create'."
},
"content": {
"type": "string",
"description": "Expected content string or JSON-serialized object. For 'file_diff', this is a unified diff."
},
"content_schema": {
"type": "object",
"description": "JSON Schema that the artifact content must satisfy. Mutually exclusive with 'content'."
},
"contains": {
"type": "array",
"items": { "type": "string" },
"description": "Substrings that must appear in the artifact. Evaluated after 'content' check."
},
"excludes": {
"type": "array",
"items": { "type": "string" },
"description": "Substrings that must NOT appear in the artifact."
}
}
},
"scoring": {
"type": "object",
"required": ["method"],
"additionalProperties": false,
"description": "How the run is graded.",
"properties": {
"method": {
"type": "string",
"enum": ["exact_match", "fuzzy_match", "schema_valid", "llm_judge", "custom"],
"description": "Primary scoring method. 'llm_judge' requires a judge_prompt. 'custom' requires a scorer_ref."
},
"fuzzy_threshold": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "Minimum similarity ratio (0–1) required to pass. Used when method = 'fuzzy_match'."
},
"judge_prompt": {
"type": "string",
"description": "System prompt for the LLM judge. Must instruct the judge to return a JSON object with a 'pass' boolean and 'reason' string. Used when method = 'llm_judge'."
},
"scorer_ref": {
"type": "string",
"description": "Module path or URI to a custom scorer callable (e.g. 'evals.scorers:check_typescript_compiles'). Used when method = 'custom'."
},
"pass_threshold": {
"type": "number",
"minimum": 0,
"maximum": 1,
"default": 1.0,
"description": "Fraction of sub-checks that must pass for the task to be marked as passed. Allows partial credit."
}
}
},
"baseline": {
"type": "object",
"additionalProperties": false,
"description": "Reference scores from the last known-good run. Used to detect regression.",
"properties": {
"model": { "type": "string" },
"date": { "type": "string", "format": "date" },
"pass_rate": { "type": "number", "minimum": 0, "maximum": 1 },
"median_turns": { "type": "number", "minimum": 0 },
"notes": { "type": "string" }
}
},
"metadata": {
"type": "object",
"description": "Free-form key-value pairs for tooling (CI pipeline ID, author, linked issue, etc.).",
"additionalProperties": { "type": "string" }
}
},
"examples": [
{
"id": "add-missing-return-type",
"version": 1,
"tags": ["regression", "typescript"],
"task": {
"prompt": "Add explicit return types to all exported functions in src/utils.ts that are missing them.",
"repo_snapshot": "fixtures/ts-project-v1.tar.gz",
"allowed_tools": ["Read", "Edit", "Bash"],
"max_turns": 10
},
"expected": {
"artifact_type": "file_diff",
"artifact_path": "src/utils.ts",
"contains": [": string", ": number", ": void"],
"excludes": ["any"]
},
"scoring": {
"method": "llm_judge",
"judge_prompt": "You are a TypeScript expert. The agent was asked to add return types. Return {\"pass\": true/false, \"reason\": \"...\"}. Pass if all exported functions now have explicit return types and no 'any' was introduced.",
"pass_threshold": 1.0
},
"baseline": {
"model": "<provider-model-slug-at-baseline-date>",
"date": "2026-04-01",
"pass_rate": 0.94,
"median_turns": 4
}
}
]
}
data/sources.json{
"metadata": {
"skill": "ai-coding-agents-observability-evals",
"title": "AI Coding Agents Observability And Evals - Sources",
"description": "Official documentation and implementation references for tracing, replay, regression evals, and cost controls in coding-agent runtimes",
"last_updated": "2026-08-21",
"updated": "2026-08-21",
"total_sources": 11,
"version": "1.4"
},
"categories": {
"official_documentation": [
{
"name": "Anthropic: Building Effective Agents",
"url": "https://www.anthropic.com/engineering/building-effective-agents",
"type": "guide",
"relevance": "High-level guidance on reliability, tool usage, and eval-driven agent improvement",
"update_frequency": "quarterly",
"access": "free",
"add_as_web_search": true
},
{
"name": "OpenTelemetry Documentation",
"url": "https://opentelemetry.io/docs/",
"type": "documentation",
"relevance": "Reference model for trace, span, log, and metric correlation across runtimes",
"update_frequency": "quarterly",
"access": "free",
"add_as_web_search": true
}
],
"implementation_references": [
{
"name": "Claude Code GitHub Repository",
"url": "https://github.com/anthropics/claude-code",
"type": "repository",
"relevance": "Primary implementation reference for session events, task boundaries, and runtime telemetry opportunities",
"update_frequency": "weekly",
"access": "free",
"add_as_web_search": false
},
{
"name": "Codex CLI Repository",
"url": "https://github.com/openai/codex",
"type": "repository",
"relevance": "Cross-runtime comparison point for agent tracing, provider usage, and operational surfaces",
"update_frequency": "monthly",
"access": "free",
"add_as_web_search": false
},
{
"name": "OpenAI Codex State and Doctor Source",
"url": "https://github.com/openai/codex/blob/7d47056ea42636271ac020b86347fbbef49490aa/codex-rs/state/src/lib.rs",
"type": "repository_source",
"relevance": "Pinned first-party source for rollout metadata extraction, SQLite state mirrors, goals/log databases, backfill metrics, and diagnostic telemetry shape",
"update_frequency": "pinned",
"access": "free",
"add_as_web_search": false
}
],
"evaluation_references": [
{
"name": "Inspect AI Documentation",
"url": "https://inspect.aisi.org.uk/",
"type": "documentation",
"relevance": "Reference for structured eval runs, scoring, and regression workflows",
"update_frequency": "quarterly",
"access": "free",
"add_as_web_search": true
},
{
"name": "OpenAI Evals Design Guide",
"url": "https://developers.openai.com/api/docs/guides/evals",
"type": "guide",
"relevance": "Reference for practical eval construction, scorecards, and release gating",
"update_frequency": "monthly",
"access": "free",
"add_as_web_search": true
},
{
"name": "Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses (arXiv 2604.25850)",
"url": "https://arxiv.org/abs/2604.25850",
"type": "paper",
"relevance": "Primary source for the harness self-evolution closed loop: component/experience/decision observability, falsifiable-contract edits, Terminal-Bench 2 and SWE-bench-verified transfer results",
"update_frequency": "static",
"access": "free",
"add_as_web_search": true
},
{
"name": "Building Effective AI Coding Agents for the Terminal (arXiv 2603.05344)",
"url": "https://arxiv.org/abs/2603.05344",
"type": "paper",
"relevance": "Independent terminal coding-agent reference architecture (workload-specialized model routing, dual-agent plan/execute, lazy tool discovery, adaptive compaction) used to corroborate runtime-decomposition completeness",
"update_frequency": "static",
"access": "free",
"add_as_web_search": true
},
{
"name": "SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (arXiv 2603.24755v1)",
"url": "https://arxiv.org/abs/2603.24755v1",
"type": "paper",
"relevance": "Primary preprint source for carried-workspace iterative evaluation, checkpoint-level correctness and structural-quality trajectories, and evidence that quality-aware prompting improves initial quality without halting degradation",
"update_frequency": "static",
"access": "free",
"add_as_web_search": true
},
{
"name": "OpenTelemetry GenAI Semantic Conventions (moved to dedicated repo, mid-2026)",
"url": "https://github.com/open-telemetry/semantic-conventions-genai",
"type": "specification",
"relevance": "Development-status OTel spec for gen_ai agent/tool spans: gen_ai.operation.name, gen_ai.agent.name/id/version, gen_ai.tool.name/tool.call.id, gen_ai.conversation.id. Requires OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. As of mid-2026 the spec split out of open-telemetry/semantic-conventions into this dedicated repo; the old opentelemetry.io/docs/specs/semconv/gen-ai/* pages now show moved notices and the attribute registry marks gen_ai.* entries deprecated-in-place (relocated, names unchanged). Contrast with Codex proprietary otel crate.",
"update_frequency": "quarterly",
"access": "free",
"add_as_web_search": true
}
]
}
}
learnings.consolidated.md# ai-coding-agents-observability-evals — Consolidated Learnings Curated, dated, committed memory for this skill. Pruned from raw `learnings.md` via `agents-skills-feedback-loop/scripts/consolidate.py`. Human-approved. Cap: 60 entries. When exceeded, promote durable rules to `references/`. ## Filter Override <!-- Add 2-4 bullets that sharpen what counts as a learning for this skill. Leave empty to use the default filter from agents-skills-feedback-loop/references/learnings-format.md. --> ## Patterns That Work ## Mistakes to Avoid ## Domain Knowledge ## Open Questions ## Consolidated Principles
learnings.md# ai-coding-agents-observability-evals — Learnings ## Patterns That Work - [2026-08-17] A deterministic cross-runtime smoke should clear inherited model overrides and exercise the real ranked search route before joining deployment and telemetry evidence. - [2026-08-15] Treat identical zero-event live-eval failures as infrastructure-invalid: execute the parent runner outside its outer sandbox only when it re-establishes a stricter nested candidate sandbox, and never score or promote those rows. - [2026-08-14] A privacy-safe routing loop can stay local by storing only versioned event types, real skill names, ranks, and outcomes while forbidding prompts, queries, arguments, outputs, and identifiers. ## Mistakes to Avoid - [2026-08-15] A live eval can verify the edit yet fail the rubric when the response schema and evidence path disagree; freeze the failed pack and validate schema-path alignment before a versioned rerun. - [2026-08-15] Version cumulative-versus-delta usage semantics in every telemetry row and make reporters reject mixed generations until an atomic rebuild completes. - [2026-08-14] Fail-open local telemetry must open FIFO and special-file targets nonblocking before descriptor validation, or the safety path can hang. ## Domain Knowledge - [2026-07-11] OTel GenAI semconv split into a dedicated semantic-conventions-genai repo mid-2026 (still Development status); re-resolve bookmarked gen-ai spec/doc URLs before citing them. ## Open Questions ## Consolidated Principles
references/evals-regression-and-cost-ops.md# Evals, Regression, And Cost Ops Treat coding-agent evals as a release discipline, not a side project. ## Contents - Core eval pack and iterative self-extension - Scoring, judge controls, and pairwise evaluation - Release gates, cost controls, and edge cases ## Core eval pack Every serious coding-agent runtime should have: - **golden coding tasks** for bounded edits - **golden review tasks** with known seeded defects - **tool-choice tasks** where the correct path depends on search or inspection before editing - **verification tasks** that punish self-approval and reward separate verification - **multi-agent tasks** when the product supports workers, teammates, or coordinator flows - **iterative self-extension tasks** that carry the agent's own workspace through evolving specifications - **cost and latency baselines** per task family ## Iterative self-extension pack Use a versioned trajectory pack for agents that repeatedly extend a repository. A trajectory starts from an empty or fixed seed workspace, then carries the candidate's own checkpoint output into the next evolved specification. Do not replace it with reference code between checkpoints: that erases the effect of the candidate's earlier design decisions. At each checkpoint, collect: - lineage: `trajectory_id`, `checkpoint_id`, `parent_checkpoint_id`, and `spec_version` - workspace provenance: stable workspace identity plus a content hash - correctness: strict, isolated, core, and regression results as separate fields - structural-quality signals: erosion and verbosity - operations: cost and duration Keep the pack's detailed task construction, fresh-context protocol, and hidden black-box testing with [`../../qa-agent-testing/SKILL.md`](../../qa-agent-testing/SKILL.md). Keep the definitions and interpretation of erosion and verbosity with [`../../software-clean-code-standard/SKILL.md`](../../software-clean-code-standard/SKILL.md). This eval-ops layer owns versioning, telemetry, comparison, and release decisions. Green tests at one checkpoint—or even a green final snapshot—do not prove extension robustness. Compare candidate and baseline across the full trajectory. Report per-checkpoint levels, candidate-versus-baseline slope, and late-checkpoint correctness regressions; do not collapse these into one final pass rate. Set gates from repeated runs on the product's own representative pack rather than importing a universal slope or metric threshold. SlopCodeBench (arXiv:2603.24755v1) provides preprint evidence for this failure mode in its Python track. Its anti-slop and plan-first prompts improved initial quality, but did not halt the quality-degradation slope or consistently improve correctness. Treat prompt changes as evaluated interventions, not as sufficient controls. ## Score more than final correctness Useful dimensions: - final output correctness - patch quality and blast radius - tool-call precision - retry discipline - escalation quality - verifier effectiveness - latency - token usage - dollar cost ## LLM-as-judge bias and flake control When a golden task uses `method: llm_judge`, the judge is itself a model with failure modes. Treat its scores as a calibrated instrument, not ground truth. - **Self-preference bias**: do not judge an agent with the same model/config that produced the output. A model rates its own style higher. Use a different judge model, or a deterministic check, for the gate that blocks release. - **Length / verbosity bias**: judges reward longer patches and longer explanations even when shorter is correct. Pin the rubric to behavior (compiles, tests pass, blast radius) and penalize unnecessary diff size explicitly so verbosity cannot buy a passing score. - **Position bias** (pairwise mode): when comparing two candidate runs, judges favor whichever is shown first. Always run both orderings and require both to agree; count disagreement as a tie, not a win. - **Flake / non-determinism**: model judges and the agents under test are both stochastic. Run each task N times (pass@k or majority vote), fix judge temperature low, and treat a task whose verdict flips run-to-run as a *broken golden task*, not a real regression — quarantine and rewrite it. For the full judge-bias taxonomy, calibration mechanics, and threshold derivation, see the `ai-evals` skill. ## Pairwise and preference evals Absolute pass/fail is not enough when you are choosing between two harness versions, two prompts, or two models. Add a pairwise track: - Show the judge both candidates' transcripts for the same golden task. - Swap order on a second pass (position-bias guard above). - Aggregate to a win rate with confidence intervals, not a single tally. - Gate on win rate **and** absolute cost/latency, so a "better" candidate that doubled cost is surfaced as a trade, not a silent win. ## Release gates Block release when any of these regress materially: - pass rate on seeded critical defects - verifier catch rate - median or p95 cost per task family - median or p95 latency for common tasks - false-positive or false-negative rate on code-review tasks - candidate-versus-baseline structural-quality slope on iterative packs - strict, isolated, core, or regression results at late checkpoints, even when the final aggregate remains acceptable ## Cost tips - Track provider cost per turn and per tool-heavy phase. - Compare candidate changes against a fixed baseline corpus before rollout. - Keep a “cheap smoke pack” and a “full release pack” so every change does not require full-cost evaluation. - Add real production failures back into the corpus after they are fixed. ## Edge cases - **Provider swaps**: Normalized pass rates can hide large cost drift, so compare quality and cost together. - **Caching changes**: Prompt-cache improvements can change cost and latency even when quality is stable; track them explicitly. - **New safety rules**: Approval or sandbox changes can reduce defect risk while increasing latency. Treat that as a deliberate trade, not noise. - **Multi-agent systems**: Grade the coordinator and workers separately so you can see whether failures come from delegation or execution. ## Practical tip If you can only afford one strict gate initially, make it: - seeded-defect catch rate for review agents - behavior-preserving pass rate for edit agents - verification catch rate for multi-agent workflows
references/harness-self-evolution.md# Harness Self-Evolution (Closed Loop)
Static evals tell you *whether* a build regressed. A harness-evolution loop uses the same trace and eval substrate to *improve the harness itself* — automatically, with attribution. As of 2026 this is the frontier addition to coding-agent observability: the eval corpus stops being only a release gate and becomes the optimizer's reward signal.
Source: *Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses*, arXiv [2604.25850](https://arxiv.org/abs/2604.25850) (2026-04). Empirical claims below are from that paper and one cross-runtime reference design (*Building Effective AI Coding Agents for the Terminal*, arXiv [2603.05344](https://arxiv.org/abs/2603.05344)). Verify numbers and method names against the current papers before treating them as fixed fact.
## Why this belongs in the observability skill
The harness — tool wiring, middleware, memory, retry/verification scaffolding, *not* the system prompt — is now the dominant lever on coding-agent performance. Manual harness tuning fails for three reasons the rest of this skill already names elsewhere: heterogeneous edit surface, trajectory volume that buries signal, and edits whose effect is hard to attribute. Those are observability problems. Solve them and harness improvement becomes a closed loop on top of the trace + eval datasets you already built.
## The three observability pillars
A self-evolution loop needs three distinct observability surfaces. Missing any one collapses it back into trial-and-error:
- **Component observability** — every editable harness component has a file-level, revertible representation. The action space is explicit, not implicit in code. No component representation → the optimizer cannot reason about *what* it changed or roll it back.
- **Experience observability** — millions of raw trajectory tokens are distilled into a layered, drill-down evidence corpus an evolving agent can actually consume. This is the eval/trace store from this skill, re-shaped for an *agent* reader, not a human dashboard.
- **Decision observability** — every edit is paired with a self-declared prediction, later verified against the next round's task-level outcomes. This converts each edit into a falsifiable contract and is what makes effects attributable.
## The loop
```text
baseline harness + eval corpus
-> agent proposes a harness edit (component observability: explicit, revertible)
-> agent self-declares a prediction about its effect (decision observability)
-> run eval pack; distill trajectories to evidence (experience observability)
-> verify prediction against task-level outcomes
prediction held -> keep edit, fold into baseline
prediction falsified -> revert; the failed contract is itself signal
-> repeat
```
The falsifiable-contract step is the load-bearing one. An optimizer that edits without a pre-declared, verified prediction is doing benchmark-chasing, not attribution — and will overfit the corpus.
## Evidence
- Terminal-Bench 2 pass@1: **69.7% → 77.0%** over **ten** AHE iterations; surpasses the **human-designed Codex-CLI harness (71.9%)** and the ACE / TF-GRPO self-evolving baselines.
- Transfer: the frozen evolved harness reaches top aggregate success on **SWE-bench-verified using ~12% fewer tokens than the seed** harness, with no re-evolution.
- Cross-model-family: **+5.1 to +10.1 pp** on Terminal-Bench 2 across three other model families.
- Ablation: gains come from **tools, middleware, and long-term memory — not the system prompt.** Structural, not prose-level, transfer. This is why prompt-only "agent tuning" plateaus.
## Pattern / Anti-pattern / Recipe
- **Pattern:** treat the eval corpus as an optimizer reward signal, not only a release gate. Make every harness component file-level and revertible, require a pre-declared prediction per edit, and verify it against the next eval round before the edit is kept.
- **Anti-pattern:** "self-improving agent" that edits the system prompt in a loop with no component model and no pre-declared prediction. It overfits the benchmark, the gains do not transfer, and you cannot attribute or revert a regression.
- **Recipe:**
1. Reuse the existing trace store (this skill's `trace-and-telemetry-model`) as the experience-observability source; add a distillation pass that produces an agent-readable evidence corpus, not a human dashboard.
2. Represent each editable harness component as a tracked file with a revert path (component observability).
3. Require every proposed edit to carry a written predicted effect on a named eval slice (decision observability).
4. Gate keep/revert on the existing release-gate machinery — the prediction must clear the same baseline-delta thresholds used for human-authored changes.
5. Keep the evolution loop and the production release gate as **separate datasets** (Core Invariant of this skill): the optimizer must never train on production telemetry directly.
## Boundary
This is an *optional, advanced* layer. The Minimal Viable Version of this skill (canonical trace ID, replay-safe storage, one golden pack, one release gate) ships without it. Add the loop only once the eval corpus is trustworthy and versioned — an evolution loop on a noisy corpus optimizes the noise.
---
*Thank you to arXiv for use of its open access interoperability.*
references/openai-codex-otel-config.md---
source_snapshot: openai/codex main branch (verified 2026-05-25)
anchors:
- codex-rs/otel/ — OtelProvider, OtelSettings, SessionTelemetry, MetricsClient, OtelExporter
- codex-rs/analytics/src/events.rs — TrackEventRequest, SkillInvocation, GuardianReviewAnalyticsResult
- codex-rs/analytics/src/facts.rs — TurnTokenUsageFact, TurnResolvedConfigFact
---
# OpenAI Codex OTel Config
## Table of Contents
- [When To Use](#when-to-use)
- [What It Covers](#what-it-covers)
- [The `codex-rs/otel` Crate](#the-codex-rsotel-crate)
- [Contrast: OTel vs Analytics Proprietary Events](#contrast-otel-vs-analytics-proprietary-events)
- [Design Rules](#design-rules)
- [Anti-Patterns](#anti-patterns)
## When To Use
Use this reference when wiring OpenTelemetry (OTel) instrumentation into a Codex-class coding-agent runtime, or when contrasting standards-based OTel telemetry with proprietary analytics events.
## What It Covers
- `codex-rs/otel` crate: structs, TOML schema, exporter options, W3C tracestate handling
- Contrast with `codex-rs/analytics` proprietary event types
- TOML config skeleton
## The `codex-rs/otel` Crate
### Key Types
| Type | Role |
|------|------|
| `OtelProvider` | Top-level provider — wires exporters to the global OTel tracer and meter |
| `OtelSettings` | TOML-deserializable settings struct; controls all exporter and span config |
| `SessionTelemetry` | Session-scoped telemetry emission (trace IDs, turn events, W3C tracestate propagation) |
| `MetricsClient` | Abstraction over the OTel metrics API; emits counters and histograms |
| `MetricsConfig` | Configures the metrics pipeline within `OtelSettings` |
| `InMemoryMetricExporter` | Test double for metrics output; used in unit tests |
### `OtelSettings` Fields
```toml
[otel]
environment = "production" # deployment environment label
service_name = "codex" # OTLP service.name attribute
service_version = "1.2.3" # OTLP service.version attribute
codex_home = "/home/user/.codex" # base path for local log files
exporter = "otlp-http" # shorthand; overridden by trace_exporter/metrics_exporter
# Exporter selection (OtelExporter enum variants):
# None — no export; useful for development
# OtlpHttp — standard OTLP over HTTP (endpoint, headers, protocol, TLS)
# Statsig — shorthand for OTLP/HTTP JSON to Statsig (Codex-internal defaults)
[otel.trace_exporter]
# endpoint, headers, protocol = "binary" | "json", tls settings
[otel.metrics_exporter]
# same fields as trace_exporter
[otel.span_attributes]
custom_key = "custom_value" # arbitrary k/v pairs added to every span
[otel.tracestate.my_vendor]
key = "my_vendor"
value = "abc123" # W3C tracestate member; propagated through async queues
```
### HTTP Protocol Options
`OtelHttpProtocol::Binary` — binary protobuf (default OTLP)
`OtelHttpProtocol::Json` — JSON-encoded OTLP (useful for human debugging or Statsig compatibility)
### W3C Tracestate Handling
`OtelSettings.tracestate` is a map of named members. Each member carries a `key:value` pair injected into the W3C `tracestate` header on outbound HTTP requests. This lets the runtime propagate vendor-specific trace metadata (e.g. Statsig experiment ID, internal request routing metadata) across async queue boundaries without polluting the standard `traceparent`.
Design rule: propagate `traceparent` and `tracestate` through async queues explicitly — they do not survive task-spawning automatically in Tokio unless carried in the span context.
## Contrast: OTel vs Analytics Proprietary Events
Codex ships two parallel telemetry systems. Understanding the boundary prevents mixing them.
| Dimension | `codex-rs/otel` | `codex-rs/analytics` |
|-----------|----------------|----------------------|
| Standard | OpenTelemetry (W3C trace context, OTLP) | Proprietary Codex event schema |
| Export target | Any OTLP-compatible backend (Jaeger, Datadog, Statsig) | Codex internal analytics pipeline |
| Primary types | Spans, metrics, tracestate | `TrackEventRequest` enum variants |
| Audience | Operators, platform teams, external observability tools | OpenAI product analytics |
| Config | `[otel]` TOML section | `analytics_client` / event dispatch in `codex-rs/core` |
### Analytics Proprietary Event Types (from `codex-rs/analytics/src/events.rs`)
Key variants of the `TrackEventRequest` enum:
- `SkillInvocation` — tracks each time a skill is rendered (via `SkillInvocationEventRequest`, includes `skill_id`, `skill_name`, thread/turn IDs, model)
- `GuardianReview` — captures guardian (safety) review outcomes (via `GuardianReviewAnalyticsResult`, includes `decision`, `terminal_status`, `failure_reason`, token usage, timing)
- `TurnEvent` — comprehensive per-turn metrics (`CodexTurnEventParams`: `total_tool_call_count`, `input_tokens`, `output_tokens`, tool-specific counters)
- `HookRun`, `CommandExecution`, `FileChange`, `McpToolCall`, `DynamicToolCall`, `WebSearch`, `Compaction`
### Analytics Fact Types (from `codex-rs/analytics/src/facts.rs`)
- `TurnTokenUsageFact` — `{ turn_id, thread_id, token_usage }` — captures token consumption per turn
- `TurnResolvedConfigFact` — per-turn resolved config snapshot including `approval_policy`, `sandbox_network_access`, `collaboration_mode`, and model metadata
## Design Rules
- Use the `[otel]` config section for any telemetry that must flow to an external observability backend.
- Do not re-implement `traceparent`/`tracestate` propagation by hand — use `SessionTelemetry`'s propagation helpers.
- Treat `TrackEventRequest` variants as internal analytics only; do not build cross-org dashboards on them.
- Keep OTel metric dimensions low-cardinality — `service_name`, `environment`, `model_slug` are safe; raw prompt text, user IDs, and file paths are not.
- The `InMemoryMetricExporter` is the correct test double for unit tests; do not spin up a real OTLP endpoint in tests.
## Anti-Patterns
- Emitting raw user prompts, provider payloads, or file paths as OTel span attributes — these become high-cardinality and may leak PII.
- Conflating the OTel trace pipeline with the analytics pipeline; they have different retention, privacy, and recipient contracts.
- Losing `tracestate` at Tokio task-spawn boundaries because the span context was not explicitly carried across.
references/openai-codex-rollout-doctor-telemetry.md# OpenAI Codex Rollout Doctor Telemetry Source snapshot: OpenAI Codex commit `7d47056ea42636271ac020b86347fbbef49490aa` (2026-05-22), especially `codex-rs/state/src/lib.rs`, `codex-rs/cli/src/doctor.rs`, `codex-rs/otel`, and session/task telemetry code under `codex-rs/core/src`. ## Table Of Contents - [Design Goal](#design-goal) - [Rollout As Replay Artifact](#rollout-as-replay-artifact) - [SQLite Mirror](#sqlite-mirror) - [Doctor Reports](#doctor-reports) - [Trace And Metric Hooks](#trace-and-metric-hooks) ## Design Goal Observability for a coding-agent runtime should support both debugging a single user session and measuring aggregate runtime health. Codex does this by combining rollout JSONL, SQLite-derived indexes, structured doctor checks, trace IDs, and metrics. ## Rollout As Replay Artifact Codex stores rich rollout items that can be replayed or mined later. The runtime can reconstruct history, seed token usage, and recover metadata from rollout items. Copy this rule: - the human transcript is not enough - persist event messages, response items, compaction markers, token counts, and task boundaries - make rollout flush behavior explicit around abort/interruption paths ## SQLite Mirror Codex mirrors rollout metadata into local SQLite databases for fast query and lifecycle state. The state crate separates: - raw rollout extraction - thread metadata - logs - goals - backfill state - telemetry around DB init, fallback, and backfill This is the right shape for long-running agent CLIs: append-only session artifacts remain canonical, while SQLite is an index/cache that can be rebuilt. ## Doctor Reports Codex's `doctor` command emits both human-readable output and a redacted JSON report. Each check has: - stable ID - category - status: ok, warning, or fail - summary - details - structured issues - remediation - duration Reuse that schema for support tooling. A good doctor report should be machine-readable first, then rendered for humans. ## Trace And Metric Hooks Codex carries W3C trace context on submissions and emits trace IDs on turn start. It also records metrics for token usage, tool calls, skill rendering, DB initialization, and runtime events. For new runtimes: - propagate `traceparent` and `tracestate` through async queues - include trace IDs in start events so UI and logs can correlate - tag token metrics by token type, not just total - measure skill/tool truncation and deferred loading because prompt budget affects behavior ## Traps - Treating SQLite as canonical session storage instead of a rebuildable index. - Emitting doctor output only as pretty terminal text. - Losing trace context at UI -> core queue boundaries. - Recording token totals without input/output/reasoning/cache breakdowns.
references/recovery-trace-events.md# Recovery Trace Events
Concrete examples of the trace events emitted during agent recovery, reconnect, and cancellation scenarios. Use these as the reference when building trace parsers, eval harnesses, or observability dashboards.
## Table of Contents
- [Event Envelope](#event-envelope)
- [Event Catalog](#event-catalog)
- [Using Recovery Events in Evals](#using-recovery-events-in-evals)
---
## Event Envelope
All events share this envelope. Fields marked `*` are required.
```jsonc
{
"event": "string*", // event name (see catalog below)
"session_id": "string*", // stable ID for the session
"task_id": "string | null", // task the event belongs to; null for session-level events
"turn": "integer | null", // agent turn number when event fired; null for async events
"ts": "string*", // ISO-8601 timestamp with milliseconds
"data": "object" // event-specific payload (see per-event schema below)
}
```
---
## Event Catalog
### 1. `agent.recovery.started`
Fired when the agent detects a recoverable error (tool failure, partial tool output, unexpected model stop) and enters the recovery branch.
```json
{
"event": "agent.recovery.started",
"session_id": "sess_01j9kx2fvg3b4h7r",
"task_id": "task_review_pr_42",
"turn": 7,
"ts": "2026-04-27T14:22:01.304Z",
"data": {
"trigger": "tool_error",
"tool_name": "Bash",
"error_code": "ETIMEDOUT",
"error_message": "Command timed out after 30s: npm test",
"recovery_strategy": "retry_with_timeout_increase",
"attempt": 1,
"max_attempts": 3
}
}
```
**Fields:**
| Field | Type | Meaning |
|-------|------|---------|
| `trigger` | string | What caused recovery: `tool_error`, `partial_output`, `model_stop`, `context_overflow` |
| `tool_name` | string | Tool that failed (if `trigger = tool_error`) |
| `error_code` | string | Machine-readable error code |
| `recovery_strategy` | string | Strategy selected: `retry_with_timeout_increase`, `fallback_tool`, `replan`, `abort` |
| `attempt` | integer | Current attempt number (1-based) |
| `max_attempts` | integer | Maximum attempts before the strategy escalates to `abort` |
---
### 2. `agent.recovery.succeeded`
Fired when the recovery attempt produced an acceptable result and the agent resumes normal execution.
```json
{
"event": "agent.recovery.succeeded",
"session_id": "sess_01j9kx2fvg3b4h7r",
"task_id": "task_review_pr_42",
"turn": 8,
"ts": "2026-04-27T14:22:14.817Z",
"data": {
"recovery_strategy": "retry_with_timeout_increase",
"attempt": 2,
"elapsed_ms": 13513,
"resumed_from_turn": 7
}
}
```
---
### 3. `agent.recovery.failed`
Fired when all recovery attempts are exhausted and the task is being escalated or aborted.
```json
{
"event": "agent.recovery.failed",
"session_id": "sess_01j9kx2fvg3b4h7r",
"task_id": "task_review_pr_42",
"turn": 9,
"ts": "2026-04-27T14:22:45.001Z",
"data": {
"trigger": "tool_error",
"recovery_strategy": "retry_with_timeout_increase",
"total_attempts": 3,
"total_elapsed_ms": 43697,
"escalation": "task_abort",
"reason": "npm test consistently times out; environment likely unhealthy"
}
}
```
---
### 4. `session.reconnect.initiated`
Fired when the transport layer detects a dropped connection and begins reconnection. Common in remote-runtime configurations (WebSocket or SSE streams).
```json
{
"event": "session.reconnect.initiated",
"session_id": "sess_01j9kx2fvg3b4h7r",
"task_id": null,
"turn": null,
"ts": "2026-04-27T14:35:08.112Z",
"data": {
"transport": "websocket",
"disconnect_reason": "network_timeout",
"reconnect_attempt": 1,
"backoff_ms": 1000,
"last_ack_turn": 12,
"resume_token": "tok_8xkq3p2"
}
}
```
**Fields:**
| Field | Type | Meaning |
|-------|------|---------|
| `transport` | string | `websocket`, `sse`, `grpc`, `http_polling` |
| `disconnect_reason` | string | `network_timeout`, `server_closed`, `client_closed`, `auth_expired` |
| `last_ack_turn` | integer | Last turn the server acknowledged; replay starts from `last_ack_turn + 1` |
| `resume_token` | string | Opaque token used to resume the session without full re-auth |
---
### 5. `session.reconnect.succeeded`
```json
{
"event": "session.reconnect.succeeded",
"session_id": "sess_01j9kx2fvg3b4h7r",
"task_id": null,
"turn": null,
"ts": "2026-04-27T14:35:09.344Z",
"data": {
"transport": "websocket",
"reconnect_attempt": 1,
"replay_turns": 0,
"session_state": "running"
}
}
```
`replay_turns`: number of turns replayed to re-sync the agent state after reconnect. Zero means the server held state and no replay was needed.
---
### 6. `session.reconnect.failed`
```json
{
"event": "session.reconnect.failed",
"session_id": "sess_01j9kx2fvg3b4h7r",
"task_id": null,
"turn": null,
"ts": "2026-04-27T14:35:59.002Z",
"data": {
"transport": "websocket",
"total_attempts": 5,
"total_elapsed_ms": 50890,
"final_reason": "auth_expired",
"session_state": "dead",
"recovery_hint": "Re-authenticate and create a new session; this session cannot be resumed."
}
}
```
---
### 7. `task.cancelled`
Fired when the user or orchestrator explicitly cancels a running task.
```json
{
"event": "task.cancelled",
"session_id": "sess_01j9kx2fvg3b4h7r",
"task_id": "task_refactor_auth",
"turn": 15,
"ts": "2026-04-27T14:41:33.780Z",
"data": {
"cancelled_by": "user",
"reason": "User pressed Ctrl+C",
"last_completed_turn": 14,
"partial_artifacts": [
{ "path": "src/auth/token.ts", "status": "modified_uncommitted" }
],
"rollback_action": "none"
}
}
```
**Fields:**
| Field | Type | Meaning |
|-------|------|---------|
| `cancelled_by` | string | `user`, `orchestrator`, `timeout`, `budget_exceeded` |
| `partial_artifacts` | array | Files that were modified but not yet committed when cancellation fired |
| `rollback_action` | string | `none`, `git_restore`, `snapshot_restore` — what the runtime did with partial artifacts |
---
### 8. `task.cancellation.completed`
Fired after the cancellation handshake is fully resolved (partial artifacts handled, cleanup done).
```json
{
"event": "task.cancellation.completed",
"session_id": "sess_01j9kx2fvg3b4h7r",
"task_id": "task_refactor_auth",
"turn": null,
"ts": "2026-04-27T14:41:33.952Z",
"data": {
"rollback_status": "skipped",
"artifacts_left_on_disk": true,
"cleanup_ms": 172
}
}
```
---
## Using Recovery Events in Evals
1. **Recovery ratio**: `agent.recovery.succeeded` / (`agent.recovery.succeeded` + `agent.recovery.failed`) — measures how often the agent self-heals.
2. **Mean recovery time**: `elapsed_ms` from `agent.recovery.started` to `agent.recovery.succeeded` per strategy.
3. **Reconnect rate**: `session.reconnect.initiated` count per session-hour — a spike indicates transport instability.
4. **Cancellation cleanliness**: `rollback_action != "none"` rate — partial artifacts left on disk should trend toward zero.
5. **Partial-artifact rate after cancel**: tasks where `partial_artifacts` is non-empty at `task.cancelled` — high values suggest the agent is not checkpointing frequently enough.
references/trace-and-telemetry-model.md# Trace And Telemetry Model Use a layered telemetry model for coding agents: 1. **Session layer** Session id, repo or workspace, runtime mode, provider config, settings snapshot, and user-visible start or stop metadata. 2. **Turn layer** Prompt input, selected context, command mode, agent identity, and final answer summary. 3. **Execution layer** Tool calls, approvals, subprocess boundaries, file edits, retries, worker spawning, and verification passes. 4. **Outcome layer** Success or failure category, latency, token usage, cost, and any regression score outputs. 5. **Trajectory layer** Iterative-eval lineage, specification version, carried-workspace identity, per-checkpoint correctness and structural quality, cost, and duration. ## Iterative checkpoint payload When an eval carries the agent's own workspace across evolving specifications, emit one record per checkpoint with: - `trajectory_id`, `checkpoint_id`, and `parent_checkpoint_id` - `spec_version` - stable workspace identity and a content hash; keep source contents in the replay-safe artifact store, not metric dimensions - separate strict, isolated, core, and regression results - erosion and verbosity values produced by the pack's version-pinned analyzers - checkpoint cost and duration Preserve an explicit parent edge rather than inferring order from timestamps. The lineage must support candidate-versus-baseline trajectory-slope comparisons and late-checkpoint regression queries. Keep identifiers or hashes out of low-cardinality metrics labels; join them in the trace or eval store instead. ## Minimum replay-safe payload Persist enough state to explain a bad run without depending on transient UI events: - user input - resolved context or file list - tool-call sequence - tool inputs and outputs after redaction - diffs or write summaries - approval requests and decisions - worker or teammate handoffs - verifier findings - final answer ## Good design rules - Use one canonical trace or session id across local UI, remote bridge, and worker tasks. - Give every tool call a stable id and parent turn id. - Keep UI rendering metadata separate from semantic execution metadata. - Store redacted but structured payloads so search and replay remain useful. - Emit event timestamps in causal order and preserve monotonic ordering when clocks differ. ## Edge cases - **Background tasks**: They should emit progress events against the same session while preserving their own task ids. - **Remote sessions**: Bridge control messages belong in the trace even when the local UI never re-renders them directly. - **Resume flows**: A resumed session should continue the same semantic session lineage while marking the restore boundary explicitly. - **Verifier passes**: Record them separately from implementation attempts so regressions can distinguish “wrong fix” from “missing verification.” - **Carried workspaces**: Hash the exact checkpoint output before the next specification begins. A resumed or retried checkpoint must identify the input workspace and its attempt, rather than silently overwriting lineage. ## Practical tip If a user reports “the agent made the wrong change,” the trace should answer four questions quickly: 1. what context it loaded 2. which tool sequence it chose 3. whether any approval or policy boundary changed the plan 4. whether a verifier saw the defect and failed to block it
SKILL.md---
name: ai-coding-agents-observability-evals
description: "Designs coding-agent observability and evals. Use when measuring traces, replay, checkpoint lineage, quality trajectories, tool grading, regression, or cost."
compatibility: Portable core. Works on Claude Code and Codex.
version: "1.2"
last_validated: 2026-08-21
---
# AI Coding Agents Observability And Evals
Use this skill to design or review the feedback loop around a coding-agent runtime: traces, replayable transcripts, eval packs, regression gates, tool-call grading, latency and cost accounting, and production failure triage.
This skill covers how you operate a coding-agent product after the core runtime exists. It does not replace the runtime skills themselves.
## ASCII Flow
```text
agent session
|
v
trace events
prompts + model turns + tool calls + permissions + file diffs + costs
|
v
replayable transcript
stable IDs + redaction + source/runtime correlation
|
v
eval pack
golden tasks + graders + regression gates + cost/latency budgets
|
v
release decision
pass | investigate | rollback | update eval coverage
```
## Quick Reference
| Question | Read | Outcome |
|----------|------|---------|
| What should the trace and telemetry model include? | [`references/trace-and-telemetry-model.md`](references/trace-and-telemetry-model.md) | Durable trace schema, session correlation, event stages, and replay boundaries |
| How should evals, regressions, and cost controls work? | [`references/evals-regression-and-cost-ops.md`](references/evals-regression-and-cost-ops.md) | Golden tasks, iterative self-extension packs, trajectory scorecards, and cost-aware release gates |
| How do I use the eval/trace substrate to improve the harness itself? | [`references/harness-self-evolution.md`](references/harness-self-evolution.md) | Closed-loop harness evolution: three observability pillars, falsifiable-contract edits, attribution |
| How does OpenAI Codex combine rollout replay, SQLite state, doctor reports, and telemetry? | [`references/openai-codex-rollout-doctor-telemetry.md`](references/openai-codex-rollout-doctor-telemetry.md) | Replay artifacts, rebuildable state indexes, redacted diagnostics, W3C traces, token metrics |
| How does Codex wire OTel exporters and what analytics events exist? | [`references/openai-codex-otel-config.md`](references/openai-codex-otel-config.md) | OtelSettings TOML schema, exporter selection, W3C tracestate, contrast with proprietary analytics events |
## When To Use
- Design tracing and replay for a coding-agent CLI
- Add regression evals for coding, review, or task-execution agents
- Evaluate whether a coding agent preserves correctness and structural quality while extending its own workspace across evolving specifications
- Grade tool calls, patch quality, verification behavior, or handoff quality
- Build latency, token, and cost accounting for agent sessions
- Review how incidents and bad runs should be debugged from stored traces
## Use Other Skills
| Need | Use Instead |
|------|-------------|
| Broader coding-agent architecture | [`../ai-coding-agents/SKILL.md`](../ai-coding-agents/SKILL.md) |
| Session persistence and transcript restore | [`../ai-coding-agents-sessions/SKILL.md`](../ai-coding-agents-sessions/SKILL.md) |
| Tool runtime design | [`../ai-coding-agents-tools/SKILL.md`](../ai-coding-agents-tools/SKILL.md) |
| Generic agent eval harnesses | [`../qa-agent-testing/SKILL.md`](../qa-agent-testing/SKILL.md) |
| Reliability and observability outside agent systems | [`../qa-observability/SKILL.md`](../qa-observability/SKILL.md) |
## Default Workflow
1. **Define the trace spine.** Session, turn, tool call, approval, worker, and verification events should share one correlation model.
2. **Store replay-safe artifacts.** Persist prompts, tool inputs, outputs, diffs, approvals, and synthesized summaries with enough structure to replay failures.
3. **Separate product telemetry from eval telemetry.** Production traces describe what happened; eval runs describe whether it was acceptable.
4. **Build golden task packs.** Keep a representative set of coding, review, debugging, multi-agent, and iterative self-extension tasks with stable scoring rubrics.
5. **Grade behavior, not just final output.** Score tool choice, verification discipline, retry loops, escalation quality, and cost efficiency.
6. **Keep telemetry cardinality under control.** Stable prompt IDs, opaque hashes, and bounded error categories belong in event payloads; high-cardinality strings do not belong in metrics dimensions.
7. **Attach release gates to deltas.** Compare candidate changes against a known baseline for quality, cost, latency, failure-mode drift, and—when work carries across checkpoints—trajectory slope and late-checkpoint regressions.
8. **Instrument incident triage.** A bad run should be trace-searchable by repo, user, session, tool, provider, worker, and error family.
9. **Review regressions continuously.** Add new real failures back into the eval corpus so the system hardens over time.
## Host Rules
- Keep one canonical trace ID across the entire session lifecycle.
- Preserve causal order for tool calls, approvals, worker messages, and verification passes.
- Keep event ordering monotonic within a session even when log sinks or transports are asynchronous.
- Store enough normalized state to debug a run without depending on transient UI rendering.
- Score traces at multiple layers: final answer, tool behavior, and workflow correctness.
- Preserve checkpoint and workspace lineage for iterative evals; a final snapshot cannot explain when extensibility was lost.
- Track token and cost usage per turn and per subsystem, not only per session total.
- Use eval results to block releases when quality or cost drift exceeds explicit thresholds.
- Hash or redact user-identifying plugin or extension data before it becomes telemetry dimensions.
## Scratch-Rebuild Coverage
- Coverage strength:
strong for trace correlation, replay-safe storage, multi-layer grading, release-gate framing, and the need to trace recovery-class events
- Missing for faithful reproduction:
low-cardinality telemetry discipline, reconnect and recovery event classes, approval-cancel telemetry, task-budget-versus-token-budget accounting, and incident-first trace queries need more explicit treatment
- Required additions:
document trace events for reconnect, cancellation, fallback activation, recovery class, worker escalation, and plugin lifecycle changes, plus the eval rubric fields that map those events back to product quality without exploding metric cardinality
## Build Order
1. Define the canonical trace and correlation model.
2. Persist replay-safe prompts, tool IO, approvals, and diffs.
3. Add event sequencing, redaction, and low-cardinality telemetry rules.
4. Add per-turn and per-subsystem usage accounting.
5. Add production search and incident-debug views over traces.
6. Build eval corpora and scoring rubrics from real tasks.
7. Attach release gates to baseline deltas in quality, cost, and failure drift.
## Core Invariants
- Every meaningful runtime action must be trace-correlated.
- Production telemetry and eval telemetry are different datasets with different purposes.
- Replay must not depend on ephemeral UI state.
- Cost accounting must explain which subsystem and provider consumed budget.
- Real failures should feed the eval corpus over time.
- Metrics dimensions must stay low-cardinality even when trace events carry richer detail.
## Failure Modes
- Trace fragments that cannot be joined across tool calls, approvals, or workers.
- Incident debugging blocked because only rendered output was stored.
- Eval suites scoring final answers while missing workflow regressions.
- Cost spikes that cannot be attributed to provider, tool, or worker class.
- Release gates based on synthetic tasks that miss real production failures.
- Green tests at a final checkpoint masking steadily worsening extension robustness or structural quality.
- Metrics or dashboards becoming unusable because free-form strings were emitted as dimensions.
## Minimal Viable Version
- One canonical trace ID and turn correlation model.
- One replay-safe storage shape for prompts, tool calls, outputs, and approvals.
- One searchable incident view over stored traces.
- One golden-task eval pack with stable rubrics.
- One carried-workspace trajectory with per-checkpoint correctness, quality, cost, and duration when the product performs repeated repository edits.
- One low-cardinality telemetry policy for event fields versus metrics dimensions.
- One explicit threshold for blocking regressions in quality or cost.
## What Strong Implementations Add
- Recovery-specific trace events for reconnect, fallback, cancellation, and continuation.
- Per-subsystem cost and latency slices.
- Twin-column telemetry patterns with redacted or hashed identifiers where needed.
- Eval grading for verification discipline, escalation quality, and retry behavior.
- Continuous ingestion of real production failures into regression packs.
- Rollout gates that compare candidate builds to known-good baselines.
- Iterative self-extension packs that compare candidate and baseline trajectories, including degradation slope and late-checkpoint behavior rather than only final scores.
- A closed-loop **harness self-evolution** layer that turns the eval corpus into an optimizer signal (see [`references/harness-self-evolution.md`](references/harness-self-evolution.md)) — advanced, not MVP.
## Known Traps
- Logging only user-visible messages and losing the tool, permission, retry, and fallback evidence needed to explain failures.
- Designing replay as a transcript export instead of a structured artifact set that can reconstruct routing, tool calls, and decision boundaries.
- Aggregating eval, runtime, and cost signals into one scoreboard and making regressions impossible to attribute.
- Tagging telemetry with raw prompts, provider payloads, or user data that should have been redacted or hashed before export.
- Shipping evaluation suites that reward benchmark gains while ignoring recoverability, debuggability, and operational failure modes.
- Treating an anti-slop or plan-first prompt as a durable quality control without measuring what happens after repeated extensions.
## Common Anti-Patterns
- Logging only the final answer and calling it observability.
- Treating replay as a transcript screenshot rather than structured artifacts.
- Mixing production telemetry and eval metrics into one undifferentiated score.
- Measuring only session-total cost with no attribution.
- Emitting raw provider, plugin, or prompt text into metrics tags.
- Shipping on benchmark wins while ignoring incident-debuggability.
## Iterative Self-Extension Trajectories
Single-shot correctness and green tests do not prove extension robustness. Add a versioned iterative pack when the agent is expected to revisit the same codebase: each checkpoint supplies an evolved external specification and the agent continues from its own prior workspace. Record fresh checkpoint context separately from the carried code so the eval measures the consequences of earlier design choices rather than conversation recall.
For every checkpoint, persist lineage plus the outcome vector: `trajectory_id`, `checkpoint_id`, `parent_checkpoint_id`, `spec_version`, workspace identity and content hash, strict/isolated/core/regression results, erosion, verbosity, cost, and duration. Compare candidate and baseline trajectories on both level and slope. A release review should surface worsening structural-quality slope or a late-checkpoint correctness regression even when the final aggregate score is green. Derive product-specific gates from a representative baseline and repeated runs; SlopCodeBench does not establish universal thresholds.
SlopCodeBench's Python experiments found that anti-slop and plan-first prompts improved initial structural quality, but did not halt the degradation slope or consistently improve correctness. Prompt-only controls are therefore insufficient: keep the trajectory pack as the control surface and treat prompt changes as candidates to evaluate. Use [`../qa-agent-testing/SKILL.md`](../qa-agent-testing/SKILL.md) for the detailed carried-workspace benchmark protocol and [`../software-clean-code-standard/SKILL.md`](../software-clean-code-standard/SKILL.md) for structural-erosion and verbosity definitions; this skill owns their telemetry and release-gate integration, not their formulas.
## OTel gen_ai Semantic Conventions
The OpenTelemetry GenAI spec (status: Development, unchanged since 2026-05) defines a standard schema for agent telemetry. Key attributes:
- `gen_ai.operation.name` — operation identifier on the root span (e.g., `invoke_agent`, `create_agent`, `execute_tool`)
- `gen_ai.agent.name`, `gen_ai.agent.id`, `gen_ai.agent.version` — agent identity
- `gen_ai.tool.name` / `gen_ai.tool.call.id` — child spans for tool invocations
- `gen_ai.conversation.id` — cross-turn conversation correlation
**Repo-split caveat (verify before citing a URL):** as of mid-2026 these conventions moved out of the main `open-telemetry/semantic-conventions` repo into a dedicated `open-telemetry/semantic-conventions-genai` repo. The old `opentelemetry.io/docs/specs/semconv/gen-ai/*` pages now render "moved" notices, and the attribute registry marks the gen_ai.* entries as deprecated-in-place (relocated, not removed — the names above are still current). Treat any bookmarked gen-ai-spans URL as unstable; re-resolve from the dedicated repo before you cite it in a runbook or dashboard link.
**Experimental status caveat:** These conventions require `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental` to activate in most SDK implementations. A repo split this late in the spec's life is itself a signal — expect further attribute churn through at least end of 2026. Design instrumentation against the attribute names, but gate production dashboards and alerting thresholds on a version-pinned snapshot, not "whatever the SDK emits today," so an upstream rename doesn't silently blank a panel.
**Contrast with Codex proprietary crate:** Codex ships a bespoke `codex-rs/otel` crate (OtelSettings TOML, OtelExporter variants, W3C tracestate propagation) that predates the gen_ai semconv. The proprietary crate maps to the same conceptual slots but uses different attribute names and has no gen_ai.tool.call.id equivalent. When building cross-runtime dashboards, normalize to gen_ai semconv attributes and treat the Codex-proprietary shape as a source adapter.
**Responses API cache-hit telemetry:** The OpenAI Responses API achieves 40-80% better cache utilization than Chat Completions in agentic workloads (per OpenAI migration guidance). Cache-hit events are a distinct cost-accounting concern and should be tracked as a separate telemetry dimension from inference cost. Do not aggregate cache hits into generic token-usage metrics; they have different cost multipliers and different debugging value.
## Cross-Platform Patterns (Goose)
Goose is now maintained under AAIF (Linux Foundation; founding contributors Block, Anthropic, OpenAI; transferred April 7, 2026). Repository: `aaif-goose/goose`.
Goose's `evals/` structure (including `open-model-gym/`) plus the `recipe-scanner/` static validator suggest two additions to how this skill frames evals and pre-release gates.
### Named eval harness with versioned model×task matrix
Goose's `open-model-gym/` is an explicit, versioned eval pack targeting open-weight models across a fixed workflow matrix. This is stronger than "we have some regression tests" — it is a named, referenceable benchmark with a stable identity.
- **Pattern:** give your eval corpus a product name, a semantic version, and a published rubric. "Did it pass `open-model-gym v2.1`?" is a more actionable release gate than "did the eval suite pass?"
- **Anti-pattern:** an eval suite that silently changes its task set between releases, so pass-rate deltas are not comparable.
- **Recipe:** publish the eval pack as a versioned artifact. Release notes cite the pack version. Add a pack-version field to eval telemetry so historical pass rates stay interpretable after the pack evolves.
### Static analysis for agent definitions (recipe-scanner)
Goose's `recipe-scanner/` is cargo-deny for YAML workflows — it catches schema violations, undeclared-extension usage, and policy-violating configurations before a recipe is allowed to ship or run. This generalizes beyond recipes to any YAML/JSON artifact your agent consumes: MCP manifests, plugin manifests, eval task definitions, skill frontmatter.
- **Pattern:** every declarative artifact an agent reads should have a static validator. Validation runs in CI, at package, at install, and at runtime-load. Each layer catches different drift.
- **Anti-pattern:** validating only at runtime. Invalid artifacts then reach the user as a cryptic crash instead of a pre-ship error.
- **Recipe:** add a `validate` command to your CLI that runs all static gates: schema conformance, dependency reachability, policy compliance, extension allowlist intersection. Wire it into CI and into the activation path.
## Harness Self-Evolution (Frontier)
Static evals tell you *whether* a build regressed. As of 2026, best-in-class implementations also close the loop: the same trace + eval substrate becomes the reward signal for automatically improving the **harness** (tool wiring, middleware, memory, retry/verification scaffolding — not the system prompt). Observability-driven harness evolution beats human-designed harnesses on Terminal-Bench 2 and transfers frozen to SWE-bench-verified at lower token cost.
This needs three distinct observability surfaces — **component** (every editable harness part is file-level and revertible), **experience** (trajectories distilled into an agent-readable evidence corpus, not a human dashboard), and **decision** (every edit carries a pre-declared prediction verified against the next eval round). The decision pillar — falsifiable contracts per edit — is what separates attributable evolution from benchmark-chasing that overfits the corpus.
Keep this strictly separate from the production release gate (Core Invariant: optimizer never trains on production telemetry), and add it only once the eval corpus is versioned and trustworthy. Full method, the loop, evidence, and the Pattern/Anti-pattern/Recipe are in [`references/harness-self-evolution.md`](references/harness-self-evolution.md).
## Expert Judgment: Where Non-Experts Get This Wrong
These are the calls a strong operator makes differently from a team that just wired up a tracer and a pass/fail suite.
- **A pass-rate number without a confidence interval is not a release gate, it's a coin flip.** A 30-task golden pack moving from 26/30 to 24/30 (87%→80%) looks like a regression but is well within noise for that sample size. Compute a Wilson or Clopper-Pearson interval per task family and require the candidate's lower bound to clear the baseline's, not just the point estimate. Teams that skip this either ship real regressions ("it was only a 3-point dip, could be noise" — and it wasn't) or block good releases on sampling variance. Grow the pack before you trust single-run deltas; below roughly 50 tasks per family, run each task N≥3 times and gate on the mean.
- **Uniform trace sampling throws away the signal you built observability for.** At scale, capturing every session at full fidelity is a cost problem, so teams sample — but uniform sampling keeps the 99% of boring successful runs and drops exactly the failed, escalated, or cancelled sessions that justify the whole pipeline. Sample on outcome, not on request count: capture 100% of failures, escalations, cancellations, and verifier rejections; sample successes at whatever rate the budget allows. This is tail-based sampling keyed on business outcome, not on span duration.
- **`contains`/`excludes` substring checks in a golden task are gameable, and agents will find the gap.** An agent optimized against a static eval pack (by you, by harness self-evolution, or by the model provider's own RL) can learn to satisfy the literal check without satisfying the intent — e.g., adding a docstring containing the word "handles error" without handling the error. Treat any eval pack that has been used as an optimization target for more than a few cycles as partially compromised: rotate a subset of golden tasks, add mutated/adversarial variants, and keep at least one grading path (compiles, tests pass, schema-valid) that is not a substring match.
- **Full-fidelity traces of proprietary source code are a data-residency liability, not just an engineering convenience.** Storing complete file diffs and raw prompts (which routinely embed customer source, secrets in comments, or internal API names) in a central trace store creates an enterprise trust problem the moment a customer asks "where does our code go and who can read it." Decide early whether traces containing file content live in customer-controlled storage/region, get truncated to diff hunks plus hashes, or get a separate, shorter retention window than metadata-only telemetry — retrofitting this after a large customer's security review is far more expensive than designing it in.
## Navigation
### References
- [`references/trace-and-telemetry-model.md`](references/trace-and-telemetry-model.md) — Trace schema, replay boundaries, and production telemetry
- [`references/evals-regression-and-cost-ops.md`](references/evals-regression-and-cost-ops.md) — Eval packs, scorecards, release gates, and cost operations
- [`references/harness-self-evolution.md`](references/harness-self-evolution.md) — Closed-loop harness evolution: three observability pillars, falsifiable-contract edits, evidence and recipe
- [`references/openai-codex-rollout-doctor-telemetry.md`](references/openai-codex-rollout-doctor-telemetry.md) — OpenAI Codex rollout replay, SQLite mirror, doctor report schema, trace propagation, and metrics
- [`references/openai-codex-otel-config.md`](references/openai-codex-otel-config.md) — OtelSettings TOML schema, OtelExporter variants, W3C tracestate members, analytics event contrast
- [`references/recovery-trace-events.md`](references/recovery-trace-events.md) — Recovery-event taxonomy and trace requirements for interrupted or resumed agent work
### Data
- [`data/sources.json`](data/sources.json) — Primary docs and implementation references for coding-agent observability and evals
### Related Skills
- [`../ai-evals/SKILL.md`](../ai-evals/SKILL.md) - Judge-bias taxonomy, pairwise/flake control, and threshold derivation for the golden-task graders here
- [`../ai-coding-agents/SKILL.md`](../ai-coding-agents/SKILL.md)
- [`../ai-coding-agents-sessions/SKILL.md`](../ai-coding-agents-sessions/SKILL.md)
- [`../ai-coding-agents-tools/SKILL.md`](../ai-coding-agents-tools/SKILL.md)
- [`../qa-agent-testing/SKILL.md`](../qa-agent-testing/SKILL.md)
## Fact-Checking
- Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
- Trace shapes, event names, and replay payloads are product-specific. Preserve the architecture, but verify the target runtime before copying field names directly.
- Evals should reflect the real failure profile of your agent. Do not ship only synthetic tasks or happy-path benchmarks.
## Learnings Loop
Before applying this skill on a non-trivial task, read `learnings.consolidated.md` in this directory (and `learnings.md` if present).
After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.