evals/application.json
{
"_doc": "The application evaluator runs each case as a baseline/no-skill arm and a with-skill arm, then grades behavior against expected_flags, expected_fix_hints, absent_signals, and the per-case criteria checklist.",
"skill_name": "diagnosis",
"subject": "Use when facing an unknown software failure, when symptoms point to different root causes, or when an initial…",
"mode": "application",
"schema_version": 1,
"adjacent_concepts": [
"code-review",
"error-tracking",
"owasp-security",
"testing-strategy",
"debugging",
"a11y"
],
"cases": [
{
"id": 1,
"scenario_type": "realistic-positive",
"criticality": "critical",
"red_herring": false,
"scenario": "A user asks: \"the agent has been chasing this bug for 30 minutes — what's the structural fix?\"",
"context": "The agent has access to the diagnosis skill and must decide whether to use it, apply its procedure, and stay inside its boundaries.",
"question": "What should the agent do, and why is diagnosis the right skill to apply?",
"expected_flags": [
"Recognizes diagnosis as applicable to this request",
"Uses the skill-specific purpose or mental model instead of generic advice",
"Names the relevant boundary against code-review, error-tracking, owasp-security",
"Turns the request into concrete steps, checks, or decisions from the skill"
],
"expected_fix_hints": [
"Apply the diagnosis workflow to the concrete scenario",
"State the next action and the evidence or verification needed",
"Call out any adjacent skill that should own a different part of the request"
],
"absent_signals": [
"Does not answer with generic advice that could ignore diagnosis",
"Does not route to code-review, error-tracking, owasp-security when the diagnosis scope is the owner",
"Does not invent certification, freshness, or runtime success claims without evidence"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes diagnosis as applicable to this request"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Uses the skill-specific purpose or mental model instead of generic advice"
},
{
"id": "c3",
"polarity": "positive",
"statement": "Turns the request into concrete steps, checks, or decisions from the skill"
},
{
"id": "c4",
"polarity": "positive",
"statement": "State the next action and the evidence or verification needed"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not answer with generic advice that could ignore diagnosis"
},
{
"id": "c6",
"polarity": "guard",
"statement": "Does not invent certification, freshness, or runtime success claims without evidence"
}
]
},
{
"id": 2,
"scenario_type": "realistic-positive",
"criticality": "critical",
"red_herring": false,
"scenario": "A user asks: \"the symptoms span data integrity and UI rendering — which is the root cause?\"",
"context": "The agent has access to the diagnosis skill and must decide whether to use it, apply its procedure, and stay inside its boundaries.",
"question": "What should the agent do, and why is diagnosis the right skill to apply?",
"expected_flags": [
"Recognizes diagnosis as applicable to this request",
"Uses the skill-specific purpose or mental model instead of generic advice",
"Names the relevant boundary against code-review, error-tracking, owasp-security",
"Turns the request into concrete steps, checks, or decisions from the skill"
],
"expected_fix_hints": [
"Apply the diagnosis workflow to the concrete scenario",
"State the next action and the evidence or verification needed",
"Call out any adjacent skill that should own a different part of the request"
],
"absent_signals": [
"Does not answer with generic advice that could ignore diagnosis",
"Does not route to code-review, error-tracking, owasp-security when the diagnosis scope is the owner",
"Does not invent certification, freshness, or runtime success claims without evidence"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes diagnosis as applicable to this request"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Uses the skill-specific purpose or mental model instead of generic advice"
},
{
"id": "c3",
"polarity": "positive",
"statement": "Turns the request into concrete steps, checks, or decisions from the skill"
},
{
"id": "c4",
"polarity": "positive",
"statement": "State the next action and the evidence or verification needed"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not answer with generic advice that could ignore diagnosis"
},
{
"id": "c6",
"polarity": "guard",
"statement": "Does not invent certification, freshness, or runtime success claims without evidence"
}
]
},
{
"id": 3,
"scenario_type": "realistic-positive",
"criticality": "critical",
"red_herring": false,
"scenario": "A user asks: \"the build fails locally but passes in CI — how do I diagnose that class first?\"",
"context": "The agent has access to the diagnosis skill and must decide whether to use it, apply its procedure, and stay inside its boundaries.",
"question": "What should the agent do, and why is diagnosis the right skill to apply?",
"expected_flags": [
"Recognizes diagnosis as applicable to this request",
"Uses the skill-specific purpose or mental model instead of generic advice",
"Names the relevant boundary against code-review, error-tracking, owasp-security",
"Turns the request into concrete steps, checks, or decisions from the skill"
],
"expected_fix_hints": [
"Apply the diagnosis workflow to the concrete scenario",
"State the next action and the evidence or verification needed",
"Call out any adjacent skill that should own a different part of the request"
],
"absent_signals": [
"Does not answer with generic advice that could ignore diagnosis",
"Does not route to code-review, error-tracking, owasp-security when the diagnosis scope is the owner",
"Does not invent certification, freshness, or runtime success claims without evidence"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes diagnosis as applicable to this request"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Uses the skill-specific purpose or mental model instead of generic advice"
},
{
"id": "c3",
"polarity": "positive",
"statement": "Turns the request into concrete steps, checks, or decisions from the skill"
},
{
"id": "c4",
"polarity": "positive",
"statement": "State the next action and the evidence or verification needed"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not answer with generic advice that could ignore diagnosis"
},
{
"id": "c6",
"polarity": "guard",
"statement": "Does not invent certification, freshness, or runtime success claims without evidence"
}
]
},
{
"id": 4,
"scenario_type": "realistic-positive",
"criticality": "high",
"red_herring": false,
"scenario": "A user asks: \"I have a stack trace and an unhandled exception — what's the cheapest technique?\"",
"context": "The agent has access to the diagnosis skill and must decide whether to use it, apply its procedure, and stay inside its boundaries.",
"question": "What should the agent do, and why is diagnosis the right skill to apply?",
"expected_flags": [
"Recognizes diagnosis as applicable to this request",
"Uses the skill-specific purpose or mental model instead of generic advice",
"Names the relevant boundary against code-review, error-tracking, owasp-security",
"Turns the request into concrete steps, checks, or decisions from the skill"
],
"expected_fix_hints": [
"Apply the diagnosis workflow to the concrete scenario",
"State the next action and the evidence or verification needed",
"Call out any adjacent skill that should own a different part of the request"
],
"absent_signals": [
"Does not answer with generic advice that could ignore diagnosis",
"Does not route to code-review, error-tracking, owasp-security when the diagnosis scope is the owner",
"Does not invent certification, freshness, or runtime success claims without evidence"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes diagnosis as applicable to this request"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Uses the skill-specific purpose or mental model instead of generic advice"
},
{
"id": "c3",
"polarity": "positive",
"statement": "Turns the request into concrete steps, checks, or decisions from the skill"
},
{
"id": "c4",
"polarity": "positive",
"statement": "State the next action and the evidence or verification needed"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not answer with generic advice that could ignore diagnosis"
},
{
"id": "c6",
"polarity": "guard",
"statement": "Does not invent certification, freshness, or runtime success claims without evidence"
}
]
},
{
"id": 5,
"scenario_type": "boundary-positive",
"criticality": "high",
"red_herring": false,
"scenario": "A user asks: \"intermittent failure that doesn't reproduce on retry — which class is this?\"",
"context": "The agent has access to the diagnosis skill and must decide whether to use it, apply its procedure, and stay inside its boundaries.",
"question": "What should the agent do, and why is diagnosis the right skill to apply?",
"expected_flags": [
"Recognizes diagnosis as applicable to this request",
"Uses the skill-specific purpose or mental model instead of generic advice",
"Names the relevant boundary against code-review, error-tracking, owasp-security",
"Turns the request into concrete steps, checks, or decisions from the skill"
],
"expected_fix_hints": [
"Apply the diagnosis workflow to the concrete scenario",
"State the next action and the evidence or verification needed",
"Call out any adjacent skill that should own a different part of the request"
],
"absent_signals": [
"Does not answer with generic advice that could ignore diagnosis",
"Does not route to code-review, error-tracking, owasp-security when the diagnosis scope is the owner",
"Does not invent certification, freshness, or runtime success claims without evidence"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes diagnosis as applicable to this request"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Uses the skill-specific purpose or mental model instead of generic advice"
},
{
"id": "c3",
"polarity": "positive",
"statement": "Turns the request into concrete steps, checks, or decisions from the skill"
},
{
"id": "c4",
"polarity": "positive",
"statement": "State the next action and the evidence or verification needed"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not answer with generic advice that could ignore diagnosis"
},
{
"id": "c6",
"polarity": "guard",
"statement": "Does not invent certification, freshness, or runtime success claims without evidence"
}
]
},
{
"id": 6,
"scenario_type": "regression-prior-failure",
"criticality": "high",
"red_herring": false,
"scenario": "A user asks: \"we ran profiling, instrumentation, and bisect — none converge. What did we misclassify?\"",
"context": "The agent has access to the diagnosis skill and must decide whether to use it, apply its procedure, and stay inside its boundaries.",
"question": "What should the agent do, and why is diagnosis the right skill to apply?",
"expected_flags": [
"Recognizes diagnosis as applicable to this request",
"Uses the skill-specific purpose or mental model instead of generic advice",
"Names the relevant boundary against code-review, error-tracking, owasp-security",
"Turns the request into concrete steps, checks, or decisions from the skill"
],
"expected_fix_hints": [
"Apply the diagnosis workflow to the concrete scenario",
"State the next action and the evidence or verification needed",
"Call out any adjacent skill that should own a different part of the request"
],
"absent_signals": [
"Does not answer with generic advice that could ignore diagnosis",
"Does not route to code-review, error-tracking, owasp-security when the diagnosis scope is the owner",
"Does not invent certification, freshness, or runtime success claims without evidence"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes diagnosis as applicable to this request"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Uses the skill-specific purpose or mental model instead of generic advice"
},
{
"id": "c3",
"polarity": "positive",
"statement": "Turns the request into concrete steps, checks, or decisions from the skill"
},
{
"id": "c4",
"polarity": "positive",
"statement": "State the next action and the evidence or verification needed"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not answer with generic advice that could ignore diagnosis"
},
{
"id": "c6",
"polarity": "guard",
"statement": "Does not invent certification, freshness, or runtime success claims without evidence"
}
]
},
{
"id": 7,
"scenario_type": "hard-negative-boundary",
"criticality": "critical",
"red_herring": true,
"scenario": "A user asks: \"actually execute scientific-method debugging on this stack trace\"",
"context": "This is a boundary test. The skill should avoid false-positive activation and route to the better owner when appropriate.",
"question": "Should diagnosis be applied here? If not, what should happen instead?",
"expected_flags": [
"Recognizes the request is outside diagnosis's ownership",
"Routes the work to code-review or asks for the missing owner instead of forcing this skill",
"Preserves any real issue in the request without applying diagnosis terminology incorrectly"
],
"expected_fix_hints": [
"Decline to apply diagnosis as the primary skill",
"Name the better owner or the boundary reason",
"Offer only a narrow handoff if diagnosis has a secondary support role"
],
"absent_signals": [
"Does not claim diagnosis should own the request",
"Does not use diagnosis's framework to solve an explicitly excluded problem",
"Does not suppress the correct adjacent owner"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes the request is outside diagnosis's ownership"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Name the better owner or the boundary reason"
},
{
"id": "c3",
"polarity": "guard",
"statement": "Does not claim diagnosis should own the request"
},
{
"id": "c4",
"polarity": "guard",
"statement": "Does not use diagnosis's framework to solve an explicitly excluded problem"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not suppress the correct adjacent owner"
}
]
},
{
"id": 8,
"scenario_type": "hard-negative-boundary",
"criticality": "critical",
"red_herring": true,
"scenario": "A user asks: \"review this AI-generated PR for correctness\"",
"context": "This is a boundary test. The skill should avoid false-positive activation and route to the better owner when appropriate.",
"question": "Should diagnosis be applied here? If not, what should happen instead?",
"expected_flags": [
"Recognizes the request is outside diagnosis's ownership",
"Routes the work to code-review or asks for the missing owner instead of forcing this skill",
"Preserves any real issue in the request without applying diagnosis terminology incorrectly"
],
"expected_fix_hints": [
"Decline to apply diagnosis as the primary skill",
"Name the better owner or the boundary reason",
"Offer only a narrow handoff if diagnosis has a secondary support role"
],
"absent_signals": [
"Does not claim diagnosis should own the request",
"Does not use diagnosis's framework to solve an explicitly excluded problem",
"Does not suppress the correct adjacent owner"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes the request is outside diagnosis's ownership"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Name the better owner or the boundary reason"
},
{
"id": "c3",
"polarity": "guard",
"statement": "Does not claim diagnosis should own the request"
},
{
"id": "c4",
"polarity": "guard",
"statement": "Does not use diagnosis's framework to solve an explicitly excluded problem"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not suppress the correct adjacent owner"
}
]
},
{
"id": 9,
"scenario_type": "hard-negative-boundary",
"criticality": "critical",
"red_herring": true,
"scenario": "A user asks: \"scan this repo for OWASP top 10 vulnerabilities\"",
"context": "This is a boundary test. The skill should avoid false-positive activation and route to the better owner when appropriate.",
"question": "Should diagnosis be applied here? If not, what should happen instead?",
"expected_flags": [
"Recognizes the request is outside diagnosis's ownership",
"Routes the work to code-review or asks for the missing owner instead of forcing this skill",
"Preserves any real issue in the request without applying diagnosis terminology incorrectly"
],
"expected_fix_hints": [
"Decline to apply diagnosis as the primary skill",
"Name the better owner or the boundary reason",
"Offer only a narrow handoff if diagnosis has a secondary support role"
],
"absent_signals": [
"Does not claim diagnosis should own the request",
"Does not use diagnosis's framework to solve an explicitly excluded problem",
"Does not suppress the correct adjacent owner"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes the request is outside diagnosis's ownership"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Name the better owner or the boundary reason"
},
{
"id": "c3",
"polarity": "guard",
"statement": "Does not claim diagnosis should own the request"
},
{
"id": "c4",
"polarity": "guard",
"statement": "Does not use diagnosis's framework to solve an explicitly excluded problem"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not suppress the correct adjacent owner"
}
]
},
{
"id": 10,
"scenario_type": "hard-negative-boundary",
"criticality": "critical",
"red_herring": true,
"scenario": "A user asks: \"design observability instrumentation for this service\"",
"context": "This is a boundary test. The skill should avoid false-positive activation and route to the better owner when appropriate.",
"question": "Should diagnosis be applied here? If not, what should happen instead?",
"expected_flags": [
"Recognizes the request is outside diagnosis's ownership",
"Routes the work to code-review or asks for the missing owner instead of forcing this skill",
"Preserves any real issue in the request without applying diagnosis terminology incorrectly"
],
"expected_fix_hints": [
"Decline to apply diagnosis as the primary skill",
"Name the better owner or the boundary reason",
"Offer only a narrow handoff if diagnosis has a secondary support role"
],
"absent_signals": [
"Does not claim diagnosis should own the request",
"Does not use diagnosis's framework to solve an explicitly excluded problem",
"Does not suppress the correct adjacent owner"
],
"criteria": [
{
"id": "c1",
"polarity": "positive",
"statement": "Recognizes the request is outside diagnosis's ownership"
},
{
"id": "c2",
"polarity": "positive",
"statement": "Name the better owner or the boundary reason"
},
{
"id": "c3",
"polarity": "guard",
"statement": "Does not claim diagnosis should own the request"
},
{
"id": "c4",
"polarity": "guard",
"statement": "Does not use diagnosis's framework to solve an explicitly excluded problem"
},
{
"id": "c5",
"polarity": "guard",
"statement": "Does not suppress the correct adjacent owner"
}
]
}
]
}
SKILL.md
---
name: diagnosis
# description: routing-facing summary of when this skill activates and what it covers.
description: "Use when facing an unknown software failure, when symptoms point to different root causes, or when an initial debugging attempt has not converged. Provides a triage-first diagnostic routing framework: classify the failure, collect the right evidence, choose a technique, track confidence, and escalate when stuck. Do NOT use for executing scientific debugging after triage (use `debugging`), code-quality review (use `code-review`), or proactive observability setup."
# license: SPDX-compatible license identifier for the skill content.
license: MIT
# compatibility: runtime and portability notes for this skill.
compatibility:
notes: "Language- and stack-agnostic. The classification taxonomy, evidence protocol, and confidence ladder apply to any software failure investigation; specific technique names (git bisect, EXPLAIN plans, HMAC verification) are illustrative — substitute the equivalents of your stack."
allowed-tools: Read Grep
# metadata: Skill Metadata Protocol fields encoded under Agent Skills-compatible frontmatter.
metadata:
# schema_version: protocol contract version this skill conforms to.
# Integer 8. Prior contract retrievable via `git show schema-v7:schemas/skill.schema.json`.
# version: skill content version (semver). Bumped when the instructional content changes.
# === v8 Classification (subject + public; polyhierarchy via subjects[]) — see ADR-0020 ===
# subject: primary browse shelf — what the skill teaches. One of twelve closed values:
# backend-engineering / frontend-engineering / software-architecture / data-engineering / agent-ops / ai-engineering /
# quality-assurance / design / reasoning-strategy / software-engineering-method / knowledge-organization / product-domain.
subject: software-engineering-method
# public: publishability / private-data gate. true = safe for public release; false = private/internal.
public: true
# scope: required free-text statement of what this skill teaches and what it does not.
scope: "Use when facing an unknown software failure, when symptoms point to different root causes, or when an initial debugging attempt has not converged. Provides a triage-first diagnostic routing framework: classify the failure, collect the right evidence, choose a technique, track confidence, and escalate when stuck. Do NOT use for executing scientific debugging after triage (use `debugging`), code-quality review (use `code-review`), or proactive observability setup."
# public: publishability / private-data gate. true = safe for public release; false = private/internal.
# Project anchoring lives in project[] and requires grounding when present.
# taxonomy_domain: optional hierarchical sub-path within `subject`. Slash-delimited
# lowercase kebab-case segments. rename of the original v8 `domain`. Remove when the flat
# `subject` is sufficient.
taxonomy_domain: engineering/debugging
# owner: team handle, GitHub username, or tool name responsible for keeping this skill current.
# freshness: ISO date the skill body was last reviewed or updated.
# drift_check: truth-source verification record. Object with required `last_verified`
# (ISO date) and optional `truth_source_hashes`. Record hashes with:
# `node scripts/skill-graph-drift.js --record --apply <skill-dir>`.
# === Evaluation Status: three orthogonal axes ===
# eval_artifacts: disk-truth — does an eval file exist on disk?
# none (no intent) / planned (intent declared, no file yet) / present (file exists).
# eval_state: runtime-truth — has the eval been run and passed?
# unverified (no run yet, or no file) / passing (one-shot green) / monitored (cadenced green).
# `monitored` is strictly stronger than `passing` — a forward state for continuous runs.
# routing_eval: routing-coverage — is the skill's activation verified by the harness?
# absent (not verified) / present (gated by lint check 12; harness must exit 0).
# stability: lifecycle marker. One of:
# experimental (active development) / stable (production-ready) /
# frozen (no further changes expected) / deprecated.
# When `deprecated`, schema's allOf REQUIRES `superseded_by: <real-skill-name>`.
stability: experimental
# keywords: semantic phrases for fuzzy router activation. v8 cap: max 10.
# Keep terms a user would actually type when starting a task in this skill's domain.
keywords: ["diagnostic triage software failure","symptom classification taxonomy","what kind of bug is this","which debugging approach","diagnostic routing framework","evidence collection before hypothesis","confidence ladder debugging","escalation criteria debugging","cascade vs coincidence failure","environment ghost"]
# examples: 2-5 realistic user prompts the skill SHOULD activate for.
# Written in the user's voice. Improves retrieval recall beyond keywords alone.
examples: ["the agent has been chasing this bug for 30 minutes — what's the structural fix?","the symptoms span data integrity and UI rendering — which is the root cause?","the build fails locally but passes in CI — how do I diagnose that class first?","I have a stack trace and an unhandled exception — what's the cheapest technique?","intermittent failure that doesn't reproduce on retry — which class is this?","we ran profiling, instrumentation, and bisect — none converge. What did we misclassify?","two engineers disagree on whether this is a config issue or a logic error — what evidence settles it?"]
# anti_examples: near-miss prompts that should route ELSEWHERE.
# Pair with relations.suppresses to indicate the confusable territory's owner.
anti_examples: ["actually execute scientific-method debugging on this stack trace","review this AI-generated PR for correctness","scan this repo for OWASP top 10 vulnerabilities","design observability instrumentation for this service","decide which agent should pick up this ticket","what's the right test pyramid for this feature"]
# relations: typed graph edges to sibling skills. Six edge types:
# related (adjacency for browse / co-routing expansion) /
# suppresses (exclude listed skills from co-routing when THIS skill wins;
# write reason as "I own this exclusively over X", not "use X instead") /
# verify_with (cross-check; co-loaded as one-hop expansion) /
# depends_on (composition; transitive — A→B→C loads all three) /
# broader / narrower (SKOS-style generalization; broader drives co-load, narrower does not).
# grounding: required when non-empty `project[]`. Declares the truth sources
# the skill anchors to and the failure modes those sources prevent. Omit when the
# skill is universal-knowledge. `subject_matter` replaces v8 `domain_object`.
grounding: "{\"subject_matter\":\"Portable software-failure diagnostic triage: evidence collection, symptom classification, technique selection, confidence tracking, escalation, and sensitive diagnostic evidence handling\",\"grounding_mode\":\"universal\",\"truth_sources\":[\"https://sre.google/sre-book/effective-troubleshooting/\",\"https://git-scm.com/docs/git-bisect\",\"https://stackoverflow.com/help/minimal-reproducible-example\",\"https://developer.chrome.com/docs/devtools/performance/overview\",\"https://www.postgresql.org/docs/current/sql-explain.html\",\"https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html\",\"https://opentelemetry.io/docs/security/handling-sensitive-data/\"],\"failure_modes\":[\"fixing_before_classification\",\"hypothesis_without_baseline_evidence\",\"wrong_technique_for_problem_class\",\"confidence_inflation_without_verification\",\"stuck_state_not_escalated_or_reclassified\",\"diagnostic_evidence_captures_sensitive_or_secret_data\",\"eval_or_routing_claim_inflated_without_run\"],\"evidence_priority\":\"equal\"}"
# portability: external-runtime export claims. Object with:
# readiness — declared (claim only) / scripted (export tooling exists) /
# verified (proven with a receipt artifact).
# targets — array; currently only `skill-md` is in the enum.
# lifecycle: maintenance policy for the drift sentinel.
# stale_after_days — skill flagged STALE when N days past `drift_check.last_verified`.
# review_cadence — process commitment (quarterly / monthly / annual), not a calendar fact.
# === Export provenance (set by the export pipeline; do not hand-author) ===
# skill_graph_protocol is a content-label claim distinct from `schema_version` semantics.
# See AGENTS.md § Version Labels Are Earned, Not Bumped.
skill_graph_source_repo: "https://github.com/jacob-balslev/skill-graph"
skill_graph_project: Skill Graph
skill_graph_canonical_skill: skills/software-engineering-method/diagnosis/SKILL.md
# === Health Block (written by the audit loop, not hand-authored) ===
# See SKILL_AUDIT_LOOP.md § The Health Block. UNVERIFIED is the honest default.
#
# structural_verdict: form/export shape (gates 1-2, 7 — external mandates only).
# PASS / PASS_WITH_FIXES / FAIL / UNVERIFIED.
# truth_verdict: truth sources vs declared hashes (gates 3-6).
# PASS / DRIFT / BROKEN / UNVERIFIED.
# comprehension_verdict: gate 8 — cheap recitation smoke test. Never alone certifies.
# PASS / SHALLOW / REDUNDANT / UNVERIFIED / PROVISIONAL / SKIPPED_BASELINE_HIGH / NA.
# application_verdict: gate 9 — the primary quality signal. APPLICABLE is the only verdict
# that certifies the skill is USEFUL (grader-confirmed). PROVISIONAL = one model self-assessed.
# APPLICABLE / REDUNDANT / HARMFUL / MIXED / FALSE_POSITIVE / PROVISIONAL / UNVERIFIED.
relations:
related: ["code-review","error-tracking","owasp-security","testing-strategy","debugging"]
suppresses: ["debugging"]
verify_with: ["debugging","a11y"]
---
# Diagnosis
## Concept of the skill
Use when facing an unknown software failure, when symptoms point to different root causes, or when an initial debugging attempt has not converged.
## Coverage
The triage-first framework that classifies a software failure into a _problem class_ and routes it to the right diagnostic technique before root-cause investigation begins. Names nine symptom classes — Logic Error, Runtime Crash, Data Integrity, Timing / Race, Performance, Configuration, Security, Integration, Tooling / Build / Script-path — and provides a classification decision tree that walks from "is there a stack trace?" to a single class. Specifies a universal evidence-collection protocol (exact error message, reproduction steps, last-known-good state, environment facts) and class-specific evidence checklists. Lays out the technique-selection matrix — stack-trace reading, data-flow tracing, git bisect, differential comparison, instrumentation, MRE isolation, profiling, boundary probing — with each technique's time cost, best-case class, and evidence prerequisite. Defines the diagnostic confidence ladder (level 0 Symptom → 1 Classified → 2 Localized → 3 Root Cause → 4 Verified Fix) with explicit "you can say / you cannot say" boundaries at each level and stuck-state checkpoints (5-min, 10-min, 15-min, oscillation). Names escalation criteria for switching approach, switching class, or escalating to a human. Covers three cross-domain patterns where multiple classes apply simultaneously: the Cascade (one root cause, many symptoms), the Coincidence (two unrelated bugs that look like one), the Environment Ghost (works in one environment, fails in another). Catalogues diagnostic anti-patterns and ships a structured diagnostic-session template.
## Philosophy of the skill
Debugging fails most often not because the engineer lacks skill, but because the wrong methodology is applied to the problem class. A timing bug needs different tools than a data-integrity bug. A scope leak needs different thinking than a rendering glitch. The most expensive debugging mistake is spending 30 minutes applying scientific-method debugging to what is actually a configuration error discoverable in 2 minutes.
This skill is the triage _nurse_, not the surgeon. A nurse does not treat the patient — they take vital signs, route to cardiology or neurology, and escalate to the attending physician when criteria are met. Software diagnosis works the same way: collect evidence, classify the symptom, route to the right specialist technique, and pivot when convergence stalls. The small cost of triage is almost always smaller than the cost of chasing a plausible but wrong cause. Skipping triage because "the cause is obvious" is a confirmation-bias trap; even seasoned engineers benefit from making the classification step explicit.
## 1. The Diagnostic Triage Protocol
Before debugging, diagnose which _kind_ of problem you have. The class determines the technique and the technique determines the time-to-fix.
```
1. Collect baseline evidence (Section 3)
2. Classify the symptom (Section 2)
3. Select the diagnostic technique (Section 4)
4. Execute using the routed technique
5. If not converging after 3 attempts, escalate (Section 6)
```
**Rule:** never start fixing before completing steps 1–3. The cost of misclassification often exceeds the cost of a short triage pass, and the written classification gives the next person something concrete to challenge.
### Diagnosis vs debugging handoff
| Surface | Diagnosis owns | Handoff signal | Next owner |
| --- | --- | --- | --- |
| Failure triage | Evidence collection, symptom class, technique choice, confidence level, escalation trigger | The failure has a primary class, a chosen technique, and enough evidence to run it | `debugging` |
| Root-cause execution | Reproduction, scope reduction, instrumentation, hypothesis testing, fix verification, regression test | The selected technique has started producing falsifiable evidence | `debugging` |
| Error capture pipeline | Whether the failure was captured, sanitized, and made observable | The problem is "this error was not reported or was reported unsafely" | `error-tracking` |
| Pre-merge quality review | Whether the code is risky before a known failure exists | The question is about correctness risk, maintainability, or review feedback rather than an observed symptom | `code-review` |
| Security investigation | Threat-model-specific analysis against an attack class | Evidence points at auth, authorization, injection, secret exposure, or data exposure | `owasp-security` |
Treat the handoff as a contract, not a vague recommendation. Diagnosis does not fix the bug; it decides which investigation path is justified by evidence.
## 2. Symptom-Classification Taxonomy
Every failure falls into one of nine classes. Each class has a primary diagnostic technique.
| Class | Symptoms | Primary technique |
| --------------------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Logic Error** | Wrong output, wrong calculation, wrong state transition | Trace data flow; compare expected vs actual at each stage |
| **Runtime Crash** | Unhandled exception, process exit, 500 error | Read stack trace; find the throwing line; check preconditions |
| **Data Integrity** | Missing records, wrong totals, duplicate entries, cross-tenant leak | Compare source data to derived data at each transform stage |
| **Timing / Race** | Intermittent failure, works on retry, order-dependent | Add timestamps to logs; look for concurrent mutations; check locks |
| **Performance** | Slow response, timeout, memory growth, CPU spike | Profile _first_ (measure before hypothesizing); find the hot path |
| **Configuration** | Works locally but not in staging / prod, env-dependent | Diff environments — env vars, versions, feature flags, DNS, SSL |
| **Security** | Auth bypass, data exposure, HMAC failure, injection | Follow data flow from untrusted input to sensitive operation |
| **Integration** | Webhook not arriving, API returning unexpected shape, sync drift | Check both sides of the boundary independently, then compare |
| **Tooling / Build / Script-path** | `Cannot find module`, wrong cwd, stale script paths, `read EIO`, `ENOENT` on a script | Verify path resolution; check cwd; verify dependency install; compare referenced path vs actual filesystem path |
### Classification decision tree
```
Is there a stack trace or error message?
YES → Does it point to a specific line?
YES → Runtime Crash (read the line; check preconditions)
NO → Is it a timeout or OOM?
YES → Performance
NO → Logic Error (the error is a symptom of wrong state)
NO → Is the output wrong but no error thrown?
YES → Is the wrongness in calculated numbers or records?
YES → Data Integrity
NO → Logic Error
NO → Is it intermittent?
YES → Timing / Race
NO → Does it depend on environment?
YES → Configuration
NO → Does the error message contain a file/module path?
YES → Tooling / Build / Script-path
NO → Does it involve external services?
YES → Integration
NO → Are there security signals
(auth failure, permission error,
unexpected data exposure, HMAC failure,
access-control bypass)?
YES → Security
NO → Unknown / Unclassified
→ restart evidence collection;
run a fresh investigative sweep
```
## 3. Evidence-Collection Protocol
Before forming any hypothesis, collect baseline evidence. The class determines the additional evidence needed beyond the universal set.
### Evidence safety rule
Diagnostic notes, logs, screenshots, and repro snippets often contain more sensitive information than the final fix. Collect enough evidence to classify the failure, but redact or replace personal data, credentials, session tokens, raw request bodies, and secret-bearing headers before copying evidence into a shared note, issue, audit artifact, or skill. Prefer internal opaque IDs, hashes, synthetic examples, and minimal reproductions over real payload dumps.
### Universal evidence (always collect)
| Evidence | How to collect | Why |
| ----------------------------------- | ------------------------------------------------------------ | ---------------------------------------- |
| Exact error message or wrong output | Copy from logs, terminal, or UI | Prevents paraphrasing errors |
| Reproduction steps | The minimal sequence that triggers the failure | Proves the bug exists and is testable |
| Last-known-good state | `git log --oneline -10`, recent deploys, recent data changes | Brackets the introduction window |
| Environment facts | Runtime version, env vars, database state, running services | Eliminates the Configuration class early |
### Class-specific evidence
| Class | Additional evidence to collect |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| Logic Error | Input data, expected output, actual output, intermediate values at key transform points |
| Runtime Crash | Full stack trace, request payload, database state at crash time |
| Data Integrity | Source record count vs derived count, sample rows from each stage, tenant / scope identifiers |
| Timing / Race | Timestamps of concurrent operations, lock state, retry behaviour, whether it reproduces under load |
| Performance | Response-time baseline, CPU / memory profile, query plans (EXPLAIN), N+1 query check |
| Configuration | Env-var diff (local vs staging vs prod), package-version diff, feature-flag state |
| Security | Auth state, session-token contents, role / permission, request headers, HMAC comparison |
| Integration | Request / response pair from both sides, delivery logs, timestamp alignment |
| Tooling / Build / Script-path | Module-resolution output, current working directory at failure, dependency-install verification, referenced path vs filesystem path |
**Rule:** if you cannot fill the universal evidence table, you are not ready to hypothesize. Collect first, think second.
### Evidence ledger
Use an evidence ledger when the investigation has more than one plausible class. This keeps assumptions separate from observations and prevents confidence inflation.
| Field | Record | Example |
| --- | --- | --- |
| Observation | Raw fact, redacted if sensitive | `POST /webhook` returns 401 in staging only |
| Source | Where the fact came from | Deployment log, stack trace, profile, sanitized request sample |
| Class signal | Which class it supports | Configuration, Integration, Security |
| Contradiction | Which class it weakens | Logic Error: same code path passes locally |
| Next test | Cheapest falsification step | Compare staging and local signing secret metadata without exposing the secret |
If an observation changes the likely class, update the class explicitly. Silent reclassification is how investigations drift into mythology.
## 4. Technique-Selection Matrix
Once the symptom is classified, pick the cheapest technique that could resolve the class.
| Technique | Best for | Time cost | Evidence required |
| -------------------------------- | -------------------------------------------- | -------------- | --------------------------------------- |
| **Stack-trace reading** | Runtime crashes, unhandled exceptions | 1–2 min | Stack trace |
| **Data-flow tracing** | Logic errors, data integrity | 5–15 min | Input + output at each stage |
| **Binary search (`git bisect`)** | Regressions with known-good state | 3–10 min | Known-good commit + reproducible test |
| **Differential comparison** | Configuration, environment-dependent failure | 2–5 min | Two environments to compare |
| **Instrumentation (logging)** | Timing / race, intermittent failures | 5–10 min setup | Hypothesis about where to instrument |
| **Isolation (MRE)** | Complex failures with many variables | 10–20 min | Reproducible failure |
| **Profiling** | Performance, memory, CPU | 5–15 min | Running system under load |
| **Boundary probing** | Integration failures | 5–10 min | Access to both sides of the integration |
### Technique-ordering principle
Always start with the cheapest technique that could resolve the class:
1. **Read the error** (~30 s) — cheapest first pass for runtime crashes
2. **Check the environment** (~1 min) — cheapest first pass for configuration issues
3. **Trace the data flow** (~5 min) — cheapest first pass for logic / data errors
4. **Isolate with MRE** (~10 min) — useful when too many variables remain in play
5. **Instrument and observe** (~10+ min) — necessary when timing / intermittent failures cannot be reproduced directly
The percentages are intentionally absent. This skill is a routing framework, not a benchmark claim. Use local incident history or an actual eval corpus before making quantified success-rate claims.
## 5. The Diagnostic Confidence Ladder
As evidence accumulates, confidence in the diagnosis should increase _monotonically_. If it doesn't, the symptom has been misclassified.
| Level | Confidence | You can say | You cannot say |
| ---------------- | ---------- | ------------------------------------------------------- | ------------------------ |
| 0 — Symptom | 0% | "Something is wrong" | Anything about the cause |
| 1 — Classified | 20% | "This is a [class] problem" | Where specifically |
| 2 — Localized | 50% | "The failure is in [module / file / function]" | What exactly is wrong |
| 3 — Root cause | 80% | "The cause is [specific condition]" | That the fix will work |
| 4 — Verified fix | 95% | "This fix resolves the root cause and does not regress" | Nothing — ship it |
### Stuck-state checkpoints
- **Stuck at level 0 for > 5 min** → you need more evidence; restart Section 3
- **Stuck at level 1 for > 10 min** → likely misclassification; re-run the classification tree
- **Stuck at level 2 for > 15 min** → the problem may be cross-domain; check whether multiple classes apply
- **Oscillating between levels** → stop. Write down what you _know_ vs what you're _assuming_. The assumption is wrong.
### Reclassification rule
Classification is provisional until the evidence keeps moving the confidence ladder upward. Re-run the classification tree when any of these happens:
| Signal | Meaning | Required action |
| --- | --- | --- |
| The selected technique produces no new evidence | The class may be wrong or the evidence prerequisite is missing | Re-check Section 3, then choose the next cheapest class-compatible technique |
| A contradiction appears | The current class does not explain all observations | Split observation from assumption in the evidence ledger and reclassify |
| Confidence decreases after a test | The hypothesis was falsified, not "almost right" | Record the falsification and move down the ladder before continuing |
| Two classes stay equally plausible | The failure may be a Cascade or Coincidence | Test the earliest shared data-flow point, then split symptoms if one fix does not affect both |
## 6. Escalation Criteria
### Switch diagnostic approach when
| Signal | Action |
| ------------------------------------------------ | --------------------------------------------------- |
| Three hypotheses tested, none confirmed | Re-classify the symptom from scratch |
| Fix works locally but not in target env | Switch to Configuration-class techniques |
| Multiple symptoms that don't share a root cause | You may have 2+ bugs; triage each independently |
| Evidence contradicts the classification | Trust the evidence; re-classify |
| Confidence has _decreased_ over the last 3 steps | Stop. You're making it worse. Fresh context needed. |
### Escalate to human when
| Signal | Why a human is needed |
| --------------------------------------------------------------------- | ------------------------------- |
| Requires access you don't have (production DB, third-party dashboard) | Authorization boundary |
| Business-logic ambiguity ("should this return 0 or null?") | Product decision, not technical |
| Fix requires a breaking change to a public API | Stakeholder alignment needed |
| Reproduction requires real user data you cannot access | Privacy / compliance boundary |
| 30 minutes of investigation with no progress | Fresh perspective needed |
## 7. Cross-Domain Patterns
Some failures span multiple classes simultaneously. These compound failures are the hardest to diagnose.
### Pattern: the Cascade
A single root cause triggers symptoms across multiple classes.
```
Root cause: missing null-check in a data transform
→ Data Integrity symptom: wrong totals
→ Logic Error symptom: UI shows negative values
→ Integration symptom: webhook payload rejected by partner
```
**Diagnostic approach:** find the _earliest_ symptom in the data flow. That's closest to the root cause.
### Pattern: the Coincidence
Two unrelated bugs appear simultaneously, creating a misleading compound symptom.
```
Bug A: CSS regression from a recent deploy (Logic Error)
Bug B: slow API from an unrelated query change (Performance)
Combined symptom: "the page is broken and slow"
```
**Diagnostic approach:** separate the symptoms. Test each independently. If fixing one doesn't affect the other, they're independent bugs.
### Pattern: the Environment Ghost
Works in one environment, fails in another, with no code difference.
```
Local: works (runtime 20.11, .env.local, fresh DB)
Staging: fails (runtime 20.9, CI env vars, migrated DB)
```
**Diagnostic approach:** diff _everything_ — runtime versions, env vars, DB state, feature flags, DNS, SSL, headers. The first difference you find is usually the cause.
## 8. Anti-Patterns
| Anti-pattern | Why it fails | Correct |
| ------------------------------------------ | ------------------------------------------------------- | ------------------------------------------------ |
| Fixing before diagnosing | Treats the symptom; root cause persists | Complete the triage protocol first |
| Hypothesis without evidence | Confirmation bias drives you toward your guess | Collect universal evidence before any hypothesis |
| Changing multiple variables at once | Cannot determine which change had the effect | One variable at a time |
| Assuming the obvious cause | "Obvious" often means "familiar," not "verified" | Verify with evidence even when "obvious" |
| Copying raw sensitive data into evidence | The diagnostic artifact becomes a privacy or secret leak | Redact, synthesize, hash, or replace with opaque IDs |
| Debugging by `printf` without a hypothesis | Random instrumentation wastes time | Instrument to test a _specific_ hypothesis |
| Applying the wrong class's technique | Performance profiling won't find a logic error | Re-classify if the technique isn't converging |
| Escalating too early | Hasn't gathered enough evidence for a useful escalation | Fill the evidence table before escalating |
| Escalating too late | Spent 45 minutes on what a human could resolve in 5 | Follow the time-based escalation triggers |
## 9. Diagnostic-Session Template
Use this template to structure a diagnostic session. It prevents skipping steps.
```markdown
## Diagnostic Session: [Brief description]
### 1. Symptom
- What: [exact error or wrong behavior]
- Where: [route / component / job]
- When: [always / intermittent / environment-specific]
- Since: [commit / deploy / data change]
### 2. Classification
- Primary class: [from taxonomy]
- Confidence: [0–4 level]
- Technique: [from technique matrix]
### 3. Evidence Collected
- [ ] Error message / wrong output (exact)
- [ ] Reproduction steps (minimal)
- [ ] Last-known-good state
- [ ] Environment facts
- [ ] Sensitive evidence redacted or replaced with safe identifiers
- [ ] Class-specific evidence: [list]
### 4. Evidence Ledger
| Observation | Source | Class signal | Contradiction | Next test |
| ----------- | ------ | ------------ | ------------- | --------- |
| | | | | |
### 5. Hypotheses Tested
| # | Hypothesis | Test | Result | Confidence after |
| --- | ---------- | ---- | ------ | ---------------- |
| 1 | | | | |
### 6. Resolution
- Root cause: [one sentence]
- Fix: [what was changed]
- Prevention: [test / guard / doc added]
```
## Grounding and Evaluation State
This skill is grounded in public troubleshooting and diagnostic-practice references: Google SRE troubleshooting guidance, `git bisect` documentation for regression bisection, Stack Overflow MRE guidance for isolation, Chrome DevTools and PostgreSQL EXPLAIN docs for measurement/profiling examples, OWASP logging guidance for diagnostic event capture, and OpenTelemetry sensitive-data guidance for safe telemetry handling.
The current eval metadata remains intentionally conservative: `eval_artifacts: planned`, `eval_state: unverified`, and `routing_eval: absent`. Do not mark this skill verified or routing-present until a real comprehension eval and routing eval include `diagnosis` and pass in the same change.
## Verification
- [ ] The symptom was classified before any debugging technique was chosen
- [ ] Baseline evidence was collected before any hypothesis was formed
- [ ] Sensitive or secret-bearing evidence was redacted, synthesized, hashed, or replaced with opaque IDs before sharing
- [ ] The cheapest technique that could resolve this class was tried first
- [ ] Confidence increased monotonically — or the symptom was re-classified the moment it didn't
- [ ] If the approach was changed, the reason was documented (which signal triggered the switch)
- [ ] The time-based stuck-state checkpoints were respected (5-min / 10-min / 15-min triggers)
- [ ] If the failure spanned multiple classes, the cross-domain pattern (Cascade / Coincidence / Environment Ghost) was named explicitly
## Do NOT Use When
| Use instead | When |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `debugging` | Actually executing scientific-method debugging on a failure that has _already_ been classified — this skill routes to debugging; it does not replace it |
| `code-review` | Reviewing code for quality / correctness _before_ a failure exists — diagnosis is downstream |
| `owasp-security` | A focused security audit against a known threat list — diagnosis only routes here when symptoms point at security |
| `testing-strategy` | Deciding what to test proactively — diagnosis is for _reactive_ investigation after a failure |
| `error-tracking` | Setting up the production-error-capture / sampling / alerting stack — diagnosis investigates a _specific_ failure already in front of you |
| `skill-router` | Choosing which agent skill activates for an arbitrary query — that's cross-skill dispatch, not failure triage |