evals-init

어떤 스킬인가요?

Use when standing up evals/{system}/ for the first time — scaffolds the EDD directory structure, picks PromptFoo or DeepEval by tech stack, and generates a security baseline.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

#!/usr/bin/env bash
# setup-evals-init.sh — Setup for evals-init (self-contained)
set -euo pipefail

resolve_project_root() {
  local dir
  dir="$(pwd)"
  while [[ "$dir" != "/" ]]; do
    if [[ -d "${dir}/.adlc" ]]; then
      echo "$dir"
      return
    fi
    dir="$(dirname "$dir")"
  done
  git rev-parse --show-toplevel 2>/dev/null || pwd
}

resolve_team_ai_directives() {
  local project_root="$1"
  local td="${TEAM_AI_DIRECTIVES:-}"
  [[ -n "$td" ]] && { echo "$td"; return; }
  if [[ -f "${project_root}/.adlc/init-options.json" ]]; then
    td=$(jq -r '.team_ai_directives // empty' "${project_root}/.adlc/init-options.json" 2>/dev/null || true)
    [[ -n "$td" ]] && { echo "$td"; return; }
  fi
  echo "${project_root}/team-ai-directives"
}

resolve_branch() {
  git branch --show-current 2>/dev/null || echo "unknown"
}

PROJECT_ROOT=$(resolve_project_root)
TEAM_AI_DIRECTIVES=$(resolve_team_ai_directives "$PROJECT_ROOT")
BRANCH=$(resolve_branch)

jq -n --arg rr "$PROJECT_ROOT" --arg td "$TEAM_AI_DIRECTIVES" --arg br "$BRANCH" \
  '{REPO_ROOT:$rr,TEAM_AI_DIRECTIVES:$td,BRANCH:$br}'

#!/usr/bin/env pwsh
# setup-evals-init.ps1 — Setup for evals-init (self-contained)
$ErrorActionPreference = "Stop"

function Resolve-ProjectRoot {
    $dir = $PSScriptRoot
    while ($dir -ne "") {
        if (Test-Path (Join-Path $dir ".adlc")) { return $dir }
        $parent = Split-Path $dir -Parent
        if ($parent -eq $dir) { break }
        $dir = $parent
    }
    $gitRoot = git rev-parse --show-toplevel 2>$null
    if ($gitRoot) { return $gitRoot }
    return (Get-Location).Path
}

function Resolve-TeamAiDirectives {
    param([string]$ProjectRoot)
    $td = $env:TEAM_AI_DIRECTIVES
    if ($td) { return $td }
    $initOpts = Join-Path $ProjectRoot ".adlc/init-options.json"
    if (Test-Path $initOpts) {
        try {
            $config = Get-Content $initOpts -Raw | ConvertFrom-Json
            if ($config.team_ai_directives) { return $config.team_ai_directives }
        } catch {}
    }
    return (Join-Path $ProjectRoot "team-ai-directives")
}

function Resolve-Branch {
    $b = git branch --show-current 2>$null
    if ($b) { return $b }
    return "unknown"
}

$ProjectRoot = Resolve-ProjectRoot
$TeamAiDirectives = Resolve-TeamAiDirectives $ProjectRoot
$Branch = Resolve-Branch

$result = @{
    REPO_ROOT = $ProjectRoot
    TEAM_AI_DIRECTIVES = $TeamAiDirectives
    BRANCH = $Branch
}

$result | ConvertTo-Json -Compress

---
name: evals-init
description: Use when standing up evals/{system}/ for the first time — scaffolds the EDD directory structure, picks PromptFoo or DeepEval by tech stack, and generates a security baseline.
disable-model-invocation: true
---

# evals-init

## What this skill does

Initialize the **project-level evaluation directory structure** following EDD (Eval-Driven Development) principles to prepare for systematic evaluation development. This is completely standalone with zero spec-kit dependencies.

**Output**:
1. **Directory Structure** - `evals/{system}/` with proper organization (promptfoo | deepeval)
2. **Security Baseline** - Auto-created graders for PII leakage, prompt injection, hallucination detection, misinformation detection
3. **Configuration Files** - Standalone config.yml and goldset templates under `.adlc/evals/`
4. **Auto-handoff** to `/evals-specify` to begin error analysis

**Key EDD Principles Applied**:
- **Principle I**: Spec-Driven Contracts - Evals validate spec compliance
- **Principle II**: Binary Pass/Fail - No Likert scales in grader templates
- **Principle IV**: Evaluation Pyramid - Tier 1 (fast) + Tier 2 (goldset) structure
- **Principle IX**: Test Data as Code - Version control setup for datasets

## When to use

- **Starting systematic evaluation**: Set up the initial evaluation harness for your application
- **EDD Adoption**: Converting from traditional testing to evaluation-driven development
- **Security-first evaluation**: Auto-generate baseline security checks from the start

## When NOT to use

- **Evals directory already exists**: Use `/evals-validate` to run tests, or `/evals-specify` to add criteria
- **Evaluating team directives**: This is for project-level application behavior testing, not directives compliance

## Process

### User Input
```text
$ARGUMENTS
```
Parse flags from the arguments first, then treat remaining text as focus areas:
- `--system SYSTEM` — Choose `promptfoo` or `deepeval`. If omitted, choose interactively based on tech stack.
- Remaining text — System description (focus setup)

### Execution Steps

#### Phase 1: Tech Stack Detection
- Scan project manifests (`package.json`, `requirements.txt`, `Cargo.toml`, `go.mod`, etc.)
- Recommends PromptFoo for mixed/JS stacks; DeepEval for Python-native stacks

#### Phase 2: Create Directory Structure
Creates:
```
evals/
├── {system}/                    # promptfoo | deepeval
│   ├── goldset.md              # Published goldset
│   ├── goldset.json            # Auto-generated for system consumption
│   ├── config.yml              # System-specific configuration
│   ├── config.{js,py}          # Generated system config (.js for promptfoo, .py for deepeval)
│   └── graders/                # Binary pass/fail graders
│       ├── check_pii_leakage.py           # Security baseline
│       ├── check_prompt_injection.py     # Security baseline
│       ├── check_hallucination.py        # Security baseline
│       └── check_misinformation.py       # Security baseline
├── results/                    # Git-ignored run outputs
└── .adlc/
    └── drafts/evals/           # Draft eval records (Markdown + YAML)
```

#### Phase 3: Configuration Copy
- Create `.adlc/evals/` if missing.
- Copy `skills/evals/evals-templates/evals-config-template.yml` to `.adlc/evals/evals-config.yml`.

#### Phase 4: Auto-Handoff
Trigger `/evals-specify` to begin error analysis.

## Verification
- `evals/{system}/goldset.md` exists (initially empty)
- `.adlc/evals/evals-config.yml` exists
- Graders directory populated with 4 security baseline python scripts
- Results directory contains `.gitignore` to prevent versioning traces
- Handover report generated with recommended framework and next steps