references/axial-coding.md
# Axial Coding
Group open-ended notes into structured failure taxonomies.
## Process
1. **Gather** - Collect open coding notes
2. **Pattern** - Group notes with common themes
3. **Name** - Create actionable category names
4. **Quantify** - Count failures per category
## Example Taxonomy
```yaml
failure_taxonomy:
content_quality:
hallucination: [invented_facts, fictional_citations]
incompleteness: [partial_answer, missing_key_info]
inaccuracy: [wrong_numbers, wrong_dates]
communication:
tone_mismatch: [too_casual, too_formal]
clarity: [ambiguous, jargon_heavy]
context:
user_context: [ignored_preferences, misunderstood_intent]
retrieved_context: [ignored_documents, wrong_context]
safety:
missing_disclaimers: [legal, medical, financial]
```
## Add Annotation (Python)
```python
from phoenix.client import Client
client = Client()
client.spans.add_span_annotation(
span_id="abc123",
annotation_name="failure_category",
label="hallucination",
explanation="invented a feature that doesn't exist",
annotator_kind="HUMAN",
sync=True,
)
```
## Add Annotation (TypeScript)
```typescript
import { addSpanAnnotation } from "@arizeai/phoenix-client/spans";
await addSpanAnnotation({
spanAnnotation: {
spanId: "abc123",
name: "failure_category",
label: "hallucination",
explanation: "invented a feature that doesn't exist",
annotatorKind: "HUMAN",
}
});
```
## Agent Failure Taxonomy
```yaml
agent_failures:
planning: [wrong_plan, incomplete_plan]
tool_selection: [wrong_tool, missed_tool, unnecessary_call]
tool_execution: [wrong_parameters, type_error]
state_management: [lost_context, stuck_in_loop]
error_recovery: [no_fallback, wrong_fallback]
```
## Transition Matrix (Agents)
Shows where failures occur between states:
```python
def build_transition_matrix(conversations, states):
matrix = defaultdict(lambda: defaultdict(int))
for conv in conversations:
if conv["failed"]:
last_success = find_last_success(conv)
first_failure = find_first_failure(conv)
matrix[last_success][first_failure] += 1
return pd.DataFrame(matrix).fillna(0)
```
## Principles
- **MECE** - Each failure fits ONE category
- **Actionable** - Categories suggest fixes
- **Bottom-up** - Let categories emerge from data
references/common-mistakes-python.md
# Common Mistakes (Python)
Patterns that LLMs frequently generate incorrectly from training data.
## Legacy Model Classes
```python
# WRONG
from phoenix.evals import OpenAIModel, AnthropicModel
model = OpenAIModel(model="gpt-4")
# RIGHT
from phoenix.evals import LLM
llm = LLM(provider="openai", model="gpt-4o")
```
**Why**: `OpenAIModel`, `AnthropicModel`, etc. are legacy 1.0 wrappers in `phoenix.evals.legacy`.
The `LLM` class is provider-agnostic and is the current 2.0 API.
## Using run_evals Instead of evaluate_dataframe
```python
# WRONG — legacy 1.0 API
from phoenix.evals import run_evals
results = run_evals(dataframe=df, evaluators=[eval1], provide_explanation=True)
# Returns list of DataFrames
# RIGHT — current 2.0 API
from phoenix.evals import evaluate_dataframe
results_df = evaluate_dataframe(dataframe=df, evaluators=[eval1])
# Returns single DataFrame with {name}_score dict columns
```
**Why**: `run_evals` is the legacy 1.0 batch function. `evaluate_dataframe` is the current
2.0 function with a different return format.
## Wrong Result Column Names
```python
# WRONG — column doesn't exist
score = results_df["relevance"].mean()
# WRONG — column exists but contains dicts, not numbers
score = results_df["relevance_score"].mean()
# RIGHT — extract numeric score from dict
scores = results_df["relevance_score"].apply(
lambda x: x.get("score", 0.0) if isinstance(x, dict) else 0.0
)
score = scores.mean()
```
**Why**: `evaluate_dataframe` returns columns named `{name}_score` containing Score dicts
like `{"name": "...", "score": 1.0, "label": "...", "explanation": "..."}`.
## Deprecated project_name Parameter
```python
# WRONG
df = client.spans.get_spans_dataframe(project_name="my-project")
# RIGHT
df = client.spans.get_spans_dataframe(project_identifier="my-project")
```
**Why**: `project_name` is deprecated in favor of `project_identifier`, which also
accepts project IDs.
## Wrong Client Constructor
```python
# WRONG
client = Client(endpoint="https://app.phoenix.arize.com")
client = Client(url="https://app.phoenix.arize.com")
# RIGHT — for remote/cloud Phoenix
client = Client(base_url="https://app.phoenix.arize.com", api_key="...")
# ALSO RIGHT — for local Phoenix (falls back to env vars or localhost:6006)
client = Client()
```
**Why**: The parameter is `base_url`, not `endpoint` or `url`. For local instances,
`Client()` with no args works fine. For remote instances, `base_url` and `api_key` are required.
## Too-Aggressive Time Filters
```python
# WRONG — often returns zero spans
from datetime import datetime, timedelta
df = client.spans.get_spans_dataframe(
project_identifier="my-project",
start_time=datetime.now() - timedelta(hours=1),
)
# RIGHT — use limit to control result size instead
df = client.spans.get_spans_dataframe(
project_identifier="my-project",
limit=50,
)
```
**Why**: Traces may be from any time period. A 1-hour window frequently returns
nothing. Use `limit=` to control result size instead.
## Not Filtering Spans Appropriately
```python
from phoenix.client.types.spans import SpanQuery
# WRONG — fetches all spans including internal LLM calls, retrievers, etc.
df = client.spans.get_spans_dataframe(project_identifier="my-project")
# RIGHT for end-to-end evaluation — filter to top-level spans
df = client.spans.get_spans_dataframe(
project_identifier="my-project",
query=SpanQuery().where("parent_id is None"),
)
# RIGHT for RAG evaluation — fetch child spans for retriever/LLM metrics
all_spans = client.spans.get_spans_dataframe(
project_identifier="my-project",
)
retriever_spans = all_spans[all_spans["span_kind"] == "RETRIEVER"]
llm_spans = all_spans[all_spans["span_kind"] == "LLM"]
```
**Why**: For end-to-end evaluation (e.g., overall answer quality), scope the query to root
spans with `SpanQuery().where("parent_id is None")`. Use `parent_span is None` instead if you
also want orphans (spans whose parent is absent) counted as roots.
For RAG systems, you often need child spans separately — retriever spans for
RetrievalRelevance and LLM spans for Faithfulness. Choose the right span level
for your evaluation target.
## Assuming Span Output is Plain Text
```python
# WRONG — output may be JSON, not plain text
df["output"] = df["attributes.output.value"]
# RIGHT — parse JSON and extract the answer field
import json
def extract_answer(output_value):
if not isinstance(output_value, str):
return str(output_value) if output_value is not None else ""
try:
parsed = json.loads(output_value)
if isinstance(parsed, dict):
for key in ("answer", "result", "output", "response"):
if key in parsed:
return str(parsed[key])
except (json.JSONDecodeError, TypeError):
pass
return output_value
df["output"] = df["attributes.output.value"].apply(extract_answer)
```
**Why**: LangChain and other frameworks often output structured JSON from root spans,
like `{"context": "...", "question": "...", "answer": "..."}`. Evaluators need
the actual answer text, not the raw JSON.
## Using @create_evaluator for LLM-Based Evaluation
```python
# WRONG — @create_evaluator doesn't call an LLM
@create_evaluator(name="relevance", kind="llm")
def relevance(input: str, output: str) -> str:
pass # No LLM is involved
# RIGHT — use ClassificationEvaluator for LLM-based evaluation
from phoenix.evals import ClassificationEvaluator, LLM
relevance = ClassificationEvaluator(
name="relevance",
prompt_template="Is this relevant?\n{{input}}\n{{output}}\nAnswer:",
llm=LLM(provider="openai", model="gpt-4o"),
choices={"relevant": 1.0, "irrelevant": 0.0},
)
```
**Why**: `@create_evaluator` wraps a plain Python function. Setting `kind="llm"`
marks it as LLM-based but you must implement the LLM call yourself.
For LLM-based evaluation, prefer `ClassificationEvaluator` which handles
the LLM call, structured output parsing, and explanations automatically.
## Using llm_classify Instead of ClassificationEvaluator
```python
# WRONG — legacy 1.0 API
from phoenix.evals import llm_classify
results = llm_classify(
dataframe=df,
template=template_str,
model=model,
rails=["relevant", "irrelevant"],
)
# RIGHT — current 2.0 API
from phoenix.evals import ClassificationEvaluator, async_evaluate_dataframe, LLM
classifier = ClassificationEvaluator(
name="relevance",
prompt_template=template_str,
llm=LLM(provider="openai", model="gpt-4o"),
choices={"relevant": 1.0, "irrelevant": 0.0},
)
results_df = await async_evaluate_dataframe(dataframe=df, evaluators=[classifier])
```
**Why**: `llm_classify` is the legacy 1.0 function. The current pattern is to create
an evaluator with `ClassificationEvaluator` and run it with `async_evaluate_dataframe()`.
## Using HallucinationEvaluator
```python
# WRONG — not a top-level export, and takes an LLM, not a model string
from phoenix.evals import HallucinationEvaluator
eval = HallucinationEvaluator(model)
# WRONG — there is no `context` field
eval.evaluate({"input": question, "output": answer, "context": retrieved_docs})
# RIGHT
from phoenix.evals.metrics import HallucinationEvaluator
from phoenix.evals import LLM
eval = HallucinationEvaluator(llm=LLM(provider="openai", model="gpt-4o"))
eval.evaluate({"input": conversation_so_far, "output": assistant_reply})
```
**Why**: pre-built evaluators live in `phoenix.evals.metrics`, not the top-level
`phoenix.evals`. `HallucinationEvaluator` grounds a response against **the conversation**
— `input` is the full history the assistant had access to (prior turns, tool calls, tool
results), `output` is the reply being judged. There is no separate `context` field: if your
source of truth is a supplied context such as retrieved documents, that is
`FaithfulnessEvaluator`, not this one.
Labels are `hallucinated`/`grounded` and the score is **minimized** — `hallucinated` is
`1.0`, `grounded` is `0.0`. Do not assume "1.0 = good"; read the returned `Score`'s
`direction` before thresholding.
references/error-analysis-multi-turn.md
# Error Analysis: Multi-Turn Conversations
Debugging complex multi-turn conversation traces.
## The Approach
1. **End-to-end first** - Did the conversation achieve the goal?
2. **Find first failure** - Trace backwards to root cause
3. **Simplify** - Try single-turn before multi-turn debug
4. **N-1 testing** - Isolate turn-specific vs capability issues
## Find First Upstream Failure
```
Turn 1: User asks about flights ✓
Turn 2: Assistant asks for dates ✓
Turn 3: User provides dates ✓
Turn 4: Assistant searches WRONG dates ← FIRST FAILURE
Turn 5: Shows wrong flights (consequence)
Turn 6: User frustrated (consequence)
```
Focus on Turn 4, not Turn 6.
## Simplify First
Before debugging multi-turn, test single-turn:
```python
# If single-turn also fails → problem is retrieval/knowledge
# If single-turn passes → problem is conversation context
response = chat("What's the return policy for electronics?")
```
## N-1 Testing
Give turns 1 to N-1 as context, test turn N:
```python
context = conversation[:n-1]
response = chat_with_context(context, user_message_n)
# Compare to actual turn N
```
This isolates whether error is from context or underlying capability.
## Checklist
1. Did conversation achieve goal? (E2E)
2. Which turn first went wrong?
3. Can you reproduce with single-turn?
4. Is error from context or capability? (N-1 test)
references/error-analysis.md
# Error Analysis
Review traces to discover failure modes before building evaluators.
## Process
1. **Sample** - 100+ traces (errors, negative feedback, random)
2. **Open Code** - Write free-form notes per trace
3. **Axial Code** - Group notes into failure categories
4. **Quantify** - Count failures per category
5. **Prioritize** - Rank by frequency × severity
## Sample Traces
### Span-level sampling (Python — DataFrame)
```python
from phoenix.client import Client
# Client() works for local Phoenix (falls back to env vars or localhost:6006)
# For remote/cloud: Client(base_url="https://app.phoenix.arize.com", api_key="...")
client = Client()
spans_df = client.spans.get_spans_dataframe(project_identifier="my-app")
# Build representative sample
sample = pd.concat([
spans_df[spans_df["status_code"] == "ERROR"].sample(30),
spans_df[spans_df["feedback"] == "negative"].sample(30),
spans_df.sample(40),
]).drop_duplicates("span_id").head(100)
```
### Span-level sampling (TypeScript)
```typescript
import { getSpans } from "@arizeai/phoenix-client/spans";
const { spans: errors } = await getSpans({
project: { projectName: "my-app" },
statusCode: "ERROR",
limit: 30,
});
const { spans: allSpans } = await getSpans({
project: { projectName: "my-app" },
limit: 70,
});
const sample = [...errors, ...allSpans.sort(() => Math.random() - 0.5).slice(0, 40)];
const unique = [...new Map(sample.map((s) => [s.context.span_id, s])).values()].slice(0, 100);
```
### Trace-level sampling (Python)
When errors span multiple spans (e.g., agent workflows), sample whole traces:
```python
from datetime import datetime, timedelta
traces = client.traces.get_traces(
project_identifier="my-app",
start_time=datetime.now() - timedelta(hours=24),
include_spans=True,
sort="latency_ms",
order="desc",
limit=100,
)
# Each trace has: trace_id, start_time, end_time, spans
```
### Trace-level sampling (TypeScript)
```typescript
import { getTraces } from "@arizeai/phoenix-client/traces";
const { traces } = await getTraces({
project: { projectName: "my-app" },
startTime: new Date(Date.now() - 24 * 60 * 60 * 1000),
includeSpans: true,
limit: 100,
});
```
## Add Notes (Python)
```python
client.spans.add_span_note(
span_id="abc123",
note="wrong timezone - said 3pm EST but user is PST"
)
```
## Add Notes (TypeScript)
```typescript
import { addSpanNote } from "@arizeai/phoenix-client/spans";
await addSpanNote({
spanNote: {
spanId: "abc123",
note: "wrong timezone - said 3pm EST but user is PST"
}
});
```
## What to Note
| Type | Examples |
| ---- | -------- |
| Factual errors | Wrong dates, prices, made-up features |
| Missing info | Didn't answer question, omitted details |
| Tone issues | Too casual/formal for context |
| Tool issues | Wrong tool, wrong parameters |
| Retrieval | Wrong docs, missing relevant docs |
## Good Notes
```
BAD: "Response is bad"
GOOD: "Response says ships in 2 days but policy is 5-7 days"
```
## Group into Categories
```python
categories = {
"factual_inaccuracy": ["wrong shipping time", "incorrect price"],
"hallucination": ["made up a discount", "invented feature"],
"tone_mismatch": ["informal for enterprise client"],
}
# Priority = Frequency × Severity
```
## Retrieve Existing Annotations
### Python
```python
# From a spans DataFrame
annotations_df = client.spans.get_span_annotations_dataframe(
spans_dataframe=sample,
project_identifier="my-app",
include_annotation_names=["quality", "correctness"],
)
# annotations_df has: span_id (index), name, label, score, explanation
# Or from specific span IDs
annotations_df = client.spans.get_span_annotations_dataframe(
span_ids=["span-id-1", "span-id-2"],
project_identifier="my-app",
)
```
### TypeScript
```typescript
import { getSpanAnnotations } from "@arizeai/phoenix-client/spans";
const { annotations } = await getSpanAnnotations({
project: { projectName: "my-app" },
spanIds: ["span-id-1", "span-id-2"],
includeAnnotationNames: ["quality", "correctness"],
});
for (const ann of annotations) {
console.log(`${ann.span_id}: ${ann.name} = ${ann.result?.label} (${ann.result?.score})`);
}
```
## Saturation
Stop when new traces reveal no new failure modes. Minimum: 100 traces.
references/evaluate-dataframe-python.md
# Batch Evaluation with evaluate_dataframe (Python)
Run evaluators across a DataFrame. The core 2.0 batch evaluation API.
## Preferred: async_evaluate_dataframe
For batch evaluations (especially with LLM evaluators), prefer the async version
for better throughput:
```python
from phoenix.evals import async_evaluate_dataframe
results_df = await async_evaluate_dataframe(
dataframe=df, # pandas DataFrame with columns matching evaluator params
evaluators=[eval1, eval2], # List of evaluators
concurrency=5, # Max concurrent LLM calls (default 3)
exit_on_error=False, # Optional: stop on first error (default True)
max_retries=3, # Optional: retry failed or timed-out LLM calls (default 10)
)
```
## Sync Version
```python
from phoenix.evals import evaluate_dataframe
results_df = evaluate_dataframe(
dataframe=df, # pandas DataFrame with columns matching evaluator params
evaluators=[eval1, eval2], # List of evaluators
exit_on_error=False, # Optional: stop on first error (default True)
max_retries=3, # Optional: retry failed or timed-out LLM calls (default 10)
)
```
## Errors and Retries
Failing rows retry up to `max_retries`, except `PhoenixException`s, which fail immediately (`RateLimitError` still retries). `exit_on_error=True` (default) stops at the first exhausted row, leaving the rest unevaluated; `False` continues. Per-row `status`/`exceptions` are in `{name}_execution_details`.
## Result Column Format
`async_evaluate_dataframe` / `evaluate_dataframe` returns a copy of the input DataFrame with added columns.
**Result columns contain dicts, NOT raw numbers.**
For each evaluator named `"foo"`, two columns are added:
| Column | Type | Contents |
| ------ | ---- | -------- |
| `foo_score` | `dict` | `{"name": "foo", "score": 1.0, "label": "True", "explanation": "...", "metadata": {...}, "kind": "code", "direction": "maximize"}` |
| `foo_execution_details` | `dict` | `{"status": "success", "exceptions": [], "execution_seconds": 0.001}` |
Only non-None fields appear in the score dict.
### Extracting Numeric Scores
```python
# WRONG — these will fail or produce unexpected results
score = results_df["relevance"].mean() # KeyError!
score = results_df["relevance_score"].mean() # Tries to average dicts!
# RIGHT — extract the numeric score from each dict
scores = results_df["relevance_score"].apply(
lambda x: x.get("score", 0.0) if isinstance(x, dict) else 0.0
)
mean_score = scores.mean()
```
### Extracting Labels
```python
labels = results_df["relevance_score"].apply(
lambda x: x.get("label", "") if isinstance(x, dict) else ""
)
```
### Extracting Explanations (LLM evaluators)
```python
explanations = results_df["relevance_score"].apply(
lambda x: x.get("explanation", "") if isinstance(x, dict) else ""
)
```
### Finding Failures
```python
scores = results_df["relevance_score"].apply(
lambda x: x.get("score", 0.0) if isinstance(x, dict) else 0.0
)
failed_mask = scores < 0.5
failures = results_df[failed_mask]
```
## Input Mapping
Evaluators receive each row as a dict. Column names must match the evaluator's
expected parameter names. If they don't match, use `.bind()` or `bind_evaluator`:
```python
from phoenix.evals import bind_evaluator, create_evaluator, async_evaluate_dataframe
@create_evaluator(name="check", kind="code")
def check(response: str) -> bool:
return len(response.strip()) > 0
# Option 1: Use .bind() method on the evaluator
check.bind(input_mapping={"response": "answer"})
results_df = await async_evaluate_dataframe(dataframe=df, evaluators=[check])
# Option 2: Use bind_evaluator function
bound = bind_evaluator(evaluator=check, input_mapping={"response": "answer"})
results_df = await async_evaluate_dataframe(dataframe=df, evaluators=[bound])
```
Or simply rename columns to match:
```python
df = df.rename(columns={
"attributes.input.value": "input",
"attributes.output.value": "output",
})
```
## DO NOT use run_evals
```python
# WRONG — legacy 1.0 API
from phoenix.evals import run_evals
results = run_evals(dataframe=df, evaluators=[eval1])
# Returns List[DataFrame] — one per evaluator
# RIGHT — current 2.0 API
from phoenix.evals import async_evaluate_dataframe
results_df = await async_evaluate_dataframe(dataframe=df, evaluators=[eval1])
# Returns single DataFrame with {name}_score dict columns
```
Key differences:
- `run_evals` returns a **list** of DataFrames (one per evaluator)
- `async_evaluate_dataframe` returns a **single** DataFrame with all results merged
- `async_evaluate_dataframe` uses `{name}_score` dict column format
- `async_evaluate_dataframe` uses `bind_evaluator` for input mapping (not `input_mapping=` param)
references/evaluators-code-python.md
# Evaluators: Code Evaluators in Python
Deterministic evaluators without LLM. Fast, cheap, reproducible.
## Basic Pattern
```python
import re
import json
from phoenix.evals import create_evaluator
@create_evaluator(name="has_citation", kind="code")
def has_citation(output: str) -> bool:
return bool(re.search(r'\[\d+\]', output))
@create_evaluator(name="json_valid", kind="code")
def json_valid(output: str) -> bool:
try:
json.loads(output)
return True
except json.JSONDecodeError:
return False
```
## Parameter Binding
| Parameter | Description |
| --------- | ----------- |
| `output` | Task output |
| `input` | Example input |
| `expected` | Expected output |
| `metadata` | Example metadata |
```python
@create_evaluator(name="matches_expected", kind="code")
def matches_expected(output: str, expected: dict) -> bool:
return output.strip() == expected.get("answer", "").strip()
```
## Common Patterns
- **Regex**: `re.search(pattern, output)`
- **JSON schema**: `jsonschema.validate()`
- **Keywords**: `keyword in output.lower()`
- **Length**: `len(output.split())`
- **Similarity**: `editdistance.eval()` or Jaccard
## Return Types
| Return type | Result |
| ----------- | ------ |
| `bool` | `True` → score=1.0, label="True"; `False` → score=0.0, label="False" |
| `float`/`int` | Used as the `score` value directly |
| `str` (short, ≤3 words) | Used as the `label` value |
| `str` (long, ≥4 words) | Used as the `explanation` value |
| `dict` with `score`/`label`/`explanation`/`metadata` | Mapped to Score fields directly |
| `Score` object | Used as-is |
## Important: Code vs LLM Evaluators
The `@create_evaluator` decorator wraps a plain Python function.
- `kind="code"` (default): For deterministic evaluators that don't call an LLM.
- `kind="llm"`: Marks the evaluator as LLM-based, but **you** must implement the LLM
call inside the function. The decorator does not call an LLM for you.
For most LLM-based evaluation, prefer `ClassificationEvaluator` which handles
the LLM call, structured output parsing, and explanations automatically:
```python
from phoenix.evals import ClassificationEvaluator, LLM
relevance = ClassificationEvaluator(
name="relevance",
prompt_template="Is this relevant?\n{{input}}\n{{output}}\nAnswer:",
llm=LLM(provider="openai", model="gpt-4o"),
choices={"relevant": 1.0, "irrelevant": 0.0},
)
```
## Pre-Built
```python
from phoenix.client.experiments import create_evaluator
from phoenix.evals.metrics import MatchesRegex
date_format = MatchesRegex(pattern=r"\d{4}-\d{2}-\d{2}")
@create_evaluator(name="contains_any_keyword", kind="code")
def contains_any_keyword(output, expected):
keywords = expected.get("keywords", [])
return any(kw.lower() in str(output).lower() for kw in keywords)
@create_evaluator(name="json_parseable", kind="code")
def json_parseable(output):
import json
try:
json.loads(output)
return True
except (json.JSONDecodeError, TypeError):
return False
```
references/evaluators-code-typescript.md
# Evaluators: Code Evaluators in TypeScript
Deterministic evaluators without LLM. Fast, cheap, reproducible.
## Basic Pattern
```typescript
import { createEvaluator } from "@arizeai/phoenix-evals";
const containsCitation = createEvaluator<{ output: string }>(
({ output }) => /\[\d+\]/.test(output) ? 1 : 0,
{ name: "contains_citation", kind: "CODE" }
);
```
The wrapped function can return a number (becomes `score`), a string (becomes `label`), or an object; from an object, `score`, `label`, `explanation`, and `metadata` (a plain object) are kept and other keys are dropped.
## With Full Results (asExperimentEvaluator)
```typescript
import { asExperimentEvaluator } from "@arizeai/phoenix-client/experiments";
const jsonValid = asExperimentEvaluator({
name: "json_valid",
kind: "CODE",
evaluate: async ({ output }) => {
try {
JSON.parse(String(output));
return { score: 1.0, label: "valid_json" };
} catch (e) {
return { score: 0.0, label: "invalid_json", explanation: String(e) };
}
},
});
```
## Parameter Types
```typescript
interface EvaluatorParams {
input: Record<string, unknown>;
output: unknown;
expected: Record<string, unknown>;
metadata: Record<string, unknown>;
}
```
## Common Patterns
- **Regex**: `/pattern/.test(output)`
- **JSON**: `JSON.parse()` + zod schema
- **Keywords**: `output.includes(keyword)`
- **Similarity**: `fastest-levenshtein`
## Built-In Classification Metrics (precision/recall/F-score)
`@arizeai/phoenix-evals/code` provides precision, recall, and F-beta (including F1) as built-in dataset-level evaluators, mirroring Python's `PrecisionRecallFScore`. `expected`/`output` are the full label sequence across a batch, not a single row.
```typescript
import {
createPrecisionEvaluator,
createRecallEvaluator,
createF1Evaluator,
createFBetaEvaluator,
createPrecisionRecallFScoreEvaluators,
} from "@arizeai/phoenix-evals/code";
// Binary classification via positiveLabel (or auto-detected for numeric {0,1}
// labels under the default "macro" average)
const precision = createPrecisionEvaluator({ positiveLabel: "spam" });
const recall = createRecallEvaluator({ positiveLabel: "spam" });
// Multi-class: average is "macro" (default, equal weight per class),
// "micro" (pool TP/FP/FN first — equals accuracy for single-label
// multi-class), or "weighted" (weight by class support)
const { precision: p, recall: r, fScore: f } =
createPrecisionRecallFScoreEvaluators({ average: "weighted" });
// F-beta: beta > 1 weights recall more (e.g. medical screening), beta < 1
// weights precision more (e.g. spam filtering); beta = 1 is F1
const f2 = createFBetaEvaluator({ beta: 2 });
const result = await createF1Evaluator().evaluate({
expected: ["cat", "dog", "cat", "bird"],
output: ["cat", "cat", "cat", "bird"],
});
// { score: 0.6 }
```
Docs: [Precision / Recall / F-Score](https://arize.com/docs/phoenix/evaluation/pre-built-metrics/precision-recall-fscore).
references/evaluators-custom-templates.md
# Evaluators: Custom Templates
Design LLM judge prompts.
## Complete Template Pattern
```python
TEMPLATE = """Evaluate faithfulness of the response to the context.
<context>{{context}}</context>
<response>{{output}}</response>
CRITERIA:
"faithful" = ALL claims supported by context
"unfaithful" = ANY claim NOT in context
EXAMPLES:
Context: "Price is $10" → Response: "It costs $10" → faithful
Context: "Price is $10" → Response: "About $15" → unfaithful
EDGE CASES:
- Empty context → cannot_evaluate
- "I don't know" when appropriate → faithful
- Partial faithfulness → unfaithful (strict)
Answer (faithful/unfaithful):"""
```
## Template Structure
1. Task description
2. Input variables in XML tags
3. Criteria definitions
4. Examples (2-4 cases)
5. Edge cases
6. Output format
## XML Tags
```
<question>{{input}}</question>
<response>{{output}}</response>
<context>{{context}}</context>
<reference>{{reference}}</reference>
```
## Common Mistakes
| Mistake | Fix |
| ------- | --- |
| Vague criteria | Define each label exactly |
| No examples | Include 2-4 cases |
| Ambiguous format | Specify exact output |
| No edge cases | Address ambiguity |
references/evaluators-llm-python.md
# Evaluators: LLM Evaluators in Python
LLM evaluators use a language model to judge outputs. Use when criteria are subjective.
## Quick Start
```python
from phoenix.evals import ClassificationEvaluator, LLM
llm = LLM(provider="openai", model="gpt-4o")
HELPFULNESS_TEMPLATE = """Rate how helpful the response is.
<question>{{input}}</question>
<response>{{output}}</response>
"helpful" means directly addresses the question.
"not_helpful" means does not address the question.
Your answer (helpful/not_helpful):"""
helpfulness = ClassificationEvaluator(
name="helpfulness",
prompt_template=HELPFULNESS_TEMPLATE,
llm=llm,
choices={"not_helpful": 0, "helpful": 1}
)
```
## Decision Models
Any `ClassificationEvaluator`, including every built-in classification evaluator, also accepts a `DecisionModel` in place of `LLM`. A decision model such as TypeSafe's Jev picks one of the labels directly instead of generating text, so it is a cheaper, faster judge for high-volume labeling. Install `typesafe-sdk` (`pip install typesafe-sdk`); extra keyword arguments such as `api_key` are forwarded to the TypeSafe client.
```python
from phoenix.evals import ClassificationEvaluator, DecisionModel
sentiment = ClassificationEvaluator(
name="sentiment",
llm=DecisionModel(provider="typesafe", model="jev-latest"),
prompt_template="Classify the sentiment: {{text}}",
choices={"positive": 1, "neutral": 0.5, "negative": 0},
)
# Score(label="positive", score=1, explanation=None,
# metadata={"model": "jev-latest", "probabilities": {...}, "model_id": "...", "confidence": ...})
```
- **No explanation.** `explanation` is always `None`; `metadata["probabilities"]` maps each label to its probability, and `metadata["confidence"]` is set when the provider reports one.
- **Plain-text prompt.** The rendered template is sent as one plain-text state; message roles are dropped and only text parts are kept.
- **Providers.** `typesafe` is the only registered provider; `from phoenix.evals.decision import show_decision_provider_availability` prints which providers are usable in the current environment.
- **Traced as DECISION spans.** Each call emits a DECISION span with `decision.*` attributes instead of an LLM span.
- **Not for calculations.** Decision models can't reason step by step; use an `LLM` for judgments that need arithmetic or unit conversion.
## Template Variables
Use XML tags to wrap variables for clarity:
| Variable | XML Tag |
| -------- | ------- |
| `{{input}}` | `<question>{{input}}</question>` |
| `{{output}}` | `<response>{{output}}</response>` |
| `{{reference}}` | `<reference>{{reference}}</reference>` |
| `{{context}}` | `<context>{{context}}</context>` |
## create_classifier (Factory)
Shorthand factory that returns a `ClassificationEvaluator`. Prefer direct
`ClassificationEvaluator` instantiation for more parameters/customization:
```python
from phoenix.evals import create_classifier, LLM
relevance = create_classifier(
name="relevance",
prompt_template="""Is this response relevant to the question?
<question>{{input}}</question>
<response>{{output}}</response>
Answer (relevant/irrelevant):""",
llm=LLM(provider="openai", model="gpt-4o"),
choices={"relevant": 1.0, "irrelevant": 0.0},
)
```
## Input Mapping
Column names must match template variables. Rename columns or use `bind_evaluator`:
```python
# Option 1: Rename columns to match template variables
df = df.rename(columns={"user_query": "input", "ai_response": "output"})
# Option 2: Use bind_evaluator
from phoenix.evals import bind_evaluator
bound = bind_evaluator(
evaluator=helpfulness,
input_mapping={"input": "user_query", "output": "ai_response"},
)
```
## Running
```python
from phoenix.evals import evaluate_dataframe
results_df = evaluate_dataframe(dataframe=df, evaluators=[helpfulness])
```
## Best Practices
1. **Be specific** - Define exactly what pass/fail means
2. **Include examples** - Show concrete cases for each label
3. **Explanations by default** - `ClassificationEvaluator` includes explanations automatically
4. **Study built-in prompts** - See
`phoenix.evals.__generated__.classification_evaluator_configs` for examples
of well-structured evaluation prompts (Faithfulness, Correctness, RetrievalRelevance, etc.)
references/evaluators-llm-typescript.md
# Evaluators: LLM Evaluators in TypeScript
LLM evaluators use a language model to judge outputs. Uses Vercel AI SDK.
## Quick Start
```typescript
import { createClassificationEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";
const helpfulness = await createClassificationEvaluator<{
input: string;
output: string;
}>({
name: "helpfulness",
model: openai("gpt-4o"),
promptTemplate: `Rate helpfulness.
<question>{{input}}</question>
<response>{{output}}</response>
Answer (helpful/not_helpful):`,
choices: { not_helpful: 0, helpful: 1 },
});
```
## Decision Models
Any classification evaluator, including every built-in, also accepts an AI SDK decision model such as OpenAI's `gpt-6-luna` or TypeSafe's Jev. A decision model picks one of the labels directly instead of generating text, so it is a cheaper, faster judge for high-volume labeling. With OpenAI, use `@ai-sdk/openai` 4.0.89 or later.
```typescript
import { openai } from "@ai-sdk/openai";
import { createCorrectnessEvaluator } from "@arizeai/phoenix-evals";
const correctness = createCorrectnessEvaluator({
model: openai.decisionModel("gpt-6-luna"),
});
// { label: "incorrect", score: 0,
// metadata: { probabilities: { correct: 0.02, incorrect: 0.98 }, confidence: 0.96, modelId: "gpt-6-luna" } }
```
- **No explanation.** `explanation` is always undefined; `metadata.probabilities` maps each label to its probability, and `metadata.confidence` is set when the provider reports one.
- **Plain-text prompt.** The template is rendered as usual, then sent as one plain-text state; message roles are dropped.
- **Refusals throw.** A refusal throws the AI SDK's `Experimental_DecisionRefusalError`; in `runExperiment` it is recorded as that evaluation's error.
- **String IDs are language models.** Wrap the ID with the provider's `decisionModel()`. `isDecisionModel(model)` and the `DecisionModel` type are exported for branching on this yourself.
- **Not for calculations.** Decision models can't reason step by step; use a language model for judgments that need arithmetic or unit conversion.
## Template Variables
Use XML tags: `<question>{{input}}</question>`, `<response>{{output}}</response>`, `<context>{{context}}</context>`
## Custom Evaluator with asExperimentEvaluator
```typescript
import { asExperimentEvaluator } from "@arizeai/phoenix-client/experiments";
const customEval = asExperimentEvaluator({
name: "custom",
kind: "LLM",
evaluate: async ({ input, output }) => {
// Your LLM call here
return { score: 1.0, label: "pass", explanation: "..." };
},
});
```
## Pre-Built Evaluators
```typescript
import { createFaithfulnessEvaluator } from "@arizeai/phoenix-evals";
const faithfulnessEvaluator = createFaithfulnessEvaluator({
model: openai("gpt-4o"),
});
```
## Best Practices
- Be specific about criteria
- Include examples in prompts
- Use `<thinking>` for chain of thought
references/evaluators-overview.md
# Evaluators: Overview
When and how to build automated evaluators.
## Decision Framework
```
Should I Build an Evaluator?
│
▼
Can I fix it with a prompt change?
YES → Fix the prompt first
NO → Is this a recurring issue?
YES → Build evaluator
NO → Add to watchlist
```
**Don't automate prematurely.** Many issues are simple prompt fixes.
## Evaluator Requirements
1. **Clear criteria** - Specific, not "Is it good?"
2. **Labeled test set** - 100+ examples with human labels
3. **Measured accuracy** - Know TPR/TNR before deploying
## Evaluator Lifecycle
1. **Discover** - Error analysis reveals pattern
2. **Design** - Define criteria and test cases
3. **Implement** - Build code or LLM evaluator
4. **Calibrate** - Validate against human labels
5. **Deploy** - Add to experiment/CI pipeline
6. **Monitor** - Track accuracy over time
7. **Maintain** - Update as product evolves
## What NOT to Automate
- **Rare issues** - <5 instances? Watchlist, don't build
- **Quick fixes** - Fixable by prompt change? Fix it
- **Evolving criteria** - Stabilize definition first
references/evaluators-pre-built.md
# Evaluators: Pre-Built
Use for exploration only. Validate before production.
Pre-built evaluators are importable from `phoenix.evals.metrics` (Python) and
`@arizeai/phoenix-evals` (TypeScript). Naming follows `<Name>Evaluator` in
Python and `create<Name>Evaluator` in TypeScript.
## The Evaluators
LLM-judged evaluators, with their required input fields, labels, and score
direction. Input field names are snake_case in Python and camelCase in
TypeScript. For `minimize` evaluators a high score is the bad outcome.
| Evaluator | Inputs (Python names) | Labels (1.0 / 0.0) | Direction |
| --------- | --------------------- | ------------------ | --------- |
| Completeness | `conversation` | `complete` / `incomplete` | maximize |
| Conciseness | `input`, `output` | `concise` / `verbose` | maximize |
| Correctness | `input`, `output` | `correct` / `incorrect` | maximize |
| Faithfulness | `input`, `output`, `context` | `faithful` / `unfaithful` | maximize |
| Hallucination | `input`, `output` | `hallucinated` / `grounded` | minimize |
| PiiDetection | `conversation` | `pii_detected` / `no_pii_detected` | minimize |
| Refusal | `input`, `output` | `refused` / `answered` | neutral |
| RetrievalRelevance | `input`, `context` | `relevant` / `irrelevant` | maximize |
| ToolInvocation | `input`, `available_tools`, `tool_selection` | `correct` / `incorrect` | maximize |
| ToolResponseHandling | `input`, `tool_call`, `tool_result`, `output` | `correct` / `incorrect` | maximize |
| ToolSelection | `input`, `available_tools`, `tool_selection` | `correct` / `incorrect` | maximize |
| Toxicity | `text` | `toxic` / `non-toxic` | minimize |
| UserFriction | `conversation`, `user_message` | `friction` / `no_friction` | minimize |
Code-based evaluators ship alongside the LLM judges: `exact_match`,
`MatchesRegex`, and `PrecisionRecallFScore` in Python; the TypeScript
equivalents live in `@arizeai/phoenix-evals/code` — see
[evaluators-code-typescript.md](evaluators-code-typescript.md).
## Python
```python
from phoenix.evals import LLM
from phoenix.evals.metrics import FaithfulnessEvaluator
llm = LLM(provider="openai", model="gpt-4o")
faithfulness_eval = FaithfulnessEvaluator(llm=llm)
```
## TypeScript
```typescript
import { createFaithfulnessEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";
const faithfulnessEval = createFaithfulnessEvaluator({ model: openai("gpt-4o") });
```
## Notes on Specific Evaluators
- **Faithfulness vs. hallucination.** Both check whether a response is
supported by a source of truth; they differ in what that source is.
Faithfulness judges against a separately supplied `context` (retrieved
documents — use it for RAG). Hallucination judges against the conversation
itself: `input` holds the full history the assistant saw, including tool
calls and results, and there is no `context` field — use it for multi-turn
agents and chat.
- **Completeness scores finished work across the whole conversation.** The
judge lists every request the user made in any turn, marks each one
`fulfilled`, `withdrawn`, or `unfulfilled`, and labels the conversation
`incomplete` if any non-withdrawn request is unfulfilled — so pass the full
history in `conversation`, not the final turn. Fulfilled means the
deliverable is visible in the record: an answer covering every part asked
for, an artifact with every required component, or an action whose success
shows up in a tool result. A refusal, an "I don't know", a clarifying
question, or a promise is unfulfilled. Correctness is out of scope, so a
wrong-but-delivered answer is still complete. For agent traces include tool
calls and results: matching tool evidence overrides the assistant's claim,
and without it the judge has to take the assistant's word. The
`explanation` is an `INTENTIONS:` block, one line per request with its
`state` and a `reason_code` (`pending`, `blocked`, `failed`, `ignored`) you
can parse to see which request was dropped and why.
- **PII detection screens the whole record.** The single `conversation` field
should include everything — system instructions, tool calls and results,
retrieved documents — not just what the user saw. The judge's `explanation`
is a structured `FINDINGS:` block (or `FINDINGS: none`) you can parse for
per-instance categories. Placeholders and redactions (`[REDACTED]`,
`555-01xx` numbers, `example.com` addresses, anything marked as a sample)
are deliberately not flagged, so don't build test cases out of dummy
identifiers.
## Retrieval relevance
`RetrievalRelevanceEvaluator` is source-agnostic and scores the retrieved
information *as a whole*: if any meaningful part of it materially helps address
the request, the step is `relevant`. Labels are `relevant` / `irrelevant`, the
score is **maximized** (`relevant` is `1.0`, `irrelevant` is `0.0`), and each
result carries an `explanation` from the judge.
Pass one retrieved document as `context` for per-document evaluation. To judge
the whole retrieval step, join all returned items into one `context` value.
Two field conventions matter, and getting them wrong quietly changes what you
measured:
- `input` should be the **user's request** — e.g. the trace root's
`input.value` — not a reformulated tool argument or a generated SQL query.
- `context` should contain the retrieved information at the scope you want to
judge: one document for per-document evaluation, or all returned items joined
together for holistic step evaluation.
Relevance is not correctness: outdated or later-contradicted information still
scores `relevant` if it was genuinely about the right subject. A failed
retrieval — an error, a timeout, or "no results found" — scores `irrelevant`.
```python
from phoenix.evals import LLM
from phoenix.evals.metrics import RetrievalRelevanceEvaluator
relevance_eval = RetrievalRelevanceEvaluator(llm=LLM(provider="openai", model="gpt-4o-mini"))
scores = relevance_eval.evaluate({
"input": "What is the capital of France?",
"context": "Paris is the capital and largest city of France.",
})
print(scores[0].label) # "relevant"
```
```typescript
import { createRetrievalRelevanceEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";
const evaluator = createRetrievalRelevanceEvaluator({ model: openai("gpt-4o-mini") });
const result = await evaluator.evaluate({
input: "What is the capital of France?",
context: "Paris is the capital and largest city of France.",
});
console.log(result.label); // "relevant"
```
`RetrievalRelevanceEvaluator` takes `llm` plus arbitrary `**kwargs` forwarded to
the LLM client (e.g. `temperature=0.0`), and requires a model that supports tool
calling or structured output. The TypeScript factory accepts optional `name`,
`choices`, `promptTemplate`, and `optimizationDirection` overrides on top of the
usual classification evaluator arguments.
## When to Use
| Situation | Recommendation |
| --------- | -------------- |
| Exploration | Find traces to review |
| Find outliers | Sort by scores |
| Production | Validate first (>80% human agreement) |
| Domain-specific | Build custom |
## Exploration Pattern
```python
from phoenix.evals import evaluate_dataframe
results_df = evaluate_dataframe(dataframe=traces, evaluators=[faithfulness_eval])
# Score columns contain dicts — extract numeric scores
scores = results_df["faithfulness_score"].apply(
lambda x: x.get("score", 0.0) if isinstance(x, dict) else 0.0
)
low_scores = results_df[scores < 0.5] # Review these
high_scores = results_df[scores > 0.9] # Also sample
```
## Validation Required
```python
# pip install scikit-learn (not a Phoenix dependency)
from sklearn.metrics import classification_report
print(classification_report(human_labels, evaluator_results["label"]))
# Target: >80% agreement
```
references/evaluators-rag.md
# Evaluators: RAG Systems
RAG has two distinct components requiring different evaluation approaches.
## Two-Phase Evaluation
```
RETRIEVAL GENERATION
───────── ──────────
Query → Retriever → Docs Docs + Query → LLM → Answer
│ │
IR Metrics LLM Judges / Code Checks
```
**Debug retrieval first** using IR metrics, then tackle generation quality.
## Retrieval Evaluation (IR Metrics)
Use traditional information retrieval metrics:
| Metric | What It Measures |
| ------ | ---------------- |
| Recall@k | Of all relevant docs, how many in top k? |
| Precision@k | Of k retrieved docs, how many relevant? |
| MRR | How high is first relevant doc? |
| NDCG | Quality weighted by position |
```python
# Requires query-document relevance labels
def recall_at_k(retrieved_ids, relevant_ids, k=5):
retrieved_set = set(retrieved_ids[:k])
relevant_set = set(relevant_ids)
if not relevant_set:
return 0.0
return len(retrieved_set & relevant_set) / len(relevant_set)
```
**No relevance labels?** IR metrics need them. When you don't have them, judge
the retrieval step with an LLM instead: `RetrievalRelevanceEvaluator`
(`createRetrievalRelevanceEvaluator` in TypeScript) scores the retrieved
information holistically against the request, and is source-agnostic — it works
for tool calls, MCP servers, web search, and database queries, not just vector
search. See [evaluators-pre-built](evaluators-pre-built.md).
## Creating Retrieval Test Data
Generate query-document pairs synthetically:
```python
# Reverse process: document → questions that document answers
def generate_retrieval_test(documents):
test_pairs = []
for doc in documents:
# Extract facts, generate questions
questions = llm(f"Generate 3 questions this document answers:\n{doc}")
for q in questions:
test_pairs.append({"query": q, "relevant_doc_id": doc.id})
return test_pairs
```
## Generation Evaluation
Use LLM judges for qualities code can't measure:
| Eval | Question |
| ---- | -------- |
| **Faithfulness** | Are all claims supported by retrieved context? |
| **Relevance** | Does answer address the question? |
| **Completeness** | Does answer cover key points from context? |
```python
from phoenix.evals import ClassificationEvaluator, LLM
FAITHFULNESS_TEMPLATE = """Given the context and answer, is every claim in the answer supported by the context?
<context>{{context}}</context>
<answer>{{output}}</answer>
"faithful" = ALL claims supported by context
"unfaithful" = ANY claim NOT in context
Answer (faithful/unfaithful):"""
faithfulness = ClassificationEvaluator(
name="faithfulness",
prompt_template=FAITHFULNESS_TEMPLATE,
llm=LLM(provider="openai", model="gpt-4o"),
choices={"unfaithful": 0, "faithful": 1}
)
```
## RAG Failure Taxonomy
Common failure modes to evaluate:
```yaml
retrieval_failures:
- no_relevant_docs: Query returns unrelated content
- partial_retrieval: Some relevant docs missed
- wrong_chunk: Right doc, wrong section
generation_failures:
- hallucination: Claims not in retrieved context
- ignored_context: Answer doesn't use retrieved docs
- incomplete: Missing key information from context
- wrong_synthesis: Misinterprets or miscombines sources
```
## Evaluation Order
1. **Retrieval first** - If wrong docs, generation will fail
2. **Faithfulness** - Is answer grounded in context?
3. **Answer quality** - Does answer address the question?
Fix retrieval problems before debugging generation.
references/experiments-datasets-python.md
# Experiments: Datasets in Python
Creating and managing evaluation datasets.
## Creating Datasets
`create_dataset()` upserts: if a dataset with the same name already exists it is updated in-place; re-running with identical inputs is a no-op.
```python
from phoenix.client import Client
client = Client()
# From examples
dataset = client.datasets.create_dataset(
name="qa-test-v1",
examples=[
{
"input": {"question": "What is 2+2?"},
"output": {"answer": "4"},
"metadata": {"category": "math"},
},
],
)
# With stable example IDs for targeted updates across uploads
dataset = client.datasets.create_dataset(
name="qa-test-v1",
examples=[
{
"id": "q-001", # stable ID — server updates this row, not inserts
"input": {"question": "What is 2+2?"},
"output": {"answer": "4"},
"metadata": {"category": "math"},
},
],
)
# From DataFrame
dataset = client.datasets.create_dataset(
dataframe=df,
name="qa-test-v1",
input_keys=["question"],
output_keys=["answer"],
metadata_keys=["category"],
split_key="split", # single split column (use this instead of deprecated split_keys)
example_id_key="id", # column containing stable example IDs
)
```
## From Production Traces
```python
spans_df = client.spans.get_spans_dataframe(project_identifier="my-app")
dataset = client.datasets.create_dataset(
dataframe=spans_df[["input.value", "output.value"]],
name="production-sample-v1",
input_keys=["input.value"],
output_keys=["output.value"],
)
```
## Retrieving Datasets
```python
dataset = client.datasets.get_dataset(name="qa-test-v1")
df = dataset.to_dataframe()
```
## Key Parameters
| Parameter | Description |
| --------- | ----------- |
| `input_keys` | Columns for task input |
| `output_keys` | Columns for expected output |
| `metadata_keys` | Additional context |
| `example_id_key` | Column with stable example IDs; server updates the matching row instead of inserting |
| `split_key` | Single column for split assignment (replaces deprecated `split_keys`) |
| `split_keys` | **Deprecated** — use `split_key` (singular) instead |
## Using Evaluators in Experiments
### Evaluators as experiment evaluators
Pass phoenix-evals evaluators directly to `run_experiment` as the `evaluators` argument:
```python
from functools import partial
from phoenix.client import AsyncClient
from phoenix.evals import ClassificationEvaluator, LLM, bind_evaluator
# Define an LLM evaluator
refusal = ClassificationEvaluator(
name="refusal",
prompt_template="Is this a refusal?\nQuestion: {{query}}\nResponse: {{response}}",
llm=LLM(provider="openai", model="gpt-4o"),
choices={"refusal": 0, "answer": 1},
)
# Bind to map dataset columns to evaluator params
refusal_evaluator = bind_evaluator(refusal, {"query": "input.query", "response": "output"})
# Define experiment task
async def run_rag_task(input, rag_engine):
return rag_engine.query(input["query"])
# Run experiment with the evaluator
experiment = await AsyncClient().experiments.run_experiment(
dataset=ds,
task=partial(run_rag_task, rag_engine=query_engine),
experiment_name="baseline",
evaluators=[refusal_evaluator],
concurrency=10,
)
```
### Evaluators as the task (meta evaluation)
Use an LLM evaluator as the experiment **task** to test the evaluator itself
against human annotations:
```python
from phoenix.evals import create_evaluator
# The evaluator IS the task being tested
def run_refusal_eval(input, evaluator):
result = evaluator.evaluate(input)
return result[0]
# A simple heuristic checks judge vs human agreement
@create_evaluator(name="exact_match")
def exact_match(output, expected):
return float(output["score"]) == float(expected["refusal_score"])
# Run: evaluator is the task, exact_match evaluates it
experiment = await AsyncClient().experiments.run_experiment(
dataset=annotated_dataset,
task=partial(run_refusal_eval, evaluator=refusal),
experiment_name="judge-v1",
evaluators=[exact_match],
concurrency=10,
)
```
This pattern lets you iterate on evaluator prompts until they align with human judgments.
See `tutorials/evals/evals-2/evals_2.0_rag_demo.ipynb` for a full worked example.
## Best Practices
- **Upsert by default**: Re-upload to the same name to update in-place; use `example_id_key` so the server targets specific rows instead of treating every upload as new data
- **Versioning**: Version with tags or new names (e.g., `qa-test-v2`) when you want a clean snapshot, not just incremental edits
- **Metadata**: Track source, category, difficulty
- **Balance**: Ensure diverse coverage across categories
- **Avoid `split_keys`**: Pass `split_key` (singular) — `split_keys` is deprecated and emits a `DeprecationWarning`
references/experiments-datasets-typescript.md
# Experiments: Datasets in TypeScript
Creating and managing evaluation datasets.
## Creating Datasets
`createDataset()` upserts: if a dataset with the same name already exists it is updated to match the provided examples. Re-running with identical inputs is a no-op.
```typescript
import { createClient } from "@arizeai/phoenix-client";
import { createDataset } from "@arizeai/phoenix-client/datasets";
const client = createClient();
const { datasetId } = await createDataset({
client,
name: "qa-test-v1",
examples: [
{
input: { question: "What is 2+2?" },
output: { answer: "4" },
metadata: { category: "math" },
},
],
});
// With stable example IDs for targeted updates across uploads
const { datasetId } = await createDataset({
client,
name: "qa-test-v1",
examples: [
{
id: "q-001", // stable ID — server updates this row, not inserts
input: { question: "What is 2+2?" },
output: { answer: "4" },
metadata: { category: "math" },
},
],
});
```
## Example Structure
```typescript
interface Example {
input: Record<string, unknown>; // Task input
output?: Record<string, unknown> | null; // Expected output
metadata?: Record<string, unknown> | null; // Additional context
splits?: string | string[] | null; // Split assignment ("train", ["train", "easy"], etc.)
spanId?: string | null; // OTEL span ID to link back to source trace
id?: string | null; // Stable user-provided ID; server updates matching row
}
```
## From Production Traces
```typescript
import { getSpans } from "@arizeai/phoenix-client/spans";
const { spans } = await getSpans({
project: { projectName: "my-app" },
parentId: null, // root spans only
limit: 100,
});
const examples = spans.map((span) => ({
input: { query: span.attributes?.["input.value"] },
output: { response: span.attributes?.["output.value"] },
metadata: { spanId: span.context.span_id },
}));
await createDataset({ client, name: "production-sample", examples });
```
## Retrieving Datasets
```typescript
import { getDataset, listDatasets } from "@arizeai/phoenix-client/datasets";
// `dataset` is `{ datasetId }` or `{ datasetName }` in every helper
const dataset = await getDataset({ client, dataset: { datasetName: "qa-test-v1" } });
const all = await listDatasets({ client });
```
## Managing Splits
`createDataset()` assigns splits per example via `splits`; these helpers edit splits on an existing dataset (server >= 19.20.0):
```typescript
import {
createDatasetSplit,
deleteDatasetSplit,
updateDatasetSplit,
} from "@arizeai/phoenix-client/datasets";
const dataset = { datasetName: "qa-test-v1" };
const split = await createDatasetSplit({ client, dataset, name: "test", exampleIds: ["ex-1", "ex-2"] });
await updateDatasetSplit({ client, dataset, splitId: split.id, addExampleIds: ["ex-3"], removeExampleIds: ["ex-1"] }); // idempotent
await deleteDatasetSplit({ client, dataset, splitId: split.id }); // keeps the examples
```
Split names are unique instance-wide (409 on conflict). Before server 20.16.0, `exampleIds` must be example `nodeId`s, not user-provided `id`s.
## Labeling Datasets
Dataset labels tag whole datasets (e.g. `golden`, `regression`) so they can be grouped and found later; splits partition examples inside one dataset. A label is a global resource — create it once, then apply it to any number of datasets:
```typescript
import {
addLabelToDataset,
createDatasetLabel,
listDatasetLabels,
listLabelsForDataset,
removeLabelFromDataset,
replaceLabelsForDataset,
} from "@arizeai/phoenix-client/datasets";
const label = await createDatasetLabel({ client, name: "golden", color: "#00cc88" }); // 409 if the name exists
const dataset = { datasetName: "qa-test-v1" };
await addLabelToDataset({ client, dataset, labelId: label.id }); // idempotent
const applied = await listLabelsForDataset({ client, dataset }); // DatasetLabel[]
await replaceLabelsForDataset({ client, dataset, labelIds: [] }); // [] removes every label
await removeLabelFromDataset({ client, dataset, labelId: label.id }); // keeps the label itself
const { datasetLabels, nextCursor } = await listDatasetLabels({ client, limit: 100 });
```
`DatasetLabel` is `{ id, name, description, color }`; `color` is a lowercase hex string. `getDatasetLabel`, `updateDatasetLabel` (partial; `description: null` clears it), and `deleteDatasetLabel` take a `labelId`. The per-dataset helpers (`listLabelsForDataset`, `addLabelToDataset`, `removeLabelFromDataset`, `replaceLabelsForDataset`) require Phoenix server >= 17.16.0. The Python client has no dataset-label helpers.
## Best Practices
- **Upsert by default**: Re-upload to the same name to update in-place; set a unique-per-dataset `id` on examples so the server targets specific rows instead of treating every upload as new data
- **Versioning**: Version with new names (e.g., `qa-test-v2`) when you want a clean snapshot, not just incremental edits
- **Metadata**: Track source, category, provenance
- **Type safety**: Use the `Example` type from `@arizeai/phoenix-client/types/datasets`
references/experiments-overview.md
# Experiments: Overview
Systematic testing of AI systems with datasets, tasks, and evaluators.
## Structure
```
DATASET → Examples: {input, expected_output, metadata}
TASK → function(input) → output
EVALUATORS → (input, output, expected) → score
EXPERIMENT → Run task on all examples, score results
```
## Basic Usage
```python
from phoenix.client import Client
client = Client()
experiment = client.experiments.run_experiment(
dataset=my_dataset,
task=my_task,
evaluators=[accuracy, faithfulness],
experiment_name="improved-retrieval-v2",
)
print(experiment.aggregate_scores)
# {'accuracy': 0.85, 'faithfulness': 0.92}
```
## Workflow
1. **Create dataset** - From traces, synthetic data, or manual curation
2. **Define task** - The function to test (your LLM pipeline)
3. **Select evaluators** - Code and/or LLM-based
4. **Run experiment** - Execute and score
5. **Analyze & iterate** - Review, modify task, re-run
## Dry Runs
Test setup before full execution:
```python
experiment = client.experiments.run_experiment(
dataset=dataset,
task=task,
evaluators=evaluators,
dry_run=3,
) # Just 3 examples
```
## Async Usage
Use `AsyncClient` when your task or evaluators make network calls and you want higher throughput:
```python
from phoenix.client import AsyncClient
client = AsyncClient()
experiment = await client.experiments.run_experiment(
dataset=my_dataset,
task=my_async_task,
evaluators=[accuracy, faithfulness],
experiment_name="improved-retrieval-v2",
)
```
## Best Practices
- **Name meaningfully**: `"improved-retrieval-v2-2024-01-15"` not `"test"`
- **Version datasets**: Don't modify existing
- **Multiple evaluators**: Combine perspectives
references/experiments-running-python.md
# Experiments: Running Experiments in Python
Execute experiments with `run_experiment`.
## Basic Usage
```python
from phoenix.client import Client
from phoenix.client.experiments import run_experiment
from openai import OpenAI
client = Client()
openai_client = OpenAI()
# Replace these sample rows with your local QA examples.
qa_examples = [
{"question": "What is 2+2?", "expected": "4"},
{"question": "What is the capital of France?", "expected": "Paris"},
]
dataset = client.datasets.create_dataset(
name="qa-test-v1",
examples=[
{"input": {"question": row["question"]}, "output": {"answer": row["expected"]}}
for row in qa_examples
],
)
def my_task(example):
response = openai_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": example.input["question"]}],
)
return response.choices[0].message.content or ""
def exact_match(output, expected):
return 1.0 if output.strip().lower() == expected["answer"].strip().lower() else 0.0
experiment = run_experiment(
dataset=dataset,
task=my_task,
evaluators=[exact_match],
experiment_name="qa-experiment-v1",
repetitions=3, # Average sampling noise from the LLM task on a small dataset
)
```
`expected` is the dataset example's entire `output` dictionary, not a flattened answer. Here, local `row["expected"]` is stored as `output["answer"]`, so the evaluator reads `expected["answer"]`. If you store it under `output["expected"]` instead, read `expected["expected"]`. When adapting an existing evaluator that takes two strings, wrap it as `def qa_evaluator(output, expected): return existing_evaluator(output, expected["answer"])` and pass the wrapper to `run_experiment`.
## Task Functions
```python
# Basic task
def task(example):
return call_llm(example.input["question"])
# With context (RAG)
def rag_task(example):
return call_llm(f"Context: {example.input['context']}\nQ: {example.input['question']}")
```
## Evaluator Parameters
| Parameter | Access |
| --------- | ------ |
| `output` | Task output |
| `expected` | Entire example output dictionary; select the key used when creating the dataset |
| `input` | Example input |
| `metadata` | Example metadata |
## Options
```python
experiment = run_experiment(
dataset=dataset,
task=my_task,
evaluators=evaluators,
experiment_name="my-experiment",
dry_run=3, # Test with 3 examples
repetitions=3, # Run each example 3 times
)
```
## Results
```python
print(experiment["experiment_id"])
for run in experiment["task_runs"]:
print(run["output"])
```
`run_experiment` prints a summary by default. View aggregate scores and compare named experiments in the Phoenix UI. The returned `RanExperiment` is a dictionary with `task_runs` and `evaluation_runs`; it has no `aggregate_scores` or `runs` attributes.
## Stability
Single-run scores are noisy when either the task or the evaluator is non-deterministic — an LLM call, tool use, streaming output, an LLM-as-judge. On a small dataset, that per-run noise can swamp the signal from a prompt change.
Averaging over repetitions lets the score you report reflect the prompt rather than the sampling noise:
```python
run_experiment(
# ...
repetitions=3,
)
```
Things to consider:
- Reach for repetitions when the task or the evaluator is an LLM call and the dataset is small.
- Prefer repetitions when per-example cost is low and you mostly want to settle the score; prefer growing the dataset when you also need to cover more behaviors.
- Skip repetitions when both the task and the evaluator are deterministic (e.g. string comparison against a ground truth) — a single run is the answer.
Consider adding stability when:
- Repeat runs of the same experiment drift in ways that feel larger than the differences you're trying to measure.
- A prompt change flips example labels in ways that don't track with how the outputs actually changed.
- The judge's reasoning on the same output reads differently from one run to the next.
Repetitions are also what `repetitions=1` (default) silently relies on — don't trust a tuning decision based on a single 10-example run.
## Add Evaluations Later
```python
from phoenix.client.experiments import evaluate_experiment
evaluate_experiment(experiment=experiment, evaluators=[new_evaluator])
```
`evaluate_experiment` scores every run again and overwrites evaluations of the same name. To score only the runs still missing an evaluation, use `resume_evaluation` (below).
## Resume an Interrupted Experiment
When a run crashed, hit a provider outage, or was interrupted, finish it instead of starting over:
```python
client = Client()
# Re-runs the task only for (example, repetition) pairs with no successful run,
# then applies evaluators to runs missing that evaluation.
client.experiments.resume_experiment(
experiment_id="experiment-id",
task=my_task,
evaluators=[exact_match],
)
# All tasks finished, but some evaluations are missing or failed —
# or you want to add a new evaluator to a finished experiment.
client.experiments.resume_evaluation(
experiment_id="experiment-id",
evaluators={"exact_match": exact_match},
)
```
- `resume_evaluation` matches evaluators to existing evaluations by name, so each evaluator should return a single result whose name matches its key. Evaluators that return a list of results re-run on every call.
- When `resume_experiment` replaces an errored run, the run keeps its ID; LLM and code evaluations on it are discarded and re-scored, human annotations are kept. Requires Phoenix server >= 20.21.3.
- Both also exist on `AsyncClient` (with a `concurrency` parameter).
references/experiments-running-typescript.md
# Experiments: Running Experiments in TypeScript
Execute experiments with `runExperiment`.
## Basic Usage
```typescript
import { createClient } from "@arizeai/phoenix-client";
import {
runExperiment,
asExperimentEvaluator,
} from "@arizeai/phoenix-client/experiments";
const client = createClient();
const task = async (example: { input: Record<string, unknown> }) => {
return await callLLM(example.input.question as string);
};
const exactMatch = asExperimentEvaluator({
name: "exact_match",
kind: "CODE",
evaluate: async ({ output, expected }) => ({
score: output === expected?.answer ? 1.0 : 0.0,
label: output === expected?.answer ? "match" : "no_match",
}),
});
const experiment = await runExperiment({
client,
experimentName: "qa-experiment-v1",
dataset: { datasetId: "your-dataset-id" },
task,
evaluators: [exactMatch],
});
```
## Task Functions
```typescript
// Basic task
const task = async (example) => await callLLM(example.input.question as string);
// With context (RAG)
const ragTask = async (example) => {
const prompt = `Context: ${example.input.context}\nQ: ${example.input.question}`;
return await callLLM(prompt);
};
```
## Tracing AI SDK Calls Inside Tasks
`@arizeai/phoenix-client` 7.x requires AI SDK v7, which **does not emit
OpenTelemetry spans through the global tracer provider on its own**. If your task
calls `generateText`/`streamText`, pass the `@ai-sdk/otel` integration per call and
**construct it inside the task**:
```typescript
import { openai } from "@ai-sdk/openai";
import { OpenTelemetry } from "@ai-sdk/otel";
import { runExperiment } from "@arizeai/phoenix-client/experiments";
import type { ExperimentTask } from "@arizeai/phoenix-client/types/experiments";
import { generateText } from "ai";
const model = openai("gpt-4o-mini");
const task: ExperimentTask = async (example) => {
const { text } = await generateText({
model,
prompt: String(example.input.question),
telemetry: { integrations: [new OpenTelemetry()] },
});
return text;
};
```
**Why inside the task:** the integration binds to whichever tracer provider is
active when it is *constructed*, and `runExperiment` mounts the experiment's
provider only while tasks run. An integration registered once at startup via
`registerTelemetry` binds before that provider exists, and the task spans are lost.
This is the opposite of ordinary application tracing, where startup registration is
correct.
Evaluators from `@arizeai/phoenix-evals` are traced automatically and need no
telemetry setup.
## Evaluator Parameters
```typescript
interface EvaluatorParams {
input: Record<string, unknown>;
output: unknown;
expected: Record<string, unknown>;
metadata: Record<string, unknown>;
}
```
## Options
```typescript
const experiment = await runExperiment({
client,
experimentName: "my-experiment",
dataset: { datasetName: "qa-test-v1" },
task,
evaluators,
repetitions: 3, // Run each example 3 times
maxConcurrency: 5, // Limit concurrent executions
});
```
## Stability
Single-run scores are noisy when either the task or the evaluator is non-deterministic — an LLM call, tool use, streaming output, an LLM-as-judge. On a small dataset, that per-run noise can swamp the signal from a prompt change.
Averaging over repetitions lets the score you report reflect the prompt rather than the sampling noise:
```typescript
await runExperiment({
// ...
repetitions: 3,
});
```
Things to consider:
- Reach for repetitions when the task or the evaluator is an LLM call and the dataset is small.
- Prefer repetitions when per-example cost is low and you mostly want to settle the score; prefer growing the dataset when you also need to cover more behaviors.
- Skip repetitions when both the task and the evaluator are deterministic (e.g. string comparison against a ground truth) — a single run is the answer.
Consider adding stability when:
- Repeat runs of the same experiment drift in ways that feel larger than the differences you're trying to measure.
- A prompt change flips example labels in ways that don't track with how the outputs actually changed.
- The judge's reasoning on the same output reads differently from one run to the next.
Repetitions are also what `repetitions: 1` (default) silently relies on — don't trust a tuning decision based on a single 10-example run.
## Add Evaluations Later
```typescript
import { evaluateExperiment } from "@arizeai/phoenix-client/experiments";
await evaluateExperiment({ client, experiment, evaluators: [newEvaluator] });
```
`evaluateExperiment` scores every run again and overwrites evaluations of the same name. To score only the runs still missing an evaluation, use `resumeEvaluation` (below).
## Resume an Interrupted Experiment
When a run crashed, hit a provider outage, or was interrupted, finish it instead of starting over:
```typescript
import {
resumeExperiment,
resumeEvaluation,
} from "@arizeai/phoenix-client/experiments";
// Re-runs the task only for (example, repetition) pairs with no successful run,
// then applies evaluators to runs missing that evaluation.
await resumeExperiment({
client,
experimentId: "experiment-id",
task,
evaluators: [exactMatch],
});
// All tasks finished, but some evaluations are missing or failed —
// or you want to add a new evaluator to a finished experiment.
await resumeEvaluation({
client,
experimentId: "experiment-id",
evaluators: [exactMatch],
});
```
- `resumeEvaluation` matches evaluators to existing evaluations by name, so each evaluator should return a single result with that name. Evaluators that return a list of results re-run on every call.
- When `resumeExperiment` replaces an errored run, the run keeps its ID; LLM and code evaluations on it are discarded and re-scored, human annotations are kept. Requires Phoenix server >= 20.21.3.
references/experiments-synthetic-python.md
# Experiments: Generating Synthetic Test Data
Creating diverse, targeted test data for evaluation.
## Dimension-Based Approach
Define axes of variation, then generate combinations:
```python
dimensions = {
"issue_type": ["billing", "technical", "shipping"],
"customer_mood": ["frustrated", "neutral", "happy"],
"complexity": ["simple", "moderate", "complex"],
}
```
## Two-Step Generation
1. **Generate tuples** (combinations of dimension values)
2. **Convert to natural queries** (separate LLM call per tuple)
```python
# Step 1: Create tuples
tuples = [
("billing", "frustrated", "complex"),
("shipping", "neutral", "simple"),
]
# Step 2: Convert to natural query
def tuple_to_query(t):
prompt = f"""Generate a realistic customer message:
Issue: {t[0]}, Mood: {t[1]}, Complexity: {t[2]}
Write naturally, include typos if appropriate. Don't be formulaic."""
return llm(prompt)
```
## Target Failure Modes
Dimensions should target known failures from error analysis:
```python
# From error analysis findings
dimensions = {
"timezone": ["EST", "PST", "UTC", "ambiguous"], # Known failure
"date_format": ["ISO", "US", "EU", "relative"], # Known failure
}
```
## Quality Control
- **Validate**: Check for placeholder text, minimum length
- **Deduplicate**: Remove near-duplicate queries using embeddings
- **Balance**: Ensure coverage across dimension values
## When to Use
| Use Synthetic | Use Real Data |
| ------------- | ------------- |
| Limited production data | Sufficient traces |
| Testing edge cases | Validating actual behavior |
| Pre-launch evals | Post-launch monitoring |
## Sample Sizes
| Purpose | Size |
| ------- | ---- |
| Initial exploration | 50-100 |
| Comprehensive eval | 100-500 |
| Per-dimension | 10-20 per combination |
references/experiments-synthetic-typescript.md
# Experiments: Generating Synthetic Test Data (TypeScript)
Creating diverse, targeted test data for evaluation.
## Dimension-Based Approach
Define axes of variation, then generate combinations:
```typescript
const dimensions = {
issueType: ["billing", "technical", "shipping"],
customerMood: ["frustrated", "neutral", "happy"],
complexity: ["simple", "moderate", "complex"],
};
```
## Two-Step Generation
1. **Generate tuples** (combinations of dimension values)
2. **Convert to natural queries** (separate LLM call per tuple)
```typescript
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
// Step 1: Create tuples
type Tuple = [string, string, string];
const tuples: Tuple[] = [
["billing", "frustrated", "complex"],
["shipping", "neutral", "simple"],
];
// Step 2: Convert to natural query
async function tupleToQuery(t: Tuple): Promise<string> {
const { text } = await generateText({
model: openai("gpt-4o"),
prompt: `Generate a realistic customer message:
Issue: ${t[0]}, Mood: ${t[1]}, Complexity: ${t[2]}
Write naturally, include typos if appropriate. Don't be formulaic.`,
});
return text;
}
```
## Target Failure Modes
Dimensions should target known failures from error analysis:
```typescript
// From error analysis findings
const dimensions = {
timezone: ["EST", "PST", "UTC", "ambiguous"], // Known failure
dateFormat: ["ISO", "US", "EU", "relative"], // Known failure
};
```
## Quality Control
- **Validate**: Check for placeholder text, minimum length
- **Deduplicate**: Remove near-duplicate queries using embeddings
- **Balance**: Ensure coverage across dimension values
```typescript
function validateQuery(query: string): boolean {
const minLength = 20;
const hasPlaceholder = /\[.*?\]|<.*?>/.test(query);
return query.length >= minLength && !hasPlaceholder;
}
```
## When to Use
| Use Synthetic | Use Real Data |
| ------------- | ------------- |
| Limited production data | Sufficient traces |
| Testing edge cases | Validating actual behavior |
| Pre-launch evals | Post-launch monitoring |
## Sample Sizes
| Purpose | Size |
| ------- | ---- |
| Initial exploration | 50-100 |
| Comprehensive eval | 100-500 |
| Per-dimension | 10-20 per combination |
references/filter-expressions.md
# Filter expressions
A filter condition is a Python boolean expression that Phoenix compiles to SQL. There are three
filter languages, one each for spans, traces, and sessions, and the argument name picks the
language. The vocabularies do not mix.
| Language | Argument | Keeps | Accepted on |
| --- | --- | --- | --- |
| Span filter | `filterCondition` | individual spans | `Project.spans`, `Trace.spans`, project aggregates (`recordCount`, `tokenCountTotal`, `costSummary`, `latencyMsQuantile`, ...), `SpanQuery().where(...)` in the Python client, the UI spans filter bar |
| Trace filter | `traceFilterCondition` | every span of a matching trace | `Project.spans`, the UI traces filter bar, the `filter` query param on `GET /v1/projects/{id}/traces` and the `filter` argument of `client.traces.get_traces(...)` (Python) and `getTraces({...})` (TypeScript) |
| Session filter | `sessionFilterCondition` | sessions | `Project.sessions`, project aggregates, the UI sessions filter bar, the `filter` query param on `GET /v1/projects/{id}/sessions` and the `filter` argument of `client.sessions.list(...)` (Python) and `listSessions({...})` (TypeScript) |
`filterCondition` and `traceFilterCondition` compose on `Project.spans`: matching spans inside
matching traces. `filterCondition` and `sessionFilterCondition` are mutually exclusive on the
aggregates that accept both.
## Checking a condition
| Need | GraphQL field on `Project` |
| --- | --- |
| Validate before running | `validateSpanFilterCondition(condition:)`, `validateTraceFilterCondition(condition:)`, `validateSessionFilterCondition(condition:)` → `{ isValid errorMessage }` |
| List bindable names | `traceFilterVocabulary`, `sessionFilterVocabulary` → `{ name type category description iterableName }` |
| Span filter names | no vocabulary field; the table under [Span filter](#span-filter) is exhaustive |
A span filter that returns no rows may be misspelled rather than unmatched. Check the spelling
against the vocabulary before concluding there is no data.
## Syntax shared by all three languages
### Operators
| Category | Accepted | Rejected |
| --- | --- | --- |
| Comparison | `==` `!=` `<` `<=` `>` `>=`; chained: `500 < latency_ms <= 2000` | `=` |
| Missing values | `is None`, `is not None` | `is null`, `null`, `is` with any other value |
| Membership | `x in [...]` exact; `'text' in field` case-insensitive substring; `not in` | `None` inside a list; a literal on the left (`1 in [1, 2]`); `like` |
| Logic | `and`, `or`, `not`, parentheses | `&&`, `\|\|`, `&`, `\|`, `!` |
| Arithmetic | `+` `-` `*` `/` `%` | `**`, `//`, bitwise operators |
| Casts | `float(x)`, `int(x)`, `str(x)` | `bool(x)`, any other function |
| Comprehensions | `any`, `all`, `len`, `sum`, `max`, `min` over a declared collection: `any(d.cost > 0 for d in cost_details)`; `len` takes a list comprehension | method calls (`name.startswith(...)`), `len(name)` |
### Literals
| Kind | Write | Not |
| --- | --- | --- |
| String | `'LLM'` or `"LLM"` | unquoted: `LLM` |
| Number | `100`, `0.5` | quoted: `'100'` |
| Boolean | `True`, `False` | `true`, `false` |
| Missing | `None` | `null`, `nil` |
| Datetime | ISO 8601 with an offset: `'2026-09-01T00:00:00Z'`, `'2026-09-01T00:00:00+00:00'` | without an offset |
| List | `['LLM', 'TOOL']`, same type throughout | mixed types; `None` elements |
### Rules that differ from Python
- A missing value fails every comparison, including `!=`. A span without `metadata['tier']`
matches neither `metadata['tier'] == 'premium'` nor `metadata['tier'] != 'premium'`. Spell the
missing case out: `metadata['tier'] != 'premium' or metadata['tier'] is None`.
- `is None` is true for an absent key and for a stored JSON `null`.
- Every operand of `and`, `or`, and `not` must itself be a condition. `name == 'x' and
metadata['flag']` is rejected; write `metadata['flag'] == True`.
- The whole expression must be a condition. A bare `True` or a bare field is rejected.
- `'text' in field` ignores case. `==` and list membership are exact.
- Annotation accessors expose `.label` (string), `.score` (number), `.explanation` (string), and
`.identifier` (string). The bare accessor is an existence check: `annotations['quality']`.
- `.identifier` is never missing: it is `''` on an annotation written without one, so test for
that with `== ''` rather than `is None`.
## Span filter
### Root spans
There is no `traces` connection and no root-span argument. Root-span scoping is a clause in
`filterCondition`, and it composes with everything else. The UI's traces table is
`spans(filterCondition: "parent_span is None", traceFilterCondition: ...)`.
| Clause | Keeps | Use when |
| --- | --- | --- |
| `parent_id is None` | spans with no parent id | the default |
| `parent_span is None` | those, plus orphans whose parent span was never received | you want every top-level span, including ones whose parent was dropped |
A root span is usually one per trace. Fragmented traces can have several.
### Vocabulary
This table is exhaustive. Every identifier not in it is read as an attribute path.
| Name | Type | Notes |
| --- | --- | --- |
| `span_id`, `trace_id`, `parent_id` | string | OpenTelemetry hex ids |
| `name` | string | span name |
| `span_kind` | enum | `'CHAIN'`, `'LLM'`, `'RETRIEVER'`, `'EMBEDDING'`, `'TOOL'`, `'AGENT'`, `'RERANKER'`, `'GUARDRAIL'`, `'EVALUATOR'`, `'PROMPT'`, `'DECISION'`, `'UNKNOWN'`; literals are uppercased for you |
| `status_code` | enum | `'OK'`, `'ERROR'`, `'UNSET'`; literals are uppercased for you |
| `status_message` | string | error text |
| `latency_ms` | number | |
| `start_time`, `end_time` | datetime | |
| `cumulative_llm_token_count_prompt`, `cumulative_llm_token_count_completion`, `cumulative_llm_token_count_total` | number | this span plus its descendants |
| `llm.token_count.prompt`, `llm.token_count.completion`, `llm.token_count.total` | number | this span alone |
| `total_cost`, `prompt_cost`, `completion_cost` | number | `0` when the span has no cost row |
| `cost_details` | collection | iterable only; elements have `token_type` (string), `is_prompt` (boolean), `cost`, `tokens`, `cost_per_token` (number) |
| `parent_span` | reserved | only `is None` / `is not None`; `parent_span.name` is rejected |
| `annotations['name']` | annotation | on the span itself; `evals['name']` is a legacy alias |
| `trace_annotations['name']` | annotation | on the span's containing trace |
| `attributes[...]`, `metadata[...]`, any other dotted name | attribute | JSON attribute path; type unknown until read |
Legacy spellings still accepted: `context.span_id`, `context.trace_id`,
`cumulative_token_count.prompt`, `cumulative_token_count.completion`,
`cumulative_token_count.total`.
### Attributes
| Write | Reads |
| --- | --- |
| `llm.model_name`, `attributes['llm.model_name']`, `attributes['llm']['model_name']` | the same attribute |
| `metadata['topic']` | `attributes['metadata']['topic']` |
| `input.value`, `output.value` | the span's input and output text |
| `input.mime_type`, `output.mime_type` | the payload type |
| `attributes['tags'][0]` | the first element of a JSON array |
| `user.id` | `attributes['user']['id']` |
An attribute's type is unknown until the row is read. Compared against a number it is read as a
number, against `True` or `False` as a boolean, otherwise as text. Rows whose value cannot be
converted drop out instead of erroring. `float(x)`, `int(x)`, and `str(x)` force a read as that
type.
### Annotations
The accessor picks the level. The wrong level compiles and matches nothing.
| Accessor | Matches annotations on |
| --- | --- |
| `annotations['name']` | the span itself |
| `evals['name']` | the span itself (legacy alias) |
| `trace_annotations['name']` | the span's containing trace |
`trace_annotations[...]` matches every span of an annotated trace. Add `parent_id is None` to
get one row per trace. Discover names with `Project.spanAnnotationNames` and
`Project.traceAnnotationNames`.
One name can cover several rows on the same span — a reviewer per pass, a run per coding
session — because annotations are keyed by `(name, span, identifier)`. `.identifier` picks out
one of them: `annotations['note'].identifier == 'coding-run:1'` reads only the row that run
wrote, and combining it with `.label` or `.score` constrains that same row.
### Spellings that compile and match nothing
Unknown names fall back to attribute paths, so these are not errors.
| Wrote | Phoenix reads | Write instead |
| --- | --- | --- |
| `span_kind == LLM` | attribute named `LLM` | `span_kind == 'LLM'` |
| `parent_id == null` | attribute named `null` | `parent_id is None` |
| `error_count > 0`, `num_spans > 3`, `token_count_total > 0` | attributes; these are trace filter names | `traceFilterCondition`, or `status_code == 'ERROR'` for the span |
| `'refund' in input`, `output is None` | the `input` / `output` attribute objects | `'refund' in input.value`, `output.value is None` |
| `annotations.q.label == 'x'` | attribute path `annotations.q.label` | `annotations['q'].label == 'x'` |
| `attributes.llm.model_name == 'x'` | attribute named `attributes` | `llm.model_name == 'x'` |
| `is_root == True`, `user_id == 'u1'` | attributes | `parent_id is None`, `user.id == 'u1'` |
### Examples
```python span-filter
parent_id is None
parent_span is None
parent_id is None and status_code == "ERROR"
parent_id is None and latency_ms > 5000
span_kind == "LLM" and "gpt-4o" in llm.model_name
span_kind in ["LLM", "RETRIEVER"]
span_kind == "TOOL" and tool.name == "search"
status_code == "ERROR" and "timeout" in status_message
"refund" in input.value
"refund" in input.value or "refund" in output.value
output.value is None
metadata["topic"] == "billing"
metadata["tier"] != "premium" or metadata["tier"] is None
user.id == "u1"
float(metadata["retry_count"]) > 1
"true" in str(metadata["flag"])
attributes["tags"][0] == "urgent"
start_time > "2026-09-01T00:00:00Z" and end_time < "2026-09-02T00:00:00Z"
500 < latency_ms <= 2000
cumulative_llm_token_count_total > 10000
llm.token_count.total > 4000
total_cost > 0.01
any(d.token_type == "input" and d.tokens > 1000 for d in cost_details)
sum(d.cost for d in cost_details) > 0.01
annotations["correctness"].label == "incorrect"
annotations["hallucination"].score > 0.5
annotations["correctness"].label is None
annotations["correctness"]
annotations["note"].identifier == "coding-run:1"
annotations["note"].identifier == "coding-run:1" and annotations["note"].label == "bad"
evals["correctness"].label == "incorrect"
trace_annotations["quality"].label == "poor"
parent_id is None and trace_annotations["quality"].score < 0.5
not span_kind == "LLM"
span_kind == "LLM" and (latency_ms > 5000 or status_code == "ERROR")
```
## Trace filter
Matches whole traces. Names are strict: an unknown name is rejected with a "did you mean"
suggestion. Span filter names (`span_kind`, `status_code`, `parent_id`, `input.value`) are
unknown here. Ask span-level questions with a comprehension over `spans`.
### Vocabulary
| Name | Type | Notes |
| --- | --- | --- |
| `trace_id` | string | |
| `start_time`, `end_time` | datetime | |
| `latency_ms` | number | |
| `num_spans`, `error_count`, `tool_span_count`, `llm_span_count` | number | rollups; `0` when empty, never null |
| `token_count_prompt`, `token_count_completion`, `token_count_total` | number | rollups |
| `prompt_cost`, `completion_cost`, `total_cost` | number | rollups |
| `input`, `output` | string | the root span's I/O text: `'refund' in input`, `output is None` |
| `user.id` | string | the root span's user id; the only dotted name |
| `metadata['k']`, `attributes['k']` | attribute | read from the root span; `attributes["input.value"]` for any other root attribute |
| `trace_annotations['name']` | annotation | `annotations[...]` is rejected here |
| `spans` | collection | element fields: `name`, `parent_id`, `span_kind`, `status_code`, `start_time`, `end_time`, `latency_ms`, `llm_token_count_prompt`, `llm_token_count_completion`, `llm_token_count_total`, `cumulative_error_count`, `cumulative_llm_token_count_prompt`, `cumulative_llm_token_count_completion`, `cumulative_llm_token_count_total`; relations `parent_span` (traversable), `children`, `siblings`, `annotations`, `cost_details` |
| `trace_annotations`, `span_annotations` | collection | elements have `name`, `label`, `score`, `identifier` |
| `span_cost_details` | collection | elements have `token_type`, `is_prompt`, `cost`, `tokens`, `cost_per_token` |
Root-span reads (`input`, `output`, `user.id`, `metadata`, `attributes`) bind to the
representative root span the traces table displays. Inside a comprehension, reference only the
loop variable's fields.
### Examples
```python trace-filter
error_count > 0 and latency_ms > 1000
num_spans > 10 and total_cost > 0.25
"refund" in input
output is None
user.id == "u1"
metadata["topic"] == "support"
attributes["llm.model_name"] == "gpt-4o"
trace_annotations["quality"].score < 0.5
trace_annotations["quality"].identifier == "review-2"
start_time >= "2026-07-01T00:00:00Z"
any(span.status_code == "ERROR" for span in spans)
any(span.span_kind == "LLM" and span.latency_ms > 5000 for span in spans)
any(span.parent_span is None and span.status_code == "ERROR" for span in spans)
any(span.parent_span.span_kind == "LLM" and span.span_kind == "TOOL" for span in spans)
any(annotation.label == "hallucinated" for annotation in span_annotations)
len([span for span in spans if span.span_kind == "TOOL"]) > 3
sum(detail.cost for detail in span_cost_details if detail.is_prompt) > 0.10
```
## Session filter
Matches sessions. A session is one conversation, and each trace in it is usually one turn.
Names are strict, as in the trace filter.
### Vocabulary
| Name | Type | Notes |
| --- | --- | --- |
| `session_id` | string | |
| `start_time`, `end_time` | datetime | |
| `duration_ms` | number | |
| `num_traces`, `num_traces_with_error`, `tool_span_count`, `llm_span_count` | number | rollups; `0` when empty |
| `token_count_prompt`, `token_count_completion`, `token_count_total` | number | rollups |
| `prompt_cost`, `completion_cost`, `total_cost` | number | rollups |
| `first_input`, `last_output` | string | the opening input and final output; `==`, `in`, `is None` |
| `any_input`, `any_output` | containment only | `'refund' in any_input`; never `any_input == ...` |
| `user.id`, `metadata['k']`, `attributes['k']` | attribute | read from the session's root spans |
| `session_annotations['name']` | annotation | |
| `traces` | collection | elements have `start_time`, `end_time`, `latency_ms`, and their own `spans` |
| `spans`, `session_annotations`, `span_annotations`, `span_cost_details` | collection | as in the trace filter |
### Examples
```python session-filter
num_traces > 5 and total_cost > 0.50
num_traces_with_error / num_traces > 0.2
duration_ms > 60000
"refund" in any_input
last_output is not None
session_annotations["Quality"].score <= 0.5
session_annotations["Quality"].identifier == "review-2"
any(span.status_code == "ERROR" for span in spans)
all(trace.latency_ms < 30000 for trace in traces)
any(len([span for span in trace.spans if span.span_kind == "TOOL"]) > 5 for trace in traces)
```
references/fundamentals-anti-patterns.md
# Anti-Patterns
Common mistakes and fixes.
| Anti-Pattern | Problem | Fix |
| ------------ | ------- | --- |
| Generic metrics | Pre-built scores don't match your failures | Build from error analysis |
| Vibe-based | No quantification | Measure with experiments |
| Ignoring humans | Uncalibrated LLM judges | Validate >80% accuracy, >70% TPR/TNR |
| Premature automation | Evaluators for imagined problems | Let observed failures drive |
| Saturation blindness | 100% pass = no signal | Keep capability evals at 50-80% |
| Similarity metrics | BERTScore/ROUGE for generation | Use for retrieval only |
| Model switching | Hoping a model works better | Error analysis first |
| Single-run scoring | LLM judges and non-deterministic tasks add per-run noise that can drown the signal from a prompt change on a small dataset | Set `repetitions` on `runExperiment` (or grow the dataset) when the task or judge is an LLM call |
## Quantify Changes
```python
from phoenix.client import Client
client = Client()
baseline = client.experiments.run_experiment(dataset=dataset, task=old_prompt, evaluators=evaluators)
improved = client.experiments.run_experiment(dataset=dataset, task=new_prompt, evaluators=evaluators)
print(f"Improvement: {improved.pass_rate - baseline.pass_rate:+.1%}")
```
## Don't Use Similarity for Generation
```python
# BAD
score = bertscore(output, reference)
# GOOD
correct_facts = check_facts_against_source(output, context)
```
## Error Analysis Before Model Change
```python
# BAD
for model in models:
results = test(model)
# GOOD
failures = analyze_errors(results)
# Then decide if model change is warranted
```
references/fundamentals-model-selection.md
# Model Selection
Error analysis first, model changes last.
## Decision Tree
```
Performance Issue?
│
▼
Error analysis suggests model problem?
NO → Fix prompts, retrieval, tools
YES → Is it a capability gap?
YES → Consider model change
NO → Fix the actual problem
```
## Judge Model Selection
| Principle | Action |
| --------- | ------ |
| Start capable | Use gpt-4o first |
| Optimize later | Test cheaper after criteria stable |
| Same model OK | Judge does different task |
```python
# Start with capable model
judge = ClassificationEvaluator(
llm=LLM(provider="openai", model="gpt-4o"),
...
)
# After validation, test cheaper
judge_cheap = ClassificationEvaluator(
llm=LLM(provider="openai", model="gpt-4o-mini"),
...
)
# Compare TPR/TNR on same test set
```
## Don't Model Shop
```python
from phoenix.client import Client
client = Client()
# BAD
for model in ["gpt-4o", "claude-3", "gemini-pro"]:
results = client.experiments.run_experiment(
dataset=dataset,
task=lambda input, _model=model: task(input, model=_model),
evaluators=evaluators,
)
# GOOD
failures = analyze_errors(results)
# "Ignores context" → Fix prompt
# "Can't do math" → Maybe try better model
```
## When Model Change Is Warranted
- Failures persist after prompt optimization
- Capability gaps (reasoning, math, code)
- Error analysis confirms model limitation
references/fundamentals.md
# Fundamentals
Application-specific tests for AI systems. Code first, LLM for nuance, human for truth.
## Evaluator Types
| Type | Speed | Cost | Use Case |
| ---- | ----- | ---- | -------- |
| **Code** | Fast | Cheap | Regex, JSON, format, exact match |
| **LLM** | Medium | Medium | Subjective quality, complex criteria |
| **Human** | Slow | Expensive | Ground truth, calibration |
**Decision:** Code first → LLM only when code can't capture criteria → Human for calibration.
## Score Structure
| Property | Required | Description |
| -------- | -------- | ----------- |
| `name` | Yes | Evaluator name |
| `kind` | Yes | `"code"`, `"llm"`, `"human"` |
| `score` | No* | 0-1 numeric |
| `label` | No* | `"pass"`, `"fail"` |
| `explanation` | No | Rationale |
*One of `score` or `label` required.
## Binary > Likert
Use pass/fail, not 1-5 scales. Clearer criteria, easier calibration.
```python
# Multiple binary checks instead of one Likert scale
evaluators = [
AnswersQuestion(), # Yes/No
UsesContext(), # Yes/No
NoHallucination(), # Yes/No
]
```
## Quick Patterns
### Code Evaluator
```python
from phoenix.evals import create_evaluator
@create_evaluator(name="has_citation", kind="code")
def has_citation(output: str) -> bool:
return bool(re.search(r'\[\d+\]', output))
```
### LLM Evaluator
```python
from phoenix.evals import ClassificationEvaluator, LLM
evaluator = ClassificationEvaluator(
name="helpfulness",
prompt_template="...",
llm=LLM(provider="openai", model="gpt-4o"),
choices={"not_helpful": 0, "helpful": 1}
)
```
### Run Experiment
```python
from phoenix.client.experiments import run_experiment
experiment = run_experiment(
dataset=dataset,
task=my_task,
evaluators=[evaluator1, evaluator2],
)
print(experiment.aggregate_scores)
```
references/integrations-pytest.md
# Integrations: pytest (Python)
Write evals as ordinary `pytest` tests that record to Phoenix as experiments and gate CI. Each marked test file → a dataset; each `parametrize` case → an example; each run → an experiment; the assert outcome → a `pass` annotation.
Requires `arize-phoenix-client>=2.10.0` (introduced `@pytest.mark.phoenix`).
## When to use vs `run_experiment`
Use the pytest plugin when each case needs different logic, you want a hard `assert` gate, or you already use pytest (keep fixtures, `parametrize`, `-k`, `pytest-xdist`, `pytest-asyncio`). Prefer `run_experiment` for large, homogeneous datasets scored the same way.
## Install
```bash
pip install "arize-phoenix-client[pytest]>=2.10.0" pytest
# add the evals extra if evaluators use arize-phoenix-evals:
pip install "arize-phoenix-client[pytest,evals]>=2.10.0" pytest
```
pytest discovers the plugin via its entry point — no `conftest.py` needed.
## Minimal suite
```python
import pytest
from phoenix.client.pytest import log_output
@pytest.mark.phoenix(dataset="qa-suite")
@pytest.mark.parametrize(
"question,expected",
[("Capital of France?", "Paris"), ("12 × 8?", "96")],
ids=["geography", "arithmetic"], # stable ids map reruns to the same example
)
def test_answers(question, expected):
result = my_app(question)
log_output(result) # pytest warns on non-None return; log instead
assert result == expected # pass=True / pass=False annotation
```
Stable `parametrize` `ids` give each case a fixed identity so runs accumulate as experiments over a fixed example set instead of duplicating.
## The `@pytest.mark.phoenix` marker
| Argument | Description |
|---|---|
| `dataset` | Dataset/experiment name. Defaults to the test file's path relative to project root. |
| `evaluators` | Evaluators run automatically against every case (hoisted). |
| `repetitions` | Run each case N times to measure non-determinism (each is its own pytest item + experiment run). |
Dataset-name precedence: `PHOENIX_TEST_DATASET` env > `phoenix_dataset` in `pytest.ini` > marker `dataset=` > file path.
## Logging helpers
- `log_output(value)` — records the system output (any JSON-serializable value).
- `log_evaluation(name, score, label?, explanation?)` — records a named inline score.
- `evaluate(evaluator, **kwargs)` — runs an evaluator callable, records its result as an annotation, and returns it so you can `assert` on the score to gate the case.
Hoisted evaluators bind arguments **by parameter name**: `output` (from `log_output`), `input` (parametrized fields), `expected`/`reference`/`metadata` (parametrized fields of that name), `trace_id`. Any `arize-phoenix-evals` evaluator (`FaithfulnessEvaluator`, etc.) or any `run_experiment` function works unchanged. Every score is wrapped in its own evaluator span linked to the trace.
## Two kinds of checks: invariants vs. signals
The decision that shapes every eval suite is which checks gate CI and which only trend. Get this split right and the rest is plumbing.
- **Hard invariants** — exactly one acceptable behavior, verifiable in code (a required refusal, valid JSON, a tool that must fire). `assert` these. A failure records `pass=False` and turns CI red, like any unit test.
- **Quality signals** — answers that live on a spectrum with no single correct string (helpfulness, groundedness, tone). Score with an LLM judge and **`log_evaluation`/`evaluate` only** — do *not* `assert` per case. LLM output is non-deterministic, so one weak answer shouldn't break the build; watch the aggregate trend in Phoenix instead. Gate on the trend separately (e.g. a nightly threshold or `run_experiment`), not on every case.
Rule of thumb: assert the behavior you'd be embarrassed to ship broken; log everything else as a signal.
## LLM-as-a-judge inside a test
A judge is just an evaluator passed to `evaluate()` (or hoisted on the marker). The cleanest judge is a `create_classifier` from `arize-phoenix-evals`: it emits a label mapped to a numeric score plus an explanation, recorded as its own annotation under a linked evaluator span. `evaluate(judge, **kwargs)` fills the prompt-template variables from `kwargs` — pass the judge only what it needs to grade. The judge runs on its own model, configured independently of the system under test (see configuring the judge LLM).
```python
import time
import pytest
from phoenix.client.pytest import evaluate, log_evaluation, log_output
from phoenix.evals import LLM, create_classifier
# Judge reads just the question + response; the evaluate() kwargs fill these vars.
helpfulness = create_classifier(
name="helpfulness",
llm=LLM(provider="anthropic", model="claude-sonnet-4-6"),
prompt_template=(
"Question: {{question}}\n\nResponse: {{response}}\n\n"
'Label "helpful" if it accurately answers the question, else "unhelpful".'
),
choices={"helpful": 1.0, "unhelpful": 0.0},
)
CASES = [
("How do I get a refund?", False),
("What's the capital of France?", True), # off-topic → must refuse
]
@pytest.mark.phoenix(dataset="support-bot")
@pytest.mark.parametrize("question,expect_refusal", CASES, ids=["refund", "offtopic"])
def test_support_response(question, expect_refusal):
t0 = time.perf_counter()
response = answer_question(question)
log_output({"response": response})
log_evaluation(name="latency_ms", score=(time.perf_counter() - t0) * 1000) # CODE signal
if expect_refusal:
assert "I don't have information on that" in response # hard invariant → gates CI
else:
# Quality signal — judged, NOT asserted. Helpfulness only means something
# for answerable questions, so judge here; it trends in Phoenix.
evaluate(helpfulness, question=question, response=response)
```
Refusal = invariant (asserted → gates CI); helpfulness + latency = signals (logged → trended). Judge only the cases where quality is meaningful. To gate CI on a judge anyway, capture its score and assert: `result = evaluate(judge, ...); assert result["score"] == 1.0` — reserve for invariants a code check can't express. For *groundedness*, add a `{{context}}` var and pass `context=...`, or use the pre-built `FaithfulnessEvaluator`.
## Environment variables
| Variable | Default | Purpose |
|---|---|---|
| `PHOENIX_ENDPOINT` | — | Phoenix base URL |
| `PHOENIX_API_KEY` | — | Bearer token |
| `PHOENIX_CLIENT_HEADERS` | — | Optional JSON headers |
| `PHOENIX_TEST_TRACKING` | `true` | Set `0`/`false` to run offline (tests execute, nothing uploaded) |
| `PHOENIX_TEST_REPETITIONS` | `1` | Default repetitions per marked test |
| `PHOENIX_TEST_DATASET` | _(file path)_ | Override dataset name for all collected tests |
Iterate locally without recording: `PHOENIX_TEST_TRACKING=0 pytest tests/evals/`.
## Dataset sync
- **Full run** (no filter): updates the dataset to match collected cases exactly, pruning examples for deleted tests.
- **Partial run** (`-k`, `-m`, a file, or `::node`): only appends, leaving unselected examples in place.
Two concurrent full runs writing the same dataset name can prune each other — give parallel CI jobs distinct names: `PHOENIX_TEST_DATASET=evals-${GIT_BRANCH}`.
## Parallel runs
`pytest -n auto` (pytest-xdist) is supported: the controller creates the dataset/experiment once and distributes IDs to workers. Exactly one experiment regardless of worker count.
## CI gate (GitHub Actions)
No extra config — pytest exits `1` on any failure. Uploads to Phoenix are best-effort (a Phoenix outage warns, never fails the build).
```yaml
name: eval-ci
on:
pull_request:
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install "arize-phoenix-client[pytest,evals]" pytest
- name: Run eval suite
env:
PHOENIX_ENDPOINT: ${{ secrets.PHOENIX_ENDPOINT }}
PHOENIX_API_KEY: ${{ secrets.PHOENIX_API_KEY }}
run: pytest tests/evals/
```
references/integrations-vitest-jest.md
# Integrations: Vitest / Jest (TypeScript)
Write evals as ordinary Vitest or Jest suites that record to Phoenix as experiments and gate CI. Each `describe()` → a dataset + experiment; each `test()` → an example + experiment run; the assert outcome → a `pass` annotation. OpenInference-instrumented LLM calls appear as child spans of each test's task.
Requires `@arizeai/phoenix-client>=6.11.1`. The testing API is **beta** and may change.
## When to use vs `runExperiment`
Use the test-runner integration when each case needs different logic, you want a hard `expect()` gate, or you want [acceptance criteria](#acceptance-criteria) to gate CI on aggregate scores — and you already use Vitest/Jest (`test.each`, `.only`/`.skip`, mocks). Prefer `runExperiment` for large, homogeneous datasets scored the same way.
## Install
```bash
npm install -D @arizeai/phoenix-client@^6.11.1 @arizeai/phoenix-evals dotenv
```
Vitest and Jest entrypoints are bundled in `@arizeai/phoenix-client`.
## Config (keep eval suites separate)
```ts
// phoenix.vitest.config.ts
import { defineConfig } from "vitest/config";
export default defineConfig({
test: {
include: ["**/*.eval.?(c|m)[jt]s"],
reporters: ["default", "@arizeai/phoenix-client/vitest/reporter"],
setupFiles: ["dotenv/config"], // loads PHOENIX_ENDPOINT, PHOENIX_API_KEY, ...
testTimeout: 30_000, // LLM calls are slow
},
});
```
Jest equivalent: `phoenix.jest.config.cjs` with `testMatch`, `reporters: ["default", "@arizeai/phoenix-client/jest/reporter"]`, `setupFiles: ["dotenv/config"]`. Add `"eval": "vitest run --config phoenix.vitest.config.ts"` (or `jest --config ...`) to `package.json`. **`jsdom` is not supported** — use `node`.
## Minimal suite
```ts
import * as px from "@arizeai/phoenix-client/vitest"; // or .../jest
import { expect } from "vitest";
px.describe("answer quality", () => {
px.test(
"capital city",
{ input: { question: "Capital of France?" }, expected: { answer: "Paris" } },
async ({ input, expected }) => {
const result = await myApp(input.question);
px.logOutput({ answer: result });
expect(result).toContain(expected?.answer ?? "");
},
);
});
```
Reference output accepts any one of `expected`, `reference`, or `output` — all arrive as `expected` in the test body. `it` is an alias for `test`.
Use `px.test.each(rows)` for larger datasets. To run against an existing Phoenix dataset, load rows with `getDatasetExamples` (top-level `await`, ESM) and keep each example's `id` so runs upsert onto the same example.
## Logging helpers
- `px.logOutput(value)` — records the run output (`ExperimentRun.output`).
- `px.logAnnotation({ name, score, label?, explanation?, metadata?, annotatorKind? })` — inline named score. `score` may be number or boolean (bools stored as 0/1); `annotatorKind` is `"LLM" | "CODE" | "HUMAN" | "DECISION" | "AGENT"` (default `"CODE"`).
- `px.evaluate(evaluator, params?)` — runs an `@arizeai/phoenix-evals` evaluator, records an annotation + linked evaluator trace. Omit `params` to auto-supply the test's `input`/`output`/`expected`/`metadata`/`traceId`.
- `px.traceEvaluator(fn, options?)` — wrap a plain grading fn; runs in its own evaluator span and captures a returned `{ name, score }` as an annotation. Passes through your args (no auto-supply).
## Two kinds of checks: invariants vs. signals
The decision that shapes every eval suite is which checks gate CI and which only trend.
- **Hard invariants** — exactly one acceptable behavior, verifiable in code (a required refusal, valid JSON, a tool that must fire). Use a per-case `expect()`. A failure fails the run and turns CI red.
- **Quality signals** — answers on a spectrum with no single correct string (helpfulness, groundedness, tone). Score with an LLM judge and record via `evaluate()`/`logAnnotation()` — do *not* `expect()` per case. Gate them at the **suite** level with `acceptanceCriteria` so aggregate quality (e.g. ≥70% helpful) holds the line while a single weak answer doesn't break the build.
This is the natural division of labor in this runner: invariants → per-case `expect()`; signals → `acceptanceCriteria`.
## LLM-as-a-judge inside a suite
The cleanest judge is a `createClassificationEvaluator` from `@arizeai/phoenix-evals`: it emits a label mapped to a numeric score (recorded as an annotation under a linked evaluator span). Pass the judge only what it needs to grade as the second arg to `px.evaluate()`, matching the template vars — no `inputMapping` needed. (For structured records or pre-built evaluators, `inputMapping` projects fields onto template vars instead.) The judge runs on its own model, configured independently of the system under test (see configuring the judge LLM).
```ts
import { anthropic } from "@ai-sdk/anthropic";
import * as px from "@arizeai/phoenix-client/vitest";
import { createClassificationEvaluator } from "@arizeai/phoenix-evals";
import { expect } from "vitest";
const helpfulness = createClassificationEvaluator({
name: "helpfulness",
model: anthropic("claude-sonnet-4-6"),
choices: { helpful: 1, unhelpful: 0 },
promptTemplate: `Question: {{question}}\n\nResponse: {{response}}\n\nLabel "helpful" if it accurately answers the question, else "unhelpful".`,
});
px.describe(
"support bot",
() => {
px.test.each(CASES)((row) => row.id, async ({ input }) => {
const start = performance.now();
const response = await answerQuestion(input.question);
px.logOutput({ response });
px.logAnnotation({ name: "latency_ms", score: performance.now() - start, annotatorKind: "CODE" });
if (input.expectRefusal) {
expect(response).toContain("I don't have information on that"); // hard invariant
} else {
// Quality signal — NOT asserted; gated by acceptanceCriteria below.
await px.evaluate(helpfulness, { question: input.question, response });
}
});
},
{
acceptanceCriteria: [
// signal gate: ≥70% of judged answers must score helpful
{ annotationName: "helpfulness", metric: "passRate", passFn: (a) => a.score === 1, minPassRate: 0.7 },
// budget gate: mean latency under 5s
{ annotationName: "latency_ms", metric: "average", threshold: 5000, direction: "minimize" },
],
},
);
```
The refusal stays a per-case `expect()` (invariant); helpfulness and latency move into `acceptanceCriteria` (signals). `passFn` receives the full annotation, so you can gate on `score`, `label`, or `explanation`. For *groundedness*, add a `{{context}}` var and pass it too, or use `createFaithfulnessEvaluator`.
## Acceptance criteria
Suite-level gates evaluated **after** all tests (every case runs first, full scorecard prints before failing). Pass as `describe`'s third arg:
```ts
px.describe("text-to-sql", () => { /* tests log token_f1, valid_sql, latency_ms */ }, {
acceptanceCriteria: [
{ annotationName: "token_f1", metric: "average", threshold: 0.8 },
{ annotationName: "token_f1", metric: "passRate",
passFn: (a) => typeof a.score === "number" && a.score >= 0.7, minPassRate: 0.9 },
{ annotationName: "valid_sql", metric: "passRate", passFn: (a) => a.score === true, minPassRate: 1 },
{ annotationName: "latency_ms", metric: "average", threshold: 800, direction: "minimize" },
],
});
```
- `metric: "average"` checks the mean against `threshold`; `direction` is `"maximize"` (default, clears at `>=`) or `"minimize"` (latency/cost/errors, clears at `<=`).
- `metric: "passRate"` counts runs where `passFn(annotation)` is true and requires that fraction to reach `minPassRate` (`1` = all). `passFn` receives the full annotation (`score`, `label`, `explanation`, `metadata`).
## Suite & test config
`describe(name, fn, opts)` opts: `datasetName`, `description`, `metadata`, `client` (custom `createClient`), `repetitions`, `dryRun`, `acceptanceCriteria`. Per-test/per-row fields alongside `input`: `id` (stable example id for upsert), `metadata`, `splits`, `repetitions`, `dryRun`, `config: { tags, metadata }`.
## Environment variables
| Variable | Purpose |
|---|---|
| `PHOENIX_ENDPOINT` | Phoenix base URL — usually the same URL as `PHOENIX_COLLECTOR_ENDPOINT`, which is used when this is unset |
| `PHOENIX_API_KEY` | Bearer token |
| `PHOENIX_CLIENT_HEADERS` | Optional JSON headers |
| `PHOENIX_TEST_TRACKING` | `false` disables sync (dry run) |
| `PHOENIX_TEST_REPETITIONS` | Default repetitions per test |
| `PHOENIX_TEST_REPORTER` | `verbose` shows every row + per-test output |
Dry run: `PHOENIX_TEST_TRACKING=false npm run eval`, or `{ dryRun: true }` per suite/test. Skipped (`.skip`) tests are never recorded; use dry-run to track locally without recording.
## CI gate (GitHub Actions)
The runner's exit code is the gate — a failed `expect()` or acceptance criterion exits nonzero.
```yaml
name: eval-ci
on:
pull_request:
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "20"
- run: npm ci
- name: Run eval suite
env:
PHOENIX_ENDPOINT: ${{ secrets.PHOENIX_ENDPOINT }}
PHOENIX_API_KEY: ${{ secrets.PHOENIX_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: npm run eval
```
references/observe-sampling-python.md
# Observe: Sampling Strategies
How to efficiently sample production traces for review.
## Strategies
### 1. Failure-Focused (Highest Priority)
```python
errors = spans_df[spans_df["status_code"] == "ERROR"]
negative_feedback = spans_df[spans_df["feedback"] == "negative"]
```
### 2. Outliers
```python
long_responses = spans_df.nlargest(50, "response_length")
slow_responses = spans_df.nlargest(50, "latency_ms")
```
### 3. Stratified (Coverage)
```python
# Sample equally from each category
by_query_type = spans_df.groupby("metadata.query_type").apply(
lambda x: x.sample(min(len(x), 20))
)
```
### 4. Metric-Guided
```python
# Review traces flagged by automated evaluators
flagged = spans_df[eval_results["label"] == "hallucinated"]
borderline = spans_df[(eval_results["score"] > 0.3) & (eval_results["score"] < 0.7)]
```
## Building a Review Queue
```python
def build_review_queue(spans_df, max_traces=100):
queue = pd.concat([
spans_df[spans_df["status_code"] == "ERROR"],
spans_df[spans_df["feedback"] == "negative"],
spans_df.nlargest(10, "response_length"),
spans_df.sample(min(30, len(spans_df))),
]).drop_duplicates("span_id").head(max_traces)
return queue
```
## Sample Size Guidelines
| Purpose | Size |
| ------- | ---- |
| Initial exploration | 50-100 |
| Error analysis | 100+ (until saturation) |
| Golden dataset | 100-500 |
| Judge calibration | 100+ per class |
**Saturation:** Stop when new traces show the same failure patterns.
## Trace-Level Sampling
When you need whole requests (all spans per trace), use `get_traces`:
```python
from phoenix.client import Client
from datetime import datetime, timedelta
client = Client()
# Recent traces with full span trees
traces = client.traces.get_traces(
project_identifier="my-app",
limit=100,
include_spans=True,
)
# Time-windowed sampling (e.g., last hour)
traces = client.traces.get_traces(
project_identifier="my-app",
start_time=datetime.now() - timedelta(hours=1),
limit=50,
include_spans=True,
)
# Filter by session (multi-turn conversations)
traces = client.traces.get_traces(
project_identifier="my-app",
session_id="user-session-abc",
include_spans=True,
)
# Sort by latency to find slowest requests
traces = client.traces.get_traces(
project_identifier="my-app",
sort="latency_ms",
order="desc",
limit=50,
)
# Server-side trace filter (requires Phoenix server >= 20.12.0)
slow_failures = client.traces.get_traces(
project_identifier="my-app",
filter="error_count > 0 and latency_ms >= 1000",
limit=50,
)
# The clean, slow traces — the ones that are wrong without crashing
quiet_and_slow = client.traces.get_traces(
project_identifier="my-app",
filter="error_count == 0 and 5000 <= latency_ms <= 30000",
limit=50,
)
```
`filter` is a trace filter expression: the same language as the UI's traces
filter bar, with rollups like `error_count`, `latency_ms`, `num_spans`, and
`total_cost` and comprehensions over `spans`. The vocabulary is in
[filter-expressions.md](filter-expressions.md) under "Trace filter". It
combines with `start_time`, `end_time`, and `session_id` using AND. Against an
older server the client raises before sending the request rather than
returning unfiltered traces.
references/observe-sampling-typescript.md
# Observe: Sampling Strategies (TypeScript)
How to efficiently sample production traces for review.
## Strategies
### 1. Failure-Focused (Highest Priority)
Use server-side filters to fetch only what you need:
```typescript
import { getSpans } from "@arizeai/phoenix-client/spans";
// Server-side filter — only ERROR spans are returned
const { spans: errors } = await getSpans({
project: { projectName: "my-project" },
statusCode: "ERROR",
limit: 100,
});
// Fetch only LLM spans
const { spans: llmSpans } = await getSpans({
project: { projectName: "my-project" },
spanKind: "LLM",
limit: 100,
});
// Filter by span name
const { spans: chatSpans } = await getSpans({
project: { projectName: "my-project" },
name: "chat_completion",
limit: 100,
});
```
### 2. Outliers
```typescript
const { spans } = await getSpans({
project: { projectName: "my-project" },
limit: 200,
});
const latency = (s: (typeof spans)[number]) =>
new Date(s.end_time).getTime() - new Date(s.start_time).getTime();
const sorted = [...spans].sort((a, b) => latency(b) - latency(a));
const slowResponses = sorted.slice(0, 50);
```
### 3. Stratified (Coverage)
```typescript
// Sample equally from each category
function stratifiedSample<T>(items: T[], groupBy: (item: T) => string, perGroup: number): T[] {
const groups = new Map<string, T[]>();
for (const item of items) {
const key = groupBy(item);
if (!groups.has(key)) groups.set(key, []);
groups.get(key)!.push(item);
}
return [...groups.values()].flatMap((g) => g.slice(0, perGroup));
}
const { spans } = await getSpans({
project: { projectName: "my-project" },
limit: 500,
});
const byQueryType = stratifiedSample(spans, (s) => s.attributes?.["metadata.query_type"] ?? "unknown", 20);
```
### 4. Metric-Guided
```typescript
import { getSpanAnnotations } from "@arizeai/phoenix-client/spans";
// Fetch annotations for your spans, then filter by label
const { annotations } = await getSpanAnnotations({
project: { projectName: "my-project" },
spanIds: spans.map((s) => s.context.span_id),
includeAnnotationNames: ["hallucination"],
});
const flaggedSpanIds = new Set(
annotations.filter((a) => a.result?.label === "hallucinated").map((a) => a.span_id)
);
const flagged = spans.filter((s) => flaggedSpanIds.has(s.context.span_id));
```
## Trace-Level Sampling
When you need whole requests (all spans in a trace), use `getTraces`:
```typescript
import { getTraces } from "@arizeai/phoenix-client/traces";
// Recent traces with full span trees
const { traces } = await getTraces({
project: { projectName: "my-project" },
limit: 100,
includeSpans: true,
});
// Filter by session (e.g., multi-turn conversations)
const { traces: sessionTraces } = await getTraces({
project: { projectName: "my-project" },
sessionId: "user-session-abc",
includeSpans: true,
});
// Time-windowed sampling
const { traces: recentTraces } = await getTraces({
project: { projectName: "my-project" },
startTime: new Date(Date.now() - 60 * 60 * 1000), // last hour
limit: 50,
includeSpans: true,
});
// Server-side trace filter (requires Phoenix server >= 20.12.0)
const { traces: slowFailures } = await getTraces({
project: { projectName: "my-project" },
filter: "error_count > 0 and latency_ms >= 1000",
limit: 50,
});
// The clean, slow traces — the ones that are wrong without crashing
const { traces: quietAndSlow } = await getTraces({
project: { projectName: "my-project" },
filter: "error_count == 0 and 5000 <= latency_ms <= 30000",
limit: 50,
});
```
`filter` is a trace filter expression: the same language as the UI's traces
filter bar, with rollups like `error_count`, `latency_ms`, `num_spans`, and
`total_cost` and comprehensions over `spans`. The vocabulary is in
[filter-expressions.md](filter-expressions.md) under "Trace filter". It
combines with `startTime`, `endTime`, and `sessionId` using AND. Against an
older server the client throws before sending the request rather than
returning unfiltered traces.
## Building a Review Queue
```typescript
// Combine server-side filters into a review queue
const { spans: errorSpans } = await getSpans({
project: { projectName: "my-project" },
statusCode: "ERROR",
limit: 30,
});
const { spans: allSpans } = await getSpans({
project: { projectName: "my-project" },
limit: 100,
});
const random = allSpans.sort(() => Math.random() - 0.5).slice(0, 30);
const combined = [...errorSpans, ...random];
const unique = [...new Map(combined.map((s) => [s.context.span_id, s])).values()];
const reviewQueue = unique.slice(0, 100);
```
## Sample Size Guidelines
| Purpose | Size |
| ------- | ---- |
| Initial exploration | 50-100 |
| Error analysis | 100+ (until saturation) |
| Golden dataset | 100-500 |
| Judge calibration | 100+ per class |
**Saturation:** Stop when new traces show the same failure patterns.
references/observe-tracing-setup.md
# Observe: Tracing Setup
Configure tracing to capture data for evaluation.
## Quick Setup
```python
# Python
from phoenix.otel import register
register(project_name="my-app", auto_instrument=True)
```
```typescript
// TypeScript
import { registerPhoenix } from "@arizeai/phoenix-otel";
registerPhoenix({ projectName: "my-app", autoInstrument: true });
```
## Essential Attributes
| Attribute | Why It Matters |
| --------- | -------------- |
| `input.value` | User's request |
| `output.value` | Response to evaluate |
| `retrieval.documents` | Context for faithfulness |
| `tool.name`, `tool.parameters` | Agent evaluation |
| `llm.model_name` | Track by model |
## Custom Attributes for Evals
```python
span.set_attribute("metadata.client_type", "enterprise")
span.set_attribute("metadata.query_category", "billing")
```
## Exporting for Evaluation
### Spans (Python — DataFrame)
```python
from phoenix.client import Client
from phoenix.client.types.spans import SpanQuery
# Client() works for local Phoenix (falls back to env vars or localhost:6006)
# For remote/cloud: Client(base_url="https://app.phoenix.arize.com", api_key="...")
client = Client()
spans_df = client.spans.get_spans_dataframe(
project_identifier="my-app", # NOT project_name= (deprecated)
query=SpanQuery().where("parent_id is None"), # top-level spans only
)
dataset = client.datasets.create_dataset(
name="error-analysis-set",
dataframe=spans_df[["input.value", "output.value"]],
input_keys=["input.value"],
output_keys=["output.value"],
)
```
#### Filtering with `SpanQuery().where(...)`
The `where` string is a span filter expression, the same language as the UI spans filter bar and
the GraphQL `filterCondition`. Root spans are a clause in it: `parent_id is None` keeps spans with
no parent id, and `parent_span is None` also keeps orphans whose parent was never received. The
full vocabulary, operators, and rules are in
[filter-expressions](filter-expressions.md); read it before writing any other condition, because
an unknown name compiles as an attribute path and silently matches nothing.
```python span-filter
parent_id is None
parent_id is None and status_code == 'ERROR'
span_kind == 'LLM' and llm.model_name == 'gpt-4o'
span_kind == 'RETRIEVER'
'refund' in input.value
metadata['client_type'] == 'enterprise'
annotations['correctness'].label is None
start_time > '2026-09-01T00:00:00Z'
```
### Spans (TypeScript)
```typescript
import { getSpans } from "@arizeai/phoenix-client/spans";
const { spans } = await getSpans({
project: { projectName: "my-app" },
parentId: null, // root spans only
limit: 100,
});
```
### Traces (Python — structured)
Use `get_traces` when you need full trace trees (e.g., multi-turn conversations, agent workflows):
```python
from datetime import datetime, timedelta
traces = client.traces.get_traces(
project_identifier="my-app",
start_time=datetime.now() - timedelta(hours=24),
include_spans=True, # includes all spans per trace
limit=100,
)
# Each trace has: trace_id, start_time, end_time, spans (when include_spans=True)
```
### Traces (TypeScript)
```typescript
import { getTraces } from "@arizeai/phoenix-client/traces";
const { traces } = await getTraces({
project: { projectName: "my-app" },
startTime: new Date(Date.now() - 24 * 60 * 60 * 1000),
includeSpans: true,
limit: 100,
});
```
## Uploading Evaluations as Annotations
### Python
```python
from phoenix.evals import evaluate_dataframe
from phoenix.evals.utils import to_annotation_dataframe
# Run evaluations
results_df = evaluate_dataframe(dataframe=spans_df, evaluators=[my_eval])
# Format results for Phoenix annotations
annotations_df = to_annotation_dataframe(results_df)
# Upload to Phoenix
client.spans.log_span_annotations_dataframe(dataframe=annotations_df)
```
### TypeScript
```typescript
import { logSpanAnnotations } from "@arizeai/phoenix-client/spans";
await logSpanAnnotations({
spanAnnotations: [
{
spanId: "abc123",
name: "quality",
label: "good",
score: 0.95,
annotatorKind: "LLM",
},
],
});
```
Annotations are visible in the Phoenix UI alongside your traces.
## Verify
Required attributes: `input.value`, `output.value`, `status_code`
For RAG: `retrieval.documents`
For agents: `tool.name`, `tool.parameters`
references/production-continuous.md
# Production: Continuous Evaluation
Capability vs regression evals and the ongoing feedback loop.
## Two Types of Evals
| Type | Pass Rate Target | Purpose | Update |
| ---- | ---------------- | ------- | ------ |
| **Capability** | 50-80% | Measure improvement | Add harder cases |
| **Regression** | 95-100% | Catch breakage | Add fixed bugs |
## Saturation
When capability evals hit >95% pass rate, they're saturated:
1. Graduate passing cases to regression suite
2. Add new challenging cases to capability suite
## Feedback Loop
```
Production → Sample traffic → Run evaluators → Find failures
↑ ↓
Deploy ← Run CI evals ← Create test cases ← Error analysis
```
## Implementation
Build a continuous monitoring loop:
1. **Sample recent traces** at regular intervals (e.g., 100 traces per hour)
2. **Run evaluators** on sampled traces
3. **Log results** to Phoenix for tracking
4. **Queue concerning results** for human review
5. **Create test cases** from recurring failure patterns
### Python
```python
from phoenix.client import Client
from datetime import datetime, timedelta
from phoenix.client.types.spans import SpanQuery
client = Client()
# 1. Sample recent spans (includes full attributes for evaluation)
spans_df = client.spans.get_spans_dataframe(
project_identifier="my-app",
start_time=datetime.now() - timedelta(hours=1),
query=SpanQuery().where("parent_id is None"), # top-level spans only
limit=100,
)
# 2. Run evaluators
from phoenix.evals import evaluate_dataframe
results_df = evaluate_dataframe(
dataframe=spans_df,
evaluators=[quality_eval, safety_eval],
)
# 3. Upload results as annotations
from phoenix.evals.utils import to_annotation_dataframe
annotations_df = to_annotation_dataframe(results_df)
client.spans.log_span_annotations_dataframe(dataframe=annotations_df)
```
### TypeScript
```typescript
import { getSpans } from "@arizeai/phoenix-client/spans";
import { logSpanAnnotations } from "@arizeai/phoenix-client/spans";
// 1. Sample recent spans
const { spans } = await getSpans({
project: { projectName: "my-app" },
startTime: new Date(Date.now() - 60 * 60 * 1000),
parentId: null, // root spans only
limit: 100,
});
// 2. Run evaluators (user-defined)
const results = await Promise.all(
spans.map(async (span) => ({
spanId: span.context.span_id,
...await runEvaluators(span, [qualityEval, safetyEval]),
}))
);
// 3. Upload results as annotations
await logSpanAnnotations({
spanAnnotations: results.map((r) => ({
spanId: r.spanId,
name: "quality",
score: r.qualityScore,
label: r.qualityLabel,
annotatorKind: "LLM" as const,
})),
});
```
For trace-level monitoring (e.g., agent workflows), use `get_traces`/`getTraces` to identify traces:
```python
# Python: identify slow traces
traces = client.traces.get_traces(
project_identifier="my-app",
start_time=datetime.now() - timedelta(hours=1),
filter="latency_ms >= 1000",
limit=50,
)
```
```typescript
// TypeScript: identify slow traces
import { getTraces } from "@arizeai/phoenix-client/traces";
const { traces } = await getTraces({
project: { projectName: "my-app" },
startTime: new Date(Date.now() - 60 * 60 * 1000),
filter: "latency_ms >= 1000",
limit: 50,
});
```
`filter` takes a trace filter expression (`error_count > 0`, `latency_ms >= 1000`,
`any(span.span_kind == "TOOL" for span in spans)`); the vocabulary is in
[filter-expressions.md](filter-expressions.md). It requires Phoenix server
>= 20.12.0, and against an older server the client raises rather than
silently returning unfiltered traces.
## Alerting
| Condition | Severity | Action |
| --------- | -------- | ------ |
| Regression < 98% | Critical | Page oncall |
| Capability declining | Warning | Slack notify |
| Capability > 95% for 7d | Info | Schedule review |
## Key Principles
- **Two suites** - Capability + Regression always
- **Graduate cases** - Move consistent passes to regression
- **Track trends** - Monitor over time, not just snapshots
references/production-guardrails.md
# Production: Guardrails vs Evaluators
Guardrails block in real-time. Evaluators measure asynchronously.
## Key Distinction
```
Request → [INPUT GUARDRAIL] → LLM → [OUTPUT GUARDRAIL] → Response
│
└──→ ASYNC EVALUATOR (background)
```
## Guardrails
| Aspect | Requirement |
| ------ | ----------- |
| Timing | Synchronous, blocking |
| Latency | < 100ms |
| Purpose | Prevent harm |
| Type | Code-based (deterministic) |
**Use for:** PII detection, prompt injection, profanity, length limits, format validation.
## Evaluators
| Aspect | Characteristic |
| ------ | -------------- |
| Timing | Async, background |
| Latency | Can be seconds |
| Purpose | Measure quality |
| Type | Can use LLMs |
**Use for:** Helpfulness, faithfulness, tone, completeness, citation accuracy.
## Decision
| Question | Answer |
| -------- | ------ |
| Must block harmful content? | Guardrail |
| Measuring quality? | Evaluator |
| Need LLM judgment? | Evaluator |
| < 100ms required? | Guardrail |
| False positives = angry users? | Evaluator |
## LLM Guardrails: Rarely
Only use LLM guardrails if:
- Latency budget > 1s
- Error cost >> LLM cost
- Low volume
- Fallback exists
**Key Principle:** Guardrails prevent harm (block). Evaluators measure quality (log).
references/production-overview.md
# Production: Overview
CI/CD evals vs production monitoring - complementary approaches.
## Two Evaluation Modes
| Aspect | CI/CD Evals | Production Monitoring |
| ------ | ----------- | -------------------- |
| **When** | Pre-deployment | Post-deployment, ongoing |
| **Data** | Fixed dataset | Sampled traffic |
| **Goal** | Prevent regression | Detect drift |
| **Response** | Block deploy | Alert & analyze |
## CI/CD Evaluations
```python
from phoenix.client import Client
client = Client()
# Fast, deterministic checks
ci_evaluators = [
has_required_format,
no_pii_leak,
safety_check,
regression_test_suite,
]
# Small but representative dataset (~100 examples)
client.experiments.run_experiment(dataset=ci_dataset, task=task, evaluators=ci_evaluators)
```
Set thresholds: regression=0.95, safety=1.0, format=0.98.
## Production Monitoring
### Python
```python
from phoenix.client import Client
from datetime import datetime, timedelta
client = Client()
# Sample recent traces (last hour)
traces = client.traces.get_traces(
project_identifier="my-app",
start_time=datetime.now() - timedelta(hours=1),
include_spans=True,
limit=100,
)
# Run evaluators on sampled traffic
for trace in traces:
results = run_evaluators_async(trace, production_evaluators)
if any(r["score"] < 0.5 for r in results):
alert_on_failure(trace, results)
```
### TypeScript
```typescript
import { getTraces } from "@arizeai/phoenix-client/traces";
import { getSpans } from "@arizeai/phoenix-client/spans";
// Sample recent traces (last hour)
const { traces } = await getTraces({
project: { projectName: "my-app" },
startTime: new Date(Date.now() - 60 * 60 * 1000),
includeSpans: true,
limit: 100,
});
// Or sample spans directly for evaluation
const { spans } = await getSpans({
project: { projectName: "my-app" },
startTime: new Date(Date.now() - 60 * 60 * 1000),
limit: 100,
});
// Run evaluators on sampled traffic
for (const span of spans) {
const results = await runEvaluators(span, productionEvaluators);
if (results.some((r) => r.score < 0.5)) {
await alertOnFailure(span, results);
}
}
```
Prioritize: errors → negative feedback → random sample.
## Feedback Loop
```
Production finds failure → Error analysis → Add to CI dataset → Prevents future regression
```
references/setup-python.md
# Setup: Python
Packages required for Phoenix evals and experiments.
## Installation
```bash
# Core Phoenix package (includes client, evals, otel)
pip install arize-phoenix
# Or install individual packages
pip install arize-phoenix-client # Phoenix client only
pip install arize-phoenix-evals # Evaluation utilities
pip install arize-phoenix-otel # OpenTelemetry integration
```
## LLM Providers
For LLM-as-judge evaluators, install your provider's SDK:
```bash
pip install openai # OpenAI
pip install anthropic # Anthropic
pip install google-genai # Google
```
## Validation (Optional)
```bash
pip install scikit-learn # For TPR/TNR metrics
```
## Quick Verify
```python
from phoenix.client import Client
from phoenix.evals import LLM, ClassificationEvaluator
from phoenix.otel import register
# All imports should work
print("Phoenix Python setup complete")
```
## Key Imports (Evals 2.0)
```python
from phoenix.client import Client
from phoenix.evals import (
ClassificationEvaluator, # LLM classification evaluator (preferred)
LLM, # Provider-agnostic LLM wrapper
async_evaluate_dataframe, # Batch evaluate a DataFrame (preferred, async)
evaluate_dataframe, # Batch evaluate a DataFrame (sync)
create_evaluator, # Decorator for code-based evaluators
create_classifier, # Factory for LLM classification evaluators
bind_evaluator, # Map column names to evaluator params
Score, # Score dataclass
)
from phoenix.evals.utils import to_annotation_dataframe # Format results for Phoenix annotations
```
**Prefer**: `ClassificationEvaluator` over `create_classifier` (more parameters/customization).
**Prefer**: `async_evaluate_dataframe` over `evaluate_dataframe` (better throughput for LLM evals).
**Do NOT use** legacy 1.0 imports: `OpenAIModel`, `AnthropicModel`, `run_evals`, `llm_classify`.
references/setup-typescript.md
# Setup: TypeScript
Packages required for Phoenix evals and experiments.
## Installation
```bash
# Using npm
npm install @arizeai/phoenix-client @arizeai/phoenix-evals @arizeai/phoenix-otel
# Using pnpm
pnpm add @arizeai/phoenix-client @arizeai/phoenix-evals @arizeai/phoenix-otel
```
## Requirements
`@arizeai/phoenix-evals` 2.x requires **Node.js >= 22.12** and AI SDK **v7**. It
depends on `ai@^7` and `@ai-sdk/otel@^1` directly, so you only need to install a
provider — but that provider must be AI SDK v7-compatible (e.g. `@ai-sdk/openai`
v4). `@arizeai/phoenix-client` 7.x also requires the `ai` peer at `^7.0.0`.
Type-checking the published `@arizeai/phoenix-client` declarations requires
**TypeScript >= 5.3** (the prompts entry uses `with { "resolution-mode": "import" }`
import attributes, which `skipLibCheck` does not suppress).
## LLM Providers
For LLM-as-judge evaluators, install a Vercel AI SDK provider:
```bash
npm install @ai-sdk/openai # OpenAI
npm install @ai-sdk/anthropic # Anthropic
npm install @ai-sdk/google # Google
```
Or use direct provider SDKs:
```bash
npm install openai # OpenAI direct
npm install @anthropic-ai/sdk # Anthropic direct
```
## Quick Verify
```typescript
import { createClient } from "@arizeai/phoenix-client";
import { createClassificationEvaluator } from "@arizeai/phoenix-evals";
import { register } from "@arizeai/phoenix-otel";
// All imports should work
console.log("Phoenix TypeScript setup complete");
```
references/validation-evaluators-python.md
# Validating Evaluators (Python)
Validate LLM evaluators against human-labeled examples. Target >80% accuracy, >70% TPR/TNR.
## Calculate Metrics
These snippets use scikit-learn, which no Phoenix package installs: `pip install scikit-learn`.
```python
from sklearn.metrics import classification_report, confusion_matrix
print(classification_report(human_labels, evaluator_predictions))
cm = confusion_matrix(human_labels, evaluator_predictions)
tn, fp, fn, tp = cm.ravel()
tpr = tp / (tp + fn)
tnr = tn / (tn + fp)
print(f"TPR: {tpr:.2f}, TNR: {tnr:.2f}")
```
## Correct Production Estimates
```python
def correct_estimate(observed, tpr, tnr):
"""Adjust observed pass rate using known TPR/TNR."""
return (observed - (1 - tnr)) / (tpr - (1 - tnr))
```
## Find Misclassified
```python
# False Positives: Evaluator pass, human fail
fp_mask = (evaluator_predictions == 1) & (human_labels == 0)
false_positives = dataset[fp_mask]
# False Negatives: Evaluator fail, human pass
fn_mask = (evaluator_predictions == 0) & (human_labels == 1)
false_negatives = dataset[fn_mask]
```
## Red Flags
- TPR or TNR < 70%
- Large gap between TPR and TNR
- Kappa < 0.6
references/validation-evaluators-typescript.md
# Validating Evaluators (TypeScript)
Validate an LLM evaluator against human-labeled examples before deploying it.
Target: **>80% accuracy, >70% TPR and >70% TNR**.
Roles are inverted compared to a normal task experiment:
| Normal experiment | Evaluator validation |
|---|---|
| Task = agent logic | Task = run the evaluator under test |
| Evaluator = judge output | Evaluator = exact-match vs human ground truth |
| Dataset = agent examples | Dataset = golden hand-labeled examples |
## Golden Dataset
Use a separate dataset name so validation experiments don't mix with task experiments in Phoenix.
Store human ground truth in `metadata.groundTruthLabel`. Aim for ~50/50 balance:
```typescript
import type { Example } from "@arizeai/phoenix-client/types/datasets";
const goldenExamples: Example[] = [
{ input: { q: "Capital of France?" }, output: { answer: "Paris" }, metadata: { groundTruthLabel: "correct" } },
{ input: { q: "Capital of France?" }, output: { answer: "Lyon" }, metadata: { groundTruthLabel: "incorrect" } },
{ input: { q: "Capital of France?" }, output: { answer: "Major city..." }, metadata: { groundTruthLabel: "incorrect" } },
];
const VALIDATOR_DATASET = "my-app-qa-evaluator-validation"; // separate from task dataset
const POSITIVE_LABEL = "correct";
const NEGATIVE_LABEL = "incorrect";
```
## Validation Experiment
```typescript
import { createClient } from "@arizeai/phoenix-client";
import { createOrGetDataset, getDatasetExamples } from "@arizeai/phoenix-client/datasets";
import { asExperimentEvaluator, runExperiment } from "@arizeai/phoenix-client/experiments";
import { myEvaluator } from "./myEvaluator.js";
const client = createClient();
const { datasetId } = await createOrGetDataset({ client, name: VALIDATOR_DATASET, examples: goldenExamples });
const { examples } = await getDatasetExamples({ client, dataset: { datasetId } });
const groundTruth = new Map(examples.map((ex) => [ex.id, ex.metadata?.groundTruthLabel as string]));
// Task: invoke the evaluator under test
const task = async (example: (typeof examples)[number]) => {
const result = await myEvaluator.evaluate({ input: example.input, output: example.output, metadata: example.metadata });
return result.label ?? "unknown";
};
// Evaluator: exact-match against human ground truth
const exactMatch = asExperimentEvaluator({
name: "exact-match", kind: "CODE",
evaluate: ({ output, metadata }) => {
const expected = metadata?.groundTruthLabel as string;
const predicted = typeof output === "string" ? output : "unknown";
return { score: predicted === expected ? 1 : 0, label: predicted, explanation: `Expected: ${expected}, Got: ${predicted}` };
},
});
const experiment = await runExperiment({
client, experimentName: `evaluator-validation-${Date.now()}`,
dataset: { datasetId }, task, evaluators: [exactMatch],
});
// Compute confusion matrix
const runs = Object.values(experiment.runs);
const predicted = new Map((experiment.evaluationRuns ?? [])
.filter((e) => e.name === "exact-match")
.map((e) => [e.experimentRunId, e.result?.label ?? null]));
let tp = 0, fp = 0, tn = 0, fn = 0;
for (const run of runs) {
if (run.error) continue;
const p = predicted.get(run.id), a = groundTruth.get(run.datasetExampleId);
if (!p || !a) continue;
if (a === POSITIVE_LABEL && p === POSITIVE_LABEL) tp++;
else if (a === NEGATIVE_LABEL && p === POSITIVE_LABEL) fp++;
else if (a === NEGATIVE_LABEL && p === NEGATIVE_LABEL) tn++;
else if (a === POSITIVE_LABEL && p === NEGATIVE_LABEL) fn++;
}
const total = tp + fp + tn + fn;
const tpr = tp + fn > 0 ? (tp / (tp + fn)) * 100 : 0;
const tnr = tn + fp > 0 ? (tn / (tn + fp)) * 100 : 0;
console.log(`TPR: ${tpr.toFixed(1)}% TNR: ${tnr.toFixed(1)}% Accuracy: ${((tp + tn) / total * 100).toFixed(1)}%`);
```
## Results & Quality Rules
| Metric | Target | Low value means |
|---|---|---|
| TPR (sensitivity) | >70% | Misses real failures (false negatives) |
| TNR (specificity) | >70% | Flags good outputs (false positives) |
| Accuracy | >80% | General weakness |
**Golden dataset rules:** ~50/50 balance · include edge cases · human-labeled only · never mutate (append new versions) · 20–50 examples is enough.
**Re-validate when:** prompt template changes · judge model changes · criteria updated · production FP/FN spike.
## See Also
- `validation.md` — Metric definitions and concepts
- `experiments-running-typescript.md` — `runExperiment` API
- `experiments-datasets-typescript.md` — `createOrGetDataset` / `getDatasetExamples`
references/validation.md
# Validation
Validate LLM judges against human labels before deploying. Target >80% agreement.
This page covers what to compute and the targets to hit. Runnable code is in
`validation-evaluators-{python|typescript}.md`.
## Requirements
| Requirement | Target |
| ----------- | ------ |
| Test set size | 100+ examples |
| Balance | ~50/50 pass/fail |
| Accuracy | >80% |
| TPR/TNR | Both >70% |
## Metrics
| Metric | Formula | Use When |
| ------ | ------- | -------- |
| **Accuracy** | (TP+TN) / Total | General |
| **TPR (Recall)** | TP / (TP+FN) | Quality assurance |
| **TNR (Specificity)** | TN / (TN+FP) | Safety-critical |
| **Cohen's Kappa** | Agreement beyond chance | Comparing evaluators |
## Human Agreement Is the Ceiling
Human labels are not perfect ground truth — labelers disagree. When two or more people label
the same examples, **their agreement with each other is the ceiling** the judge is measured
against: a judge cannot be expected to agree with the labels more often than the labelers agree
with one another.
- Compute inter-annotator agreement (raw agreement rate, or Cohen's kappa for two labelers) and
treat *that* as the target the judge is approaching, not 100%.
- Where labelers disagree, the "true" label is itself uncertain — adjudicate it to a single gold
label or drop the example before scoring the judge; never count the judge "wrong" on an
example the humans themselves split on.
- If inter-annotator agreement is low, the criteria are unclear — fix the labeling guidelines
before trusting any judge number.
## Quick Validation
Requires scikit-learn, which no Phoenix package installs: `pip install scikit-learn`.
```python
from sklearn.metrics import classification_report, confusion_matrix, cohen_kappa_score
print(classification_report(human_labels, evaluator_predictions))
print(f"Kappa: {cohen_kappa_score(human_labels, evaluator_predictions):.3f}")
# Get TPR/TNR
cm = confusion_matrix(human_labels, evaluator_predictions)
tn, fp, fn, tp = cm.ravel()
tpr = tp / (tp + fn)
tnr = tn / (tn + fp)
```
## Inspect the Errors, Not Just the Score
An aggregate accuracy or TPR/TNR number tells you *how often* the judge disagrees with humans,
never *why*. Always pull out the disagreements and read a handful of each:
- **False positives** — judge says pass, humans say fail (the judge is too lenient).
- **False negatives** — judge says fail, humans say pass (the judge is too strict).
Reading them tells you whether the judge is miscalibrated, the prompt is ambiguous, or the
labels themselves are wrong — and that is what drives the next prompt revision.
`validation-evaluators-{python|typescript}.md` has the code to isolate these cases (`fp_mask` /
`fn_mask`) and to correct an observed production pass rate for known TPR/TNR (`correct_estimate`).
## Golden Dataset Structure
```python
golden_example = {
"input": "What is the capital of France?",
"output": "Paris is the capital.",
"ground_truth_label": "correct",
}
```
## Building Golden Datasets
1. Sample production traces (errors, negative feedback, edge cases)
2. Balance ~50/50 pass/fail
3. Expert labels each example
4. Version datasets (never modify existing)
```python
# GOOD - create new version
golden_v2 = golden_v1 + [new_examples]
# BAD - never modify existing
golden_v1.append(new_example)
```
## Warning Signs
- All pass or all fail → too lenient/strict
- Random results → criteria unclear
- TPR/TNR < 70% → needs improvement
## Re-Validate When
- Prompt template changes
- Judge model changes
- Criteria changes
- Monthly
## See Also
- [validation-evaluators-python](validation-evaluators-python.md), [validation-evaluators-typescript](validation-evaluators-typescript.md)
SKILL.md
---
name: phoenix-evals
description: Build, run, and validate evaluators for AI/LLM applications with Phoenix, in Python or TypeScript (including Vercel AI SDK apps) — code-based checks and LLM-as-a-judge, run over exported traces or dataframes or as Phoenix experiments, gate them in CI, and validate judges against human-labeled data.
license: Apache-2.0
compatibility: Requires Phoenix server. Python skills need phoenix and openai packages; TypeScript skills need @arizeai/phoenix-client.
metadata:
targets: "agents mcp"
author: oss@arize.com
version: "1.0.0"
languages: "Python, TypeScript"
---
# Phoenix Evals
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
## Quick Reference
| Task | Files |
| ---- | ----- |
| Setup | [setup-python](references/setup-python.md), [setup-typescript](references/setup-typescript.md) |
| Decide what to evaluate | [evaluators-overview](references/evaluators-overview.md) |
| Choose a judge model | [fundamentals-model-selection](references/fundamentals-model-selection.md) |
| Use pre-built evaluators | [evaluators-pre-built](references/evaluators-pre-built.md) |
| Build code evaluator | [evaluators-code-python](references/evaluators-code-python.md), [evaluators-code-typescript](references/evaluators-code-typescript.md) |
| Build LLM evaluator | [evaluators-llm-python](references/evaluators-llm-python.md), [evaluators-llm-typescript](references/evaluators-llm-typescript.md), [evaluators-custom-templates](references/evaluators-custom-templates.md) |
| Batch evaluate DataFrame | [evaluate-dataframe-python](references/evaluate-dataframe-python.md) |
| Run experiment | [experiments-running-python](references/experiments-running-python.md), [experiments-running-typescript](references/experiments-running-typescript.md) |
| Run evals in a test runner (CI gate) | [integrations-pytest](references/integrations-pytest.md), [integrations-vitest-jest](references/integrations-vitest-jest.md) |
| Create dataset | [experiments-datasets-python](references/experiments-datasets-python.md), [experiments-datasets-typescript](references/experiments-datasets-typescript.md) |
| Generate synthetic data | [experiments-synthetic-python](references/experiments-synthetic-python.md), [experiments-synthetic-typescript](references/experiments-synthetic-typescript.md) |
| Validate evaluator accuracy | [validation](references/validation.md), [validation-evaluators-python](references/validation-evaluators-python.md), [validation-evaluators-typescript](references/validation-evaluators-typescript.md) |
| Export spans | [observe-tracing-setup](references/observe-tracing-setup.md) |
| Write a span filter (`SpanQuery().where`) | [filter-expressions](references/filter-expressions.md) |
| Sample traces for review | [observe-sampling-python](references/observe-sampling-python.md), [observe-sampling-typescript](references/observe-sampling-typescript.md) |
| Analyze errors | [error-analysis](references/error-analysis.md), [error-analysis-multi-turn](references/error-analysis-multi-turn.md), [axial-coding](references/axial-coding.md) |
| RAG evals | [evaluators-rag](references/evaluators-rag.md) |
| Avoid common mistakes | [common-mistakes-python](references/common-mistakes-python.md), [fundamentals-anti-patterns](references/fundamentals-anti-patterns.md) |
| Production | [production-overview](references/production-overview.md), [production-guardrails](references/production-guardrails.md), [production-continuous](references/production-continuous.md) |
## Workflows
**Starting Fresh:**
[observe-tracing-setup](references/observe-tracing-setup.md) → [error-analysis](references/error-analysis.md) → [axial-coding](references/axial-coding.md) → [evaluators-overview](references/evaluators-overview.md)
**Building Evaluator:**
[fundamentals](references/fundamentals.md) → [common-mistakes-python](references/common-mistakes-python.md) → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
**RAG Systems:**
[evaluators-rag](references/evaluators-rag.md) → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
**Running Experiments:**
Read experiments-datasets-{python|typescript} and experiments-running-{python|typescript} together. Map local rows to dataset input/output fields, then match the task and evaluator parameter names to those fields. For small datasets with LLM tasks or judges, use repetitions to compare runs with less sampling noise.
**Gating CI:**
evaluators-{code|llm}-{python|typescript} → integrations-{pytest|vitest-jest} → [production-continuous](references/production-continuous.md)
**Production:**
[production-overview](references/production-overview.md) → [production-guardrails](references/production-guardrails.md) → [production-continuous](references/production-continuous.md)
## Reference Categories
| Prefix | Description |
| ------ | ----------- |
| `fundamentals-*` | Types, scores, anti-patterns |
| `observe-*` | Tracing, sampling |
| `error-analysis-*` | Finding failures |
| `axial-coding-*` | Categorizing failures |
| `evaluators-*` | Code, LLM, RAG evaluators |
| `experiments-*` | Datasets, running experiments |
| `integrations-*` | Run evals from test runners (pytest, Vitest, Jest) as a CI gate |
| `validation-*` | Validating evaluator accuracy against human labels |
| `production-*` | CI/CD, monitoring |
## Key Principles
| Principle | Action |
| --------- | ------ |
| Error analysis first | Can't automate what you haven't observed |
| Custom > generic | Build from your failures |
| Code first | Deterministic before LLM |
| Validate judges | >80% accuracy, >70% TPR/TNR |
| Binary > Likert | Pass/fail, not 1-5 |
| Invariants gate, signals trend | `assert`/`expect` hard invariants (CI red); log LLM-judge quality signals and gate the aggregate (acceptance criteria), not every case |