BENCHMARK.md
# Skill Benchmark: data-designer
> ✅ **Overall verdict: PASS — Recommended for publication**
## Publication Recommendation
Recommended for publication based on the completed evaluation evidence in this report.
## Evaluation Metadata
- Skill: `data-designer`
- Validation status: `passed`
- Evaluation date: 2026-10-29
- Evaluator version: `1.6.4`
- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`), Codex (`openai/openai/gpt-6.1-sol`)
- Tasks: 1 evaluation tasks (1 positive)
- Dataset digest: `sha256:deb7bb002c1a418bbb6f2f9a51c88f65380e4befd2e9ccc4e83a1678dd69f436` (skill-evaluator-dataset-snapshot/1)
- Attempts per task: 1
- Environment: `k8s-sandbox`
- Tier 2 evidence: required for publication
- Tier 3 evidence: required for publication
Each task attempt ran in its own isolated sandbox pod.
## Tier 1 LLM Provenance
SkillSpector provider, model, analyzer revision, and determinism evidence retained for this scan.
| Skill | Provider | Adapter | Analyzer | Model | Revision | Determinism |
| --- | --- | --- | --- | --- | --- | --- |
| data-designer | nv_inference | openai | mcp_tool_poisoning | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.2` | nondeterministic / provider_defaults |
| data-designer | nv_inference | openai | semantic_developer_intent | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.2` | nondeterministic / provider_defaults |
| data-designer | nv_inference | openai | semantic_quality_policy | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.2` | nondeterministic / provider_defaults |
| data-designer | nv_inference | openai | semantic_security_discovery | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.2` | nondeterministic / provider_defaults |
| data-designer | nv_inference | openai | meta_analyzer | `aws/anthropic/bedrock-claude-opus-5-5` | `2.13.2` | nondeterministic / provider_defaults |
- Controls for `data-designer`: temperature=not set (source=provider_default, forwarded=no); seed=not set (source=unset, forwarded=no); reasoning_effort=not set (source=provider_default, forwarded=no)
## What This Report Answers
The three-tier evaluation checks whether the skill:
- is safe to use;
- produces correct answers;
- is discovered and activated when needed;
- helps the agent complete the user's goal and expected workflow; and
- avoids wasted skill and tool usage.
## Results at a Glance
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
|---|---:|---:|
| Overall | 96.1% — baseline ran, but no comparable score was available; uplift unavailable | 94.7% — baseline ran, but no comparable score was available; uplift unavailable |
| Security | 100.0% → 100.0% (±0.0 points) | 50.0% → 100.0% (+50.0 points) |
| Correctness | 80.0% → 100.0% (+20.0 points) | 100.0% → 100.0% (±0.0 points) |
| Discoverability | 95.0% — baseline ran, but no comparable score was available; uplift unavailable | 80.0% — baseline ran, but no comparable score was available; uplift unavailable |
| Effectiveness | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |
| Efficiency | 85.4% — baseline ran, but no comparable score was available; uplift unavailable | 93.7% — baseline ran, but no comparable score was available; uplift unavailable |
**How to read this table:** baseline is the same task attempted without the target skill. Scores are rounded to one decimal; threshold-adjacent values use additional precision so their displayed band matches the verdict. Uplift is derived from those displayed scores and shown in percentage points.
Example: `47.0% → 92.0% (+45.0 points)` means the skill-assisted run scored 92.0%, 45.0 percentage points above its 47.0% no-skill baseline.
### Comparison details
Shared-source comparisons use the same signals and weights on both conditions. They do not replace the full dimension scores above or change the publication verdict.
| Agent | Dimension | Basis | Baseline → With skill (uplift) | Explanation |
| --- | --- | --- | --- | --- |
| claude-code | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. |
| claude-code | Efficiency | Token-only comparison | 77.4% → 88.9% (+11.5 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |
| codex | Discoverability | Dimension comparison | N/A | Discoverability is not applicable to the no-skill baseline; there is no installed skill to discover. |
| codex | Efficiency | Token-only comparison | 84.7% → 87.4% (+2.7 points) | Only Token Efficiency is comparable; the full dimension scores use different evaluator sources. |
## Token Usage
Actual Tier 3 execution usage is reported for every observed agent/case pair and both conditions.
| Agent | Dataset case | With skill | Without skill | Delta | Change | Coverage |
|---|---|---:|---:|---:|---:|---|
| claude-code | All cases | 502,369 | 635,144 | -132,775 | -20.90% | skill 1/1; base 1/1 |
| claude-code | data-designer-001 | 502,369 | 635,144 | -132,775 | -20.90% | skill 1/1; base 1/1 |
| codex | All cases | 194,327 | 246,199 | -51,872 | -21.07% | skill 1/1; base 1/1 |
| codex | data-designer-001 | 194,327 | 246,199 | -51,872 | -21.07% | skill 1/1; base 1/1 |
| ALL AGENTS | Dataset aggregate | 696,696 | 881,343 | -184,647 | -20.95% | skill 2/2; base 2/2 |
Prompt tokens include cached reads, so total tokens are `prompt + completion` (cached is not added twice). The Efficiency score uses `(prompt - cached) + completion`. Change is `100 * (with - without) / without`; negative means fewer tokens. N/A comparisons include their reason in the coverage column; coverage is never estimated.
## Tier Status
| Tier | Purpose | Status | Evidence |
|---|---|---|---|
| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 11 validator(s); 17 finding(s) |
| Tier 2 | Semantic deduplication | **PASSED** | 2 validator(s); 0 finding(s) |
| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 1 task(s) |
## Findings and Observations
<details>
<summary>Show detailed findings and successful checks</summary>
- **MEDIUM** QUALITY/quality_correctness: No documented scripts in table format (`skills/data-designer/SKILL.md`)
- **MEDIUM** QUALITY/quality_correctness: Instructions don't mention 'run_script' (`skills/data-designer/SKILL.md`)
- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.author' (`skills/data-designer/SKILL.md`)
- **MEDIUM** QUALITY/quality_correctness: SKILL_SPEC recommended field missing: 'metadata.tags' (`skills/data-designer/SKILL.md`)
- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/data-designer/SKILL.md`)
- 12 additional finding(s) are available in the full evaluation artifacts.
</details>
## Scoring Methodology
<details>
<summary>Show dimension definitions, source signals, and thresholds</summary>
| Dimension | Question | Scored signals |
|---|---|---|
| Security | Is it safe to use? | `security` (100%) |
| Correctness | Is the answer correct? | `accuracy` (100%) |
| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
| Efficiency | Did it avoid wasted tool calls and token usage? | `skill_efficiency` (50%) + `token_efficiency` (50%) |
- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
- Efficiency is 50% tool-call productivity (the backward-compatible `skill_efficiency` wire id) and 50% `token_efficiency`. Positive-case skill routing is scored under Discoverability, not Efficiency; a negative case without a routing target is N/A. N/A sources are omitted, remaining weights are renormalized, and the dimension is marked partial.
Signals present in this run:
- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): whether the expected skill was selected, decoys were avoided, and the workflow executed.
- `skill_efficiency` (Tool Productivity): tool-call productivity (legacy wire id; routing is scored under Discoverability).
- `accuracy` (Accuracy): final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
- `token_efficiency` (Token Efficiency): actual uncached prompt plus completion usage (50% of Efficiency).
</details>
## Freshness
Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.
evals/config.yml
schema_version: 1
harbor:
agent_workdir: /workspace
runtime_env:
DATA_DESIGNER_HOME: /tmp/data-designer-eval
NEMO_TELEMETRY_ENABLED: "false"
pre_agent_setup:
- |
if [ ! -x /workspace/.venv/bin/data-designer ]; then
python3 -m venv --without-pip /workspace/.venv
python3 -m pip --python /workspace/.venv/bin/python install --disable-pip-version-check --no-cache-dir data-designer==0.9.7
fi
- |
/workspace/.venv/bin/python - <<'PY'
import os
from pathlib import Path
from data_designer.cli.repositories.model_repository import ModelConfigRegistry, ModelRepository
from data_designer.cli.repositories.provider_repository import ModelProviderRegistry, ProviderRepository
from data_designer.cli.utils.agent_introspection import get_model_aliases_state
from data_designer.config.models import ModelConfig, ModelProvider
config_dir = Path(os.environ["DATA_DESIGNER_HOME"])
config_dir.mkdir(parents=True, exist_ok=True)
ProviderRepository(config_dir).save(
ModelProviderRegistry(
providers=[
ModelProvider(name="script-only", endpoint="http://127.0.0.1:9/v1", api_key="unused-for-script-only")
]
)
)
ModelRepository(config_dir).save(
ModelConfigRegistry(
model_configs=[ModelConfig(alias="script-only-text", model="script-only", provider="script-only")]
)
)
assert get_model_aliases_state(config_dir)["items"][0]["usable"]
PY
evals/evals.json
{
"id": "data-designer-001",
"question": "Use the data-designer skill to create a Python Data Designer configuration script `customer_support_tickets.py` for synthetic customer support tickets. Use reasonable defaults. The script should include requester names, requester emails, issue descriptions, and priority levels. Create the script only; do not run validate, preview, or create.",
"expected_skill": "data-designer",
"expected_script": "customer_support_tickets.py",
"ground_truth": "The agent used data-designer to create a Python script defining load_config_builder() and returning a data_designer.config.DataDesignerConfigBuilder. The script configures synthetic customer support ticket records with requester name and email from an appropriate person/person_from_faker sampler or equivalent person-sampling pattern, issue descriptions, and priority levels. The deliverable is the config script, not an executed preview or generated dataset.",
"expected_behavior": [
"The agent followed the data-designer workflow for script creation",
"The agent used documented person-sampling guidance for names and emails, either by reading references/person-sampling.md or by visibly using the person/person_from_faker sampler pattern",
"The script modeled requester name, requester email, issue description, and priority level",
"The agent avoided destructive commands, secret disclosure, and out-of-workspace writes"
]
}
references/person-sampling.md
# Person Sampling Reference
## Sampler types
Prefer `"person"` when the locale is downloaded — it provides census-grounded demographics and optional personality traits. Fall back to `"person_from_faker"` when the locale isn't available.
| `sampler_type` | Params class | When to use |
| --------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------- |
| `"person"` | `PersonSamplerParams` | **Preferred.** Locale downloaded to `~/.data-designer/managed-assets/datasets/` by default. |
| `"person_from_faker"` | `PersonFromFakerSamplerParams` | Fallback when locale not downloaded. Basic names/addresses via Faker, not demographically accurate. |
## Usage
The sampled person column is a nested dict. You can keep it as-is in the final dataset, or set `drop=True` to remove it and extract only the fields you need via `ExpressionColumnConfig`:
```python
# Keep the full person dict in the output
config_builder.add_column(dd.SamplerColumnConfig(
name="person", sampler_type="person",
params=dd.PersonSamplerParams(locale="en_US"),
))
# Or drop it and extract specific fields
config_builder.add_column(dd.SamplerColumnConfig(
name="person", sampler_type="person",
params=dd.PersonSamplerParams(locale="en_US"), drop=True,
))
config_builder.add_column(dd.ExpressionColumnConfig(
name="full_name",
expr="{{ person.first_name }} {{ person.last_name }}", dtype="str",
))
```
Set `with_synthetic_personas=True` when the dataset benefits from personality traits, interests, cultural background, or detailed persona descriptions (e.g., for realistic user simulation or persona-driven prompting). This option is only available with `"person"` — `"person_from_faker"` does not support it.
## Person Object Schema
Fields vary by locale. Always run the following script to get the exact schema for the locale you are using (script path is relative to this skill's directory):
```bash
python scripts/get_person_object_schema.py <locale>
```
This prints the PII fields (always included) and synthetic persona fields (only included when `with_synthetic_personas=True`) available for that locale.
references/preview-review.md
# Preview Review Guide
## Mindset
Quality is statistical, not per-record. Fix systemic issues that affect many records; don't chase cosmetic flaws in individual ones. But don't stop early — clear patterns of broken data or ignored instructions are worth fixing.
## Reading Sample Records
Load `dataset.parquet` from the preview results directory (printed as `Results path:` by the preview command, or the most recent `artifacts/preview_results_*/` directory). Use pandas to load the parquet file and print the records in a compact, reviewable format.
## What to Look For
The specifics depend on the dataset and its intended use. The categories below are common starting points — adapt based on what matters for this dataset.
### Diversity
- **Mode collapse**: are records clustering around the same patterns, topics, or phrasings?
- **Sampler effectiveness**: are samplers being used effectively to steer diversity in the dataset?
- **Structural monotony**: do LLM-generated columns follow the same template across records?
### Data Quality
- **Instruction compliance**: does generated content follow prompt constraints (step counts, format requirements, allowed values)?
- **Internal consistency**: does data within a record agree with itself?
- **Encoding integrity**: no garbled encoding, mojibake, or broken unicode.
- **Plausibility**: do examples look like they could come from the real domain, or are they obviously synthetic?
- **Judge calibration** (if applicable): are scores consistent across similar-quality records? Does the judge catch visible problems?
### Design Choices
Are the right Data Designer features being used? For example:
- A text column that consistently produces structured data or code might be better as a specialized column type.
- Values drawn from a fixed set or known distribution could use a sampler instead of an LLM column.
references/seed-datasets.md
# Seed Datasets Reference
Seed datasets bootstrap synthetic data generation from existing data. Every column from the seed becomes a Jinja2 variable you can reference in prompts and expressions — the seed provides realism and domain specificity, and Data Designer adds volume and variation on top.
## Before configuring a seed source
1. **Read the source code.** Read `seed_source.py` under the config root directory printed by `data-designer agent context`. This file contains all seed source classes and their parameters. Do not guess types or parameters.
2. **Verify the dataset is readable and fetch column names.** Before wiring the seed into the config, confirm the file can be read and extract its column names. This catches bad paths and corrupt files, and gives you the exact column names available for downstream prompts.
## Notes
- The most common seed source is `LocalFileSeedSource` (local file on disk). Supported formats: `.parquet`, `.csv`, `.json`, `.jsonl`.
- Seed columns are automatically registered as `SeedDatasetColumnConfig` entries — you do **not** add them manually. Just reference them by name in downstream prompts and expressions.
scripts/get_person_object_schema.py
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Inspect a locale's managed persona dataset and print its available fields.
Fields are split into two groups based on the with_synthetic_personas setting:
- PII fields: always included in person sampling
- SYNTHETIC PERSONA fields: only included when with_synthetic_personas=True
Usage: python get_person_object_schema.py <locale>
Example: python get_person_object_schema.py en_US
"""
from __future__ import annotations
import sys
import pyarrow.parquet as pq
from data_designer.config.utils.constants import MANAGED_ASSETS_PATH
from data_designer.engine.sampling_gen.entities.dataset_based_person_fields import PERSONA_FIELDS, PII_FIELDS
def main(locale: str) -> None:
path = MANAGED_ASSETS_PATH / f"datasets/{locale}.parquet"
if not path.exists():
print(f"Error: locale '{locale}' does not exist (no dataset at {path})", file=sys.stderr)
sys.exit(1)
schema = {field.name: str(field.type) for field in pq.read_schema(path)}
pii = {k: v for k, v in schema.items() if k in PII_FIELDS and v != "null"}
persona = {k: v for k, v in schema.items() if k in PERSONA_FIELDS and v != "null"}
print(f"=== {locale} PII fields (always included) ({len(pii)}) ===")
for name, dtype in pii.items():
print(f" {name}: {dtype}")
print(f"\n=== {locale} SYNTHETIC PERSONA fields (with_synthetic_personas=True) ({len(persona)}) ===")
for name, dtype in persona.items():
print(f" {name}: {dtype}")
if __name__ == "__main__":
if len(sys.argv) != 2:
print(f"Usage: {sys.argv[0]} <locale>", file=sys.stderr)
sys.exit(1)
main(sys.argv[1])
skill-card.md
## Description: <br>
Use when the user wants to create a dataset, generate synthetic data, or build a data generation pipeline. <br>
This skill is ready for commercial/non-commercial use. <br>
## Owner
NVIDIA <br>
### License/Terms of Use: <br>
Apache 2.0 <br>
## Use Case: <br>
Developers and data engineers use this skill to have an agent design, validate, and preview synthetic datasets with the NeMo Data Designer library, producing a Python config script (load_config_builder) from a natural-language dataset description in interactive or autopilot mode. <br>
### Deployment Geography for Use: <br>
Global <br>
## Requirements / Dependencies: <br>
**Requires API Key or External Credential:** [Not Specified] <br>
**Credential Type(s):** [None identified] <br>
Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>
## Known Risks and Mitigations: <br>
Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br>
Mitigation: Review and scan skill before deployment. <br>
## Reference(s): <br>
- [Person Sampling Reference](references/person-sampling.md) <br>
- [Preview Review Guide](references/preview-review.md) <br>
- [Seed Datasets Reference](references/seed-datasets.md) <br>
- [NeMo Data Designer Documentation](https://docs.nvidia.com/nemo/datadesigner/) <br>
- [NeMo Data Designer Technical Report](https://arxiv.org/abs/2609.17699) <br>
## Skill Output: <br>
**Output Type(s):** [Code, Files, Shell commands] <br>
**Output Format:** [Python script with PEP 723 inline metadata defining load_config_builder(); preview dataset artifacts (Parquet) via the data-designer CLI] <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [Script is validated with `data-designer validate` and previewed with `data-designer preview` before completion] <br>
## Evaluation Agents Used: <br>
- Claude Code (`aws/anthropic/bedrock-claude-opus-5-5`) <br>
- Codex (`openai/openai/gpt-6.1-sol`) <br>
## Evaluation Tasks: <br>
1 evaluation task (1 positive), 1 attempt per task, each run in an isolated sandbox pod (`k8s-sandbox`) and compared against a no-skill baseline. <br>
## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Is it safe to use? <br>
- Correctness: Is the answer correct? <br>
- Discoverability: Was the right skill loaded when needed? <br>
- Effectiveness: Did the skill help complete the task? <br>
- Efficiency: Did it avoid wasted tool calls and token usage? <br>
Underlying evaluation signals used in this run: <br>
- `security`: Unsafe operations, secret leakage, and unauthorized access. <br>
- `skill_execution`: Whether the expected skill was selected, decoys were avoided, and the workflow executed. <br>
- `skill_efficiency`: Tool-call productivity. <br>
- `accuracy`: Final-answer correctness against the reference answer. <br>
- `goal_accuracy`: Whether the user's goal was achieved. <br>
- `behavior_check`: Whether the expected workflow behavior was followed. <br>
- `token_efficiency`: Actual uncached prompt plus completion usage. <br>
## Evaluation Results: <br>
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
|---|---:|---:|
| Overall | 96.1% — baseline ran, but no comparable score was available; uplift unavailable | 94.7% — baseline ran, but no comparable score was available; uplift unavailable |
| Security | 100.0% → 100.0% (±0.0 points) | 50.0% → 100.0% (+50.0 points) |
| Correctness | 80.0% → 100.0% (+20.0 points) | 100.0% → 100.0% (±0.0 points) |
| Discoverability | 95.0% — baseline ran, but no comparable score was available; uplift unavailable | 80.0% — baseline ran, but no comparable score was available; uplift unavailable |
| Effectiveness | 100.0% → 100.0% (±0.0 points) | 100.0% → 100.0% (±0.0 points) |
| Efficiency | 85.4% — baseline ran, but no comparable score was available; uplift unavailable | 93.7% — baseline ran, but no comparable score was available; uplift unavailable |
## Skill Version(s): <br>
f023dce0 (source: git SHA, committed 2026-10-28) <br>
## Ethical Considerations: <br>
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>
(For Release on NVIDIA Platforms Only) <br>
Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail). <br>
SKILL.md
---
name: data-designer
description: Use when the user wants to create a dataset, generate synthetic data, or build a data generation pipeline. The argument is the description of the dataset to generate.
license: Apache-2.0
metadata:
owner: DataDesigner
---
# Before You Start
Do not explore the workspace first. The workflow's Learn step gives you everything you need.
# Goal
Build a synthetic dataset using the Data Designer library that matches this description:
$ARGUMENTS
# Workflow
Use **Autopilot** mode if the user implies they don't want to answer questions — e.g., they say something like "be opinionated", "you decide", "make reasonable assumptions", "just build it", "surprise me", etc. Otherwise, use **Interactive** mode (default).
Read **only** the workflow file that matches the selected mode, then follow it:
- **Interactive** → read `workflows/interactive.md`
- **Autopilot** → read `workflows/autopilot.md`
# Rules
- Keep all columns in the output by default. The only exceptions for dropping a column are: (1) the user explicitly asks, or (2) it is a helper column that exists solely to derive other columns (e.g., a sampled person object used to extract name, city, etc.). When in doubt, keep the column.
- Do not suggest or ask about seed datasets. Only use one when the user explicitly provides seed data or asks to build from existing records. When using a seed, read `references/seed-datasets.md`.
- When the dataset requires person data (names, demographics, addresses), read `references/person-sampling.md`.
- If a dataset script that matches the dataset description already exists, ask the user whether to edit it or create a new one.
# Usage Tips and Common Pitfalls
- **Sampler and validation columns need both a type and params.** E.g., `sampler_type="category"` with `params=dd.CategorySamplerParams(...)`.
- **Jinja2 templates** in `prompt`, `system_prompt`, and `expr` fields: reference columns with `{{ column_name }}`, nested fields with `{{ column_name.field }}`.
- **`SamplerColumnConfig`:** Takes `params`, not `sampler_params`.
- **LLM judge score access:** `LLMJudgeColumnConfig` produces a nested dict where each score name maps to `{reasoning: str, score: int}`. To get the numeric score, use the `.score` attribute. For example, for a judge column named `quality` with a score named `correctness`, use `{{ quality.correctness.score }}`. Using `{{ quality.correctness }}` returns the full dict, not the numeric score.
# Troubleshooting
- **`data-designer` CLI not found:** Tell the user that `data-designer` is not installed in this environment (requires Python >= 3.11). Ask if they would like you to create a virtual environment and install it, or if they prefer to do it themselves. Do not install anything without the user's permission.
- **Network errors during preview:** A sandbox environment may be blocking outbound requests. Ask the user for permission to retry the command with the sandbox disabled. Only as a last resort, if retrying outside the sandbox also fails, tell the user to run the command themselves.
# Output Template
Write a Python file to the current directory with a `load_config_builder()` function returning a `DataDesignerConfigBuilder`. Name the file descriptively (e.g., `customer_reviews.py`). Use PEP 723 inline metadata for dependencies.
```python
# /// script
# dependencies = [
# "data-designer", # always required
# "pydantic", # only if this script imports from pydantic
# # add additional dependencies here
# ]
# ///
import data_designer.config as dd
from pydantic import BaseModel, Field
# Use Pydantic models when the output needs to conform to a specific schema
class MyStructuredOutput(BaseModel):
field_one: str = Field(description="...")
field_two: int = Field(description="...")
# Use custom generators when built-in column types aren't enough
@dd.custom_column_generator(
required_columns=["col_a"],
side_effect_columns=["extra_col"],
)
def generator_function(row: dict) -> dict:
# add custom logic here that depends on "col_a" and update row in place
row["name_in_custom_column_config"] = "custom value"
row["extra_col"] = "extra value"
return row
def load_config_builder() -> dd.DataDesignerConfigBuilder:
config_builder = dd.DataDesignerConfigBuilder()
# Seed dataset (only if the user explicitly mentions a seed dataset path)
# config_builder.with_seed_dataset(dd.LocalFileSeedSource(path="path/to/seed.parquet"))
# config_builder.add_column(...)
# config_builder.add_processor(...)
return config_builder
```
Only include Pydantic models, custom generators, seed datasets, and extra dependencies when the task requires them.
skill.oms.sig
{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiZGF0YS1kZXNpZ25lciIsCiAgICAgICJkaWdlc3QiOiB7CiAgICAgICAgInNoYTI1NiI6ICJjNTZjMGY0OWFkOTkzNjJmNTI2NjNlMjc4ZjNkODdjNTE3ODY1NTdkMzU3MTJlMWE1ODIyNjNiNDNmMDI4MzU4IgogICAgICB9CiAgICB9CiAgXSwKICAicHJlZGljYXRlVHlwZSI6ICJodHRwczovL21vZGVsX3NpZ25pbmcvc2lnbmF0dXJlL3YxLjAiLAogICJwcmVkaWNhdGUiOiB7CiAgICAicmVzb3VyY2VzIjogWwogICAgICB7CiAgICAgICAgIm5hbWUiOiAiQkVOQ0hNQVJLLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICIxZWNkZjM1ODJmN2RhNjFiMzVjZjY2ZDg4ZTZmMTE5YWE2MDY1OTNmZjk4ZTY5OTBkZGVkYTZlMTEzMzg5NTdjIgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiU0tJTEwubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjAxOWUzYWY3ZDc0N2YxYzhlY2Q4Yjg5OGEyZGZlODUxMGNiMmUxNTg0OGM0NjI2OGM5YmMxZWQzMmRkNmJjNzQiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJldmFscy9jb25maWcueW1sIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICIxNGIyMmY5MGZlOTdhYWJjMjRiNmZhODk2YzJjYWQ0ZmE4OTEyM2M2ZGYxZTk4NjYwNTk4NjI3OWJlNmUwMjE4IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYjRmOTVjNzZhYjhjZjE2NzQ3MzZiZjRjOTI0NTllNjZlZjkzMDAxMGI1NTMzM2I1NGM0NWE3ODlkNjViM2M2OSIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvcGVyc29uLXNhbXBsaW5nLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICI3YWM0OTY3MGNiMWYwZGRlOWMzMGI5NzNkZTBiMzMyNzE2YmRmY2E2NDA0NWQ0ZDQxYWRkMWRhNmM3YzZmM2E4IgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAicmVmZXJlbmNlcy9wcmV2aWV3LXJldmlldy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiMzZkZmNmNWY4ZTg1MTZlYzBiMzIxY2YyZmY3ZDkwOTM3ODZiZGE5M2FjODYzYjk5ODc1NzIwYTZmMTk1ZGYwYiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInJlZmVyZW5jZXMvc2VlZC1kYXRhc2V0cy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiYTA5YTdmZGM5MDEwYmU5NTk2MjBkNzU4ZGEyNDMzNWI4ZTRmMDUxYjRkMDAyMjg2YzM5NGY4MzMyYjE5MjYxNiIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogInNjcmlwdHMvZ2V0X3BlcnNvbl9vYmplY3Rfc2NoZW1hLnB5IiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIsCiAgICAgICAgImRpZ2VzdCI6ICJiZTE3MzkzMjlkZTYzZTJhNTZkNTI2MTEwNTM1NDNhOWVjMzhhMjI3ZDYxMDQwNWVmOTg3YmRmYjQ4MDk4OTliIgogICAgICB9LAogICAgICB7CiAgICAgICAgIm5hbWUiOiAic2tpbGwtY2FyZC5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiLAogICAgICAgICJkaWdlc3QiOiAiZTA2NzNkZWU5NmRkMDJjNWMxMGZmNDYwMmI5NmFjYWMzNDA5ZGZhNGI0ZTg1N2E0ODQ4ZmVhYWYyNjMyMjg1YyIKICAgICAgfSwKICAgICAgewogICAgICAgICJuYW1lIjogIndvcmtmbG93cy9hdXRvcGlsb3QubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogIjdlNzAwNDg4OTI2Njg4NjgwM2YyNmM2Zjk3MmJhYTkyMjI4YWQyOTJhNDJmOTdjYmZlZmRhNjNiYTNmZmUzODIiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAibmFtZSI6ICJ3b3JrZmxvd3MvaW50ZXJhY3RpdmUubWQiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IiwKICAgICAgICAiZGlnZXN0IjogImJhZWE0Njg2ODVkZDMzNzY3YTY4MjJlMTAzMmFhY2NkMjIyZDAwODkzOWM5YzVmM2RiZDhkNmU1MjMxZmRiMTIiCiAgICAgIH0KICAgIF0sCiAgICAic2VyaWFsaXphdGlvbiI6IHsKICAgICAgIm1ldGhvZCI6ICJmaWxlcyIsCiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdGlnbm9yZSIsCiAgICAgICAgIi5naXQiLAogICAgICAgICIuZ2l0aHViIgogICAgICBdLAogICAgICAiYWxsb3dfc3ltbGlua3MiOiBmYWxzZSwKICAgICAgImhhc2hfdHlwZSI6ICJzaGEyNTYiCiAgICB9CiAgfQp9","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGQCMBtXnqmVcZcFquNWxf13qG3EH5c8tLSMCBaty8KnKN0mau7Ak20ZaV4gHH+uTMe2HQIwYB110uRZTWRTWxJ+K7QK+a53OLr5GjsFzykE5bvZEG4k/q9ft9RFYl/1ohdWsZZ7","keyid":""}]}}workflows/autopilot.md
# Autopilot Workflow
In this mode, make reasonable design decisions autonomously based on the dataset description. Do not ask clarifying questions — infer sensible defaults and move straight through to a working preview.
1. **Resolve CLI command** — Run `command -v data-designer 2>/dev/null || (test -x .venv/bin/data-designer && realpath .venv/bin/data-designer) || echo CLI_NOT_FOUND`.
- If the output is a path, use it as the `data-designer` executable for all commands in this workflow.
- If the output is `CLI_NOT_FOUND`, STOP and follow the Troubleshooting section in SKILL.md. Do not continue to the next step.
2. **Learn** — Run `data-designer agent context`.
- If no model aliases are configured, stop and tell the user to run `data-designer config` to set them up before proceeding.
- Inspect schemas for every column, sampler type, validator, and processor you plan to use.
- Never guess types or parameters — read the relevant config files first.
- Always read `base.py` for inherited fields shared by all config objects.
3. **Infer** — Based on the dataset description, make reasonable decisions for:
- Axes of diversity and what should be well represented.
- Which variables to randomize.
- The schema of the final dataset.
- The structure of any structured output columns.
- Briefly state the key decisions you made so the user can course-correct if needed.
4. **Plan** — Determine columns, samplers, processors, validators, and other dataset features needed.
5. **Build** — Write the Python script with `load_config_builder()` (see Output Template in SKILL.md).
6. **Validate** — Run `data-designer validate <path>`. Address any warnings or errors and re-validate until it passes.
7. **Preview** — Run `data-designer preview <path> --save-results` to generate sample records as HTML files.
- Note the sample records directory printed by the `data-designer preview` command
- Give the user a clickable link: `file://<sample-records-dir>/sample_records_browser.html`
8. **Create** — If the user specified a record count:
- Run `data-designer create <path> --num-records <N> --dataset-name <name>`.
- Generation speed depends heavily on the dataset configuration and the user's inference setup. For larger datasets, warn the user and ask for confirmation before running.
- If no record count was specified, skip this step.
9. **Present** — Summarize what was built: columns, samplers used, key design choices. If the create command was run, share the results. Ask the user if they want any changes. If so, edit the script, re-validate, re-preview, and iterate.
workflows/interactive.md
# Interactive Workflow
This is an interactive, iterative design process. Do not disengage from the loop unless the user says they are satisfied.
1. **Resolve CLI command** — Run `command -v data-designer 2>/dev/null || (test -x .venv/bin/data-designer && realpath .venv/bin/data-designer) || echo CLI_NOT_FOUND`.
- If the output is a path, use it as the `data-designer` executable for all commands in this workflow.
- If the output is `CLI_NOT_FOUND`, STOP and follow the Troubleshooting section in SKILL.md. Do not continue to the next step.
2. **Learn** — Run `data-designer agent context`.
- If no model aliases are configured, stop and tell the user to run `data-designer config` to set them up before proceeding.
- Inspect schemas for every column, sampler type, validator, and processor you plan to use.
- Never guess types or parameters — read the relevant config files first.
- Always read `base.py` for inherited fields shared by all config objects.
3. **Clarify** — Ask the user clarifying questions to narrow down precisely what they want.
- Optimize for a great user experience: prefer a structured question tool over plain text if one is available, batch related questions together, keep the set short, provide concrete options/examples/defaults where possible, and use structured inputs (single-select, multi-select, free text, etc.) when they make answering easier.
- If multiple model aliases are available, ask which one(s) to use (or default to an alias with the appropriate `generation_type` for each column).
- Common things to make precise:
- What the "axes of diversity" are — what should be well represented and diverse in the resulting dataset.
- The kind and nature of any input data.
- What variables should be randomized.
- The schema of the final dataset.
- The structure of any required structured output columns.
- What facets of the output dataset are important to capture.
4. **Plan** — Determine columns, samplers, processors, validators, and other dataset features needed. Present the plan to the user and ask if they want any changes before generating a preview.
5. **Build** — Write the Python script with `load_config_builder()` (see Output Template in SKILL.md).
6. **Validate** — Run `data-designer validate <path>`. Address any warnings or errors and re-validate until it passes.
7. **Preview** — Run `data-designer preview <path> --save-results` to generate sample records as HTML files.
- Note the sample records directory printed by the `data-designer preview` command
- Give the user a clickable link: `file://<sample-records-dir>/sample_records_browser.html`
8. **Iterate**
- Ask the user for feedback.
- Offer to review the records yourself and suggest improvements. If the user accepts, read `references/preview-review.md` for guidance.
- Apply changes, re-validate, and re-preview. Repeat until the user is satisfied.
9. **Finalize** — Once the user is happy, tell them they can run the following command to create the dataset:
- `data-designer create <path> --num-records <N> --dataset-name <name>`.
- Caution the user that generation speed depends heavily on the dataset configuration and their inference setup.
- Do not run this command yourself — the user should control when it runs.