SKILL.md
---
name: paper-reproduction-assistant
description: Plan, execute, debug, validate, and report safe reproductions of computer science papers in artificial intelligence, machine learning, software engineering, systems, security, HCI, algorithms, and related fields. Use when working from a paper, repository, artifact package, dataset, model checkpoint, environment file, README, command log, result table, or benchmark to reproduce claims, reconstruct environments, audit code and data, create run manifests, track deviations, diagnose mismatches, compare reproduced results with reported results, or write replication and reproducibility reports.
---
# Paper Reproduction Assistant
Reproduce scientific claims through an auditable chain from source material to environment, execution, results, and interpretation. Treat “the code ran” and “the claim reproduced” as different outcomes.
## Route the task
Read only the references required:
- Read [references/reproduction-planning.md](references/reproduction-planning.md) to define target claims, reproduction type, success criteria, resources, and staged protocols.
- Read [references/artifact-and-security.md](references/artifact-and-security.md) before obtaining, inspecting, building, or executing third-party code, models, containers, datasets, or scripts.
- Read [references/environment-and-execution.md](references/environment-and-execution.md) when reconstructing environments, preparing data, running experiments, recording manifests, or controlling compute.
- Read [references/validation-and-comparison.md](references/validation-and-comparison.md) when checking intermediate outputs, comparing reported and reproduced numbers, quantifying uncertainty, or classifying outcomes.
- Read [references/diagnosis-and-reporting.md](references/diagnosis-and-reporting.md) when debugging failures, managing deviations, contacting authors, or writing a reproduction report.
## Define the reproduction contract
Before changing or running anything, record:
1. Paper identity and exact artifact versions.
2. Target claim, table cell, figure, theorem, system behavior, or qualitative finding.
3. Reproduction type: exact, computational, robustness, conceptual, independent reimplementation, or artifact verification.
4. Success criteria and acceptable tolerance.
5. Required data, checkpoints, hardware, software, credentials, licenses, and expected cost.
6. Allowed actions, execution environment, time and compute budget, and stop conditions.
7. Known missing information and planned substitutions.
Do not define success after seeing the result. Separate essential claim tests from optional extensions.
## Build a claim-to-artifact map
Trace each target result through:
```text
paper claim
→ table / figure / theorem / example
→ metric and aggregation
→ experiment or proof configuration
→ code entry point and commit
→ data / model / dependency versions
→ command and environment
→ raw output
→ processing and plotting
→ reported value
```
Mark every link as verified, inferred, missing, incompatible, or substituted. Do not assume the repository default branch matches the published paper.
## Inspect before execution
Treat external artifacts as untrusted. Read licenses, provenance, setup files, dependency locks, scripts, workflows, download commands, container definitions, and entry points before running them.
Identify network access, credential use, shell execution, destructive file operations, privileged operations, telemetry, arbitrary deserialization, post-install hooks, large downloads, and resource-intensive jobs. Use the least privilege and the narrowest writable workspace.
Never execute unreviewed code merely because the paper links it. Never expose local secrets or reuse production credentials for a reproduction.
## Preserve originals and versions
Keep source artifacts immutable. Record paper version, repository URL, commit hash, release tag, submodules, patch set, dataset and checkpoint identifiers, checksums when available, license, download date, and access constraints.
Place every required change in a visible patch or deviation log. Distinguish:
- compatibility fix;
- bug fix;
- parameter recovery;
- environment substitution;
- dataset or model substitution;
- scientific protocol change;
- exploratory extension.
Do not silently “fix” code until it produces the published result.
## Reconstruct in stages
Use the smallest informative sequence:
1. static artifact inspection;
2. environment resolution without running the experiment;
3. import, build, or unit smoke test;
4. tiny synthetic or documented example;
5. one real sample or short run;
6. one baseline and one target condition;
7. complete target result;
8. repeated runs and uncertainty;
9. optional robustness or extension.
Validate outputs at each stage. Stop when a safety boundary, license, access requirement, or resource budget is exceeded.
## Record every run
Use a unique run identifier and log:
- code commit and local patch hash;
- command, arguments, configuration, environment variables by name but not secret values;
- dependency and driver versions;
- data, split, preprocessing, model, and checkpoint versions;
- hardware, precision, seeds, locale, timezone, concurrency, and cache state;
- start/end time, exit status, stdout/stderr locations, artifact paths, and resource use;
- whether the run is planned, diagnostic, or exploratory.
Retain failed and partial runs. Do not overwrite results or reuse an output directory ambiguously.
## Validate before scaling
Check data counts, label mappings, splits, leakage, normalization, tensor shapes, randomization, checkpoint loading, metric direction, denominators, units, preprocessing, evaluation mode, and aggregation against the paper and code.
Reproduce intermediate values when available. Confirm the baseline before interpreting a target method. A matching final number can still be accidental if the intermediate pipeline differs.
## Compare evidence fairly
Use the paper's reported uncertainty and repetition policy when available. Otherwise define and disclose a defensible repetition and comparison method before full execution.
Report absolute and relative differences, uncertainty, run-to-run variation, missing and failed runs, and the tolerance rationale. Avoid declaring success from one favorable seed or failure from one unfavorable run.
Do not treat a reported standard deviation, standard error, or confidence-interval width as an automatic equivalence margin. If no defensible tolerance can be set before execution, use descriptive comparison and an inconclusive classification rather than inventing a binary threshold.
Compare only compatible datasets, splits, models, metrics, checkpoints, budgets, and hardware conditions. When substitutions are unavoidable, classify the result as partial or conceptual rather than exact.
## Diagnose without outcome chasing
Classify a mismatch before modifying the setup:
- acquisition or integrity failure;
- environment or build failure;
- data or checkpoint mismatch;
- undocumented parameter or preprocessing;
- nondeterminism or insufficient repetitions;
- hardware or numerical difference;
- implementation defect;
- reporting or aggregation ambiguity;
- evidence inconsistent with the reported claim.
Change one factor at a time, state the hypothesis, predict the diagnostic outcome, and log the result. Do not search hyperparameters against the published test result unless explicitly labeled as post hoc recovery.
## Classify the outcome precisely
Use one of these conclusions:
- reproduced within predeclared tolerance;
- partially reproduced;
- qualitatively reproduced but quantitatively different;
- not reproduced under matched conditions;
- not comparable because of substitutions;
- inconclusive because information, access, compute, or repetitions are insufficient;
- artifact could not be executed;
- claim was not tested.
Do not write “the paper is wrong” when the environment failed, the protocol differs, or critical information is missing.
## Deliver an auditable package
Provide the applicable subset of:
- reproduction contract and claim-to-artifact map;
- artifact inventory, provenance, licenses, checksums, and security findings;
- environment specification and setup log;
- run matrix, manifest, commands, and resource estimate;
- patch and deviation log;
- raw-output inventory and transformation path;
- reported-versus-reproduced comparison with uncertainty;
- diagnosis history and unresolved questions;
- outcome classification, limitations, and next actions;
- concise reproduction report suitable for a thesis, lab record, issue, or artifact review.
## Non-negotiable boundaries
- Never execute unreviewed third-party code with broad permissions or live secrets.
- Never bypass licenses, authentication, paywalls, access controls, or data-use restrictions.
- Never fabricate unavailable parameters, data, checkpoints, runs, logs, or reproduced values.
- Never silently modify code, data, metrics, exclusions, or evaluation conditions.
- Never cherry-pick seeds, checkpoints, subsets, or hyperparameters to force agreement.
- Never discard failed runs that affect reliability.
- Never claim exact reproduction after substituting essential data, models, or protocols.
- Never equate a setup failure with scientific non-reproducibility.
- Preserve source artifacts, provenance, uncertainty, deviations, and unresolved ambiguity.
README.md
# Paper Reproduction Assistant Skill
面向计算机科学、人工智能、计算机技术与软件工程研究生的论文复现 Skill。它帮助 Codex 和 Claude Code 从论文、代码仓库、artifact、数据集、模型、环境文件和结果表出发，建立目标主张、审计第三方代码、重建环境、分阶段运行、记录偏差、比较复现结果，并生成可审计的复现报告。
- Skill 名称：`paper-reproduction-assistant`
- GitHub 仓库：`paper-reproduction-assistant-skill`
## 为什么需要这个 Skill
科研复现不等于执行 README 中的一条命令。常见风险包括：
- 仓库默认分支和论文使用的提交不是同一个版本；
- 论文、代码、数据、模型和评测脚本之间无法建立对应关系；
- 未审查第三方安装脚本、容器或模型文件就直接执行；
- 将本机密钥、SSH 配置或云凭据暴露给不可信代码；
- 为了跑通而静默升级依赖、修改代码或替换数据；
- 没有先核验 baseline 和中间输出，就直接运行昂贵实验；
- 只选择最接近论文结果的种子、checkpoint 或超参数；
- 把一个偶然接近的最终数字当成完整复现；
- 把数据集、模型或协议替换后的结果称为“精确复现”；
- 把环境安装失败误写成“论文不可复现”；
- 把单次不一致结果直接写成“论文结论错误”；
- 不保留失败运行、命令、日志、补丁和偏差记录。
这个 Skill 把复现拆解为可审计的主张、artifact、环境、执行、验证和解释链条。
## 核心能力
| 能力 | 说明 |
| --- | --- |
| 复现契约 | 明确目标主张、复现类型、成功阈值、资源预算和停止条件 |
| Claim-to-artifact map | 从论文表图追踪到配置、代码、数据、命令、原始输出和处理脚本 |
| Artifact 清单 | 记录论文、仓库、提交、数据、模型、依赖、许可证和校验标识 |
| 安全审计 | 检查安装钩子、shell、容器、下载、反序列化、权限、网络和凭据风险 |
| 环境重建 | 固定 OS、驱动、运行时、依赖、容器、硬件和环境差异 |
| 分阶段执行 | 从静态检查、smoke test、单样本、baseline 到完整实验逐步放大 |
| Run manifest | 保存每次运行的命令、配置、版本、种子、环境、日志和输出 |
| 偏差管理 | 区分兼容性修复、bug 修复、参数恢复、科学替换与探索扩展 |
| 中间验证 | 检查数据数量、split、标签、预处理、checkpoint、指标和中间结果 |
| 结果比较 | 对齐设置后报告差值、不确定性、失败运行和预声明容差 |
| 系统诊断 | 按 acquisition、环境、数据、配置、数值、实现和报告问题分类 |
| 复现报告 | 精确区分已复现、部分复现、不一致、不可比较、不确定和未测试 |
## 支持的场景
- 复现 AI/ML 论文的主结果、消融实验、鲁棒性或效率；
- 检查论文开源仓库是否能在本机或容器中运行；
- 重建老旧 CUDA、Python、PyTorch 或系统依赖环境；
- 对齐数据集、模型 checkpoint、提示词、split 和评测脚本；
- 复现软件工程 benchmark、代码生成或程序分析实验；
- 复现数据库、网络、分布式系统和推理服务性能结果；
- 验证 artifact evaluation 包的安装、功能与论文主张；
- 对论文算法进行独立重新实现；
- 在原设置不可获得时执行概念复现并明确替换边界；
- 诊断复现结果与论文数字不一致的原因；
- 为毕业论文、实验室记录、GitHub issue 或复现论文编写报告。
## 适用的计算机研究方向
- 机器学习、深度学习、自然语言处理、计算机视觉和生成式 AI；
- 大语言模型、RAG、智能体、多模态与模型评测；
- 软件工程、程序分析、软件测试、代码生成和 benchmark；
- 数据库、分布式系统、网络、操作系统和云计算；
- 网络安全、隐私、可靠性和对抗性评测；
- 人机交互、用户研究和混合方法研究；
- 算法、优化、理论证明和形式化验证；
- 数据集、模型、工具、artifact 和负结果研究。
## 复现类型
| 类型 | 含义 |
| --- | --- |
| 精确复现 | 使用相同 artifact、版本、数据和关键条件 |
| 计算复现 | 重跑作者代码，允许已记录的平台差异 |
| Artifact 验证 | 检查安装、功能、文档和主张支持程度 |
| 鲁棒性复现 | 在额外种子或明确扰动下重复结果 |
| 概念复现 | 用实质不同的实现或条件测试同一科学主张 |
| 独立重实现 | 主要根据论文描述重新实现方法 |
复现类型必须在看到结果前确定，不能因为结果接近就把概念复现升级为精确复现。
## 工作流程
```text
论文、仓库或 artifact
→ 冻结目标主张、复现类型和成功标准
→ 建立 claim-to-artifact map 与资源预算
→ 审计代码、依赖、数据、模型、许可证和安全风险
→ 固定版本并建立环境、补丁和偏差记录
→ 按 smoke test → 单样本 → baseline → 目标条件分阶段运行
→ 为所有成功、失败和诊断运行建立 manifest
→ 核验数据、指标、中间输出与最终聚合
→ 按预声明容差比较论文结果和复现结果
→ 分类不一致原因并执行单因素诊断
→ 交付精确结论、限制和可审计复现包
```
## 平台兼容性
仓库采用通用 `SKILL.md` 和按需加载的 `references/`，可同时用于 Codex 和 Claude Code。
| 文件或目录 | Codex | Claude Code |
| --- | --- | --- |
| `SKILL.md` | 使用 | 使用 |
| `references/` | 使用 | 使用 |
| `agents/openai.yaml` | Codex/ChatGPT 界面元数据 | 可忽略，不影响执行 |
| `README.md` | 面向仓库使用者 | 面向仓库使用者 |
frontmatter 仅使用通用的 `name` 和 `description`。实际执行第三方代码时，仍受宿主平台的权限、安全策略和用户授权范围限制。
## 在 Codex 中安装和使用
### 个人级安装
Windows PowerShell：
```powershell
git clone https://github.com/wupa12/paper-reproduction-assistant-skill.git "$HOME\.agents\skills\paper-reproduction-assistant"
```
macOS 或 Linux：
```bash
git clone https://github.com/wupa12/paper-reproduction-assistant-skill.git ~/.agents/skills/paper-reproduction-assistant
```
### 项目级安装
```bash
git clone https://github.com/wupa12/paper-reproduction-assistant-skill.git .agents/skills/paper-reproduction-assistant
```
如果不希望形成嵌套 Git 仓库，可以下载源码并复制到 `.agents/skills/paper-reproduction-assistant/`。目标目录必须直接包含 `SKILL.md`。
### 在 Codex 中调用
显式调用：
```text
使用 $paper-reproduction-assistant，为这篇 RAG 论文建立复现计划。
先映射表 2 的主结果到仓库、数据、模型、配置和脚本，不要在审计代码前执行任何命令。
```
Codex CLI 或 IDE 扩展可运行 `/skills` 查看已发现的 Skills，也可输入 `$` 选择 `paper-reproduction-assistant`。
隐式调用：
```text
请检查这个论文仓库能否安全复现。先读取安装脚本、Dockerfile、下载代码和依赖，
列出权限、网络、凭据、反序列化和资源风险，然后设计最小 smoke test。
```
Codex 会根据 `description` 判断是否自动加载。若刚创建顶层 Skills 目录后没有出现，请重启 Codex，并确认目标目录根部直接包含 `SKILL.md`。
## 在 Claude Code 中安装和使用
### 个人级安装
Windows PowerShell：
```powershell
git clone https://github.com/wupa12/paper-reproduction-assistant-skill.git "$HOME\.claude\skills\paper-reproduction-assistant"
```
macOS 或 Linux：
```bash
git clone https://github.com/wupa12/paper-reproduction-assistant-skill.git ~/.claude/skills/paper-reproduction-assistant
```
### 项目级安装
```bash
git clone https://github.com/wupa12/paper-reproduction-assistant-skill.git .claude/skills/paper-reproduction-assistant
```
项目级目录可提交到版本控制供团队共享，并且必须直接包含 `SKILL.md`。
### 在 Claude Code 中调用
显式调用：
```text
/paper-reproduction-assistant 审计当前仓库，并为论文 Figure 4 建立 claim-to-artifact map、
环境计划、分阶段命令、run manifest 字段和成功标准。未经确认不要启动完整训练。
```
隐式调用：
```text
这次实验使用了不同的数据集和更大的模型，但结果方向与原论文相同。
请判断能否称为复现，并生成准确的结果分类和限制说明。
```
可以询问 Claude Code 当前有哪些 Skills，或直接输入 `/paper-reproduction-assistant` 验证。Claude Code 会监听已有目录中的 `SKILL.md` 更新；如果会话启动时顶层 Skills 目录不存在，创建后需要重启。
> 本机 `~/.claude/skills/` 中的 Skill 不会自动进入远程云会话。远程使用时，应将 Skill 提交到项目 `.claude/skills/`，或通过账户级 Skill/插件启用。
## 更新和卸载
进入安装目录后更新：
```bash
git pull --ff-only
```
卸载时只删除对应目录：
```text
~/.agents/skills/paper-reproduction-assistant
~/.claude/skills/paper-reproduction-assistant
```
项目级安装同理，删除 `.agents/skills/paper-reproduction-assistant` 或 `.claude/skills/paper-reproduction-assistant`。删除前确认准确路径，避免误删整个 Skills 目录或用户实验产物。
## 使用示例
### 制定复现计划
```text
使用 $paper-reproduction-assistant，目标是复现论文 Table 3 中方法 A 在 Dataset X 上的 accuracy。
建立复现契约、成功容差、claim-to-artifact map、资源估算和最小运行矩阵，列出缺失参数。
```
### 审计第三方代码
```text
/paper-reproduction-assistant 检查 setup.py、pyproject.toml、requirements.txt、Dockerfile、
shell 脚本、模型下载和评测入口。不要执行，先报告 install hook、凭据、网络、删除、挂载和反序列化风险。
```
### 重建环境
```text
使用 $paper-reproduction-assistant，根据论文年份、仓库提交、锁文件和 release notes 重建环境。
区分作者明确给出的版本与推断版本；所有兼容性补丁单独记录，不要静默升级科学依赖。
```
### 分阶段运行
```text
/paper-reproduction-assistant 先运行 import/build 测试、一个合成样本、一个真实样本和 baseline。
每一步检查数据数量、输出 schema、checkpoint、指标和资源，再决定是否运行完整矩阵。
```
### 诊断数字不一致
```text
使用 $paper-reproduction-assistant，论文报告 82.4±0.6，我们得到 78.9±1.1。
不要立即调参。先按数据、split、预处理、checkpoint、评测、聚合、种子、硬件和代码版本分类诊断，
每次只改变一个因素，并更新偏差日志。
```
### 判断是否真正复现
```text
/paper-reproduction-assistant 原论文使用 Dataset A、Model 7B 和 5 seeds；
我们使用 Dataset B、Model 13B，只报告最佳 seed，结果方向相同。
请审计可比性，并写一段不会夸大结论的复现报告。
```
### 编写复现报告
```text
使用 $paper-reproduction-assistant，根据 paper.md、artifact-inventory.md、runs/、patch.diff 和 results.csv，
生成目标主张、环境、偏差、失败记录、结果比较、结论分类和下一步建议。
```
## 建议提供的输入
- 论文 PDF、标题、版本、DOI、arXiv 或正式出版信息；
- 目标表格、图、结论、算法、系统行为或定性发现；
- 代码仓库 URL、release、tag、commit 和 submodule；
- README、安装脚本、Dockerfile、lockfile、CI 和配置文件；
- 数据集、split、模型、checkpoint、索引、提示词和许可证；
- 论文中的硬件、软件、种子、指标、聚合和误差条说明；
- 本机或集群的环境、硬件、存储、时间和预算限制；
- 已经执行的命令、日志、错误、补丁、输出和结果表；
- 允许联网、下载、构建、执行或访问外部服务的范围；
- 期望的复现类型、容差和报告用途。
如果关键 artifact 或权限缺失，Skill 会给出最安全的部分复现方案，而不会绕过限制。
## 预期输出
根据任务可以生成：
- 复现契约、目标主张和成功标准；
- claim-to-artifact map；
- artifact、版本、来源、许可证和 checksum 清单；
- 安全审计和最小权限运行建议；
- 环境规范、依赖解析与兼容性计划；
- 分阶段运行矩阵、资源估算和停止条件；
- run manifest、命令、配置和日志索引；
- patch、参数恢复、替换和偏差记录；
- 中间输出与 baseline 验证报告；
- 论文结果和复现结果的可比性与差值分析；
- 系统化诊断历史；
- 已复现、部分复现、不可比较、不确定或未测试的精确结论；
- 适用于毕业论文、实验室记录、issue 或 artifact review 的复现报告。
## 安全、诚信与操作边界
Skill 明确要求：
- 不在未审查第三方代码前直接执行安装、脚本、容器或模型；
- 不向不可信环境挂载主目录、SSH 密钥、云凭据、浏览器配置或生产数据；
- 不绕过许可证、身份验证、付费访问、数据用途或安全限制；
- 不编造参数、数据、checkpoint、命令、日志、运行或结果；
- 不静默修改代码、数据、指标、排除规则或评测条件；
- 不为了接近论文数字而挑选种子、checkpoint、subset 或超参数；
- 不删除影响可靠性的失败、超时或不收敛运行；
- 不在关键数据、模型或协议被替换后声称精确复现；
- 不把安装失败、缺少信息或资源不足写成科学结论失败；
- 不根据一次不一致结果推断论文错误或研究不端；
- 不超出用户授权的网络、计算、费用、权限和文件修改范围；
- 对每项补丁、替换、偏差和未解决问题保留可审计记录。
对外部代码的实际执行必须遵守当前环境的安全策略，并在出现权限、许可、敏感数据或显著成本问题时停止并请求用户决定。
## 仓库结构
```text
paper-reproduction-assistant/
├── SKILL.md
├── README.md
├── agents/
│ └── openai.yaml
└── references/
├── reproduction-planning.md
├── artifact-and-security.md
├── environment-and-execution.md
├── validation-and-comparison.md
└── diagnosis-and-reporting.md
```
- `SKILL.md`：复现主流程、结果分类、交付物和不可妥协边界；
- `agents/openai.yaml`：Codex/ChatGPT 的可选界面元数据；
- `reproduction-planning.md`：复现类型、目标、成功标准、可行性和冻结协议；
- `artifact-and-security.md`：artifact、许可证、依赖、代码、数据与执行安全；
- `environment-and-execution.md`：环境、数据、分阶段运行、随机性和 manifest；
- `validation-and-comparison.md`：pipeline、baseline、中间值、容差和公平比较；
- `diagnosis-and-reporting.md`：失败分类、单因素诊断、偏差、作者沟通和报告。
## 验证状态
当前版本已经完成：
- Skill frontmatter、命名和目录结构校验；
- `SKILL.md` 引用完整性与 README 链接检查；
- 模板残留、编码、平台路径和仓库 URL 检查；
- 缺少关键 artifact 的复现规划独立前向测试：能够阻止未经审计的 shell 与 pickle 执行，区分精确、计算、概念和部分验证，并在 12 小时预算内分阶段规划；
- 不同数据、模型、指标和最佳 seed 被误称为复现的审计独立前向测试：能够判定原主张尚未被测试，并拒绝“已复现”与“原论文错误”两种越界结论；
- 针对前向测试修订容差规则：禁止把论文的标准差、标准误或置信区间宽度机械地当作等效性容差；
- 修订后的容差独立测试：能够拒绝“均值差小于论文误差条即复现成功”，并要求预先定义的科学容差、等效性/非劣效设计或不确定结论。
## 相关标准与官方文档
- [OpenAI：Build skills](https://learn.chatgpt.com/docs/build-skills)
- [Anthropic：Extend Claude with skills](https://code.claude.com/docs/en/skills)
- [Agent Skills 开放规范](https://agentskills.io/specification)
官方文档确认 Codex 用户级路径为 `$HOME/.agents/skills`、仓库级路径为 `.agents/skills`；Claude Code 个人级路径为 `~/.claude/skills`、项目级路径为 `.claude/skills`。平台更新后请以官方文档为准。
## 开发与验证
修改后使用 Codex `skill-creator` 自带脚本校验：
```bash
python /path/to/skill-creator/scripts/quick_validate.py /path/to/paper-reproduction-assistant
```
提交前检查：
1. frontmatter 是否只包含 `name` 和 `description`；
2. `description` 是否覆盖新的 artifact、环境、执行和报告场景；
3. 所有参考文件是否能从 `SKILL.md` 直接访问；
4. 复现类型和成功标准是否在结果前冻结；
5. 第三方 artifact 是否在执行前完成安全与许可审计；
6. 所有代码、数据、模型、依赖、命令和补丁是否有版本记录；
7. 失败运行、偏差和科学替换是否完整保留；
8. 复现结论是否与实际可比性和证据强度一致；
9. Codex 与 Claude Code 的安装路径和调用语法是否正确；
10. README 仓库 URL 和目标目录是否一致；
11. 新规则是否经过无答案泄漏的真实复现任务测试。
## 贡献建议
欢迎改进：
- AI/ML、系统、软件工程、安全和 HCI 的复现模式；
- CUDA、驱动、编译器和老旧依赖环境恢复；
- artifact 安全、供应链、许可证和敏感数据审计；
- 数据、模型、checkpoint 和 benchmark 版本追踪；
- run manifest、偏差日志和结果血缘模板；
- 性能、随机性和硬件差异的可比性分析；
- 作者沟通、复现报告和负结果表达；
- 基于公开、安全、小规模 artifact 的前向测试案例。
提交修改时，请说明适用复现类型、风险边界、协议变化和验证方式，避免加入只针对单篇论文或单台机器的临时规则。
references/diagnosis-and-reporting.md
# Diagnosis and Reporting
## Diagnose systematically
For every mismatch write:
- observed symptom;
- first failing stage;
- relevant evidence;
- ranked hypotheses;
- one-factor diagnostic change;
- predicted outcomes under each hypothesis;
- actual result;
- next decision.
Start with acquisition, versions, data, configuration, checkpoint, metric, and aggregation before tuning model behavior.
## Separate failure classes
- Setup failure: dependency, build, import, device, or permission problem.
- Artifact failure: missing, corrupt, incompatible, or undocumented asset.
- Execution failure: crash, timeout, out-of-memory, or nonconvergence.
- Validation failure: wrong counts, shapes, values, labels, or intermediate outputs.
- Result mismatch: pipeline runs but differs beyond tolerance.
- Comparability failure: essential scientific substitution prevents direct comparison.
Keep the class visible in logs and conclusions.
## Manage deviations
Record original protocol, changed element, reason, time, whether target results were already seen, scientific impact, and affected runs. Re-run dependent stages after a material change.
Maintain separate branches or patches for compatibility, bug fixes, and scientific changes where practical. Do not combine them into an opaque final state.
## Contact authors responsibly
Before contacting authors, provide a concise target claim, exact artifact versions, environment, command, observed output, expected output, checks already performed, minimal reproduction of the issue, and one focused question.
Do not demand unpublished data, private credentials, or assistance beyond reasonable clarification. Record responses with permission and distinguish private guidance from public documentation.
## Write the reproduction report
Use this structure:
1. paper and artifact identity;
2. target claims and reproduction type;
3. success criteria and resource scope;
4. artifact, license, and security review;
5. environment, data, and protocol;
6. deviations and substitutions;
7. execution record and failures;
8. reported-versus-reproduced results;
9. diagnostics and robustness checks;
10. outcome classification;
11. limitations and next steps;
12. availability of code, manifests, logs, and outputs.
Use exact language: “not executed,” “not tested,” “inconclusive,” “not comparable,” and “not reproduced under matched conditions” have different meanings.
## Report negative outcomes fairly
State what was successfully verified before describing the mismatch. Identify whether the target protocol was matched, how many runs were completed, and which alternative explanations remain.
Do not infer misconduct or invalid science from incomplete artifacts. Escalate stronger conclusions only when matched, auditable evidence rules out material setup explanations.
references/environment-and-execution.md
# Environment and Execution
## Capture the environment
Record OS and kernel, architecture, CPU, GPU or accelerator, memory, storage, drivers, compiler, runtime, package manager, language versions, dependencies, container digest, environment variables by name, locale, timezone, and relevant system limits.
Prefer a lockfile, container digest, or explicit package export over prose such as “latest PyTorch.” Keep the original environment specification even when a compatibility version is created.
## Resolve versions deliberately
Map requirements from paper, repository, release notes, commit history, and artifact metadata. Distinguish author-specified versions from inferred versions.
When the original stack is obsolete, first preserve an exact specification, then create the smallest compatibility patch. Do not upgrade multiple scientific libraries at once without diagnostics.
## Prepare data audibly
Record download source, version, checksum, extraction, filtering, split identifiers, preprocessing, label mapping, normalization, tokenization, deduplication, and excluded items. Verify counts and representative samples before training or evaluation.
Keep raw data immutable. Write processed data to versioned destinations and preserve the transformation command.
## Use staged execution
1. Resolve dependencies without running arbitrary project hooks when possible.
2. Test imports, compilation, and help commands.
3. Run unit tests or a tiny synthetic case.
4. Run one documented real sample.
5. Run one baseline and inspect intermediate artifacts.
6. Run one target condition.
7. Scale to the frozen matrix.
At every stage check exit status, logs, output schema, data counts, numerical sanity, storage growth, memory, and runtime.
## Control randomness
Record every seed and random source: language runtime, numerical library, framework, data loader, sampler, accelerator, and external service. Enable deterministic modes when they match the original protocol, and record operations that remain nondeterministic.
Do not promise bitwise determinism across incompatible hardware or software. Evaluate scientific tolerance instead.
## Create a run manifest
For every run record:
- immutable run ID and parent diagnostic run;
- target claim and condition;
- commit, patch, configuration, command, and working directory;
- data, model, checkpoint, split, and seed;
- environment and hardware identifier;
- timestamps, exit status, failure class, and rerun reason;
- raw outputs, logs, processed outputs, metrics, and checksums;
- planned, diagnostic, recovery, or exploratory status.
Never reuse a run ID or overwrite an output path.
## Control resources
Estimate storage, download size, runtime, accelerator hours, API cost, and energy-sensitive settings before scaling. Enforce timeouts, disk thresholds, checkpoint intervals, and stop conditions.
Use a representative subset to validate runtime estimates, but do not present subset results as full reproduction.
references/artifact-and-security.md
# Artifact and Security Review
## Inventory artifacts
Record the paper version, repository, release, commit, branches, submodules, packages, containers, datasets, models, weights, prompts, configuration files, pretrained assets, evaluation harnesses, plotting scripts, and external services.
For each item capture provenance, license, checksum or immutable identifier, size, access method, and relationship to a target claim.
## Inspect the repository statically
Read README and documentation, then inspect:
- package manifests, lockfiles, build files, and post-install hooks;
- shell, PowerShell, Makefile, task runner, and workflow scripts;
- Dockerfiles, compose files, entrypoints, and mounted paths;
- download, extraction, cleanup, migration, and upload code;
- pickle, joblib, torch, YAML, archive, or other deserialization paths;
- telemetry, analytics, experiment tracking, cloud SDKs, and API clients;
- subprocess, shell interpolation, `eval`, dynamic imports, and generated code;
- recursive deletion, permission changes, privileged or device access.
Trace the documented command to its actual entry point. Do not assume a wrapper is harmless.
## Apply least privilege
Use a disposable environment or container when feasible. Restrict writable directories, network access, device access, environment variables, mounted paths, and credentials. Use test accounts and task-specific tokens only when authorized.
Never mount home directories, SSH keys, cloud credentials, browser profiles, production datasets, or unrestricted sockets into untrusted containers.
## Handle dependencies
Prefer pinned, verifiable dependencies from authoritative registries. Review direct URLs, Git dependencies, abandoned packages, compiled extensions, and install scripts. Record vulnerability or provenance concerns without silently upgrading scientific dependencies.
If a dependency must be patched for safety, record the patch and assess whether it changes the scientific result.
## Handle data and models
Verify license, consent, intended use, privacy, export, and redistribution constraints. Scan archives and validate extraction destinations before unpacking. Treat model weights and serialized objects as potentially executable or malformed.
Do not upload restricted data to third-party services without authorization. Do not publish checkpoints, prompts, datasets, or outputs whose license forbids release.
## Stop conditions
Stop and request direction if execution requires unauthorized credentials, privilege escalation, bypassing access controls, destructive operations outside the workspace, unclear licensing, sensitive data exposure, or a meaningful expansion of compute or cost.
Report the exact blocker and the safest available partial reproduction.
agents/openai.yaml
interface:
display_name: "Paper Reproduction Assistant"
short_description: "Reproduce computer science papers safely and rigorously"
default_prompt: "Use $paper-reproduction-assistant to plan, execute, validate, and report this computer science paper reproduction."
references/reproduction-planning.md
# Reproduction Planning
## Select the reproduction type
- Exact reproduction: use the same artifacts, versions, data, and conditions.
- Computational reproduction: rerun the authors' code and pipeline, allowing documented platform differences.
- Artifact verification: check installation, functionality, documentation, and support for stated claims.
- Robustness reproduction: repeat under deliberate perturbations or additional seeds.
- Conceptual reproduction: test the same claim with materially different artifacts or conditions.
- Independent reimplementation: recreate the method from the paper without relying on the released implementation.
Name the type before execution. Do not upgrade a conceptual result to exact reproduction after it looks favorable.
## Identify target claims
Choose concrete targets such as a table row, figure curve, theorem, latency percentile, ablation, user-study theme, or qualitative example. Record the exact metric, population, aggregation, baseline, conditions, and reported uncertainty.
Prioritize targets by:
1. importance to the paper's contribution;
2. availability of required artifacts;
3. diagnostic value;
4. compute, time, and safety cost.
## Set success criteria
Define a tolerance from measurement precision, reported variability, metric scale, and scientific relevance. Example forms include:
- confidence intervals overlap under a comparable design;
- paired effect has the same direction and a predeclared magnitude range;
- performance is within an absolute metric tolerance;
- system throughput and tail latency fall within a hardware-normalized range;
- proof checker accepts the theorem under the stated assumptions;
- qualitative codebook recovers the declared themes with an agreed procedure.
Avoid using rounded table values as exact ground truth.
Do not mechanically reuse a reported standard deviation, standard error, or confidence-interval half-width as the reproduction tolerance. These quantities describe different uncertainty targets and are not equivalence margins. Base a binary success threshold on a scientifically meaningful difference, the original run-level variation and design when available, measurement precision, or an accepted domain protocol. If the necessary information is unavailable, predeclare a descriptive comparison and classify the result as inconclusive rather than inventing a pass/fail margin.
## Estimate feasibility
Inventory data access, model access, storage, memory, accelerator type, runtime, cost, network use, special hardware, proprietary services, human participants, and licenses.
Separate blocking requirements from desirable fidelity. Create fallback levels before running:
- Level A: published exact setup;
- Level B: documented compatible setup;
- Level C: explicit scientific substitution;
- Level D: static or partial artifact verification only.
## Plan the run matrix
Start with a minimal baseline and target condition. Add repetitions, datasets, models, or ablations only after the pipeline passes checks.
For each planned run specify condition, seed, input, expected output, resource budget, validation check, and stop condition. Include failed-run and rerun rules.
## Freeze the protocol
Version the target claims, success criteria, run matrix, environment plan, substitution policy, and analysis before full results. Log later changes with time and whether target outcomes were already visible.
references/validation-and-comparison.md
# Validation and Comparison
## Validate the pipeline
Check data counts, splits, class balance, representative samples, preprocessing, tensor shapes, checkpoint keys, model mode, precision, loss direction, metric implementation, denominator, aggregation, and output schema.
Compare intermediate artifacts such as tokenized examples, retrieved documents, logits, loss curves, predictions, confusion matrices, generated samples, resource counters, or proof obligations when available.
## Establish the baseline
Reproduce the simplest reported baseline under the same local pipeline before judging the target method. A broken baseline often reveals data, metric, or environment mismatch.
Distinguish a locally rerun baseline from a number copied from the paper. Do not compare a local method against a published baseline under different settings without warning.
## Match the reported result
Confirm:
- dataset version, split, sample count, and exclusions;
- model and checkpoint;
- training, tuning, and evaluation configuration;
- metric implementation, direction, scale, and rounding;
- seed and repetition policy;
- hardware-sensitive settings;
- table or figure aggregation and error bars.
Trace a reproduced number from raw output through processing to the final table or plot.
## Quantify the difference
Report the published value, reproduced point estimate, absolute difference, relative difference when meaningful, uncertainty, number of successful and failed runs, and tolerance decision.
When raw reported runs are unavailable, do not infer a distribution from mean and standard deviation alone. State which comparisons cannot be tested.
## Avoid false agreement
A close final metric does not establish reproduction when:
- different data or labels were used;
- a stronger model or larger budget substituted for the original;
- preprocessing or metric implementation differs;
- favorable seeds or checkpoints were selected;
- offsetting errors happen to produce a similar average.
Require pipeline fidelity and intermediate checks, not only numerical proximity.
## Avoid false failure
A mismatch does not refute the claim when:
- the environment or artifact is incomplete;
- essential parameters are unknown;
- the setup includes material substitutions;
- too few repetitions were run;
- hardware or numerical differences exceed the declared scope;
- the reported value is rounded or ambiguously aggregated.
Classify these outcomes as incomparable or inconclusive until resolved.
## Test robustness separately
After the main target, vary one declared factor such as seed, hardware, library version, dataset shift, preprocessing, model scale, or hyperparameter. Label robustness results separately from the reproduction result.
Do not use post hoc robustness sweeps to select the configuration closest to the paper.