ai-scientist-v2-guide/SKILL.md
---
name: ai-scientist-v2-guide
description: "Automated scientific discovery via agentic tree search by Sakana AI"
metadata:
openclaw:
emoji: "🧪"
category: "research"
subcategory: "automation"
keywords: ["scientific-discovery", "automation", "tree-search", "paper-generation", "experiment-design", "sakana-ai"]
source: "https://github.com/SakanaAI/AI-Scientist-v2"
---
# AI Scientist v2 Guide
## Overview
AI-Scientist-v2 is an open-source system developed by Sakana AI with over 2,000 GitHub stars that automates the full scientific research pipeline -- from idea generation through experimentation to paper writing. Building on the original AI Scientist, version 2 introduces an agentic tree search approach that systematically explores the space of research ideas, designs and runs experiments, analyzes results, and produces workshop-level scientific papers with minimal human intervention.
The key innovation in v2 is the tree search mechanism. Rather than pursuing a single research direction linearly, the system maintains a tree of possible research trajectories. At each node, the agent can branch into multiple experimental variations, evaluate the results, and prune unpromising directions while doubling down on successful ones. This mirrors how experienced researchers navigate the research landscape -- exploring broadly at first, then focusing resources on the most promising leads.
AI-Scientist-v2 has demonstrated the ability to generate novel, valid research papers in machine learning subfields including diffusion models, language model training, and optimization. While the generated papers are currently at workshop acceptance level, the system represents a significant step toward autonomous scientific discovery and is an invaluable tool for researchers looking to automate the more mechanical aspects of their research workflow.
## Installation and Setup
```bash
# Clone the repository
git clone https://github.com/SakanaAI/AI-Scientist-v2.git
cd AI-Scientist-v2
# Create a conda environment
conda create -n ai-scientist python=3.11
conda activate ai-scientist
# Install dependencies
pip install -r requirements.txt
```
### Prerequisites
AI-Scientist-v2 requires several components:
```bash
# LLM API access (required for ideation, analysis, and writing)
export OPENAI_API_KEY=$OPENAI_API_KEY
# Or Anthropic
export ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY
# GPU access for running ML experiments
# Recommended: at least one NVIDIA GPU with 24GB+ VRAM
# LaTeX installation for paper compilation
# Ubuntu/Debian
sudo apt-get install texlive-full
# macOS
brew install --cask mactex
```
### Configuration
Set up your research configuration:
```python
# config.yaml
llm:
provider: "openai"
model: "gpt-4o"
temperature: 0.7
search:
max_depth: 5 # Maximum tree depth
branching_factor: 3 # Number of branches per node
pruning_threshold: 0.3 # Prune branches below this score
experiment:
gpu_ids: [0, 1] # Available GPUs
timeout_hours: 2 # Max time per experiment
num_seeds: 3 # Random seeds per experiment
paper:
template: "icml" # Paper template (icml, neurips, iclr)
max_pages: 8 # Maximum paper length
```
## Core Research Pipeline
### Phase 1: Idea Generation
The system generates research ideas by analyzing existing literature and identifying gaps or extensions:
```python
from ai_scientist import IdeaGenerator
generator = IdeaGenerator(
research_area="efficient_transformers",
seed_papers=[
"path/to/related_paper_1.pdf",
"path/to/related_paper_2.pdf",
],
num_ideas=10,
)
ideas = generator.generate()
for idea in ideas:
print(f"Title: {idea.title}")
print(f"Hypothesis: {idea.hypothesis}")
print(f"Novelty score: {idea.novelty_score}")
print(f"Feasibility score: {idea.feasibility_score}")
```
### Phase 2: Agentic Tree Search
The tree search mechanism explores the research space systematically:
```python
from ai_scientist import TreeSearchResearcher
researcher = TreeSearchResearcher(
idea=ideas[0], # Start with the top-ranked idea
base_code="templates/efficient_transformer/",
config="config.yaml",
)
# Run the tree search
result = researcher.run()
# The search tree tracks all explorations
print(f"Tree depth reached: {result.max_depth}")
print(f"Total experiments run: {result.total_experiments}")
print(f"Best result: {result.best_node.metrics}")
```
The tree search works as follows:
1. **Root node**: The initial research idea and baseline implementation
2. **Expansion**: At each node, the agent proposes 2-4 modifications (hyperparameter changes, architectural tweaks, new training strategies)
3. **Evaluation**: Each modification is implemented and evaluated experimentally
4. **Selection**: Promising branches are selected for further exploration using UCB (Upper Confidence Bound) or similar strategies
5. **Pruning**: Branches that underperform the baseline or show diminishing returns are pruned
### Phase 3: Experiment Execution
Experiments are executed in isolated environments with proper controls:
```python
# Each experiment node contains:
class ExperimentNode:
hypothesis: str # What we're testing
code_changes: list # Specific code modifications
config_changes: dict # Hyperparameter changes
results: dict # Experimental results
analysis: str # LLM-generated analysis
children: list # Branch experiments
```
The system automatically handles experiment boilerplate including random seed management, metric logging, checkpoint saving, and result visualization. Each experiment is run with multiple seeds to ensure statistical significance.
### Phase 4: Paper Generation
After the tree search completes, the system generates a scientific paper:
```python
from ai_scientist import PaperWriter
writer = PaperWriter(
research_result=result,
template="neurips",
sections=[
"introduction",
"related_work",
"method",
"experiments",
"analysis",
"conclusion",
],
)
# Generate the paper
paper = writer.write()
# Compile to PDF
paper.compile_latex("output/paper.pdf")
# The paper includes:
# - Abstract summarizing key findings
# - Introduction with motivation and contributions
# - Related work section with citations
# - Method description with equations
# - Experiment section with tables and figures
# - Analysis of results with ablation studies
# - Conclusion with future work directions
```
## Research Templates
AI-Scientist-v2 includes several research templates that define the experimental domain:
### NanoGPT Template
Train and evaluate small language models with various architectural modifications:
```bash
python run_scientist.py \
--template nanoGPT \
--idea "Investigate the effect of rotary position embeddings on small-scale language model training" \
--max_experiments 20
```
### Diffusion Model Template
Experiment with diffusion model architectures and training strategies:
```bash
python run_scientist.py \
--template diffusion \
--idea "Compare noise schedules for conditional image generation"
```
### Creating Custom Templates
Define your own research template for your specific domain:
```python
# templates/my_domain/template.py
class MyDomainTemplate:
name = "my_research_domain"
base_metrics = ["accuracy", "f1_score", "inference_time"]
def setup_baseline(self):
"""Set up the baseline experiment."""
pass
def evaluate(self, model, data):
"""Evaluate a model configuration."""
pass
def get_modification_space(self):
"""Define the space of possible modifications."""
return {
"architecture": ["transformer", "lstm", "mamba"],
"learning_rate": [1e-4, 3e-4, 1e-3],
"batch_size": [32, 64, 128],
}
```
## Automated Paper Review
AI-Scientist-v2 includes an automated reviewer that evaluates generated papers using criteria from top ML venues:
```python
from ai_scientist import PaperReviewer
reviewer = PaperReviewer(
venue="neurips",
review_criteria=[
"novelty",
"significance",
"clarity",
"correctness",
"reproducibility",
],
)
review = reviewer.review("output/paper.pdf")
print(f"Overall score: {review.overall_score}/10")
print(f"Strengths: {review.strengths}")
print(f"Weaknesses: {review.weaknesses}")
print(f"Questions: {review.questions}")
```
## Ethical Considerations and Limitations
When using AI-Scientist-v2, keep these considerations in mind:
- **Human oversight**: Always review generated papers for correctness before submission. The system can produce plausible-sounding but incorrect analyses.
- **Attribution**: If using AI-Scientist-v2 outputs in publications, disclose the use of automated research tools per venue guidelines.
- **Scope**: The system works best for incremental research within well-defined experimental frameworks. Breakthrough conceptual contributions still require human creativity.
- **Compute cost**: Tree search with multiple seeds per experiment can require substantial GPU time. Set appropriate budgets and timeouts.
- **Reproducibility**: All experiments are logged with seeds, configurations, and code versions for full reproducibility.
## References
- Repository: https://github.com/SakanaAI/AI-Scientist-v2
- Original AI Scientist paper: https://arxiv.org/abs/2408.06292
- Sakana AI: https://sakana.ai/
- AI Scientist v1: https://github.com/SakanaAI/AI-Scientist
aim-experiment-guide/SKILL.md
---
name: aim-experiment-guide
description: "Track and compare research experiments with Aim experiment tracker"
metadata:
openclaw:
emoji: "🎯"
category: "research"
subcategory: "automation"
keywords: ["experiment-tracking", "visualization", "mlops", "reproducibility", "metrics", "hyperparameters"]
source: "https://github.com/aimhubio/aim"
---
# Aim Experiment Tracker Guide
## Overview
Aim is an open-source experiment tracking platform designed for researchers and ML engineers who need to log, compare, and analyze large numbers of experiments. Unlike cloud-based tracking services that require sending data to external servers, Aim runs entirely on your own infrastructure, making it suitable for research environments with data privacy requirements or institutional restrictions on external services.
The core problem Aim solves is experiment management at scale. A typical research project involves hundreds or thousands of training runs with different hyperparameters, data splits, model architectures, and random seeds. Without systematic tracking, researchers lose track of which configurations produced which results, leading to wasted computation and unreproducible findings. Aim provides a high-performance storage backend and a rich web UI for logging, querying, and visualizing experiment metadata and metrics.
With over 6,000 GitHub stars, Aim has established itself as a compelling self-hosted alternative to tools like Weights and Biases and MLflow. Its Python-native API integrates with minimal friction into existing training loops, and the query language enables sophisticated filtering across thousands of runs.
## Installation and Setup
Install Aim via pip:
```bash
pip install aim
```
Initialize an Aim repository in your project directory:
```bash
cd /path/to/research-project
aim init
```
This creates a `.aim` directory that stores all experiment data locally. Launch the web UI:
```bash
aim up
```
The dashboard becomes available at `http://localhost:43800`, providing interactive visualizations of all tracked experiments.
For remote server deployment:
```bash
aim up --host 0.0.0.0 --port 43800
```
## Core Features
**Experiment Logging**: Integrate Aim tracking into your training scripts with minimal code changes:
```python
from aim import Run
# Initialize a tracked run
run = Run(experiment="protein_folding_v2")
# Log hyperparameters
run["hparams"] = {
"learning_rate": 0.001,
"batch_size": 64,
"model": "transformer",
"num_layers": 6,
"hidden_dim": 256,
"dropout": 0.1,
"optimizer": "adamw",
"weight_decay": 0.01,
"seed": 42,
}
# Log dataset information
run["dataset"] = {
"name": "protein_benchmark_v3",
"train_size": 50000,
"val_size": 5000,
"test_size": 5000,
}
# Track metrics during training
for epoch in range(num_epochs):
train_loss = train_one_epoch(model, train_loader)
val_loss, val_accuracy = evaluate(model, val_loader)
run.track(train_loss, name="loss", context={"subset": "train"})
run.track(val_loss, name="loss", context={"subset": "val"})
run.track(val_accuracy, name="accuracy", context={"subset": "val"})
```
**Framework Integrations**: Aim provides built-in callbacks for popular training frameworks:
```python
# PyTorch Lightning integration
from aim.pytorch_lightning import AimLogger
aim_logger = AimLogger(experiment="lightning_exp")
trainer = pl.Trainer(logger=aim_logger, max_epochs=100)
# Hugging Face Transformers integration
from aim.hugging_face import AimCallback
aim_callback = AimCallback(experiment="hf_training")
trainer = Trainer(
model=model,
args=training_args,
callbacks=[aim_callback],
)
# Keras integration
from aim.keras import AimCallback as KerasAimCallback
model.fit(
x_train, y_train,
callbacks=[KerasAimCallback(experiment="keras_exp")],
epochs=50,
)
```
**Powerful Query Language**: Filter and retrieve experiments programmatically:
```python
from aim import Repo
repo = Repo("/path/to/research-project")
# Query runs matching specific criteria
query = """
run.experiment == "protein_folding_v2"
and run.hparams.learning_rate < 0.01
and run.hparams.model == "transformer"
"""
for run in repo.query_runs(query).iter_runs():
print(f"Run: {run.hash}")
print(f" LR: {run['hparams']['learning_rate']}")
print(f" Final val loss: {run['loss']}")
```
**Rich Visualizations**: The web UI provides interactive charts for comparing experiments:
- Line charts for metric trajectories across epochs
- Parallel coordinates plots for hyperparameter exploration
- Scatter plots correlating hyperparameters with final metrics
- Distribution plots for metric analysis across run groups
- Image and audio tracking for multimedia experiments
## Research Workflow Integration
**Hyperparameter Search Analysis**: After running grid search or random search experiments, use Aim to identify the best configurations:
```python
from aim import Repo
repo = Repo(".")
# Find the best run by validation accuracy
best_run = None
best_acc = 0.0
for run_metrics in repo.query_metrics(
"metric.name == 'accuracy' and metric.context.subset == 'val'"
).iter_runs():
for metric in run_metrics:
final_val = list(metric.values.values())[-1]
if final_val > best_acc:
best_acc = final_val
best_run = metric.run.hash
print(f"Best run: {best_run} with accuracy {best_acc:.4f}")
```
**Reproducibility Documentation**: Every tracked run captures the full hyperparameter configuration, making it straightforward to include exact experimental details in paper methods sections and supplementary materials.
**Ablation Studies**: Tag runs with ablation group identifiers and use the comparison UI to visualize the impact of each component:
```python
run = Run(experiment="ablation_study")
run["hparams"] = config
run["ablation"] = {
"group": "attention_mechanism",
"variant": "multi_head",
"description": "Standard multi-head attention vs. linear attention",
}
```
**Lab Notebook Integration**: Export experiment summaries for inclusion in electronic lab notebooks. The query API enables automated report generation:
```python
import pandas as pd
from aim import Repo
repo = Repo(".")
records = []
for run_metrics in repo.query_metrics(
"metric.name == 'accuracy'"
).iter_runs():
run = run_metrics.run
for metric in run_metrics:
values = list(metric.values.values())
records.append({
"run_hash": run.hash[:8],
"model": run["hparams"].get("model"),
"lr": run["hparams"].get("learning_rate"),
"final_accuracy": values[-1] if values else None,
})
df = pd.DataFrame(records)
df.to_csv("experiment_summary.csv", index=False)
```
## Storage and Performance
Aim uses a custom high-performance storage engine optimized for time-series metrics data. The storage scales to millions of tracked values across thousands of runs without significant degradation in query performance.
Data is stored locally in the `.aim` directory. Back up this directory to preserve your experiment history. For team settings, the Aim server can be deployed as a shared service accessible to multiple researchers.
```bash
# Check storage usage
du -sh .aim/
# Export data for archival
aim storage --repo . upgrade 3.0
```
## References
- Aim repository: https://github.com/aimhubio/aim
- Aim documentation: https://aimstack.readthedocs.io/
- Aim UI demo and screenshots in the repository wiki
- Comparison with MLflow and Weights and Biases in the documentation
claude-academic-workflow-guide/SKILL.md
---
name: claude-academic-workflow-guide
description: "Claude Code template for LaTeX, Beamer, and R research workflows"
metadata:
openclaw:
emoji: "📚"
category: "research"
subcategory: "automation"
keywords: ["Claude Code", "academic workflow", "LaTeX", "Beamer", "R", "multi-agent review"]
source: "https://github.com/pedrohcgs/claude-code-my-workflow"
---
# Claude Code Academic Workflow Guide
## Overview
A template and workflow guide for using Claude Code in academic research — managing LaTeX papers, Beamer presentations, R analysis scripts, and multi-agent peer review. Provides structured CLAUDE.md configurations, project templates, and automation patterns for common academic tasks. Designed for economists, social scientists, and quantitative researchers.
## Project Structure
```
research-project/
├── CLAUDE.md # Claude Code instructions
├── paper/
│ ├── main.tex # Main LaTeX document
│ ├── references.bib # Bibliography
│ ├── sections/ # LaTeX sections
│ └── figures/ # Generated figures
├── slides/
│ ├── presentation.tex # Beamer slides
│ └── figures/
├── code/
│ ├── analysis.R # Main analysis
│ ├── data_clean.R # Data preparation
│ └── figures.R # Figure generation
├── data/
│ ├── raw/ # Original data
│ └── processed/ # Cleaned data
└── output/
├── tables/ # LaTeX tables
└── figures/ # PDF/PNG figures
```
## CLAUDE.md Configuration
```markdown
# Project: [Your Paper Title]
## Instructions
- This is an academic research project in economics
- LaTeX compiler: pdflatex (paper) or xelatex (if CJK)
- R version: 4.3+ with tidyverse, fixest, ggplot2
- Citation style: natbib, authoryear
- Always compile paper after LaTeX changes
- Run R scripts from project root
## Paper Conventions
- Use \input{sections/intro} for section includes
- Tables: booktabs package, generated from R
- Figures: PDF format, width=\textwidth
- Cross-refs: \label{sec:}, \label{tab:}, \label{fig:}
## R Conventions
- Style: tidyverse style guide
- Data: read from data/processed/
- Output: tables/ (LaTeX), figures/ (PDF)
- Reproducibility: set.seed(42) for all random ops
## Build Commands
- Paper: `cd paper && pdflatex main && bibtex main && pdflatex main && pdflatex main`
- Slides: `cd slides && pdflatex presentation`
- Analysis: `cd code && Rscript analysis.R`
```
## LaTeX Paper Workflow
```bash
# Claude Code can manage the full LaTeX workflow:
# 1. Draft a section
# "Write the methodology section for our diff-in-diff analysis"
# 2. Generate tables from R output
# "Create a LaTeX table from the regression results in output/tables/"
# 3. Fix compilation errors
# "The paper won't compile — fix the LaTeX errors"
# 4. Update bibliography
# "Add the Callaway & Sant'Anna (2021) reference"
# 5. Format for submission
# "Format the paper for AER submission guidelines"
```
## Beamer Presentations
```latex
% Template for academic presentations
\documentclass[aspectratio=169]{beamer}
\usetheme{metropolis}
\title{Your Presentation Title}
\subtitle{Conference/Seminar Name}
\author{Author Name}
\institute{University}
\date{\today}
\begin{document}
\maketitle
\begin{frame}{Motivation}
\begin{itemize}
\item Research question
\item Why it matters
\item What we do
\end{itemize}
\end{frame}
\begin{frame}{Data}
\input{figures/summary_stats_table}
\end{frame}
\begin{frame}{Results}
\centering
\includegraphics[width=0.8\textwidth]{figures/main_result.pdf}
\end{frame}
\end{document}
```
## R Analysis Integration
```r
# analysis.R — Main analysis script
library(tidyverse)
library(fixest)
library(modelsummary)
# Load cleaned data
df <- read_csv("data/processed/analysis_data.csv")
# Main regression
model1 <- feols(outcome ~ treatment | year + state, data = df)
model2 <- feols(outcome ~ treatment + controls | year + state,
data = df, cluster = ~state)
# Export table for LaTeX
modelsummary(
list("(1)" = model1, "(2)" = model2),
output = "output/tables/main_results.tex",
stars = c("*" = 0.1, "**" = 0.05, "***" = 0.01),
gof_map = c("nobs", "r.squared", "FE: year", "FE: state"),
)
# Export figure
ggplot(df, aes(x = year, y = outcome, color = treated)) +
geom_point(alpha = 0.3) +
geom_smooth(method = "loess") +
theme_minimal() +
labs(x = "Year", y = "Outcome", color = "Treatment Group")
ggsave("output/figures/treatment_trends.pdf", width = 8, height = 5)
```
## Multi-Agent Review
```markdown
### Self-Review Workflow
Use Claude Code to simulate peer review:
1. "Review this paper as a critical referee for AER"
2. "Check all mathematical derivations in section 3"
3. "Verify that all tables match the R code output"
4. "Check for consistency between text claims and results"
5. "List potential referee objections and how to address them"
```
## Common Tasks
```markdown
### Things to ask Claude Code:
- "Compile the paper and fix any errors"
- "Add robustness check using propensity score matching"
- "Create a Beamer slide summarizing Table 2"
- "Generate event study plot from the regression results"
- "Convert this Word draft to LaTeX format"
- "Check all cross-references are correct"
- "Format references in AEA style"
```
## Use Cases
1. **Paper writing**: LaTeX drafting and compilation workflow
2. **Data analysis**: R script development and debugging
3. **Presentations**: Beamer slide creation from paper content
4. **Self-review**: Multi-agent review simulation
5. **Submission prep**: Format conversion for journal submission
## References
- [claude-code-my-workflow](https://github.com/pedrohcgs/claude-code-my-workflow)
- [Metropolis Beamer Theme](https://github.com/matze/mtheme)
- [modelsummary](https://modelsummary.com/)
data-collection-automation/SKILL.md
---
name: data-collection-automation
description: "Automate survey deployment, data collection, and pipeline management"
metadata:
openclaw:
emoji: "🤖"
category: "research"
subcategory: "automation"
keywords: ["data collection", "survey automation", "pipeline", "Qualtrics API", "research automation", "ETL"]
source: "wentor-research-plugins"
---
# Data Collection Automation Guide
A skill for automating research data collection, survey deployment, and data pipeline management. Covers survey platform APIs, automated data retrieval, quality checks, ETL pipelines, and scheduling for longitudinal studies.
## Survey Platform APIs
### Qualtrics API
```python
import os
import json
import urllib.request
import time
def export_qualtrics_responses(survey_id: str,
file_format: str = "csv") -> str:
"""
Export survey responses from Qualtrics via API.
Args:
survey_id: The Qualtrics survey ID (SV_...)
file_format: Export format (csv, json, spss)
"""
api_token = os.environ["QUALTRICS_API_TOKEN"]
data_center = os.environ["QUALTRICS_DATACENTER"]
base_url = f"https://{data_center}.qualtrics.com/API/v3"
headers = {
"X-API-TOKEN": api_token,
"Content-Type": "application/json"
}
# Step 1: Start export
export_data = json.dumps({
"format": file_format,
"compress": False
}).encode("utf-8")
req = urllib.request.Request(
f"{base_url}/surveys/{survey_id}/export-responses",
data=export_data,
headers=headers
)
response = json.loads(urllib.request.urlopen(req).read())
progress_id = response["result"]["progressId"]
# Step 2: Poll for completion
status = "inProgress"
while status == "inProgress":
time.sleep(2)
req = urllib.request.Request(
f"{base_url}/surveys/{survey_id}/export-responses/{progress_id}",
headers=headers
)
check = json.loads(urllib.request.urlopen(req).read())
status = check["result"]["status"]
file_id = check["result"]["fileId"]
# Step 3: Download file
req = urllib.request.Request(
f"{base_url}/surveys/{survey_id}/export-responses/{file_id}/file",
headers=headers
)
file_data = urllib.request.urlopen(req).read()
output_path = f"responses_{survey_id}.{file_format}"
with open(output_path, "wb") as f:
f.write(file_data)
return output_path
```
### REDCap API
```python
def export_redcap_records(api_url: str, fields: list[str] = None) -> list:
"""
Export records from a REDCap project.
Args:
api_url: REDCap API endpoint URL
fields: List of field names to export (None = all fields)
"""
api_token = os.environ["REDCAP_API_TOKEN"]
data = {
"token": api_token,
"content": "record",
"format": "json",
"type": "flat"
}
if fields:
data["fields"] = ",".join(fields)
encoded = urllib.parse.urlencode(data).encode("utf-8")
req = urllib.request.Request(api_url, data=encoded)
response = urllib.request.urlopen(req)
return json.loads(response.read())
```
## Automated Data Quality Checks
### Validation Pipeline
```python
import pandas as pd
from datetime import datetime
def validate_survey_data(df: pd.DataFrame,
rules: dict) -> dict:
"""
Run automated data quality checks on collected data.
Args:
df: DataFrame of survey responses
rules: Dict of column -> validation rule pairs
"""
issues = []
# Check for duplicates
dupes = df.duplicated(subset=["respondent_id"]).sum()
if dupes > 0:
issues.append(f"Found {dupes} duplicate respondent IDs")
# Check completion rates
completion = df.notna().mean()
low_completion = completion[completion < 0.5]
for col in low_completion.index:
issues.append(f"Column '{col}' has {low_completion[col]:.0%} completion")
# Check value ranges
for col, rule in rules.items():
if col not in df.columns:
continue
if "min" in rule:
violations = (df[col] < rule["min"]).sum()
if violations > 0:
issues.append(f"{violations} values below minimum in '{col}'")
if "max" in rule:
violations = (df[col] > rule["max"]).sum()
if violations > 0:
issues.append(f"{violations} values above maximum in '{col}'")
# Check for speeding (unusually fast completion)
if "duration_seconds" in df.columns:
median_time = df["duration_seconds"].median()
speeders = (df["duration_seconds"] < median_time * 0.3).sum()
if speeders > 0:
issues.append(f"{speeders} respondents completed in <30% of median time")
return {
"n_records": len(df),
"n_issues": len(issues),
"issues": issues,
"timestamp": datetime.now().isoformat()
}
```
## ETL Pipeline for Research Data
### Scheduled Data Retrieval
```python
def research_etl_pipeline(sources: list[dict],
output_dir: str) -> dict:
"""
Extract, transform, and load research data from multiple sources.
Args:
sources: List of data source configurations
output_dir: Directory to save processed data
"""
results = {}
for source in sources:
name = source["name"]
# Extract
if source["type"] == "qualtrics":
raw_path = export_qualtrics_responses(source["survey_id"])
df = pd.read_csv(raw_path)
elif source["type"] == "redcap":
records = export_redcap_records(source["api_url"])
df = pd.DataFrame(records)
elif source["type"] == "csv_url":
df = pd.read_csv(source["url"])
else:
continue
# Transform
df = df.dropna(how="all")
df.columns = [c.strip().lower().replace(" ", "_") for c in df.columns]
# Load
timestamp = datetime.now().strftime("%Y%m%d")
output_path = f"{output_dir}/{name}_{timestamp}.csv"
df.to_csv(output_path, index=False)
results[name] = {
"records": len(df),
"columns": len(df.columns),
"output": output_path
}
return results
```
## Scheduling and Monitoring
### Cron-Based Scheduling
```bash
# Run data collection pipeline daily at 6 AM
# crontab -e
0 6 * * * cd /path/to/project && python collect_data.py >> logs/collection.log 2>&1
```
### Monitoring Checklist
```
For longitudinal studies, automate monitoring of:
- Response rates per wave (alert if below threshold)
- Data quality metrics (completion, speeding, straight-lining)
- API quota usage (stay within rate limits)
- Storage usage and backup status
- Participant dropout patterns
```
## Ethical Considerations
Always ensure automated data collection complies with your IRB/ethics board approval. Store API tokens securely using environment variables, never in code. Implement data encryption at rest. Log all data access for audit trails. Respect rate limits on external APIs. Include automated checks for consent status before processing participant data.
datagen-research-guide/SKILL.md
---
name: datagen-research-guide
description: "AI-driven multi-agent research assistant for end-to-end studies"
source: https://github.com/DATAGEN-AI/DATAGEN
metadata:
openclaw:
category: "research"
subcategory: "automation"
emoji: "⚙️"
keywords: [multi-agent, research-assistant, data-generation, study-automation, pipeline-orchestration, ai-research]
---
# DATAGEN Research Guide
A skill for orchestrating AI-driven multi-agent research workflows that handle literature review, hypothesis generation, experiment design, data analysis, and report writing. Based on the DATAGEN project (2K stars), this skill provides structured guidance on building automated research pipelines using collaborative agent architectures.
## Overview
Modern research increasingly benefits from AI assistance at every stage. DATAGEN's approach uses multiple specialized agents that collaborate on a research task, each handling a different aspect of the workflow. This skill teaches the agent how to coordinate such multi-agent pipelines, ensuring quality control at each handoff point and maintaining scientific rigor throughout.
The multi-agent paradigm is particularly powerful for research tasks that span multiple competencies: a literature agent gathers relevant prior work, a methodology agent designs appropriate experiments, a data agent handles collection and cleaning, an analysis agent runs statistical tests, and a writing agent produces publication-ready text.
## Multi-Agent Architecture
The research pipeline employs these specialized agent roles:
**Literature Agent**
- Conducts systematic literature searches across academic databases
- Filters results by relevance, recency, and citation impact
- Extracts key findings and methodological details from selected papers
- Identifies research gaps that motivate the current study
- Produces structured literature summaries with citation metadata
**Hypothesis Agent**
- Generates testable hypotheses based on literature gaps
- Evaluates feasibility of proposed hypotheses given available resources
- Ranks hypotheses by potential impact and testability
- Defines operationalizations for abstract constructs
- Produces formal hypothesis statements with predicted effect directions
**Experiment Agent**
- Designs experimental protocols appropriate to the hypotheses
- Selects control conditions and randomization strategies
- Calculates sample size requirements and power estimates
- Identifies potential confounds and proposes mitigation strategies
- Generates detailed protocol documents suitable for pre-registration
**Analysis Agent**
- Selects statistical methods aligned with the experimental design
- Implements analysis pipelines with documented parameters
- Runs assumption checks before applying parametric tests
- Produces visualization of results with appropriate uncertainty measures
- Generates analysis reports with effect sizes and confidence intervals
**Writing Agent**
- Drafts sections following target journal formatting guidelines
- Integrates results from analysis into coherent narratives
- Ensures claims are proportional to the evidence strength
- Manages references and in-text citations consistently
- Produces abstracts, summaries, and highlight points
## Pipeline Orchestration
Coordinating multiple agents requires careful orchestration:
**Task Decomposition**
- Break the overall research question into sub-tasks aligned with agent capabilities
- Define clear input-output contracts between agents
- Establish quality gates at each pipeline stage
- Allow for iterative refinement when downstream agents identify issues
- Maintain a shared context document accessible to all agents
**Quality Control**
- Each agent output passes through a validation checkpoint
- Cross-reference literature findings with known databases
- Verify statistical analyses meet the assumptions of chosen tests
- Check written outputs against reporting guidelines (APA, CONSORT, etc.)
- Flag inconsistencies between sections for human review
**Error Recovery**
- Define fallback strategies when an agent cannot complete its task
- Allow agents to request clarification from upstream agents
- Implement retry logic with modified parameters for failed steps
- Escalate to human oversight when confidence is below threshold
- Log all decisions and their rationale for audit trails
## Data Generation Workflows
The DATAGEN approach excels at synthetic data generation for research:
- Generate synthetic datasets matching real-world statistical properties
- Create simulation-based datasets for power analysis and method testing
- Produce augmented training data for machine learning experiments
- Build synthetic control groups when ethical constraints limit real data
- Validate analysis pipelines on known ground truth before applying to real data
## Research Domain Applications
This skill adapts to multiple research contexts:
**Social Sciences** - Survey design, factor analysis, structural equation modeling
**Natural Sciences** - Experimental protocols, measurement validation, replication studies
**Computer Science** - Benchmark design, ablation studies, performance evaluation
**Health Sciences** - Clinical trial design, meta-analysis, systematic reviews
**Engineering** - Design of experiments, optimization, reliability testing
## Integration with Research-Claw
This skill coordinates with other Research-Claw capabilities:
- Literature search skills feed the Literature Agent
- Statistical analysis skills power the Analysis Agent
- Writing and citation skills support the Writing Agent
- Domain-specific skills provide specialized knowledge to all agents
- The orchestration layer uses Research-Claw's task management for pipeline control
## Best Practices
- Always maintain human oversight at critical decision points
- Document every automated decision with its reasoning
- Validate automated outputs against domain expert judgment periodically
- Start with simpler single-agent workflows before scaling to multi-agent pipelines
- Use version control for all generated artifacts (data, analyses, drafts)
- Ensure reproducibility by logging all random seeds and model versions
kedro-pipeline-guide/SKILL.md
---
name: kedro-pipeline-guide
description: "Build reproducible data science pipelines with Kedro for research projects"
metadata:
openclaw:
emoji: "🔧"
category: "research"
subcategory: "automation"
keywords: ["pipeline", "reproducibility", "data-science", "workflow", "automation", "mlops"]
source: "https://github.com/kedro-org/kedro"
---
# Kedro Pipeline Guide
## Overview
Kedro is an open-source Python framework for creating reproducible, maintainable, and modular data science pipelines. Developed originally at McKinsey's QuantumBlack labs, Kedro provides an opinionated project structure and a set of conventions that transform ad-hoc analysis scripts into production-quality code that can be tested, versioned, and shared across research teams.
In academic research, reproducibility is both a scientific imperative and a practical challenge. Jupyter notebooks and standalone scripts often become tangled webs of dependencies that are difficult to re-run months later when responding to reviewer comments or extending prior work. Kedro addresses this by separating data processing logic from data access, enforcing explicit pipeline definitions, and providing built-in data versioning and experiment tracking.
With over 11,000 GitHub stars, Kedro has gained adoption across industry and academia. Its design philosophy aligns naturally with the needs of computational research: clear data lineage, parameterized experiments, and the ability to scale from a laptop to a cluster without rewriting code.
## Installation and Setup
Install Kedro via pip:
```bash
pip install kedro
```
Create a new project using the Kedro starter:
```bash
kedro new --name my-research-project --tools lint,test,docs
cd my-research-project
```
This generates a standardized project structure:
```
my-research-project/
conf/
base/
catalog.yml # Data source definitions
parameters.yml # Experiment parameters
local/ # Local overrides (gitignored)
src/
my_research_project/
pipelines/
data_processing/
nodes.py # Pure Python functions
pipeline.py # Pipeline definition
modeling/
nodes.py
pipeline.py
pipeline_registry.py
data/ # Local data directory
notebooks/ # Jupyter notebooks
tests/ # Unit tests
```
Install project dependencies:
```bash
pip install -e ".[dev]"
```
## Core Concepts
**Nodes**: The fundamental units of computation in Kedro. Each node is a pure Python function with explicitly declared inputs and outputs:
```python
# src/my_research_project/pipelines/data_processing/nodes.py
import pandas as pd
from sklearn.preprocessing import StandardScaler
def clean_raw_data(raw_data: pd.DataFrame) -> pd.DataFrame:
"""Remove missing values and outliers from raw experimental data."""
cleaned = raw_data.dropna(subset=["measurement", "condition"])
q1 = cleaned["measurement"].quantile(0.01)
q99 = cleaned["measurement"].quantile(0.99)
return cleaned[cleaned["measurement"].between(q1, q99)]
def normalize_features(
cleaned_data: pd.DataFrame, parameters: dict
) -> pd.DataFrame:
"""Standardize feature columns specified in parameters."""
feature_cols = parameters["feature_columns"]
scaler = StandardScaler()
result = cleaned_data.copy()
result[feature_cols] = scaler.fit_transform(cleaned_data[feature_cols])
return result
```
**Pipelines**: Chains of nodes connected through named datasets:
```python
# src/my_research_project/pipelines/data_processing/pipeline.py
from kedro.pipeline import Pipeline, node, pipeline
from .nodes import clean_raw_data, normalize_features
def create_pipeline(**kwargs) -> Pipeline:
return pipeline([
node(
func=clean_raw_data,
inputs="raw_experiment_data",
outputs="cleaned_data",
name="clean_data_node",
),
node(
func=normalize_features,
inputs=["cleaned_data", "params:preprocessing"],
outputs="normalized_data",
name="normalize_node",
),
])
```
**Data Catalog**: A declarative registry that maps logical dataset names to physical storage:
```yaml
# conf/base/catalog.yml
raw_experiment_data:
type: pandas.CSVDataset
filepath: data/01_raw/experiment_results.csv
cleaned_data:
type: pandas.ParquetDataset
filepath: data/02_intermediate/cleaned.parquet
normalized_data:
type: pandas.ParquetDataset
filepath: data/03_primary/normalized.parquet
versioned: true
```
The `versioned: true` flag automatically creates timestamped versions of outputs, enabling exact reproduction of prior runs.
**Parameters**: Experiment configuration separated from code:
```yaml
# conf/base/parameters.yml
preprocessing:
feature_columns:
- temperature
- pressure
- concentration
outlier_method: iqr
modeling:
algorithm: random_forest
n_estimators: 500
max_depth: 10
test_size: 0.2
random_seed: 42
```
## Running and Visualizing Pipelines
Execute the full pipeline:
```bash
kedro run
```
Run a specific pipeline or node:
```bash
kedro run --pipeline data_processing
kedro run --nodes clean_data_node
```
Visualize the pipeline dependency graph:
```bash
pip install kedro-viz
kedro viz run
```
This launches an interactive web visualization showing the complete data flow, making it easy to understand and communicate your analytical pipeline to collaborators and reviewers.
## Research Workflow Integration
**Experiment Reproducibility**: Every Kedro run uses explicit parameters and versioned data. Store parameter files in Git alongside code to create a complete record of every experiment configuration.
**Reviewer Response**: When peer reviewers request additional analyses or modified parameters, change `parameters.yml` and re-run. The pipeline automatically reprocesses only affected downstream nodes.
**Team Collaboration**: Multiple researchers can work on different pipeline modules simultaneously. The explicit input/output contracts between nodes prevent integration conflicts.
**Scaling Computation**: Kedro pipelines can be deployed to distributed computing platforms without code changes using runners:
```bash
# Run with parallel execution
kedro run --runner=ParallelRunner
# Deploy to Airflow, Prefect, or other orchestrators
pip install kedro-airflow
kedro airflow create
```
**Integration with Jupyter**: Use Kedro notebooks for exploration while maintaining the pipeline for production runs:
```bash
kedro jupyter notebook
```
The Kedro Jupyter integration automatically loads the project catalog and parameters, bridging the gap between interactive exploration and pipeline execution.
## References
- Kedro repository: https://github.com/kedro-org/kedro
- Kedro documentation: https://docs.kedro.org/
- Kedro-Viz for pipeline visualization: https://github.com/kedro-org/kedro-viz
- QuantumBlack blog on reproducible data science
mle-agent-guide/SKILL.md
---
name: mle-agent-guide
description: "Intelligent companion for ML engineering with arXiv integration"
source: https://github.com/MLSys-Tools/MLE-agent
metadata:
openclaw:
category: "research"
subcategory: "automation"
emoji: "🔬"
keywords: [machine-learning, ml-engineering, arxiv-integration, experiment-tracking, model-development, ai-engineering]
---
# MLE Agent Guide
A skill for using an intelligent ML engineering companion that integrates arXiv paper discovery with experiment implementation, tracking, and iteration. Based on MLE-agent (2K stars), this skill helps researchers bridge the gap between reading about new ML techniques and implementing them in their own projects.
## Overview
Machine learning research moves at an extraordinary pace, with hundreds of new papers appearing on arXiv daily. Researchers struggle not just to keep up with the literature but to translate promising ideas into working implementations. MLE-agent addresses this by combining paper discovery, technique extraction, implementation assistance, and experiment management into a unified workflow.
This skill is designed for ML researchers and engineers who want to quickly prototype ideas from papers, systematically compare approaches, and maintain organized experiment records throughout the research process.
## arXiv Integration
The skill provides sophisticated arXiv paper discovery and analysis:
**Paper Discovery**
- Monitor arXiv categories relevant to the researcher's interests (cs.LG, cs.CL, cs.CV, stat.ML, etc.)
- Filter new papers by keyword relevance, author familiarity, and citation velocity
- Rank papers by potential applicability to current research projects
- Generate daily or weekly digests of the most relevant new papers
- Track papers from specific authors or research groups
**Paper Analysis**
- Extract the core technical contribution from each paper
- Identify the proposed method's key components and hyperparameters
- Compare the method against baselines reported in the paper
- Note the datasets and evaluation metrics used
- Assess reproducibility based on available code, data, and method description detail
**Technique Extraction**
- Distill the algorithmic steps from the paper's method section
- Identify required input data formats and preprocessing steps
- Map the technique's computational requirements (GPU memory, training time)
- Note dependencies on specific frameworks or libraries
- Extract training recipes including learning rate schedules, batch sizes, and augmentation strategies
## Experiment Workflow
The core experiment management workflow:
**Project Setup**
- Initialize a structured project directory with standard ML conventions
- Set up experiment configuration files (YAML or JSON)
- Configure logging and metric tracking infrastructure
- Establish a baseline model and dataset pipeline
- Create a reproducibility checklist (random seeds, library versions, hardware specs)
**Implementation Assistance**
- Translate extracted techniques from papers into implementation plans
- Generate code scaffolds for new model architectures
- Suggest appropriate loss functions and optimization strategies
- Help debug common ML issues (gradient problems, data loading bottlenecks, memory issues)
- Provide code review focused on ML best practices
**Experiment Execution**
- Configure hyperparameter search spaces based on paper recommendations
- Set up systematic ablation studies to understand component contributions
- Track metrics across training runs with consistent logging
- Generate comparison tables and learning curve plots
- Flag anomalous training behavior (loss spikes, metric plateaus, divergence)
**Result Analysis**
- Compare results across experiment runs with statistical tests
- Identify which hyperparameters have the strongest effect on performance
- Generate publication-ready result tables with confidence intervals
- Produce ablation study summaries highlighting each component's contribution
- Assess whether results match or deviate from paper-reported numbers
## ML Engineering Best Practices
The skill enforces ML engineering standards throughout the workflow:
**Reproducibility**
- Log all random seeds and ensure deterministic operations where possible
- Record exact library versions in requirements files
- Version control all configuration files alongside code
- Store model checkpoints with their corresponding configurations
- Document any manual steps in the pipeline
**Code Quality**
- Separate data loading, model definition, training, and evaluation into distinct modules
- Use configuration files rather than hardcoded hyperparameters
- Implement proper validation splits independent of test sets
- Add assertions for tensor shapes and value ranges at module boundaries
- Write unit tests for custom layers and data transformations
**Resource Management**
- Estimate computational costs before launching large experiments
- Use gradient accumulation to train with limited GPU memory
- Implement early stopping to avoid wasting compute on unpromising runs
- Profile code to identify and optimize bottlenecks
- Clean up intermediate artifacts and checkpoints to manage storage
## Common ML Research Patterns
The skill recognizes and supports common research patterns:
**Baseline Comparison** - Implement and evaluate standard baselines before proposing improvements
**Ablation Study** - Systematically remove or vary components to understand contributions
**Scaling Analysis** - Test how performance changes with model size, data size, or compute
**Transfer Learning** - Adapt pretrained models to new tasks with appropriate fine-tuning strategies
**Ensemble Methods** - Combine multiple models for improved and more robust performance
## Integration with Research-Claw
This skill connects with the Research-Claw ecosystem:
- Use literature search skills to find relevant papers for the current project
- Feed experiment results to writing skills for paper drafting
- Connect with analysis skills for statistical evaluation of results
- Store successful experiment configurations as reusable templates
- Share experiment logs and results with collaborators via the platform
## Best Practices
- Always establish baselines before implementing novel techniques
- Read the paper's appendix and supplementary materials for implementation details often missing from the main text
- Start with the paper's reported hyperparameters, then tune for your specific setup
- Keep a research log documenting what was tried, what worked, and what failed
- Verify your reimplementation on the paper's original dataset before applying to new data
- Track compute costs alongside performance metrics to evaluate efficiency trade-offs
paper-to-agent-guide/SKILL.md
---
name: paper-to-agent-guide
description: "Transform research papers into interactive AI agents for exploration"
source: https://github.com/paper2agent/Paper2Agent
metadata:
openclaw:
category: "research"
subcategory: "automation"
emoji: "📄"
keywords: [paper-parsing, agent-generation, interactive-papers, research-automation, knowledge-extraction]
---
# Paper-to-Agent Guide
A skill for transforming published research papers into interactive AI agents that can answer questions, explain methodology, and help replicate findings. Based on Paper2Agent (2K stars), this skill guides the agent through extracting structured knowledge from academic papers and creating conversational interfaces for deep exploration.
## Overview
Traditional paper reading is linear and passive. Paper-to-Agent converts this into an active, queryable experience. By parsing a paper's structure, extracting key claims, methodology details, and results, the agent becomes an expert on that specific paper, ready to answer follow-up questions, explain complex sections, and connect findings to the broader literature.
This approach is especially valuable for interdisciplinary researchers who need to quickly understand papers outside their primary expertise, for journal clubs seeking deeper discussion, and for students learning to critically evaluate published research.
## Paper Parsing Workflow
The agent should follow this structured workflow when converting a paper to an interactive agent:
**Step 1: Structure Extraction**
- Identify the paper's sections (abstract, introduction, methods, results, discussion, references)
- Extract the title, authors, affiliations, and publication venue
- Identify figure and table captions along with their referenced locations
- Note supplementary materials and their availability
- Detect the paper type (empirical, theoretical, review, meta-analysis)
**Step 2: Claim Extraction**
- Identify the primary research question or hypothesis
- Extract all major claims made in the paper
- Map each claim to its supporting evidence (data, citations, arguments)
- Note the strength of evidence for each claim (strong, moderate, suggestive)
- Identify limitations acknowledged by the authors
**Step 3: Methodology Mapping**
- Document the complete experimental or analytical pipeline
- Extract parameter values, dataset descriptions, and evaluation metrics
- Identify software tools and libraries used
- Note any preprocessing or data cleaning steps
- Map the methodology to established frameworks in the field
## Interactive Exploration Capabilities
Once a paper has been parsed, the agent can support these interaction patterns:
**Question-Answering**
- Answer specific questions about the paper's content with source references
- Explain technical terms in context of how the paper uses them
- Compare the paper's approach to common alternatives
- Identify what the paper does and does not address
- Generate summaries at different levels of detail (tweet-length, abstract, detailed)
**Critical Analysis**
- Evaluate the validity of statistical analyses
- Identify potential confounds not addressed by the authors
- Assess whether conclusions follow from the presented evidence
- Compare results to related work in the field
- Suggest follow-up experiments that would strengthen the findings
**Replication Assistance**
- Generate step-by-step replication guides from the methods section
- Identify missing details needed for exact replication
- Suggest parameter ranges for robustness checks
- Create data collection templates based on the paper's design
- List required resources (compute, data, equipment) for replication
## Knowledge Graph Construction
The skill supports building knowledge graphs from processed papers:
- Extract entities (methods, datasets, metrics, tools, concepts)
- Map relationships between entities (uses, extends, contradicts, supports)
- Link to external knowledge bases (OpenAlex, CrossRef, DOI)
- Track citation chains for key claims
- Identify research lineages and methodological evolution
## Multi-Paper Analysis
When multiple papers have been processed, the agent can:
- Compare methodologies across papers addressing similar questions
- Identify consensus findings and areas of disagreement
- Trace the evolution of a research direction over time
- Build synthesis summaries combining evidence from multiple sources
- Detect gaps in the literature that no existing paper addresses
## Integration with Research-Claw
This skill connects with other Research-Claw capabilities:
- Use literature search skills to find papers for processing
- Feed extracted knowledge into writing skills for literature reviews
- Connect methodology details to analysis skills for replication
- Store parsed papers in the local knowledge base for future reference
- Generate citation entries compatible with reference management tools
## Practical Tips
- Start with the abstract and conclusion to determine if full parsing is worthwhile
- Focus deep extraction on methods and results sections for empirical papers
- For theoretical papers, prioritize definitions, theorems, and proof sketches
- Always verify extracted claims against the original text before presenting them
- Flag areas where the paper's writing is ambiguous or inconsistent
- Use the parsed representation to generate discussion questions for journal clubs
rd-agent-guide/SKILL.md
---
name: rd-agent-guide
description: "Microsoft AI-driven R&D agent for automated data and model development"
metadata:
openclaw:
emoji: "🤖"
category: "research"
subcategory: "automation"
keywords: ["r-and-d", "microsoft", "automation", "model-development", "data-science", "experiment-automation"]
source: "https://github.com/microsoft/RD-Agent"
---
# RD-Agent Guide
## Overview
RD-Agent is an open-source AI-powered research and development automation framework developed by Microsoft Research, with over 12,000 stars on GitHub. It automates key steps in the R&D lifecycle -- including hypothesis generation, experiment design, code implementation, and result analysis -- enabling researchers and data scientists to accelerate their development cycles significantly.
The framework implements a closed-loop R&D automation pipeline where an AI agent iteratively proposes hypotheses, implements experiments, evaluates results, and refines its approach based on feedback. This mirrors the scientific method but operates at machine speed, allowing researchers to explore a much larger space of ideas and configurations than would be feasible manually.
RD-Agent is particularly valuable for researchers working in quantitative finance, data science, and machine learning, where the development process involves iterating on feature engineering, model architectures, and hyperparameter configurations. The framework has demonstrated the ability to autonomously develop competitive machine learning models and trading strategies, achieving results comparable to experienced human practitioners.
## Installation and Setup
```bash
# Clone the repository
git clone https://github.com/microsoft/RD-Agent.git
cd RD-Agent
# Install dependencies
pip install -e .
# Or install from PyPI
pip install rdagent
```
### Environment Configuration
```bash
# LLM configuration (required)
export OPENAI_API_KEY=$OPENAI_API_KEY
export CHAT_MODEL=gpt-4o
# Or use Azure OpenAI
export AZURE_OPENAI_API_KEY=$AZURE_OPENAI_API_KEY
export AZURE_OPENAI_ENDPOINT=$AZURE_OPENAI_ENDPOINT
export AZURE_OPENAI_DEPLOYMENT=$AZURE_OPENAI_DEPLOYMENT
# Docker is required for sandboxed code execution
# Ensure Docker is installed and running
docker --version
```
RD-Agent uses Docker containers to execute generated code safely, ensuring that automatically generated experiments cannot affect the host system. This sandboxed execution is critical for an autonomous agent that writes and runs arbitrary code.
## Core Concepts
### The R&D Loop
RD-Agent implements a continuous improvement loop with four phases:
1. **Proposal**: The agent analyzes the current state and proposes new hypotheses or improvements
2. **Implementation**: Hypotheses are translated into executable code (feature engineering, model changes, etc.)
3. **Evaluation**: The implemented changes are executed in a sandbox and results are measured against defined metrics
4. **Feedback**: Results are analyzed and used to inform the next round of proposals
```python
from rdagent.core.runner import RDRunner
from rdagent.scenarios.data_science import DataScienceScenario
# Define the research scenario
scenario = DataScienceScenario(
task="tabular_classification",
dataset_path="path/to/dataset.csv",
target_column="label",
metric="auc",
)
# Create and run the R&D agent
runner = RDRunner(
scenario=scenario,
max_iterations=50,
llm_model="gpt-4o",
)
# Start the autonomous R&D loop
results = runner.run()
# Review the best solution found
print(f"Best metric: {results.best_score}")
print(f"Iterations: {results.total_iterations}")
print(f"Solutions explored: {results.num_solutions}")
```
### Scenario Types
RD-Agent supports multiple R&D scenarios out of the box:
#### Data Science / Kaggle Competitions
Automatically engineer features, select models, and tune hyperparameters for tabular data tasks:
```python
from rdagent.scenarios.data_science import DataScienceScenario
scenario = DataScienceScenario(
task="tabular_regression",
dataset_path="data/housing.csv",
target_column="price",
metric="rmse",
time_budget_hours=4,
)
```
#### Quantitative Finance
Develop and backtest trading factors and strategies:
```python
from rdagent.scenarios.qlib import QlibScenario
scenario = QlibScenario(
market="csi300",
task="alpha_factor_mining",
backtest_start="2020-01-01",
backtest_end="2024-12-31",
metric="information_coefficient",
)
```
#### Model Development
Iterate on model architectures and training procedures:
```python
from rdagent.scenarios.model_dev import ModelDevScenario
scenario = ModelDevScenario(
task="image_classification",
base_model="resnet50",
dataset="cifar100",
optimization_target="accuracy",
)
```
## Advanced Features
### Experiment Tracking and Analysis
RD-Agent maintains detailed logs of all experiments, enabling post-hoc analysis of the R&D process:
```python
# Access experiment history
for experiment in results.history:
print(f"Iteration {experiment.iteration}:")
print(f" Hypothesis: {experiment.hypothesis}")
print(f" Changes: {experiment.code_changes}")
print(f" Metric: {experiment.score}")
print(f" Analysis: {experiment.feedback}")
```
### Custom Evaluation Functions
Define custom evaluation metrics for domain-specific research:
```python
from rdagent.core.evaluation import EvaluationFunction
class CustomMetric(EvaluationFunction):
def evaluate(self, predictions, ground_truth, **kwargs):
# Your custom metric computation
score = compute_domain_specific_metric(predictions, ground_truth)
return {
"primary_metric": score,
"secondary_metrics": {
"precision": compute_precision(predictions, ground_truth),
"recall": compute_recall(predictions, ground_truth),
}
}
scenario = DataScienceScenario(
evaluation_function=CustomMetric(),
# ... other config
)
```
### Human-in-the-Loop Mode
Guide the agent with human feedback at key decision points:
```python
runner = RDRunner(
scenario=scenario,
human_in_the_loop=True,
review_frequency=5, # Review every 5 iterations
)
# The agent will pause for human review at specified intervals
# You can approve, reject, or modify proposed experiments
```
## Research Applications
### Ablation Studies at Scale
Use RD-Agent to systematically explore which components contribute most to model performance:
```python
# Define ablation study
ablation_config = {
"base_model": "your_full_model",
"components_to_ablate": [
"attention_mechanism",
"residual_connections",
"layer_normalization",
"data_augmentation",
],
"metric": "accuracy",
"num_seeds": 5, # Run each configuration with 5 seeds
}
```
### Automated Feature Engineering
Let the agent discover and implement novel features for your dataset:
```python
scenario = DataScienceScenario(
task="feature_engineering",
dataset_path="data/research_data.csv",
existing_features=["feature_a", "feature_b", "feature_c"],
target="outcome",
max_new_features=20,
)
```
### Reproducibility
Every experiment run by RD-Agent is fully reproducible. The framework saves the complete experiment specification including code, data transformations, random seeds, and environment details, enabling other researchers to reproduce and build upon the results.
## References
- Repository: https://github.com/microsoft/RD-Agent
- Microsoft Research blog post: https://www.microsoft.com/en-us/research/project/rd-agent/
- Qlib quantitative platform: https://github.com/microsoft/qlib
- Documentation: https://microsoft.github.io/RD-Agent/
research-workflow-automation/SKILL.md
---
name: research-workflow-automation
description: "Automate repetitive research tasks with pipelines, schedulers, and scripting"
metadata:
openclaw:
emoji: "⚙️"
category: "research"
subcategory: "automation"
keywords: ["workflow management", "pipeline scheduler", "research automation", "scientific workflow", "task automation"]
source: "wentor"
---
# Research Workflow Automation
A skill for automating repetitive research tasks using workflow managers, pipeline tools, and scripting. Covers data pipeline design, experiment tracking, automated reporting, and reproducible research workflows.
## Workflow Management Tools
### Tool Comparison
| Tool | Language | Best For | Complexity | License |
|------|----------|----------|-----------|---------|
| Snakemake | Python | Bioinformatics, data pipelines | Medium | MIT |
| Nextflow | Groovy/DSL | Genomics, HPC | Medium | Apache 2.0 |
| Prefect | Python | Data engineering, ML | Medium | Apache 2.0 |
| Airflow | Python | Scheduled ETL pipelines | High | Apache 2.0 |
| Make | Makefile | Simple file-based pipelines | Low | GPL |
| DVC | YAML/CLI | ML experiment tracking | Low | Apache 2.0 |
### Snakemake: Scientific Workflow Example
```python
# Snakefile for a research data pipeline
# Configuration
configfile: "config.yaml"
# Define the final outputs
rule all:
input:
"results/figures/main_figure.pdf",
"results/tables/summary_table.csv",
"results/manuscript_stats.json"
# Step 1: Download and preprocess data
rule download_data:
output:
"data/raw/{dataset}.csv"
params:
url = lambda wildcards: config["datasets"][wildcards.dataset]["url"]
shell:
"curl -L {params.url} -o {output}"
rule clean_data:
input:
"data/raw/{dataset}.csv"
output:
"data/cleaned/{dataset}.parquet"
script:
"scripts/clean_data.py"
# Step 2: Run analysis
rule statistical_analysis:
input:
expand("data/cleaned/{dataset}.parquet",
dataset=config["datasets"].keys())
output:
"results/analysis/statistics.json",
"results/analysis/model_fits.pkl"
threads: 4
resources:
mem_mb = 8000
script:
"scripts/run_analysis.py"
# Step 3: Generate figures
rule create_figures:
input:
"results/analysis/statistics.json"
output:
"results/figures/main_figure.pdf"
script:
"scripts/create_figures.py"
# Step 4: Generate summary table
rule summary_table:
input:
"results/analysis/statistics.json"
output:
"results/tables/summary_table.csv"
script:
"scripts/create_tables.py"
```
```bash
# Execute the full pipeline
snakemake --cores 8 --use-conda
# Visualize the workflow DAG
snakemake --dag | dot -Tpdf > workflow.pdf
# Dry run to see what would be executed
snakemake -n
```
## Make-Based Pipelines
### Simple Makefile for Research
```makefile
# Makefile for a research project
.PHONY: all clean data analysis figures paper
# Default target
all: paper
# Data acquisition and cleaning
data/cleaned/dataset.parquet: data/raw/dataset.csv scripts/clean.py
python scripts/clean.py --input $< --output $@
# Analysis
results/statistics.json: data/cleaned/dataset.parquet scripts/analyze.py
python scripts/analyze.py --input $< --output $@
# Figures
results/figures/%.pdf: results/statistics.json scripts/plot_%.py
python scripts/plot_$*.py --input $< --output $@
# Compile paper
paper: results/figures/main.pdf results/figures/supplement.pdf
cd paper && latexmk -pdf main.tex
# Clean all generated files
clean:
rm -rf data/cleaned/ results/ paper/*.pdf paper/*.aux paper/*.log
```
## Experiment Tracking
### MLflow for Research Experiments
```python
import mlflow
import json
def track_experiment(experiment_name: str, params: dict,
metrics: dict, artifacts: list[str] = None):
"""
Track a research experiment with MLflow.
Args:
experiment_name: Name of the experiment series
params: Hyperparameters or configuration
metrics: Results metrics
artifacts: Paths to output files to log
"""
mlflow.set_experiment(experiment_name)
with mlflow.start_run():
# Log parameters
for key, value in params.items():
mlflow.log_param(key, value)
# Log metrics
for key, value in metrics.items():
mlflow.log_metric(key, value)
# Log artifacts (figures, data files, etc.)
if artifacts:
for artifact_path in artifacts:
mlflow.log_artifact(artifact_path)
# Log the full configuration as JSON
mlflow.log_dict(params, "config.json")
run_id = mlflow.active_run().info.run_id
print(f"Experiment logged: {run_id}")
return run_id
# Example: track a statistical analysis
track_experiment(
experiment_name="treatment_effect_study",
params={
'model': 'linear_regression',
'covariates': 'age,sex,baseline_score',
'alpha': 0.05,
'data_version': 'v2.3'
},
metrics={
'r_squared': 0.42,
'treatment_effect': 0.35,
'p_value': 0.003,
'n_subjects': 245
},
artifacts=['results/figures/main.pdf']
)
```
## Automated Reporting
### Generate Reports from Analysis Results
```python
from jinja2 import Template
from datetime import datetime
def generate_report(results: dict, template_path: str,
output_path: str):
"""
Auto-generate a research report from analysis results.
"""
report_template = Template("""
# Analysis Report
Generated: {{ timestamp }}
## Summary Statistics
- Sample size: {{ results.n }}
- Mean outcome: {{ "%.2f"|format(results.mean) }}
- Standard deviation: {{ "%.2f"|format(results.std) }}
## Main Results
- Treatment effect: {{ "%.3f"|format(results.effect) }}
(95% CI: {{ "%.3f"|format(results.ci_lower) }} to {{ "%.3f"|format(results.ci_upper) }})
- p-value: {{ "%.4f"|format(results.p_value) }}
- Effect size (Cohen's d): {{ "%.2f"|format(results.cohens_d) }}
## Interpretation
{% if results.p_value < 0.05 %}
The treatment effect is statistically significant at the 5% level.
{% else %}
The treatment effect is not statistically significant at the 5% level.
{% endif %}
""")
report = report_template.render(
results=results,
timestamp=datetime.now().strftime('%Y-%m-%d %H:%M')
)
with open(output_path, 'w') as f:
f.write(report)
return output_path
```
## Scheduling and Cron Jobs
### Automated Data Collection
```bash
# Crontab entry: run daily at 6 AM
0 6 * * * cd /home/researcher/project && python scripts/daily_data_fetch.py >> logs/fetch.log 2>&1
# Weekly analysis update (every Monday at 9 AM)
0 9 * * 1 cd /home/researcher/project && snakemake --cores 4 >> logs/pipeline.log 2>&1
```
## Best Practices
1. **Version everything**: Code, data, configurations, and environments
2. **Idempotent pipelines**: Running the same pipeline twice produces the same output
3. **Fail fast**: Validate inputs early; do not process bad data silently
4. **Log everything**: Record timestamps, parameters, and random seeds
5. **Separate configuration from code**: Use YAML/JSON config files, not hardcoded values
6. **Test with small data first**: Use a 1% sample to verify the pipeline before full runs
7. **Document the workflow**: A README explaining how to run the full pipeline from scratch
SKILL.md
---
name: automation-skills
description: "10 research automation skills. Trigger: automating experiments, tracking results, reproducible pipelines. Design: ML experiment management, workflow orchestration, and lab automation tools."
---
# Research Automation — 10 Skills
Select the skill matching the user's need, then `read` its SKILL.md.
| Skill | Description |
|-------|-------------|
| [ai-scientist-v2-guide](./ai-scientist-v2-guide/SKILL.md) | Automated scientific discovery via agentic tree search by Sakana AI |
| [aim-experiment-guide](./aim-experiment-guide/SKILL.md) | Track and compare research experiments with Aim experiment tracker |
| [claude-academic-workflow-guide](./claude-academic-workflow-guide/SKILL.md) | Claude Code template for LaTeX, Beamer, and R research workflows |
| [data-collection-automation](./data-collection-automation/SKILL.md) | Automate survey deployment, data collection, and pipeline management |
| [datagen-research-guide](./datagen-research-guide/SKILL.md) | AI-driven multi-agent research assistant for end-to-end studies |
| [kedro-pipeline-guide](./kedro-pipeline-guide/SKILL.md) | Build reproducible data science pipelines with Kedro for research projects |
| [mle-agent-guide](./mle-agent-guide/SKILL.md) | Intelligent companion for ML engineering with arXiv integration |
| [paper-to-agent-guide](./paper-to-agent-guide/SKILL.md) | Transform research papers into interactive AI agents for exploration |
| [rd-agent-guide](./rd-agent-guide/SKILL.md) | Microsoft AI-driven R&D agent for automated data and model development |
| [research-workflow-automation](./research-workflow-automation/SKILL.md) | Automate repetitive research tasks with pipelines, schedulers, and scripting |