vasilyu1983/ai-agents-public · GitHub
Builds ML, responsible-AI, and multimodal models. Use when doing data science or explaining fairness, privacy, speech, vision-language, or diffusion mechanics.
프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.
npx skills add vasilyu1983/ai-agents-public --skill ai-ml-data-science설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.
agents/openai.yamlinterface: display_name: "Data Science Engineering Suite - Quick Reference" short_description: "ML, responsible-AI, and multimodal modelling workflows" default_prompt: "Use $ai-ml-data-science to build ML models or explain responsible-AI and multimodal mechanics: fairness, privacy, speech, vision-language, and diffusion."
assets/eda/template-eda.md# EDA Template A reusable structure for exploratory data analysis with explicit leakage, versioning, and decision-readiness checks. --- ## 1. Run Context - Dataset version or snapshot - Environment or entrypoint (`uv`, script, notebook) - Prediction target - Prediction timestamp rule - Key business question --- ## 2. Dataset Summary - Shape - Columns - Dtypes - Unique key check - Memory usage - Time range --- ## 3. Missingness Analysis | Column | % Missing | Pattern | Action | |--------|-----------|---------|--------| | | | | | --- ## 4. Outlier Analysis - Outlier definition: <method> - Detected outliers: <summary> - Treatment plan: <method> - Illegal value checks: <notes> --- ## 5. Distribution Analysis ### Numeric - Histograms - Boxplots - Quantiles ### Categorical - Category counts - Rare category scan ### Target - Distribution - Class imbalance or skew - Baseline expectation --- ## 6. Feature Availability And Leakage - Features available at prediction time - Features that need lagging or windowing - Global statistics that must be fit on train only - Columns to drop due to leakage risk - Entity or time-based split concerns --- ## 7. Important Slices - Geography or market segments - User or account cohorts - Product or category groups - Rare or high-risk subsets --- ## 8. EDA Deliverables - Summary of findings - Risks - Recommended split strategy - Recommended next modeling step
assets/evaluation/template-evaluation-report.md# Model Evaluation Report Use this report when deciding whether a model candidate should deploy, iterate, or be rejected. --- ## 1. Executive Summary - Recommendation: <Deploy / Iterate / Reject> - Primary reason: <short explanation> - Baseline beaten: <yes/no> - Main open risk: <short explanation> --- ## 2. Objective And Decision Context - Business context - Intended action triggered by the model - Task type - Prediction timestamp / decision point - Success criteria --- ## 3. Data Description | Dataset | Version / Snapshot | Size | Time Range | Notes | |--------|---------------------|------|------------|-------| **Known Data Risks** - <risk 1> - <risk 2> **Validation / Contract Checks** - Schema validation: <details> - Freshness expectation: <details> - Leakage checks completed: <details> --- ## 4. Feature Engineering Summary - Key numeric features - Categorical encodings - Text / embedding features - Datetime / event features - Train / serve parity notes - Prediction-time availability notes --- ## 5. Model Experiments | Model | Split Strategy | Primary Metric | Notes | |-------|----------------|----------------|-------| **Final Candidate Chosen Because** - <reason 1> - <reason 2> --- ## 6. Metrics, Threshold, And Calibration - Primary metric: <value> - Guardrails: <list> - Threshold / ranking policy: <details> - Calibration status: <checked / adjusted / not applicable> - Calibration metric(s): <Brier / ECE / note> --- ## 7. Slice Analysis | Slice | N | Metric | Gap vs Overall | Action | |-------|---|--------|----------------|--------| --- ## 8. Error Analysis - Systematic error patterns - Representative failure cases - Hypothesized causes - Candidate fixes --- ## 9. Uncertainty - Metric confidence interval: <value> - Method: <bootstrap / repeated CV / other> - Prediction interval or conformal method: <details / n/a> - Interpretation for stakeholders: <short explanation> --- ## 10. Risks And Mitigations | Risk | Mitigation | |------|------------| | | | --- ## 11. Deployment Readiness - [ ] Beats baseline meaningfully - [ ] Threshold or ranking rule defined - [ ] Calibration reviewed - [ ] Uncertainty treatment reviewed - [ ] Weak slices documented - [ ] Owner and monitoring expectations defined --- ## 12. Appendix - Hyperparameters - Seeds - Environment info - Reproduction command(s)
assets/evaluation/template-model-card.md# Model Card A concise, production-grade model card for stakeholder communication and governance. --- ## Model Overview - Model name - Version - Date - Owners - Intended use - Out-of-scope uses - Prediction target and timestamp rule --- ## Data Summary - Sources - Dataset version(s) - Date ranges - Known biases - Data limitations - Sensitive or regulated fields --- ## Performance Summary | Metric | Value | Notes | |--------|-------|-------| | | | | Decision policy: - Operating threshold: <value or policy> - Calibration status: <not checked / pending / complete> - Uncertainty treatment: <CI / conformal / interval / none> Slice performance: - <slice>: <metric> - <slice>: <metric> --- ## Safety And Fairness Considerations - Sensitive attributes - Bias risks - Failure cases - Non-approved uses or misuse risks - Mitigations --- ## Operational Contract - Input schema - Output schema - Score interpretation - Expected latency - Training environment - Hardware requirements - Upstream dependencies --- ## Maintenance Plan - Monitoring strategy - Retraining cadence - Ownership and support - Rollback or fallback plan
assets/features/template-feature-engineering.md# Feature Engineering Template Use this document to define, track, and validate engineered features with point-in-time correctness. --- ## 1. Overview **Target Variable:** <describe> **Prediction Timestamp Rule:** <describe what information is available at scoring time> **Dataset Version:** <snapshot or id> **Feature Set Version:** <vX.Y> --- ## 2. Raw -> Engineered Feature Mapping | Raw Column | Transformation | Output Feature | Available At Prediction Time? | Notes | |------------|----------------|----------------|-------------------------------|-------| | | | | | | --- ## 3. Numeric Features **Scaling:** - <method> **Outlier Handling:** - <method> **Transformations:** - log(x) - sqrt(x) - binning --- ## 4. Categorical Features **Encoding Types:** - One-hot - Frequency - Target (with CV) - Native categorical **Rules:** - Handle rare categories - Map unseen categories - Version encoder logic --- ## 5. Text Features **Preprocessing:** - Lowercase or preserve case - Strip HTML - Remove obvious noise **Representations:** - TF-IDF - Pretrained embeddings --- ## 6. Datetime Features - Day of week - Hour of day - Weekend flag - Holiday flag - Lag and rolling windows **Leakage Checks:** - No use of future information - Timezone alignment confirmed --- ## 7. Contracts And Parity - Schema validation: <Pandera / GX Core / other> - Shared train/serve transforms: <yes/no> - Backfill or recompute notes: <notes> - Sensitive feature handling: <notes> --- ## 8. Final Feature List | Feature | Type | Description | Owner | |---------|------|-------------|-------| | | | | | --- ## 9. Validation Checklist - [ ] Deterministic transformations - [ ] Leakage reviewed - [ ] Train/serve parity ensured - [ ] Dataset and feature versions recorded - [ ] Sensitive features reviewed - [ ] Validation checks implemented
assets/project/template-quick.md# Quick DS Workflow Template Use this for short experiments, feasibility checks, or fast iteration cycles. --- ## Objective <Define the problem in 2-3 sentences.> **Decision Triggered:** <what will the model output change?> **Prediction Timestamp:** <what information is available at scoring time?> --- ## Data - Dataset(s): <names> - Version / snapshot: <id> - Rows / columns: <shape> - Main quality issues: <list> --- ## EDA Notes - Top findings - Missingness summary - Leakage risks - Key slices to watch --- ## Features - Key engineered features - Encoders / transforms used - Point-in-time safeguards --- ## Models Tried - Baseline: <model + metric> - Candidate(s): <models> - Split strategy: <temporal / group / random> --- ## Evaluation - Primary metric: <value> - Threshold / ranking rule: <value> - Calibration checked: <yes/no + note> - Uncertainty checked: <yes/no + note> --- ## Best Current Candidate - Model: <name> - Why it wins: <short reason> - Main limitation: <short reason> --- ## Recommendation - <Deploy / Iterate / Reject> - Next step: <single next action>
assets/project/template-standard.md# Standard Data Science Project Template Use this template for a full DS project plan, experiment package, or model-candidate handoff. --- ## 1. Project Overview **Objective:** <Describe the business problem, target outcome, and decision impact.> **Decision Triggered By Model:** <What action will a prediction or score cause?> **Task Type:** <classification / regression / ranking / forecasting-like event modelling / other> **Prediction Timestamp / Decision Point:** <Exactly what information is available when the prediction is made?> --- ## 2. Success Criteria - Primary metric: <metric> - Baseline to beat: <baseline> - Minimum acceptable threshold for success: <value> **Guardrails** - Calibration: <required / not required> - Uncertainty or interval requirement: <required / not required> - Fairness or sensitive-slice constraints: <list> - Latency / cost / size constraints: <list> --- ## 3. Reproducible Environment - Runtime manager: <uv / other> - Python version: <version> - Entry points: <scripts / marimo notebook / package module> - Key dependencies: <list> - Experiment tracking: <MLflow / W&B / other> --- ## 4. Data Summary | Dataset | Source | Version / Snapshot | Time Range | Rows | Notes | |--------|--------|--------------------|------------|------|-------| | | | | | | | **Data Risks** - <coverage gap> - <label quality issue> - <freshness issue> **Contracts / Validation** - Schema validation: <Pandera / GX Core / other> - Freshness expectation: <details> - Duplicate / entity checks: <details> --- ## 5. EDA Summary - Top findings - Missingness overview - Outlier patterns - Leakage risks - Slice risks - Target distribution notes --- ## 6. Feature Engineering Plan **Feature Set Version:** <vX.Y> **Numeric Features:** - Scaling: <method> - Outlier handling: <method> **Categorical Features:** - Encoding method: <one-hot / target / frequency / CatBoost-native / other> **Text / Embeddings:** - Representation: <tfidf / embeddings / other> **Datetime / Event Features:** - Extracted: <list> - Timezone handling: <details> - Point-in-time safeguards: <details> **Train / Serve Parity** - Offline transform path: <details> - Serving-time or shared transform path: <details> --- ## 7. Modelling Plan **Baselines** - Simple baseline: <name> - Interpretable baseline: <name> **Candidate Models** - Candidate 1: <model> - Candidate 2: <model> **Validation Strategy** - Split type: <temporal / group / random> - Validation design: <holdout / CV / rolling window> - Final test set: <size and rationale> - Seed policy: <single / repeated seeds> **Tuning Budget** - Search method: <manual / random / Bayesian> - Trial budget: <count or compute budget> --- ## 8. Evaluation Plan **Primary Metric:** <metric> **Threshold / Decision Policy:** <operating threshold, top-k rule, or interval policy> **Calibration Plan:** <how calibration will be checked or improved> **Uncertainty Plan:** <bootstrap CI / prediction intervals / conformal / none> **Slice Evaluation** - geography / market: <yes/no> - user or product segment: <yes/no> - recency / time period: <yes/no> - sensitive features: <yes/no and conditions> --- ## 9. Deliverables - Reproducible analysis entrypoint - Data validation checks - Feature engineering specification - Training pipeline or script - Evaluation report - Model card - Production handoff package --- ## 10. Deployment Readiness Gate - [ ] Beats baseline meaningfully - [ ] Threshold or ranking policy defined - [ ] Calibration reviewed - [ ] Uncertainty treatment reviewed - [ ] Weak slices documented with actions - [ ] Risks and rollback notes documented - [ ] Owner assigned --- ## 11. Risks & Mitigations | Risk | Mitigation | Owner | |------|------------|-------| | | | | --- ## 12. Final Recommendation - Recommendation: <Deploy / Iterate / Reject> - Reason: <short explanation> - Next owner: <team / person>
assets/review/experiment-review-template.md# ML Experiment Review Template **Purpose**: validate methodology, prevent leakage, and document whether the experiment is ready for production handoff. --- ## Template Contract ### Goals - Validate methodology, prevent leakage, and document decisions. - Make results reproducible and interpretable for reviewers and stakeholders. ### Inputs - Problem statement and success criteria. - Dataset version(s), split strategy, and feature definitions. - Experiment config (code commit, environment, seeds). ### Decisions - Baseline and final model selection, threshold policy, calibration status, and deployment recommendation. ### Risks - Leakage, metric gaming, overfitting narratives, unstable calibration, and non-reproducible runs. ### Metrics - Primary and secondary metrics with confidence intervals or intervals where applicable. - Slice performance, calibration, and threshold tradeoffs. ## 1. Experiment Metadata ```yaml experiment_id: "" created: "YYYY-MM-DD" author: "" hypothesis: "" status: "planning | running | completed | abandoned" repository: "" commit_hash: "" environment: "" dataset_version: "" feature_set_version: "" prediction_timestamp_rule: "" ``` --- ## 2. Problem Definition ### Business Context - **Problem statement**: _______________ - **Success criteria**: _______________ - **Stakeholder**: _______________ - **Timeline**: _______________ ### ML Framing - **Task type**: [ ] Classification [ ] Regression [ ] Ranking [ ] Clustering [ ] Other - **Target variable**: _______________ - **Prediction horizon / timestamp**: _______________ - **Baseline to beat**: _______________ --- ## 3. Data Review ### Dataset Summary | Attribute | Value | |-----------|-------| | Source | | | Version | | | Rows | | | Columns | | | Time range | | | Target distribution | | | Missing rate (overall) | | ### Leakage Check (CRITICAL) | Check | Status | Notes | |-------|--------|-------| | No features derived from target | [ ] Pass [ ] Fail | | | No future data in features | [ ] Pass [ ] Fail | | | Train/test split is appropriate | [ ] Pass [ ] Fail | Temporal if time-based | | Global statistics on train only | [ ] Pass [ ] Fail | Scalers, encoders | | No test data in validation | [ ] Pass [ ] Fail | | ### Data Quality | Check | Result | Action Taken | |-------|--------|--------------| | Missing values | ___% | | | Duplicates | ___% | | | Outliers | ___ detected | | | Class balance | Ratio: ___ | | | Feature types correct | [ ] Yes [ ] No | | ### Sensitive Features And Slices - [ ] Sensitive attributes identified - [ ] High-risk slices identified - [ ] Exclusions documented --- ## 4. Feature Engineering ### Feature Summary | Feature | Type | Source | Available At Prediction Time? | Rationale | |---------|------|--------|-------------------------------|-----------| | | | | | | ### Feature Validation | Check | Status | Notes | |-------|--------|-------| | No target leakage | [ ] Pass | | | Temporal validity | [ ] Pass | All features available at prediction time | | Missing handled | [ ] Pass | Imputation strategy: ___ | | Encoding appropriate | [ ] Pass | | | Train/serve parity | [ ] Pass | | --- ## 5. Model Review ### Candidate Models | Model | Purpose | Primary Metric | Notes | |-------|---------|----------------|-------| | | | | | ### Decision Policy - **Selected threshold or policy**: _______________ - **Calibration status**: _______________ - **Uncertainty treatment**: _______________ ### Stability - [ ] Seeds logged - [ ] Multiple runs compared where needed - [ ] Environment captured --- ## 6. Evaluation Review | Check | Status | Notes | |-------|--------|-------| | Primary metric justified | [ ] Pass | | | Guardrails defined | [ ] Pass | | | Threshold tradeoff documented | [ ] Pass | | | Calibration reviewed | [ ] Pass | | | Uncertainty documented | [ ] Pass | | | Slice analysis completed | [ ] Pass | | | Error review completed | [ ] Pass | | ### Final Recommendation - [ ] Deploy - [ ] Iterate - [ ] Reject **Conditions before deploy:** __________________________________ **Fallback or rollback candidate:** _____________________________
data/sample-model-spec.json{
"model_name": "customer-churn-predictor",
"version": "1.4.0",
"model_type": "classification",
"task_description": "Predict whether a B2B SaaS customer will churn within the next 30 days, scored daily at midnight UTC.",
"intended_use": "Trigger proactive customer success interventions for accounts with churn probability >= 0.35. Not intended for automated account termination or pricing decisions.",
"training_data": {
"source": "data-warehouse.customers.events_mart",
"date_range": {"start": "2023-01-01", "end": "2025-09-30"},
"row_count": 284500,
"feature_count": 47,
"target_variable": "churned_within_30d",
"temporal_column": "snapshot_date",
"data_collection_date": "2025-10-15"
},
"features": [
{"name": "days_since_last_login", "type": "numeric", "description": "Days elapsed since the account last had any user login, as of snapshot_date"},
{"name": "active_seats_last_30d", "type": "numeric", "description": "Number of distinct user seats with at least one session in the prior 30-day window"},
{"name": "support_tickets_open", "type": "numeric", "description": "Count of unresolved support tickets at snapshot time"},
{"name": "mrr_usd", "type": "numeric", "description": "Monthly recurring revenue in USD at snapshot time"},
{"name": "contract_months_remaining", "type": "numeric", "description": "Months left on current contract term at snapshot_date"},
{"name": "nps_score_last", "type": "numeric", "description": "Most recent NPS survey score (-100 to 100); null if no survey in last 90d"},
{"name": "feature_adoption_pct", "type": "numeric", "description": "Fraction of paid feature modules with at least one event in the last 30d"},
{"name": "plan_tier", "type": "categorical", "description": "Subscription tier: starter, growth, enterprise"},
{"name": "industry_vertical", "type": "categorical", "description": "Account industry classification"},
{"name": "account_age_days", "type": "numeric", "description": "Days since account creation at snapshot_date"},
{"name": "api_calls_last_7d", "type": "numeric", "description": "Total API calls originating from the account in the 7-day window ending at snapshot_date"},
{"name": "billing_overdue_flag", "type": "binary", "description": "1 if any invoice is past due at snapshot_date, else 0"},
{"name": "cs_health_score", "type": "numeric", "description": "Composite customer success health score (0-100) set by CS team, updated weekly"},
{"name": "rollup_logins_prev_quarter", "type": "numeric", "description": "Total login events in the calendar quarter prior to snapshot_date"},
{"name": "export_events_last_30d", "type": "numeric", "description": "Number of data-export events (CSV/API bulk) in the 30-day window; elevated counts may signal offboarding"},
{"name": "integrations_active_count", "type": "numeric", "description": "Number of third-party integrations in a connected state at snapshot_date"}
],
"performance_metrics": [
{"metric": "ROC-AUC", "value": 0.847, "benchmark": 0.800, "split": "temporal-holdout-2025-Q3"},
{"metric": "PR-AUC", "value": 0.612, "benchmark": 0.550, "split": "temporal-holdout-2025-Q3"},
{"metric": "Precision@0.35", "value": 0.701, "benchmark": 0.650, "split": "temporal-holdout-2025-Q3"},
{"metric": "Recall@0.35", "value": 0.583, "benchmark": 0.500, "split": "temporal-holdout-2025-Q3"},
{"metric": "Brier Score", "value": 0.094, "benchmark": 0.120, "split": "temporal-holdout-2025-Q3"},
{"metric": "ECE (calibration)","value": 0.031, "benchmark": 0.050, "split": "temporal-holdout-2025-Q3"}
],
"limitations": [
"Model performance degrades for accounts with fewer than 60 days of history — flag these as low-confidence predictions.",
"NPS score is missing for ~38% of accounts; the model imputes median; predictions for this segment are less reliable.",
"Trained only on B2B SaaS accounts; not validated for B2C or marketplace customers.",
"Seasonal effects (e.g., end-of-year budget cycles) are partially captured but not explicitly modelled.",
"Model does not account for macroeconomic shocks or sudden product outages that would shift baseline churn rates."
],
"ethical_considerations": [
"Churn scores must not be used to deprioritize support for accounts predicted to churn — this creates a self-fulfilling prophecy.",
"Score distributions should be audited quarterly across industry_vertical to detect disparate false-positive rates.",
"CS interventions triggered by this model must be logged for feedback collection and bias monitoring.",
"Do not surface raw probability scores to customers; communicate only through internal tooling."
],
"prediction_timestamp_defined": true,
"prediction_timestamp_field": "snapshot_date",
"label_timestamp_field": "churn_event_date",
"train_val_test_split_method": "temporal",
"split_config": {
"train_end": "2025-06-30",
"val_end": "2025-08-31",
"test_end": "2025-09-30"
},
"lineage": {
"experiment_id": "exp-churn-v14-20251010",
"git_commit": "a3f9c12",
"feature_store_version": "feast-churn-2025-10",
"mlflow_run_id": "8b2e44f1dc2a4c60b7f3a591e1c8d0ae",
"model_artifact": "s3://ml-models/churn/v1.4.0/model.pkl"
}
}
data/sources.json{
"metadata": {
"skill": "ai-ml-data-science",
"updated": "2026-08-24",
"total_sources": 55,
"description": "Curated sources for practical data science workflows: framing, EDA, validation, responsible-AI mechanics, multimodal and speech modelling, evaluation, reproducibility, and production handoff.",
"version": "4.6",
"title": "AI Ml Data Science - Sources",
"last_updated": "2026-08-24"
},
"categories": {
"foundational_papers_and_books": [
{
"name": "The Elements of Statistical Learning",
"url": "https://hastie.su.domains/ElemStatLearn/",
"type": "book",
"relevance": "Core reference for statistical learning concepts and evaluation foundations.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "An Introduction to Statistical Learning",
"url": "https://www.statlearning.com/",
"type": "book",
"relevance": "Practical ML and statistics reference for model selection, validation, and interpretation.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Datasheets for Datasets",
"url": "https://arxiv.org/abs/1803.09010",
"type": "research",
"relevance": "Dataset documentation framework for governance, provenance, and risk assessment.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Model Cards for Model Reporting",
"url": "https://arxiv.org/abs/1810.03993",
"type": "research",
"relevance": "Standardized reporting template for model performance, intended use, and limitations.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Hidden Technical Debt in Machine Learning Systems",
"url": "https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html",
"type": "research",
"relevance": "Classic reference on production ML failure modes and maintenance costs.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Hands-On Large Language Models (Alammar & Grootendorst, O'Reilly 2024)",
"url": "https://www.oreilly.com/library/view/hands-on-large-language/9781098150952/",
"type": "book",
"relevance": "Chapter 5 is the primary-adjacent source for the modular text-clustering and topic-modeling pipeline (embed -> UMAP -> HDBSCAN -> c-TF-IDF -> representation models); Grootendorst authored BERTopic.",
"update_frequency": "static",
"access": "paid",
"add_as_web_search": false
},
{
"name": "BERTopic: Neural topic modeling with a class-based TF-IDF procedure",
"url": "https://arxiv.org/abs/2203.05794",
"type": "research",
"relevance": "Defines c-TF-IDF class-based term weighting and the modular clustering-plus-representation topic pipeline.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction",
"url": "https://arxiv.org/abs/1802.03426",
"type": "research",
"relevance": "Dimensionality-reduction stage of the text-clustering pipeline; nonlinear structure handling versus PCA.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "hdbscan: Hierarchical density based clustering (JOSS 2017)",
"url": "https://joss.theoj.org/papers/10.21105/joss.00205",
"type": "research",
"relevance": "Density clustering stage: cluster count not specified in advance, explicit outlier handling.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
}
],
"python_workflow_and_compute": [
{
"name": "uv Documentation",
"url": "https://docs.astral.sh/uv/",
"type": "documentation",
"relevance": "Modern Python project, dependency, and script management for reproducible DS workflows.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "marimo Documentation",
"url": "https://docs.marimo.io/",
"type": "documentation",
"relevance": "Git-friendly, reactive notebooks for exploratory analysis and reviewable DS workflows.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "pandas Documentation",
"url": "https://pandas.pydata.org/docs/",
"type": "documentation",
"relevance": "Core tabular data manipulation and analysis reference.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "Polars Documentation",
"url": "https://docs.pola.rs/",
"type": "documentation",
"relevance": "High-performance dataframe engine with lazy execution for scalable transforms.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "DuckDB Python Documentation",
"url": "https://duckdb.org/docs/current/clients/python/overview.html",
"type": "documentation",
"relevance": "Embedded analytical SQL engine for local reproducible DS analysis.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
}
],
"core_ml_and_evaluation": [
{
"name": "scikit-learn Documentation",
"url": "https://scikit-learn.org/stable/",
"type": "documentation",
"relevance": "Core preprocessing, model selection, metrics, threshold tuning, and calibration utilities. Verified current stable release is 1.9.0 (2026-06-02); check installed version before citing API specifics.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "LightGBM Documentation",
"url": "https://lightgbm.readthedocs.io/",
"type": "documentation",
"relevance": "Strong tabular baseline for fast iteration and structured-data experiments.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "CatBoost Documentation",
"url": "https://catboost.ai/en/docs/",
"type": "documentation",
"relevance": "Gradient boosting with strong categorical support and competitive tabular performance.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "PyTorch Documentation",
"url": "https://pytorch.org/docs/stable/",
"type": "documentation",
"relevance": "Deep learning framework reference when neural approaches are justified.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "MAPIE Documentation",
"url": "https://mapie.readthedocs.io/",
"type": "documentation",
"relevance": "Prediction intervals and conformal prediction utilities for uncertainty-aware model outputs.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
}
],
"data_quality_and_validation": [
{
"name": "Pandera Documentation",
"url": "https://pandera.readthedocs.io/en/stable/",
"type": "documentation",
"relevance": "Schema and dataframe validation for DS pipelines and feature contracts.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "Great Expectations Core Documentation",
"url": "https://docs.greatexpectations.io/docs/core/introduction/",
"type": "documentation",
"relevance": "Expectation-based validation workflows for dataset and pipeline checks.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "Evidently Documentation",
"url": "https://docs.evidentlyai.com/",
"type": "documentation",
"relevance": "Data quality, drift, and monitoring metrics used in DS handoff and validation reviews.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
}
],
"experimentation_and_automation": [
{
"name": "karpathy/autoresearch",
"url": "https://github.com/karpathy/autoresearch",
"type": "tool",
"relevance": "Autonomous ML experiment agent: bounded modification surface, fixed eval metric (val_bpb), git-as-experiment-ledger, 5-min time budget per run. Reference for Pattern 8.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
}
],
"reproducibility_and_handoff": [
{
"name": "MLflow Documentation",
"url": "https://mlflow.org/docs/latest/",
"type": "documentation",
"relevance": "Experiment tracking, model registry, and artifact lineage for reproducible DS work.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "Weights & Biases Documentation",
"url": "https://docs.wandb.ai/",
"type": "documentation",
"relevance": "Experiment tracking, comparisons, artifact lineage, and collaborative reviews.",
"update_frequency": "continuous",
"access": "free_tier",
"add_as_web_search": true
},
{
"name": "DVC Documentation",
"url": "https://dvc.org/doc",
"type": "documentation",
"relevance": "Data and pipeline version tracking for reproducible experiments.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "lakeFS Documentation",
"url": "https://docs.lakefs.io/",
"type": "documentation",
"relevance": "Git-like versioning for data lakes and branchable data experiments.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "Feast Documentation",
"url": "https://docs.feast.dev/getting-started/concepts/feature-retrieval",
"type": "documentation",
"relevance": "Feature-store reference for train/serve parity and reusable feature definitions.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
},
{
"name": "SQLMesh Documentation",
"url": "https://sqlmesh.readthedocs.io/en/stable/",
"type": "documentation",
"relevance": "SQL transformation patterns for staging, intermediate, and marts layers with testing.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true
}
],
"llm_data_pipelines": [
{
"name": "TabPFN-2.5 (arXiv 2511.08667)",
"url": "https://arxiv.org/abs/2511.08667",
"type": "research",
"relevance": "TabPFN-2.5 extends the TabPFN family to 50k rows and 2k features. Confirmed resolving as of 2026-05-17. Win-rate figures are model-version-specific; hedge before asserting.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false,
"updated": "2026-05-17"
},
{
"name": "text-dedup (MinHash/LSH deduplication library)",
"url": "https://github.com/ChenghaoMou/text-dedup",
"type": "tool",
"relevance": "Production-grade MinHash LSH and exact dedup for large text corpora. Used in LLM data pipeline reference.",
"update_frequency": "continuous",
"access": "free",
"add_as_web_search": true,
"updated": "2026-05-17"
},
{
"name": "Min-K% Prob — Contamination Detection (arXiv 2310.16789)",
"url": "https://arxiv.org/abs/2310.16789",
"type": "research",
"relevance": "Post-training benchmark contamination detection via minimum-k% token log-probability. Cited in evaluation-patterns.md §10 and llm-data-pipeline.md §4.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false,
"updated": "2026-05-17"
},
{
"name": "Cleanlab Documentation",
"url": "https://docs.cleanlab.ai/",
"type": "documentation",
"relevance": "Confident learning and label noise detection for supervised ML datasets. Referenced in data-contracts-lineage.md §7 annotation-quality section.",
"update_frequency": "continuous",
"access": "free_tier",
"add_as_web_search": true,
"updated": "2026-05-17"
}
],
"responsible_ai_and_multimodal_primary_sources": [
{
"name": "Fairness and Abstraction in Sociotechnical Systems",
"url": "https://doi.org/10.1145/3287560.3287598",
"type": "research",
"relevance": "Primary source for connecting fairness definitions to sociotechnical system boundaries and affected populations.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "The Algorithmic Foundations of Differential Privacy",
"url": "https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf",
"type": "book",
"relevance": "Foundational treatment of differential privacy, composition, and privacy-utility reasoning.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Why Should I Trust You? Explaining the Predictions of Any Classifier",
"url": "https://arxiv.org/abs/1602.04938",
"type": "research",
"relevance": "Primary LIME paper for local post-hoc explanation and its assumptions.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "A Unified Approach to Interpreting Model Predictions",
"url": "https://arxiv.org/abs/1705.07874",
"type": "research",
"relevance": "Primary SHAP paper for additive feature-attribution explanations.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Deep Leakage from Gradients",
"url": "https://arxiv.org/abs/1906.08935",
"type": "research",
"relevance": "Demonstrates privacy leakage from shared gradients, important for federated-learning threat models.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Extracting Training Data from Large Language Models",
"url": "https://arxiv.org/abs/2012.07805",
"type": "research",
"relevance": "Primary evidence for memorization and training-data extraction risk.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Learning Transferable Visual Models From Natural Language Supervision",
"url": "https://arxiv.org/abs/2103.00020",
"type": "research",
"relevance": "Primary CLIP paper for dual-encoder image-text contrastive pretraining and zero-shot transfer.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Sigmoid Loss for Language Image Pre-Training",
"url": "https://arxiv.org/abs/2303.15343",
"type": "research",
"relevance": "Primary SigLIP paper for pairwise sigmoid image-text alignment without global softmax normalization.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "LayoutLM: Pre-training of Text and Layout for Document Image Understanding",
"url": "https://arxiv.org/abs/1912.13318",
"type": "research",
"relevance": "Primary document-understanding paper combining text and two-dimensional layout.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "High-Resolution Image Synthesis with Latent Diffusion Models",
"url": "https://arxiv.org/abs/2112.10752",
"type": "research",
"relevance": "Primary latent-diffusion source for compressed-space generation and cross-attention conditioning.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Classifier-Free Diffusion Guidance",
"url": "https://arxiv.org/abs/2207.12598",
"type": "research",
"relevance": "Primary source for conditional-unconditional guidance and the fidelity-diversity trade-off.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "On Distillation of Guided Diffusion Models",
"url": "https://arxiv.org/abs/2210.03142",
"type": "research",
"relevance": "Primary source for progressive distillation of guided diffusion sampling.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Robust Speech Recognition via Large-Scale Weak Supervision",
"url": "https://arxiv.org/abs/2212.04356",
"type": "research",
"relevance": "Primary Whisper paper for multilingual encoder-decoder ASR, translation, timestamps, and weak supervision.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Connectionist Temporal Classification",
"url": "https://www.cs.toronto.edu/~graves/icml_2006.pdf",
"type": "research",
"relevance": "Foundational CTC objective for monotonic sequence alignment with a blank symbol.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Sequence Transduction with Recurrent Neural Networks",
"url": "https://arxiv.org/abs/1211.3711",
"type": "research",
"relevance": "Foundational transducer objective combining acoustic encoding with output-history prediction for streaming recognition.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions",
"url": "https://arxiv.org/abs/1712.05884",
"type": "research",
"relevance": "Primary Tacotron 2 source for text-to-mel acoustic modelling followed by a neural vocoder.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "WaveNet: A Generative Model for Raw Audio",
"url": "https://arxiv.org/abs/1609.03499",
"type": "research",
"relevance": "Foundational autoregressive neural waveform model and vocoder reference.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech",
"url": "https://arxiv.org/abs/2105.06337",
"type": "research",
"relevance": "Primary diffusion acoustic-model source for TTS quality and sampling-speed trade-offs.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
}
],
"governance_and_security": [
{
"name": "NIST AI Risk Management Framework 1.0",
"url": "https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf",
"type": "specification",
"relevance": "Governance baseline for AI risk management, documentation, accountability, and controls.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
},
{
"name": "EU AI Act (Regulation (EU) 2024/1689)",
"url": "https://eur-lex.europa.eu/eli/reg/2024/1689/oj",
"type": "specification",
"relevance": "Regulatory baseline affecting documentation, monitoring, and high-risk model governance.",
"update_frequency": "quarterly",
"access": "free",
"add_as_web_search": true
},
{
"name": "NIST Secure Software Development Framework (SSDF)",
"url": "https://csrc.nist.gov/pubs/sp/800/218/final",
"type": "specification",
"relevance": "Secure development practices relevant for DS pipelines and service handoff.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": true
},
{
"name": "NIST AI 600-1 Generative AI Profile",
"url": "https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf",
"type": "specification",
"relevance": "Useful governance reference when DS work overlaps with GenAI systems or agentic evaluation.",
"update_frequency": "static",
"access": "free",
"add_as_web_search": false
}
]
}
}
learnings.consolidated.md# ai-ml-data-science — Consolidated Learnings Curated, dated, committed memory for this skill. Pruned from raw `learnings.md` via `agents-skills-feedback-loop/scripts/consolidate.py`. Human-approved. Cap: 60 entries. When exceeded, promote durable rules to `references/`. ## Filter Override <!-- Add 2-4 bullets that sharpen what counts as a learning for this skill. Leave empty to use the default filter from agents-skills-feedback-loop/references/learnings-format.md. --> ## Patterns That Work ## Mistakes to Avoid ## Domain Knowledge - **2026-05-17** — Neptune.ai hosted service shut down 2026-03-05 following OpenAI acquisition; all hosted data was deleted. Remove from any live tool recommendation; replace with MLflow (open-source default) or Comet ML / ClearML as cloud alternatives. - **2026-05-17** — TabPFN v2/v2.5 is a viable candidate for small-medium tabular datasets (≤10k: add to comparison set; ≤50k: TabPFN-2.5 supports this range). Does NOT replace LightGBM/CatBoost at scale (>50k) and should be framed as an additional comparison candidate, not a default. Specific win-rate figures from arXiv 2511.08667 are model-version-specific; always verify against current paper before asserting numbers. - **2026-05-17** — CQF (classifier quality filtering) limitation: filtering pretraining data toward a quality classifier's distribution improves downstream benchmark scores, but does not necessarily improve LM performance on the reference corpus itself. Benchmark gains may reflect distributional alignment, not genuine capability gains. Hedge any claim about classifier filtering benefits when the evidence is benchmark-only. - **2026-05-17** — Pure-synthetic pretraining does not consistently match natural-text pretraining. Synthetic data helps most as a targeted supplement (rare domains, instruction formats, reasoning chains) or in fine-tuning mixtures. No universal optimal mixing ratio is established across settings; treat published ratios as dataset- and task-specific. - **2026-05-17** — Benchmark contamination from large crawled corpora is a systematic risk for LLM evaluation. MinHash near-dup (Jaccard 0.5–0.7) + 13-gram exact matching are the standard detection pair. Min-K% Prob (arXiv 2310.16789) is the standard post-training detection method. Always decontaminate synthetic data against target benchmarks before including in training. ## Open Questions ## Consolidated Principles
learnings.md# ai-ml-data-science — Learnings ## Patterns That Work ## Mistakes to Avoid - [2026-07-11] SHAP >=0.45.0 returns shap_values() as an ndarray, not a per-class list; shap_values[1] on binary classifiers now indexes sample 1, not the positive class. Use explainer(X) Explanation objects. ## Domain Knowledge - [2026-07-11] 2026-07 versions: Optuna 4.9.x (v4 removed suggest_uniform family), scikit-learn 1.9.0, imbalanced-learn 0.14.x, LightGBM 4.6.x, XGBoost 3.2. Verify before citing. ## Open Questions ## Consolidated Principles
references/class-imbalance-patterns.md# Class Imbalance Patterns
> Operational guide for handling imbalanced datasets in classification tasks. Covers sampling strategies, loss reweighting, threshold tuning, and evaluation metrics that actually reflect minority-class performance.
**Freshness anchor:** verified 2026-07-11 — imbalanced-learn 0.14+, scikit-learn 1.9+, LightGBM 4.6+. imbalanced-learn 0.14 tracks scikit-learn 1.9 compatibility; if pinned to an older scikit-learn, pin imbalanced-learn to a matching older release rather than assuming latest-always-works.
---
## Table of Contents
- [Decision Tree: Choosing an Imbalance Strategy](#decision-tree-choosing-an-imbalance-strategy)
- [Quick Reference: Sampling Methods](#quick-reference-sampling-methods)
- [Operational Patterns](#operational-patterns)
- [Pattern 1: Class Weights (Simplest First)](#pattern-1-class-weights-simplest-first)
- [scikit-learn](#scikit-learn)
- [LightGBM — two options](#lightgbm-—-two-options)
- [XGBoost](#xgboost)
- [Pattern 2: SMOTE Oversampling](#pattern-2-smote-oversampling)
- [CRITICAL: SMOTE inside CV, never before split](#critical-smote-inside-cv-never-before-split)
- [Pattern 3: Undersampling with Ensembles](#pattern-3-undersampling-with-ensembles)
- [Option A: Balanced Random Forest](#option-a-balanced-random-forest)
- [Option B: EasyEnsemble (AdaBoost on balanced subsets)](#option-b-easyensemble-adaboost-on-balanced-subsets)
- [Pattern 4: Threshold Tuning via PR Curve](#pattern-4-threshold-tuning-via-pr-curve)
- [F1-optimal threshold](#f1-optimal-threshold)
- [F-beta for recall-heavy use cases (e.g., fraud)](#f-beta-for-recall-heavy-use-cases-eg-fraud)
- [Pattern 5: Cost-Sensitive Learning](#pattern-5-cost-sensitive-learning)
- [Custom sample weights reflecting business cost](#custom-sample-weights-reflecting-business-cost)
- [For LightGBM: per-instance weighting](#for-lightgbm-per-instance-weighting)
- [Pattern 6: Hybrid Sampling](#pattern-6-hybrid-sampling)
- [SMOTE + Tomek (moderate cleanup)](#smote-tomek-moderate-cleanup)
- [SMOTE + ENN (aggressive cleanup — better boundaries, fewer samples)](#smote-enn-aggressive-cleanup-—-better-boundaries-fewer-samples)
- [Evaluation Metrics for Imbalanced Data](#evaluation-metrics-for-imbalanced-data)
- [Metrics Decision Table](#metrics-decision-table)
- [Metric Implementation](#metric-implementation)
- [Anti-Patterns](#anti-patterns)
- [Validation Checklist](#validation-checklist)
- [Cross-References](#cross-references)
## Decision Tree: Choosing an Imbalance Strategy
```
START
│
├─ Imbalance ratio < 5:1?
│ ├─ YES → Class weights usually sufficient
│ │ └─ Try `class_weight='balanced'` first
│ └─ NO → Continue
│
├─ Imbalance ratio 5:1 – 50:1?
│ ├─ Dataset > 50k rows?
│ │ ├─ YES → Undersampling + ensemble (EasyEnsemble, BalancedRF)
│ │ └─ NO → SMOTE or ADASYN oversampling
│ └─ Tree-based model?
│ ├─ YES → `scale_pos_weight` or `is_unbalance` first
│ └─ NO → Sampling + class weights combined
│
├─ Imbalance ratio > 50:1?
│ ├─ Anomaly detection framing viable?
│ │ ├─ YES → Switch to One-Class SVM / Isolation Forest
│ │ └─ NO → Hybrid sampling + cost-sensitive learning
│ └─ Sufficient minority samples (>500)?
│ ├─ YES → SMOTE + Tomek links cleanup
│ └─ NO → Data collection > algorithmic tricks
│
└─ Always: tune decision threshold via PR curve, not default 0.5
```
---
## Quick Reference: Sampling Methods
| Method | Type | Use When | Pitfall |
|--------|------|----------|---------|
| Random oversampling | Over | Quick baseline, < 10k rows | Overfitting on duplicates |
| SMOTE | Over | Continuous features, ratio 5:1–50:1 | Noisy with high dimensionality |
| ADASYN | Over | Hard minority examples matter | Amplifies noise near boundary |
| BorderlineSMOTE | Over | Decision boundary is key | Slower than vanilla SMOTE |
| Random undersampling | Under | Large dataset (>100k), fast iteration | Loses majority-class information |
| Tomek links | Under | Cleaning noisy boundary | Removes too few samples alone |
| NearMiss-1 | Under | Want majority near minority | Aggressive — validate carefully |
| NearMiss-3 | Under | Moderate cleaning | Better than NearMiss-1 for most cases |
| SMOTE + Tomek | Hybrid | Best general-purpose combo | Two-step tuning required |
| SMOTE + ENN | Hybrid | Cleaner boundaries than SMOTE+Tomek | More aggressive cleaning |
---
## Operational Patterns
### Pattern 1: Class Weights (Simplest First)
- **Use when:** Imbalance ratio < 10:1, tree-based or linear model
- **Implementation:**
```python
# scikit-learn
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(class_weight='balanced', n_estimators=300)
# LightGBM — two options
params_a = {'is_unbalance': True} # auto-computes weight
params_b = {'scale_pos_weight': neg_count / pos_count} # manual
# XGBoost
params_xgb = {'scale_pos_weight': neg_count / pos_count}
```
- **Validation:** compare PR-AUC with and without weights
- **Gotcha:** `class_weight='balanced'` uses `n_samples / (n_classes * class_counts)` — verify the math matches your expectations
### Pattern 2: SMOTE Oversampling
- **Use when:** Dataset 1k–50k rows, continuous features, ratio 5:1–100:1
- **Implementation:**
```python
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline
pipeline = ImbPipeline([
('smote', SMOTE(
sampling_strategy=0.5, # target minority:majority ratio
k_neighbors=5, # lower for very small minorities
random_state=42
)),
('clf', RandomForestClassifier(n_estimators=300))
])
# CRITICAL: SMOTE inside CV, never before split
from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipeline, X, y, cv=5, scoring='average_precision')
```
- **Key rule:** NEVER apply SMOTE before train/test split — causes data leakage
- **k_neighbors tuning:** if minority class < 20 samples, set `k_neighbors=3` or lower
### Pattern 3: Undersampling with Ensembles
- **Use when:** Large dataset (>50k rows), need fast training
- **Implementation:**
```python
from imblearn.ensemble import BalancedRandomForestClassifier
from imblearn.ensemble import EasyEnsembleClassifier
# Option A: Balanced Random Forest
brf = BalancedRandomForestClassifier(
n_estimators=300,
sampling_strategy='all',
replacement=False,
random_state=42
)
# Option B: EasyEnsemble (AdaBoost on balanced subsets)
ee = EasyEnsembleClassifier(
n_estimators=20,
random_state=42
)
```
- **Advantage:** retains all minority samples, subsamples majority per tree
- **Gotcha:** BalancedRF can be slower than regular RF due to resampling overhead
### Pattern 4: Threshold Tuning via PR Curve
- **Use when:** Always — default 0.5 threshold is almost never optimal for imbalanced data
- **Implementation:**
```python
from sklearn.metrics import precision_recall_curve
import numpy as np
y_proba = clf.predict_proba(X_test)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_test, y_proba)
# F1-optimal threshold
f1_scores = 2 * (precision * recall) / (precision + recall + 1e-8)
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]
# F-beta for recall-heavy use cases (e.g., fraud)
beta = 2
fbeta = (1 + beta**2) * (precision * recall) / (beta**2 * precision + recall + 1e-8)
best_threshold_fbeta = thresholds[np.argmax(fbeta)]
```
- **Business alignment:** choose beta based on cost of FN vs FP
- `beta=2` — missing positives is 4x worse than false alarms (fraud, medical)
- `beta=0.5` — false alarms are 4x worse than misses (spam, content moderation)
### Pattern 5: Cost-Sensitive Learning
- **Use when:** Business has explicit cost matrix (cost of FN != cost of FP)
- **Implementation:**
```python
# Custom sample weights reflecting business cost
sample_weights = np.where(y_train == 1, cost_fn, cost_fp)
clf.fit(X_train, y_train, sample_weight=sample_weights)
# For LightGBM: per-instance weighting
train_data = lgb.Dataset(X_train, label=y_train, weight=sample_weights)
```
- **Cost matrix example:**
| | Predicted Positive | Predicted Negative |
|---|---|---|
| Actual Positive | 0 (TP) | $500 (FN — missed fraud) |
| Actual Negative | $10 (FP — investigation cost) | 0 (TN) |
- Weight ratio: `cost_fn / cost_fp = 50` → use as `scale_pos_weight`
### Pattern 6: Hybrid Sampling
- **Use when:** Ratio > 20:1, need clean decision boundaries
- **Implementation:**
```python
from imblearn.combine import SMOTETomek, SMOTEENN
# SMOTE + Tomek (moderate cleanup)
smt = SMOTETomek(
smote=SMOTE(sampling_strategy=0.5, k_neighbors=5),
random_state=42
)
# SMOTE + ENN (aggressive cleanup — better boundaries, fewer samples)
smenn = SMOTEENN(
smote=SMOTE(sampling_strategy=0.5, k_neighbors=5),
random_state=42
)
```
---
## Evaluation Metrics for Imbalanced Data
### Metrics Decision Table
| Metric | Use When | Do NOT Use When |
|--------|----------|-----------------|
| **PR-AUC** | Primary metric for imbalanced data | Balanced datasets |
| **F1** | Need single threshold, equal FP/FN cost | Costs are asymmetric |
| **F-beta** | Asymmetric FP/FN costs | Costs are equal |
| **MCC** | Want single metric accounting for all quadrants | Need threshold-free metric |
| **ROC-AUC** | Comparing models (not evaluating performance) | Severe imbalance (>100:1) — misleading |
| **Accuracy** | NEVER for imbalanced data | Always — it lies |
| **Cohen's Kappa** | Comparing to random baseline | Need interpretable business metric |
### Metric Implementation
```python
from sklearn.metrics import (
average_precision_score, # PR-AUC
f1_score,
fbeta_score,
matthews_corrcoef, # MCC
classification_report
)
y_proba = clf.predict_proba(X_test)[:, 1]
y_pred = (y_proba >= best_threshold).astype(int)
metrics = {
'PR-AUC': average_precision_score(y_test, y_proba),
'F1': f1_score(y_test, y_pred),
'F2': fbeta_score(y_test, y_pred, beta=2),
'MCC': matthews_corrcoef(y_test, y_pred),
}
```
---
## Anti-Patterns
| Anti-Pattern | Why It Fails | Fix |
|---|---|---|
| Applying SMOTE before train/test split | Data leakage — synthetic samples bleed into test set | Use `imblearn.pipeline.Pipeline` inside CV |
| Using accuracy as primary metric | 99% accuracy with 99:1 ratio means predicting all majority | Switch to PR-AUC or F-beta |
| Using ROC-AUC as sole metric at >50:1 ratio | ROC-AUC inflated by easy TN predictions | Use PR-AUC instead |
| Oversampling to 1:1 ratio | Overfitting + slow training | Target 0.3–0.5 ratio with `sampling_strategy` |
| SMOTE on categorical features | SMOTE interpolates — meaningless for categories | Use SMOTENC or encode first |
| SMOTE on high-dimensional sparse data | Generates noisy synthetic points in sparse space | Reduce dimensions first, then SMOTE |
| Ignoring threshold tuning | Default 0.5 wastes model capability | Always tune via PR curve |
| Resampling test set | Evaluation on resampled test is meaningless | ONLY resample training data |
| Using NearMiss without validation | NearMiss can destroy useful majority patterns | Compare holdout performance with/without |
| Combining multiple strategies blindly | Stacking SMOTE + weights + threshold = unpredictable | Add one technique at a time, measure impact |
---
## Validation Checklist
- [ ] Imbalance ratio measured and documented
- [ ] Baseline established with no correction (to measure improvement)
- [ ] Sampling applied ONLY inside cross-validation folds
- [ ] Test set left untouched (original distribution)
- [ ] PR-AUC or F-beta used as primary metric (NOT accuracy or ROC-AUC)
- [ ] Decision threshold tuned on validation set, evaluated on test set
- [ ] Confusion matrix reviewed at chosen threshold
- [ ] Business cost alignment verified (FN cost vs FP cost)
- [ ] Stratified splits used (`StratifiedKFold`)
- [ ] Results stable across multiple random seeds
---
## Cross-References
- `ai-ml-data-science/references/hyperparameter-optimization.md` — tuning `scale_pos_weight` and sampling params
- `ai-ml-data-science/references/interpretability-explainability.md` — explaining minority-class predictions
- `ai-mlops/references/automated-retraining-patterns.md` — monitoring class distribution drift
- `ai-mlops/references/experiment-tracking-patterns.md` — logging imbalance metrics per run
references/data-contracts-lineage.md# Data Contracts, Lineage & Feature Store Operations
Operational patterns for managing data contracts, lineage tracking, and feature store hygiene in production ML systems.
---
## Table of Contents
- [Overview](#overview)
- [1. Data Contracts](#1-data-contracts)
- [1.1 Contract Components](#11-contract-components)
- [1.2 Contract Enforcement](#12-contract-enforcement)
- [1.3 Schema Evolution](#13-schema-evolution)
- [2. Data Lineage Tracking](#2-data-lineage-tracking)
- [2.1 What to Track](#21-what-to-track)
- [2.2 Lineage Implementation Patterns](#22-lineage-implementation-patterns)
- [3. Feature Store Hygiene](#3-feature-store-hygiene)
- [3.1 Materialization Cadence](#31-materialization-cadence)
- [3.2 Backfill & Replay](#32-backfill-&-replay)
- [3.3 Encoder & Mapping Versioning](#33-encoder-&-mapping-versioning)
- [4. Train-Serve Parity](#4-train-serve-parity)
- [4.1 Common Parity Issues](#41-common-parity-issues)
- [4.2 Ensuring Parity](#42-ensuring-parity)
- [5. Monitoring & Alerts](#5-monitoring-&-alerts)
- [5.1 Key Metrics](#51-key-metrics)
- [5.2 Alert Strategy](#52-alert-strategy)
- [6. Governance & Compliance](#6-governance-&-compliance)
- [6.1 PII Handling](#61-pii-handling)
- [6.2 Data Residency](#62-data-residency)
- [Related Resources](#related-resources)
## Overview
Data contracts and lineage tracking are critical for production ML systems. They ensure data quality, enable debugging, and maintain train-serve consistency. This guide covers modern best practices for feature store operations and data governance.
---
## 1. Data Contracts
### 1.1 Contract Components
A robust data contract defines:
- **Schema**: Column names, types, nullability constraints
- **Ranges & Constraints**: Min/max values, allowed categorical values, regex patterns
- **Freshness SLAs**: Maximum acceptable data lag
- **Versioning**: Contract version number and compatibility rules
### 1.2 Contract Enforcement
**When to check contracts:**
- At data ingestion (source -> feature store)
- Before training (feature store -> training pipeline)
- At serving time (feature store -> production model)
**Enforcement strategy:**
- **Fail fast**: Block pipeline on critical contract violations
- **Warn**: Log non-critical violations but continue
- **Degrade gracefully**: Use fallback values for optional fields
### 1.3 Schema Evolution
**Backward-compatible changes (safe):**
- Adding optional fields
- Relaxing constraints (e.g., widening ranges)
- Adding new enum values
**Breaking changes (require coordination):**
- Removing fields
- Changing data types
- Renaming columns
- Tightening constraints
**Migration strategy:**
1. Version the contract (v1 -> v2)
2. Run shadow mode (dual write to v1 and v2)
3. Validate v2 data quality matches v1
4. Gradual cutover with rollback plan
5. Deprecate v1 after validation period
**Checklist: Schema Evolution**
- [ ] Contract version incremented
- [ ] Backward/forward compatibility assessed
- [ ] Shadow run completed successfully
- [ ] Rollback artifacts preserved
- [ ] Deprecation timeline communicated
---
## 2. Data Lineage Tracking
### 2.1 What to Track
**Essential lineage metadata:**
- Source system and extraction timestamp
- Feature store write timestamp and version
- Training run ID and model version
- Feature transformation code version (git commit)
- Data quality metrics at each stage
**Why it matters:**
- Debug data quality issues
- Audit compliance (GDPR, SOC2)
- Root cause analysis for model degradation
- Reproduce experiments
### 2.2 Lineage Implementation Patterns
**Storage:**
- Structured logs (JSON lines)
- Metadata stores (MLflow, DVC, Feast)
- Graph databases (Neo4j for complex lineage)
**Tagging convention:**
```python
lineage_metadata = {
"source": "postgres://db/table",
"extraction_ts": "2024-11-22T10:00:00Z",
"feature_store_version": "v2.1",
"git_commit": "a1b2c3d4",
"run_id": "train-20241122-001",
"model_version": "v1.5.2",
"feature_set_hash": "sha256:abcd1234..."
}
```
**Checklist: Lineage Implemented**
- [ ] Source -> feature store -> train -> serve path tracked
- [ ] Run IDs and model versions logged
- [ ] Git commits captured for reproducibility
- [ ] Lineage queryable (e.g., "which training runs used this data version?")
- [ ] Retention policy defined (how long to keep lineage)
---
## 3. Feature Store Hygiene
### 3.1 Materialization Cadence
**Document and enforce:**
- Batch update frequency (hourly, daily, weekly)
- Streaming update latency targets
- Backfill procedures for historical data
**Monitoring:**
- Freshness lag (source timestamp -> feature store timestamp)
- Materialization job success rate
- Data volume anomalies
### 3.2 Backfill & Replay
**Requirements for safe backfill:**
- Idempotent writes (duplicate runs produce same result)
- Timestamp-based partitioning
- Preserved input data snapshots
- Validation against production data
**Replay scenarios:**
- Bug fix in feature transformation logic
- Schema migration
- Historical model training
- Audit requirements
**Checklist: Backfill Ready**
- [ ] Backfill procedure documented and tested
- [ ] Idempotency verified (run twice -> same output)
- [ ] Validation metrics defined (replay vs original)
- [ ] Impact assessment for downstream consumers
### 3.3 Encoder & Mapping Versioning
**What to version:**
- Categorical encoders (target, frequency, hash)
- Normalization parameters (mean, std, min, max)
- Embedding models and weights
- Lookup tables and dictionaries
**Storage strategy:**
- Store alongside model artifacts
- Use feature store's versioning system
- Tag with training run ID
- Keep rollback versions
**Checklist: Encoders Versioned**
- [ ] All transformations serialized
- [ ] Encoder versions match training config
- [ ] Unseen category handling defined
- [ ] Serving uses same encoder version as training
---
## 4. Train-Serve Parity
### 4.1 Common Parity Issues
**Sources of divergence:**
- Different feature computation logic (Python vs SQL)
- Timezone mismatches
- Rounding/precision differences
- Async updates (training uses stale data)
**Detection:**
- Shadow mode: run both pipelines, compare features
- Synthetic tests with known inputs
- Production monitoring: track distribution drift
### 4.2 Ensuring Parity
**Best practices:**
- Single source of truth: shared feature transformation code
- Use feature store for both training and serving
- Integration tests: compare training vs serving features
- Monitor drift between training and production feature distributions
**Checklist: Parity Validated**
- [ ] Shared transformation code between train/serve
- [ ] Feature store used for both pipelines
- [ ] Integration tests pass (feature equality within tolerance)
- [ ] Drift monitoring active (KL divergence, PSI)
- [ ] Serving feature distributions match training
---
## 5. Monitoring & Alerts
### 5.1 Key Metrics
**Data quality:**
- Null rate per feature
- Out-of-range values
- Distribution shift (KL divergence, KS test, PSI)
**Freshness:**
- Data lag (source -> feature store)
- Staleness alerts (SLA violations)
**Operational:**
- Materialization job failures
- Query latency (p50, p99)
- Storage costs
### 5.2 Alert Strategy
**Critical (page on-call):**
- Contract violation blocking production
- Freshness SLA breach > 2x threshold
- Materialization job failures
**Warning (Slack/email):**
- Non-critical contract violations
- Minor distribution drift
- Performance degradation
**Checklist: Monitoring Active**
- [ ] Freshness SLAs defined and monitored
- [ ] Distribution drift alerts configured
- [ ] Contract violation alerts active
- [ ] Runbooks documented for common alerts
- [ ] False positive rate < 5%
---
## 6. Governance & Compliance
### 6.1 PII Handling
**Requirements:**
- PII identified and tagged in metadata
- Access controls enforced (RBAC)
- Audit logs for PII access
- Hard delete capability (GDPR right to erasure)
### 6.2 Data Residency
**Multi-region strategy:**
- Segment feature stores by region
- Enforce data sovereignty rules
- Replicate non-sensitive features
- Document cross-border data flows
**Checklist: Governance Ready**
- [ ] PII fields identified and tagged
- [ ] Access controls implemented
- [ ] Hard delete procedure tested
- [ ] Data residency requirements documented
- [ ] Audit trail enabled
---
## 7. Annotation Quality
Annotation quality gates apply when training data includes human or LLM-generated labels. Poor label quality is a silent source of model degradation that contracts and lineage tracking alone cannot catch.
### 7.1 Inter-Annotator Agreement (IAA)
Use IAA metrics as a go/no-go gate before committing an annotation batch to training:
- **Cohen's κ (kappa)**: pairwise agreement corrected for chance. Use for two annotators, categorical labels. κ < 0.6 is typically a reject signal; κ ≥ 0.8 is strong agreement.
- **Krippendorff's α**: generalizes to N annotators, ordinal/interval/ratio scales, and missing data. Preferred when annotator count varies or labels are not purely nominal. α < 0.667 is commonly treated as insufficient reliability.
- Run IAA on a statistically meaningful sample before scaling annotation (50–200 examples minimum for initial calibration).
**Checklist: IAA Gate**
- [ ] Agreement metric chosen (κ or α) and justified for the label type
- [ ] IAA computed on calibration sample before full annotation run
- [ ] Threshold documented and enforced as a pipeline gate
- [ ] Disagreements reviewed for guideline gaps, not just averaged away
### 7.2 LLM-as-Annotator Validation
When using LLMs to generate or validate labels:
- Do not assume LLM-generated labels are interchangeable with expert human labels without validation
- Validate LLM labels against a human-annotated gold set (IAA between LLM and human ≥ threshold)
- Document the model, prompt version, and temperature used for annotation — these are part of data lineage
- For high-stakes labels (safety, regulatory, clinical), treat LLM annotation as a first pass requiring human review
### 7.3 Noisy-Label Handling
When annotation quality is unavoidably low or labels are weakly supervised:
- **Label smoothing**: reduces overconfidence on noisy labels; use conservatively (0.05–0.1 for categorical)
- **Confident learning** (Cleanlab): estimates noise transition matrix and identifies likely mislabeled examples; useful for detection, not a substitute for clean labels
- **Robust loss functions**: mean absolute error (MAE) or Huber loss are less sensitive to label noise in regression; symmetric cross-entropy for classification
- **Re-annotation trigger**: if model error analysis reveals systematic failure modes concentrated in one annotator or batch, flag for re-annotation rather than noise-robustness tricks
**Checklist: Noisy Labels**
- [ ] IAA run and documented before training
- [ ] LLM annotation validated against human gold if used
- [ ] Label noise handling strategy documented and proportionate to noise level
- [ ] Annotation batch IDs tracked in data lineage
---
## Related Resources
- [Feature Engineering Patterns](feature-engineering-patterns.md) - Feature transformation techniques
- [Reproducibility Checklist](reproducibility-checklist.md) - Experiment tracking and versioning
- [Feature Freshness & Streaming](feature-freshness-streaming.md) - Real-time feature updates
- [Production Feedback Loops](production-feedback-loops.md) - Online learning and model updates
- [LLM Data Pipelines](llm-data-pipeline.md) - Dedup, quality filtering, and decontamination for LLM training data
references/eda-best-practices.md# EDA Best Practices This guide provides a structured, repeatable workflow for exploratory data analysis with explicit checks, patterns, and decision rules. It is designed for fast onboarding and consistent DS project execution. --- ## Table of Contents - [1. Initial Scan Checklist](#1-initial-scan-checklist) - [2. Data Quality Assessment](#2-data-quality-assessment) - [2.1 Missingness](#21-missingness) - [2.2 Outliers](#22-outliers) - [3. Distribution Analysis](#3-distribution-analysis) - [Numeric](#numeric) - [Categorical](#categorical) - [4. Target Variable Analysis](#4-target-variable-analysis) - [Classification targets:](#classification-targets) - [Regression targets:](#regression-targets) - [5. Leakage Detection](#5-leakage-detection) - [High-Risk Leakage Types:](#high-risk-leakage-types) - [6. EDA Deliverables](#6-eda-deliverables) ## 1. Initial Scan Checklist Perform immediately after loading the dataset. - [ ] Print shape (rows, columns) - [ ] Inspect dtypes and nullable fields - [ ] Identify primary keys or unique identifier candidates - [ ] Check for duplicate rows and duplicate keys - [ ] Evaluate memory usage - [ ] Validate expected ranges for numeric columns - [ ] Confirm presence/absence of target variable **Pattern: Schema Validation** df.info() df.describe(include='all') df.isna().sum() --- ## 2. Data Quality Assessment ### 2.1 Missingness - Identify missingness patterns by: - Row - Column - Groups (user, product, geography) - Evaluate mechanisms: - MCAR (random) - MAR (depends on other features) - MNAR (depends on itself; dangerous) **Checklist - Missingness Strategy** - [ ] Strategy per field documented - [ ] No target leakage introduced by imputation - [ ] Imputation pipelines reproducible --- ### 2.2 Outliers **Detection methods (choose at least one):** - Z-score - IQR - Winsorization scan - Domain-rule scans (e.g., speed < 0 impossible) **Checklist - Outlier Review** - [ ] Extreme values inspected manually - [ ] Outlier handling strategy defined (cap/remove/flag) - [ ] Illegal values corrected or removed --- ## 3. Distribution Analysis Perform both univariate and bivariate analysis. ### Numeric - Histograms - Boxplots - Quantile tables - Skewness/kurtosis review ### Categorical - Frequency distributions - Top-N categories report - Rare category detection (<1% threshold) **Checklist - Distribution Health** - [ ] Long tails annotated - [ ] Rare categories flagged - [ ] Highly skewed features documented for potential transforms --- ## 4. Target Variable Analysis ### Classification targets: - Class imbalance - Rare event frequency - Conditional distributions ### Regression targets: - Scale and skew - Outliers - Zero-inflation **Checklist - Target Evaluation** - [ ] Imbalance noted - [ ] Appropriate metric selection influenced (e.g., PR-AUC for imbalance) - [ ] Target leakage checks started --- ## 5. Leakage Detection Leakage is the leading cause of unrealistic performance. ### High-Risk Leakage Types: - Timestamps after event date - IDs encoding target - Aggregates computed using full window - Target visible in free text - Future features used in temporal splits ### Expert Instincts (what a non-expert misses) - **Suspiciously perfect features are the first suspect, not a lucky find.** A single feature with near-perfect separation (AUC > 0.98 alone, or a feature that alone beats the eventual model) is almost always a leak, not a signal — treat it as a bug report before treating it as a discovery. - **Leakage hides in joins, not just columns.** A feature computed correctly in isolation can still leak if the join key resolves differently at train time (full history available) versus serve time (only history up to the request). Ask "what did this join actually see" per row, not just "is this column defined before the label." - **Group leakage survives a correct time split.** A time-based split stops future-timestamp leakage but not entity leakage — the same user/account/household appearing in both train and validation with correlated behavior still inflates the score. When entities repeat across time, combine a time split with a group split (or use blocked time-series CV grouped by entity). - **Aggregation window off-by-one is the most common silent leak.** "Rolling 7-day spend" computed with a window that includes the prediction day itself (not just the 7 days before it) is leakage that will not show up in a schema check — verify window boundaries against the actual prediction timestamp, not just that a window exists. - **A metric that looks "too good for the domain" is evidence, not a compliment.** If domain experts would be surprised by the reported performance, investigate before presenting it — believable-but-wrong numbers get shipped more often than obviously-broken ones. **Checklist - Leakage Review** - [ ] Time-based checks performed - [ ] ID/cardinality checks performed - [ ] No future window features in train set - [ ] Free text screened for target bleed --- ## 6. EDA Deliverables A complete EDA must include: - Profile report (summary tables + visualizations) - Data dictionary draft - Issue register (severity, owner, fix plan) - List of known risks - Candidate hypotheses
references/evaluation-patterns.md# Evaluation Patterns Operational guidance for deciding whether an ML candidate is ready to deploy, iterate, or reject. Focus on metrics, thresholds, calibration, uncertainty, slice analysis, and explicit recommendation criteria. --- ## Table of Contents - [1. Start With The Decision](#1-start-with-the-decision) - [2. Metric, Threshold, And Calibration Selection](#2-metric-threshold-and-calibration-selection) - [2.1 Primary Metrics](#21-primary-metrics) - [2.2 Guardrail Metrics](#22-guardrail-metrics) - [2.3 Threshold Selection](#23-threshold-selection) - [3. Slice, Error, And Temporal Analysis](#3-slice-error-and-temporal-analysis) - [3.1 Slice Analysis](#31-slice-analysis) - [3.2 Error Analysis](#32-error-analysis) - [3.3 Temporal Robustness](#33-temporal-robustness) - [4. Uncertainty And Confidence](#4-uncertainty-and-confidence) - [4.1 Classification](#41-classification) - [4.2 Regression](#42-regression) - [4.3 What To Report](#43-what-to-report) - [5. Recommendation Logic](#5-recommendation-logic) - [5.1 Deployment Readiness Gate](#51-deployment-readiness-gate) - [6. Evaluation Report Structure](#6-evaluation-report-structure) - [7. Model Card Structure](#7-model-card-structure) - [8. Common Failure Modes](#8-common-failure-modes) - [9. Practical Defaults](#9-practical-defaults) ## 1. Start With The Decision Before choosing metrics, write down: - the action the model output will trigger - whether the output is a score, thresholded label, top-k ranking, or interval - the baseline to beat - the cost of false positives, false negatives, or over/under-estimation **Rule:** evaluation is incomplete if it reports only an aggregate offline score and ignores the downstream decision rule. **Checklist: Decision Context** - [ ] Decision or workflow triggered by the model is documented - [ ] Baseline and minimum acceptable performance are documented - [ ] Output type is explicit: score, label, rank, or interval - [ ] Business or operational costs are listed --- ## 2. Metric, Threshold, And Calibration Selection ### 2.1 Primary Metrics **Classification** - **ROC-AUC**: useful ranking metric, but insufficient when the decision uses a threshold - **PR-AUC**: preferred for imbalanced positive classes - **Log loss**: useful when probability quality matters - **F1 / F-beta**: acceptable when the business truly values a specific precision/recall tradeoff **Regression** - **MAE**: robust default when median-like error matters - **RMSE**: use when large misses are materially worse - **Pinball loss / interval coverage**: use when quantiles or decision bands matter **Ranking** - **NDCG** - **MAP** - **Recall@K** ### 2.2 Guardrail Metrics Track these alongside the primary metric: - **Calibration**: Brier score, calibration curve, expected calibration error - **Fairness / segment parity**: performance gaps across sensitive or operationally important slices - **Latency / size / cost**: inference constraints still matter in offline evaluation - **Stability**: variance across folds, time splits, or seeds ### 2.3 Threshold Selection Document threshold choice explicitly. Common patterns: 1. **Cost-sensitive thresholding**: convert FP/FN costs to an operating point 2. **Capacity-based thresholding**: score top N or top X% 3. **Utility-maximizing thresholding**: optimize profit, save rate, or review yield 4. **Calibrated policy thresholding**: choose cutoff only after probabilities are calibrated **Checklist: Metrics And Thresholds** - [ ] Primary metric chosen and justified - [ ] Guardrails chosen and justified - [ ] Threshold rule documented - [ ] Threshold compared against baseline policy - [ ] Metric definitions are reproducible --- ## 3. Slice, Error, And Temporal Analysis ### 3.1 Slice Analysis Minimum slices usually include: - geography or market - user/account/product segment - time period or recency cohort - confidence bucket - common vs rare cases - sensitive features where allowed and appropriate For each slice, record: - sample size - primary metric - thresholded outcome metric if relevant - calibration or interval behavior where relevant - action required if the slice is weak ### 3.2 Error Analysis Review concrete examples, not just tables: - false positives and false negatives separately - highest-loss regression examples - low-confidence correct predictions - drifted or recently failing cases Cluster failure modes into a short taxonomy such as: - label quality issue - missing feature - stale data - threshold problem - calibration problem - coverage gap ### 3.3 Temporal Robustness If data changes over time, inspect: - metric drift by month or quarter - calibration drift by period - threshold stability over recent windows **Checklist: Slice And Error Review** - [ ] Core operational slices reviewed - [ ] Weak slices explained with hypotheses - [ ] Example-level failures reviewed - [ ] Temporal robustness checked when relevant - [ ] Remediation ideas recorded --- ## 4. Uncertainty And Confidence Use uncertainty-aware reporting when decisions are risk-sensitive. ### 4.1 Classification - confidence intervals on metrics via bootstrap or repeated CV - calibrated probabilities when a score is treated as a likelihood - optional conformal or abstention policies when low-confidence cases are routed to review ### 4.2 Regression - prediction intervals or quantile estimates - coverage checks on held-out data - interval width analysis by slice ### 4.3 What To Report At minimum, include: - point estimate - interval or variance estimate - method used - practical interpretation for decision-makers **Checklist: Uncertainty** - [ ] Metric uncertainty reported for key claims - [ ] Probability calibration checked when probabilities are used - [ ] Prediction intervals or quantiles checked when outputs need ranges - [ ] Uncertainty communication is understandable to non-DS readers --- ## 5. Recommendation Logic End every serious evaluation with one of: - **Deploy**: beats baseline, threshold/calibration are acceptable, key slices are understood, and risks are manageable - **Iterate**: promising, but blocked by specific weaknesses such as calibration, slice gaps, or instability - **Reject / Hold**: does not beat baseline meaningfully, fails critical constraints, or risks are not acceptable ### 5.1 Deployment Readiness Gate A candidate is not deployment-ready unless all of these are answered: - What baseline did it beat? - What threshold or ranking policy will production use? - Is probability calibration acceptable? - Are uncertainty bounds acceptable? - Which slices are weakest, and what is the mitigation? - What inputs/outputs/versioning assumptions must MLOps preserve? **Checklist: Final Recommendation** - [ ] Recommendation is explicit: deploy, iterate, or reject - [ ] Recommendation references evidence, not intuition - [ ] Open risks and mitigations are listed - [ ] Handoff notes are complete enough for operations or reviewers --- ## 6. Evaluation Report Structure Use this order: 1. Objective and decision context 2. Dataset versions, time windows, and limitations 3. Feature and prediction-time assumptions 4. Baselines and candidate models 5. Metrics, threshold policy, and calibration 6. Slice and error analysis 7. Uncertainty and confidence 8. Risks, mitigations, and recommendation The report should support a reviewer answering: "Should we act on this model now?" --- ## 7. Model Card Structure Keep model cards shorter than full reports, but include: - intended use and out-of-scope use - dataset and version summary - prediction-time assumptions - core performance metrics - threshold or decision policy - calibration / uncertainty summary - sensitive-slice or fairness notes - owner, maintenance plan, and contact path --- ## 8. Common Failure Modes - High ROC-AUC but poor threshold performance at the actual operating point - Good aggregate metrics with unacceptable weak slices - Strong ranking metrics with poor calibration - Narrow intervals with poor actual coverage - Repeated CV improvement that disappears on the true holdout - "Deploy" recommendation without threshold, owner, or rollback expectations --- ## 9. Practical Defaults - If the model triggers an action, always report thresholded performance alongside ranking metrics. - If the model emits probabilities, always inspect calibration. - If the output is used for planning or budgeting, report intervals, not just point estimates. - If the data is time-sensitive, always review recent-window performance separately. - If the result is close to baseline, default to `iterate`, not `deploy`. --- ## 10. Benchmark Contamination **Scope:** apply this section when evaluating LLM-based models, or any model trained on large crawled corpora where test data may have appeared in pretraining or fine-tuning data. For classical ML with small curated datasets, standard train/test split hygiene (§2 of `modelling-patterns.md`) is sufficient. ### 10.1 Detection **MinHash near-duplicate matching:** - Compute MinHash signatures for benchmark examples and training documents - Use LSH at Jaccard similarity threshold 0.5–0.7 to catch near-matches - Flag training documents that overlap with any benchmark split; remove before final training run or report results with and without contaminated examples **Exact n-gram matching:** - 13-gram overlap is a common threshold (used in LLaMA evaluations and similar work) - Fast and interpretable; use as a first pass before MinHash **Min-K% Prob (post-training detection):** - Extract the k% of tokens with lowest log-probability under the model for a given input - Contaminated examples tend to have higher min-k% probability than held-out examples - Applicable when training corpus access is limited; does not require corpus re-scanning - Reference: arXiv 2310.16789 (verify before citing specific figures) ### 10.2 Contamination-Resistant Benchmark Choice When contamination risk is non-trivial: - Prefer recently released benchmarks not present in the training data window - Prefer benchmarks with procedural or dynamic generation (new instances per run) - Consider private held-out test sets for high-stakes comparisons - Report contamination analysis results alongside benchmark scores — do not omit when detected **Checklist: Contamination** - [ ] Contamination scope assessed: is training data large/crawled enough to warrant checking? - [ ] MinHash or n-gram check run between training corpus and evaluation benchmarks - [ ] Contaminated examples quarantined and results reported with/without them - [ ] Benchmark choice accounts for contamination risk; newer or procedural benchmarks preferred when risk is high
references/feature-engineering-patterns.md# Feature Engineering Patterns A collection of operational patterns for transforming raw data into model-ready features. --- ## Table of Contents - [1. Numeric Feature Patterns](#1-numeric-feature-patterns) - [1.1 Standardization](#11-standardization) - [1.2 Outlier Handling](#12-outlier-handling) - [2. Categorical Feature Patterns](#2-categorical-feature-patterns) - [2.1 Low Cardinality](#21-low-cardinality) - [2.2 High Cardinality](#22-high-cardinality) - [3. Text Feature Patterns](#3-text-feature-patterns) - [3.1 Cleaning](#31-cleaning) - [3.2 Representations](#32-representations) - [4. Time-Based Features](#4-time-based-features) - [4.1 Datetime Decomposition](#41-datetime-decomposition) - [4.2 Lag Features](#42-lag-features) - [5. Interaction Patterns](#5-interaction-patterns) - [6. Train/Serve Consistency Patterns](#6-trainserve-consistency-patterns) - [Ensuring parity between offline and production pipelines](#ensuring-parity-between-offline-and-production-pipelines) ## 1. Numeric Feature Patterns ### 1.1 Standardization Use when units vary or model sensitive to scale. - z-score - min-max - robust scaling (median/IQR) ### 1.2 Outlier Handling - Winsorize top/bottom 1% - Cap values at domain limits - Log transform long-tailed distributions **Checklist - Numeric Features** - [ ] Consistent units - [ ] Outliers handled - [ ] Skew addressed --- ## 2. Categorical Feature Patterns ### 2.1 Low Cardinality - One-hot encoding - Ordinal encoding (only when true order exists) ### 2.2 High Cardinality - Target encoding (use CV to avoid leakage) - Frequency encoding - Hashing **Checklist - Categorical Features** - [ ] Rare categories grouped/flagged - [ ] Clear mapping for unseen categories - [ ] Encoders versioned for training/serving parity --- ## 3. Text Feature Patterns ### 3.1 Cleaning - Strip HTML - Lowercase or case-preserve based on domain - Remove excessive whitespace ### 3.2 Representations - TF-IDF - Pretrained embeddings - Keyword densities - Text length signals **Checklist - Text Features** - [ ] Deterministic preprocessing - [ ] PII removed where required - [ ] Embedding models versioned --- ## 4. Time-Based Features ### 4.1 Datetime Decomposition - Year, month, day - Day of week - Hour, minute - Boolean flags (weekend, holiday) ### 4.2 Lag Features - lag_1, lag_7, lag_28 - Rolling windows **Checklist - Time Features** - [ ] Timezone alignment validated - [ ] Features do not leak future information --- ## 5. Interaction Patterns Use carefully to avoid explosion. - Crossed categorical features - Numeric x categorical interactions - Polynomial features (2nd/3rd degree) **Checklist - Interaction Features** - [ ] Interaction justified - [ ] No combinatorial blow-up - [ ] Feature importance reviewed --- ## 6. Train/Serve Consistency Patterns ### Ensuring parity between offline and production pipelines - Use shared **feature store** when possible - Encode with version-pinned transformers - Enforce dtype consistency **Checklist - Consistency** - [ ] Single source of truth for transformations - [ ] Serving pipeline tested with training artifacts
references/feature-freshness-streaming.md# Feature Freshness, Streaming & Schema Evolution
Operational patterns for managing real-time features, streaming pipelines, and schema changes in production ML systems.
---
## Table of Contents
- [Overview](#overview)
- [1. Freshness Contracts & SLAs](#1-freshness-contracts-&-slas)
- [1.1 Defining Freshness Requirements](#11-defining-freshness-requirements)
- [1.2 Freshness Monitoring](#12-freshness-monitoring)
- [2. Batch + Stream Parity](#2-batch-stream-parity)
- [2.1 The Parity Challenge](#21-the-parity-challenge)
- [2.2 Shared Feature Logic Patterns](#22-shared-feature-logic-patterns)
- [Same code for batch and stream](#same-code-for-batch-and-stream)
- [features/user_metrics.py (shared by Spark batch + Flink stream)](#featuresusermetricspy-shared-by-spark-batch-flink-stream)
- [3. Schema Evolution Strategies](#3-schema-evolution-strategies)
- [3.1 Compatible Changes](#31-compatible-changes)
- [3.2 Migration Process](#32-migration-process)
- [4. Data Quality Gates](#4-data-quality-gates)
- [4.1 Quality Checks](#41-quality-checks)
- [4.2 Gate Enforcement](#42-gate-enforcement)
- [5. Late-Arriving Data Handling](#5-late-arriving-data-handling)
- [5.1 Patterns](#51-patterns)
- [5.2 Implementation](#52-implementation)
- [6. Streaming Architecture Patterns](#6-streaming-architecture-patterns)
- [6.1 Lambda Architecture](#61-lambda-architecture)
- [6.2 Kappa Architecture](#62-kappa-architecture)
- [7. Testing Strategies](#7-testing-strategies)
- [7.1 Unit Tests](#71-unit-tests)
- [7.2 Integration Tests](#72-integration-tests)
- [7.3 Chaos Testing](#73-chaos-testing)
- [8. Operational Runbooks](#8-operational-runbooks)
- [8.1 Common Issues](#81-common-issues)
- [Related Resources](#related-resources)
## Overview
Modern ML systems increasingly require real-time features and streaming data pipelines. This guide covers best practices for maintaining freshness SLAs, ensuring batch-stream parity, and safely evolving schemas.
---
## 1. Freshness Contracts & SLAs
### 1.1 Defining Freshness Requirements
**Questions to answer:**
- What is the maximum acceptable lag between source update and feature availability?
- Are there different SLAs for different features?
- What happens when freshness SLA is violated?
**Common SLA tiers:**
- **Real-time**: < 1 minute lag (streaming features)
- **Near real-time**: 1-15 minutes lag (micro-batch)
- **Hourly**: < 1 hour lag (batch with frequent updates)
- **Daily**: < 24 hours lag (overnight batch jobs)
### 1.2 Freshness Monitoring
**Metrics to track:**
- **Lag**: `current_time - source_timestamp`
- **Staleness**: Time since last successful update
- **Update frequency**: Updates per hour/day
**Alerting thresholds:**
- **Critical**: Lag > 2x SLA threshold
- **Warning**: Lag > 1.5x SLA threshold
- **Info**: Lag approaching SLA threshold
**Checklist: Freshness Monitoring**
- [ ] Freshness SLAs defined per feature or feature group
- [ ] Lag metrics collected and dashboarded
- [ ] Alerts configured with appropriate thresholds
- [ ] Fallback strategy documented for stale data
- [ ] Historical lag trends analyzed (p50, p95, p99)
---
## 2. Batch + Stream Parity
### 2.1 The Parity Challenge
**Problem:**
- Batch pipelines use different code/frameworks than streaming
- Results can diverge due to:
- Rounding differences
- Aggregation window boundaries
- Late-arriving data handling
- Order-dependent operations
**Solution:**
- Use **shared feature transformation logic**
- Implement **idempotent upserts**
- Handle **late-arriving data** consistently
- Test parity with **synthetic replay**
### 2.2 Shared Feature Logic Patterns
**Option 1: Feature store abstraction**
```python
# Same code for batch and stream
@feature_definition
def user_7day_spend(events):
return events.filter(
lambda e: e.timestamp > now() - timedelta(days=7)
).sum("amount")
```
**Option 2: Shared libraries**
```python
# features/user_metrics.py (shared by Spark batch + Flink stream)
def compute_rolling_spend(events_df, window_days=7):
# Deterministic logic works in both contexts
return events_df.groupBy("user_id").agg(...)
```
**Checklist: Batch-Stream Parity**
- [ ] Feature logic shared between batch and streaming pipelines
- [ ] Idempotent upserts implemented (duplicate events handled)
- [ ] Late-arriving data strategy defined and consistent
- [ ] Parity tests: batch vs stream results match within tolerance
- [ ] Differences documented (if unavoidable due to windowing)
---
## 3. Schema Evolution Strategies
### 3.1 Compatible Changes
**Backward compatible (safe):**
- Adding new optional fields
- Widening numeric types (int32 -> int64)
- Relaxing constraints (nullable = true)
**Forward compatible (safe):**
- Removing optional fields (old code ignores)
- Adding default values for new fields
**Incompatible (breaking):**
- Renaming fields
- Changing types (string -> int)
- Removing required fields
- Changing semantics
### 3.2 Migration Process
**Step 1: Version schemas**
```json
{
"schema_version": "v2.1",
"fields": [...],
"compatible_with": ["v2.0"]
}
```
**Step 2: Dual write/read**
- Write to both old and new schema
- Read from new schema with fallback to old
- Validate consistency
**Step 3: Backfill**
- Run backfill jobs to populate new schema
- Use idempotent writes
- Validate against original data
**Step 4: Cutover**
- Monitor error rates
- Gradual rollout (1% -> 10% -> 50% -> 100%)
- Keep rollback artifacts
**Checklist: Schema Evolution**
- [ ] Schema versioned with compatibility metadata
- [ ] Dual-write phase completed successfully
- [ ] Backfill job run and validated
- [ ] Rollback plan documented and tested
- [ ] Deprecated schemas marked with sunset date
---
## 4. Data Quality Gates
### 4.1 Quality Checks
**PII/Format checks:**
- Regex validation (email, phone, SSN patterns)
- PII detection and redaction
- Encoding validation (UTF-8)
**Range checks:**
- Min/max bounds per feature
- Enum membership checks
- Cross-field constraints (end_date > start_date)
**Distribution checks:**
- KL divergence vs training distribution
- Kolmogorov-Smirnov test
- Population Stability Index (PSI)
### 4.2 Gate Enforcement
**At ingestion:**
- Block writes on critical violations
- Log warnings on minor violations
- Quarantine invalid records
**At training:**
- Fail pipeline on distribution drift > threshold
- Require manual approval for major changes
**At serving:**
- Reject invalid requests
- Apply fallback values
- Log and alert
**Checklist: Quality Gates Active**
- [ ] PII detection rules configured
- [ ] Range checks defined per feature
- [ ] Distribution drift thresholds set
- [ ] Gates active in CI/CD pipeline
- [ ] Gates active in production serving
- [ ] Quarantine process for invalid data
---
## 5. Late-Arriving Data Handling
### 5.1 Patterns
**Pattern 1: Allowed lateness window**
- Accept events up to N hours late
- Drop events beyond window
- Trade-off: completeness vs complexity
**Pattern 2: Watermarks**
- Track event time watermark
- Trigger computations when watermark advances
- Handle stragglers with side outputs
**Pattern 3: Reprocessing**
- Periodically recompute features with full data
- Upsert corrected values
- Idempotent operations required
### 5.2 Implementation
**Flink watermark example:**
```java
env.fromSource(source)
.assignTimestampsAndWatermarks(
WatermarkStrategy
.<Event>forBoundedOutOfOrderness(Duration.ofMinutes(5))
.withTimestampAssigner((event, ts) -> event.timestamp)
)
```
**Checklist: Late Data Handling**
- [ ] Allowed lateness window defined
- [ ] Watermark strategy configured
- [ ] Reprocessing cadence determined
- [ ] Idempotency verified for reprocessing
- [ ] Metrics track late arrival rates
---
## 6. Streaming Architecture Patterns
### 6.1 Lambda Architecture
**Components:**
- **Batch layer**: Complete, accurate, slow
- **Speed layer**: Approximate, fast, handles recent data
- **Serving layer**: Merges batch + speed results
**When to use:**
- Need both accuracy (batch) and low latency (streaming)
- Can tolerate eventual consistency
### 6.2 Kappa Architecture
**Components:**
- **Single streaming pipeline** for all data
- Reprocessing = replay from log (Kafka)
**When to use:**
- Can express all logic in streaming framework
- Want to avoid maintaining two pipelines
**Checklist: Architecture Selection**
- [ ] Latency requirements documented
- [ ] Accuracy vs speed trade-offs evaluated
- [ ] Replay/reprocessing needs assessed
- [ ] Framework capabilities validated (Flink, Spark Streaming, Kafka Streams)
- [ ] Operational complexity considered
---
## 7. Testing Strategies
### 7.1 Unit Tests
**Test feature transformations:**
```python
def test_rolling_spend_deterministic():
events = create_test_events()
result_batch = compute_rolling_spend_batch(events)
result_stream = compute_rolling_spend_stream(events)
assert result_batch == result_stream
```
### 7.2 Integration Tests
**Test end-to-end pipeline:**
- Inject synthetic events
- Wait for processing
- Verify output matches expected
### 7.3 Chaos Testing
**Simulate failures:**
- Late-arriving data
- Out-of-order events
- Duplicate events
- Network partitions
- Service restarts
**Checklist: Testing Complete**
- [ ] Unit tests for feature transformations (batch-stream parity)
- [ ] Integration tests for full pipeline
- [ ] Chaos tests for failure scenarios
- [ ] Performance tests (throughput, latency)
- [ ] Regression tests for schema changes
---
## 8. Operational Runbooks
### 8.1 Common Issues
**Issue: Features are stale**
- Check: Streaming job running?
- Check: Source producing events?
- Check: Network connectivity?
- Mitigation: Restart job, trigger backfill
**Issue: Batch-stream parity violation**
- Check: Transformation code versions match?
- Check: Late-arriving data handled?
- Mitigation: Align code, replay stream
**Issue: Schema mismatch errors**
- Check: Producers using latest schema?
- Check: Consumers handle old schema?
- Mitigation: Dual-write mode, schema registry
**Checklist: Runbooks Ready**
- [ ] Incident response procedures documented
- [ ] Common failure modes catalogued
- [ ] Escalation paths defined
- [ ] Rollback procedures tested
- [ ] On-call training completed
---
## Related Resources
- [Data Contracts & Lineage](data-contracts-lineage.md) - Schema versioning and lineage tracking
- [Reproducibility Checklist](reproducibility-checklist.md) - Experiment versioning
- [Production Feedback Loops](production-feedback-loops.md) - Online learning patterns
references/hyperparameter-optimization.md# Hyperparameter Optimization
> Operational guide for systematic hyperparameter tuning using Optuna, Ray Tune, and Bayesian optimization. Covers search space design, pruning, multi-objective optimization, and reproducible tuning recipes for common model families.
**Freshness anchor:** verified 2026-07-11 — Optuna 4.9+ (v4 removed `suggest_uniform`/`suggest_loguniform`/`suggest_discrete_uniform`; use `suggest_float(..., log=True)` as shown below — code in this file already uses the current API), Ray Tune 2.9+, scikit-learn 1.9+, LightGBM 4.6+. Optuna crossed a major version (3.x -> 4.x) since this file was last checked — re-verify sampler defaults and any deprecated-API usage against the [v4 migration guide](https://github.com/optuna/optuna/discussions/5573) before trusting older tutorials or blog posts.
---
## Table of Contents
- [Decision Tree: Choosing an Optimization Strategy](#decision-tree-choosing-an-optimization-strategy)
- [Quick Reference: Sampler Selection](#quick-reference-sampler-selection)
- [Operational Patterns](#operational-patterns)
- [Pattern 1: Optuna Basic Setup](#pattern-1-optuna-basic-setup)
- [Pattern 2: Pruning for Expensive Models](#pattern-2-pruning-for-expensive-models)
- [Pattern 3: LightGBM Tuning Recipe](#pattern-3-lightgbm-tuning-recipe)
- [Pattern 4: scikit-learn Tuning Recipe](#pattern-4-scikit-learn-tuning-recipe)
- [Pattern 5: Multi-Objective Optimization](#pattern-5-multi-objective-optimization)
- [Get Pareto front](#get-pareto-front)
- [Pattern 6: Warmstarting with Known-Good Configs](#pattern-6-warmstarting-with-known-good-configs)
- [Seed with known-good config](#seed-with-known-good-config)
- [Pattern 7: Ray Tune for Distributed Tuning](#pattern-7-ray-tune-for-distributed-tuning)
- [Reproducibility Checklist](#reproducibility-checklist)
- [1. Fix all random seeds](#1-fix-all-random-seeds)
- [2. Use seeded sampler](#2-use-seeded-sampler)
- [3. Log environment](#3-log-environment)
- [4. Store study to DB](#4-store-study-to-db)
- [5. Export best trial](#5-export-best-trial)
- [Anti-Patterns](#anti-patterns)
- [Convergence Analysis](#convergence-analysis)
- [Check if study has converged](#check-if-study-has-converged)
- [Plot optimization history](#plot-optimization-history)
- [Parameter importance (which params matter most)](#parameter-importance-which-params-matter-most)
- [Slice plot (effect of each param)](#slice-plot-effect-of-each-param)
- [Rule of thumb: if best value hasn't improved in last 30% of trials, stop](#rule-of-thumb-if-best-value-hasnt-improved-in-last-30%-of-trials-stop)
- [Cross-References](#cross-references)
## Decision Tree: Choosing an Optimization Strategy
```
START
│
├─ < 10 hyperparameters?
│ ├─ YES → Optuna with TPE sampler (default)
│ └─ NO → Continue
│
├─ 10–30 hyperparameters?
│ ├─ Training time < 5 min per trial?
│ │ ├─ YES → Optuna TPE, 100–300 trials
│ │ └─ NO → Optuna with pruning (MedianPruner)
│ └─ Distributed cluster available?
│ ├─ YES → Ray Tune + Optuna integration
│ └─ NO → Optuna with SQLite storage for resumability
│
├─ Multi-objective (e.g., accuracy + latency)?
│ └─ Optuna with NSGAIISampler → Pareto front
│
├─ Need warmstarting from prior runs?
│ └─ Optuna with enqueue_trial for known-good configs
│
└─ Very expensive trials (>1 hour each)?
└─ Bayesian optimization (GP) with <50 trials
OR early stopping with aggressive pruning
```
---
## Quick Reference: Sampler Selection
| Sampler | Trials Needed | Best For | Avoid When |
|---------|--------------|----------|------------|
| TPE (default) | 50–300 | General purpose, mixed types | Very few trials (<20) |
| GP (Gaussian Process) | 10–50 | Expensive evaluations | High-dimensional (>15 params) |
| CMA-ES | 50–200 | Continuous params, neural nets | Categorical-heavy spaces |
| NSGA-II | 100–500 | Multi-objective | Single objective |
| Random | 20–100 | Baseline comparison, parallel | Always outperformed by TPE |
| Grid | all combos | Exhaustive, < 4 params | > 4 params (combinatorial explosion) |
---
## Operational Patterns
### Pattern 1: Optuna Basic Setup
- **Use when:** Starting any tuning task
- **Implementation:**
```python
import optuna
def objective(trial):
params = {
'n_estimators': trial.suggest_int('n_estimators', 100, 1000, step=100),
'max_depth': trial.suggest_int('max_depth', 3, 12),
'learning_rate': trial.suggest_float('learning_rate', 1e-3, 0.3, log=True),
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),
}
# Cross-validation inside objective
scores = cross_val_score(model_cls(**params), X, y, cv=5, scoring='average_precision')
return scores.mean()
study = optuna.create_study(
direction='maximize',
sampler=optuna.samplers.TPESampler(seed=42),
study_name='lgbm_tuning_v1',
storage='sqlite:///optuna_studies.db', # resumable
)
study.optimize(objective, n_trials=200, timeout=3600)
```
- **Key rules:**
- Always use `log=True` for learning rates, regularization
- Always set `seed` in sampler for reproducibility
- Use SQLite storage for runs > 30 minutes (crash recovery)
### Pattern 2: Pruning for Expensive Models
- **Use when:** Single trial takes > 2 minutes
- **Implementation:**
```python
from optuna.pruners import MedianPruner, HyperbandPruner
study = optuna.create_study(
direction='maximize',
pruner=MedianPruner(
n_startup_trials=10, # don't prune first 10
n_warmup_steps=20, # don't prune before 20 epochs
interval_steps=5, # check every 5 epochs
),
)
def objective(trial):
params = {... } # suggest params
for epoch in range(100):
train_one_epoch(model, params)
val_score = evaluate(model)
trial.report(val_score, epoch)
if trial.should_prune():
raise optuna.TrialPruned()
return val_score
```
- **Pruner selection:**
| Pruner | Aggression | Use When |
|--------|-----------|----------|
| MedianPruner | Moderate | Default choice |
| HyperbandPruner | Aggressive | Deep learning, many epochs |
| PercentilePruner | Configurable | Fine-tune aggression |
| ThresholdPruner | Fixed | Known minimum acceptable score |
### Pattern 3: LightGBM Tuning Recipe
- **Use when:** Tuning LightGBM for tabular data
- **Search space (battle-tested ranges):**
```python
def lgbm_objective(trial):
params = {
'objective': 'binary',
'metric': 'average_precision',
'verbosity': -1,
'boosting_type': 'gbdt',
# Tier 1: Highest impact
'learning_rate': trial.suggest_float('learning_rate', 0.005, 0.2, log=True),
'n_estimators': trial.suggest_int('n_estimators', 100, 2000, step=100),
'num_leaves': trial.suggest_int('num_leaves', 15, 255),
'max_depth': trial.suggest_int('max_depth', 3, 12),
# Tier 2: Regularization
'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),
'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True),
'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True),
# Tier 3: Stochastic
'subsample': trial.suggest_float('subsample', 0.5, 1.0),
'colsample_bytree': trial.suggest_float('colsample_bytree', 0.4, 1.0),
'subsample_freq': trial.suggest_int('subsample_freq', 1, 7),
# Tier 4: Fine-tuning
'min_split_gain': trial.suggest_float('min_split_gain', 0.0, 1.0),
'max_bin': trial.suggest_int('max_bin', 63, 511),
}
cv_result = lgb.cv(params, train_set, nfold=5, stratified=True,
return_cvbooster=True)
return cv_result['valid average_precision-mean'][-1]
```
- **Tuning order:** Tune Tier 1 first (50 trials), freeze best, then add Tier 2–4
### Pattern 4: scikit-learn Tuning Recipe
- **Use when:** Tuning RandomForest, GradientBoosting, SVM, or other sklearn models
```python
def sklearn_rf_objective(trial):
params = {
'n_estimators': trial.suggest_int('n_estimators', 100, 1000, step=50),
'max_depth': trial.suggest_int('max_depth', 3, 30),
'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),
'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 15),
'max_features': trial.suggest_categorical('max_features', ['sqrt', 'log2', 0.3, 0.5, 0.7]),
'class_weight': trial.suggest_categorical('class_weight', ['balanced', 'balanced_subsample', None]),
}
clf = RandomForestClassifier(**params, random_state=42, n_jobs=-1)
scores = cross_val_score(clf, X, y, cv=5, scoring='average_precision')
return scores.mean()
```
### Pattern 5: Multi-Objective Optimization
- **Use when:** Trading off accuracy vs latency, accuracy vs model size, etc.
- **Implementation:**
```python
study = optuna.create_study(
directions=['maximize', 'minimize'], # accuracy UP, latency DOWN
sampler=optuna.samplers.NSGAIISampler(seed=42),
)
def multi_objective(trial):
params = {... }
score = cross_val_score(model(**params), X, y, cv=3).mean()
latency = measure_inference_latency(model(**params), X[:100])
return score, latency
study.optimize(multi_objective, n_trials=200)
# Get Pareto front
pareto_trials = study.best_trials
for t in pareto_trials:
print(f"Score: {t.values[0]:.4f}, Latency: {t.values[1]:.2f}ms")
```
### Pattern 6: Warmstarting with Known-Good Configs
- **Use when:** You have prior knowledge or production configs to start from
```python
study = optuna.create_study(direction='maximize')
# Seed with known-good config
study.enqueue_trial({
'learning_rate': 0.05,
'n_estimators': 500,
'max_depth': 7,
'num_leaves': 63,
})
study.optimize(objective, n_trials=150)
```
### Pattern 7: Ray Tune for Distributed Tuning
- **Use when:** Cluster available, need to parallelize across GPUs/nodes
```python
from ray import tune
from ray.tune.search.optuna import OptunaSearch
search_space = {
'learning_rate': tune.loguniform(1e-4, 1e-1),
'batch_size': tune.choice([32, 64, 128, 256]),
'hidden_size': tune.choice([128, 256, 512]),
}
analysis = tune.run(
train_fn,
config=search_space,
search_alg=OptunaSearch(metric='val_loss', mode='min'),
num_samples=200,
resources_per_trial={'cpu': 4, 'gpu': 1},
scheduler=tune.schedulers.ASHAScheduler(
metric='val_loss', mode='min',
max_t=100, grace_period=10,
),
)
```
---
## Reproducibility Checklist
```python
# 1. Fix all random seeds
import numpy as np, random, torch
random.seed(42)
np.random.seed(42)
torch.manual_seed(42)
# 2. Use seeded sampler
sampler = optuna.samplers.TPESampler(seed=42)
# 3. Log environment
import optuna, sklearn, lightgbm
env_info = {
'optuna': optuna.__version__,
'sklearn': sklearn.__version__,
'lgbm': lightgbm.__version__,
'python': sys.version,
}
# 4. Store study to DB
study = optuna.create_study(storage='sqlite:///studies.db', study_name='exp_v1')
# 5. Export best trial
best = study.best_trial
print(f"Best value: {best.value}")
print(f"Best params: {best.params}")
```
---
## Anti-Patterns
| Anti-Pattern | Why It Fails | Fix |
|---|---|---|
| Grid search with > 5 params | Combinatorial explosion (3^10 = 59k combos) | Use TPE or Bayesian optimization |
| Not using `log=True` for learning rate | Wastes trials in high range, under-explores low range | Always `log=True` for rates, regularization |
| Tuning all params simultaneously from start | High-dimensional space, slow convergence | Tune in tiers: most impactful first |
| No pruning for expensive trials | Wasting compute on clearly bad configs | Add MedianPruner or ASHA scheduler |
| Tuning on test set | Overfitting to test data | Tune on validation, evaluate once on test |
| Fixed number of CV folds regardless of dataset size | 5-fold on 500 rows = noisy; 10-fold on 1M rows = slow | Scale folds: 10 for small, 3–5 for large |
| Ignoring study persistence | Lose progress on crash | Use `storage='sqlite:///...'` |
| Not comparing to random baseline | Can't tell if TPE is actually helping | Run 50 random trials first as reference |
| Copy-pasting search spaces across projects | Different data needs different ranges | Start from recipes, adjust based on data |
| Running 1000 trials without analysis | Diminishing returns after ~100–200 for TPE | Check convergence plots, stop early |
---
## Convergence Analysis
```python
# Check if study has converged
import optuna.visualization as vis
# Plot optimization history
vis.plot_optimization_history(study)
# Parameter importance (which params matter most)
vis.plot_param_importances(study)
# Slice plot (effect of each param)
vis.plot_slice(study)
# Rule of thumb: if best value hasn't improved in last 30% of trials, stop
```
---
## Cross-References
- `ai-ml-data-science/references/class-imbalance-patterns.md` — tuning `scale_pos_weight` and sampling ratio
- `ai-ml-data-science/references/interpretability-explainability.md` — interpreting tuned models
- `ai-mlops/references/experiment-tracking-patterns.md` — logging Optuna studies to MLflow/W&B
- `ai-mlops/references/automated-retraining-patterns.md` — scheduling tuning runs in pipelines
references/interpretability-explainability.md# Interpretability and Explainability
> Operational guide for explaining ML model predictions. Covers SHAP, LIME, permutation importance, partial dependence, and audience-appropriate communication of model behavior. Focus on actionable interpretation, not theory.
**Freshness anchor:** verified 2026-07-11 — SHAP 0.52+ (requires Python >=3.12), LIME 0.2+, scikit-learn 1.9+, LightGBM 4.6+. SHAP 0.45.0 changed multi-output `shap_values()` return type from a Python list to a single `np.ndarray` — see the Pattern 1 gotcha below before indexing with `shap_values[1]`.
---
## Table of Contents
- [Decision Tree: Choosing an Explanation Method](#decision-tree-choosing-an-explanation-method)
- [Quick Reference: Methods Comparison](#quick-reference-methods-comparison)
- [Operational Patterns](#operational-patterns)
- [Pattern 1: TreeSHAP for Tree-Based Models](#pattern-1-treeshap-for-tree-based-models)
- [Train model](#train-model)
- [Create explainer (auto-detects tree type)](#create-explainer-auto-detects-tree-type)
- [For binary classification: shap_values is [neg_class, pos_class]](#for-binary-classification-shapvalues-is-negclass-posclass)
- [Use shap_values[1] for positive class explanations](#use-shapvalues1-for-positive-class-explanations)
- [Global: Summary plot (feature importance + distribution)](#global-summary-plot-feature-importance-distribution)
- [Global: Bar plot (mean absolute SHAP per feature)](#global-bar-plot-mean-absolute-shap-per-feature)
- [Local: Single prediction waterfall](#local-single-prediction-waterfall)
- [Pattern 2: KernelSHAP for Black-Box Models](#pattern-2-kernelshap-for-black-box-models)
- [Background data: use k-means summary for speed](#background-data-use-k-means-summary-for-speed)
- [Compute on subset (KernelSHAP is slow)](#compute-on-subset-kernelshap-is-slow)
- [Pattern 3: LIME for Quick Local Explanations](#pattern-3-lime-for-quick-local-explanations)
- [Explain single prediction](#explain-single-prediction)
- [Or export: exp.as_html(), exp.as_list()](#or-export-expashtml-expaslist)
- [Pattern 4: Permutation Importance](#pattern-4-permutation-importance)
- [Sort by importance](#sort-by-importance)
- [Pattern 5: Partial Dependence and ICE Plots](#pattern-5-partial-dependence-and-ice-plots)
- [PDP for top features](#pdp-for-top-features)
- [Pattern 6: Feature Importance Stability Analysis](#pattern-6-feature-importance-stability-analysis)
- [Bootstrap SHAP importance stability](#bootstrap-shap-importance-stability)
- [Compute rank stability per feature](#compute-rank-stability-per-feature)
- [> 0.9 = stable rankings; < 0.7 = unstable, report with caveats](#09-=-stable-rankings-07-=-unstable-report-with-caveats)
- [Audience-Appropriate Explanations](#audience-appropriate-explanations)
- [Technical Audience (Data Scientists)](#technical-audience-data-scientists)
- [Business Stakeholders](#business-stakeholders)
- [Regulatory / Audit](#regulatory-audit)
- [Model Explanation Report](#model-explanation-report)
- [Anti-Patterns](#anti-patterns)
- [Model Card Template (Interpretability Section)](#model-card-template-interpretability-section)
- [Interpretability](#interpretability)
- [Explanation Method](#explanation-method)
- [Top Features (Stable)](#top-features-stable)
- [Stability](#stability)
- [Limitations](#limitations)
- [Validation Checklist](#validation-checklist)
- [Cross-References](#cross-references)
## Decision Tree: Choosing an Explanation Method
```
START
│
├─ Model type?
│ ├─ Tree-based (LightGBM, XGBoost, RF, CatBoost)
│ │ └─ Use TreeSHAP (exact, fast, O(TLD))
│ │
│ ├─ Linear (LogisticRegression, Lasso, Ridge)
│ │ └─ Use LinearSHAP or direct coefficient interpretation
│ │
│ ├─ Neural network
│ │ ├─ Tabular → KernelSHAP (slow) or DeepSHAP
│ │ └─ Image/text → GradientSHAP, Integrated Gradients
│ │
│ └─ Black-box / API-only
│ └─ KernelSHAP or LIME (model-agnostic)
│
├─ Explanation scope?
│ ├─ Global (overall model behavior)
│ │ ├─ Feature importance ranking → SHAP summary plot
│ │ ├─ Feature effect curves → PDP or SHAP dependence
│ │ └─ Feature interactions → SHAP interaction values
│ │
│ └─ Local (single prediction)
│ ├─ Detailed breakdown → SHAP waterfall
│ ├─ Quick approximation → LIME
│ └─ Contrastive ("why not X?") → SHAP force plot
│
└─ Audience?
├─ Data scientist → Full SHAP values, interaction plots
├─ Business stakeholder → Top 3 drivers, bar charts
└─ Regulatory / audit → Model cards, stability analysis
```
---
## Quick Reference: Methods Comparison
| Method | Scope | Speed (10k rows) | Consistency | Model Types |
|--------|-------|-------------------|-------------|-------------|
| TreeSHAP | Global + Local | < 1 min | Exact | Tree ensembles only |
| KernelSHAP | Global + Local | 10–60 min | Approximate | Any model |
| DeepSHAP | Global + Local | 2–10 min | Approximate | Neural networks |
| LIME | Local only | ~1 sec/instance | Unstable across runs | Any model |
| Permutation Importance | Global only | 1–5 min | Stable with enough reps | Any model |
| PDP | Global only | 1–5 min | Exact (for model) | Any model |
| ICE Plots | Local curves | 1–5 min | Exact (for model) | Any model |
---
## Operational Patterns
### Pattern 1: TreeSHAP for Tree-Based Models
- **Use when:** Using LightGBM, XGBoost, CatBoost, RandomForest
- **Implementation:**
```python
import shap
# Train model
model = lgb.LGBMClassifier(**params).fit(X_train, y_train)
# Create explainer (auto-detects tree type)
explainer = shap.TreeExplainer(model)
explanation = explainer(X_test) # returns a shap.Explanation, not a raw array/list
# Global: Summary plot (feature importance + distribution)
shap.summary_plot(explanation)
# Global: Bar plot (mean absolute SHAP per feature)
shap.plots.bar(explanation)
# Local: Single prediction waterfall
shap.plots.waterfall(explanation[0])
```
- **Version gotcha (verify against your installed SHAP version before trusting older tutorials):** the old pattern `shap_values = explainer.shap_values(X_test); shap_values[1]` assumed multi-output classification always returned a Python list indexed by class. As of SHAP 0.45.0, `shap_values()` for multi-output/scikit-learn-style classifiers returns a single `np.ndarray` of shape `(n_samples, n_features, n_classes)` instead — `shap_values[1]` on that array now selects **sample 1**, not the positive class, and silently produces a wrong explanation with no error. For binary classification, index the last axis (`shap_values[..., 1]`) or, better, use the modern `explainer(X)` call shown above, which always returns a `shap.Explanation` object with `.values`/`.base_values` regardless of model type. LightGBM/XGBoost with the default raw-margin output instead return a plain `(n_samples, n_features)` array with no class axis at all — check `.shape` before indexing rather than assuming a fixed convention.
- **Performance tip:** For large datasets, compute SHAP on a representative sample (5k–10k rows)
- **Gotcha:** TreeSHAP with `feature_perturbation='interventional'` gives causal-style attribution but requires background data
### Pattern 2: KernelSHAP for Black-Box Models
- **Use when:** Model is an API, neural network, or ensemble of mixed types
- **Implementation:**
```python
# Background data: use k-means summary for speed
background = shap.kmeans(X_train, 100)
explainer = shap.KernelExplainer(model.predict_proba, background)
# Compute on subset (KernelSHAP is slow)
shap_values = explainer.shap_values(X_test[:500], nsamples=500)
```
- **Speed tradeoff:** `nsamples` controls accuracy vs speed
- `nsamples=100` — fast, rough approximation
- `nsamples=500` — good balance
- `nsamples=2048` — high accuracy, slow
### Pattern 3: LIME for Quick Local Explanations
- **Use when:** Need fast, single-prediction explanation for stakeholders
- **Implementation:**
```python
from lime.lime_tabular import LimeTabularExplainer
lime_exp = LimeTabularExplainer(
training_data=X_train.values,
feature_names=X_train.columns.tolist(),
class_names=['Negative', 'Positive'],
mode='classification',
discretize_continuous=True,
)
# Explain single prediction
exp = lime_exp.explain_instance(
X_test.iloc[0].values,
model.predict_proba,
num_features=10,
num_samples=5000,
)
exp.show_in_notebook()
# Or export: exp.as_html(), exp.as_list()
```
- **Stability check:** Run LIME 5 times on same instance — if top features change, results are unreliable
- **Gotcha:** LIME fits a local linear model — fails for highly non-linear local behavior
### Pattern 4: Permutation Importance
- **Use when:** Need global feature ranking, model-agnostic, simple to explain
- **Implementation:**
```python
from sklearn.inspection import permutation_importance
result = permutation_importance(
model, X_test, y_test,
n_repeats=30,
random_state=42,
scoring='average_precision',
n_jobs=-1,
)
# Sort by importance
sorted_idx = result.importances_mean.argsort()[::-1]
for idx in sorted_idx[:15]:
print(f"{X_test.columns[idx]:30s}: "
f"{result.importances_mean[idx]:.4f} +/- {result.importances_std[idx]:.4f}")
```
- **Key advantage:** Measures importance on unseen data (test set) — avoids overfitting bias
- **Gotcha:** Correlated features split importance — consider grouping correlated features
### Pattern 5: Partial Dependence and ICE Plots
- **Use when:** Need to show how a feature affects predictions across its range
- **Implementation:**
```python
from sklearn.inspection import PartialDependenceDisplay
# PDP for top features
features = ['age', 'income', ('age', 'income')] # single + interaction
PartialDependenceDisplay.from_estimator(
model, X_train, features,
kind='both', # PDP (average) + ICE (individual)
subsample=500, # ICE lines to plot
grid_resolution=50,
n_jobs=-1,
)
```
- **PDP vs ICE:**
- PDP = average effect (can hide heterogeneity)
- ICE = individual curves (reveals subgroups with different effects)
- **Always plot both** — if ICE lines are parallel, PDP is reliable; if they cross, PDP is misleading
### Pattern 6: Feature Importance Stability Analysis
- **Use when:** Regulatory or audit context, need confidence in feature rankings
- **Implementation:**
```python
import numpy as np
# Bootstrap SHAP importance stability
n_bootstrap = 20
importance_ranks = []
for i in range(n_bootstrap):
sample_idx = np.random.choice(len(X_test), size=len(X_test), replace=True)
X_sample = X_test.iloc[sample_idx]
exp_sample = explainer(X_sample) # shap.Explanation; check exp_sample.values.shape once, don't assume
# exp_sample.values.shape is (n_samples, n_features, n_classes) for scikit-learn-style
# binary classifiers, or (n_samples, n_features) for raw-margin LightGBM/XGBoost — branch on it
vals = exp_sample.values[..., 1] if exp_sample.values.ndim == 3 else exp_sample.values
mean_abs = np.abs(vals).mean(axis=0)
ranks = np.argsort(-mean_abs) # descending
importance_ranks.append(ranks)
# Compute rank stability per feature
from scipy.stats import kendalltau
stability_scores = []
for i in range(n_bootstrap):
for j in range(i+1, n_bootstrap):
tau, _ = kendalltau(importance_ranks[i], importance_ranks[j])
stability_scores.append(tau)
print(f"Mean rank correlation: {np.mean(stability_scores):.3f}")
# > 0.9 = stable rankings; < 0.7 = unstable, report with caveats
```
---
## Audience-Appropriate Explanations
### Technical Audience (Data Scientists)
- Full SHAP summary plots with distributions
- Interaction values and dependence plots
- Permutation importance with confidence intervals
- Raw SHAP values for downstream analysis
### Business Stakeholders
- Top 3–5 drivers as horizontal bar chart
- Natural language: "This customer was flagged primarily because their account age (2 months) is unusually short, and their transaction frequency (47/day) is 5x the average"
- Avoid: SHAP values, log-odds, probability scores
- Use: directional language ("increases risk", "decreases likelihood")
### Regulatory / Audit
- Model card documenting: features used, protected attributes, fairness metrics
- Stability analysis across bootstrap samples
- Monotonicity checks for regulated features
- Feature importance consistency across time periods
- Documentation template:
```markdown
## Model Explanation Report
- Model type: [type]
- Training date: [date]
- Explanation method: [TreeSHAP/KernelSHAP]
- Top 10 features (stable across 20 bootstrap runs): [list]
- Protected attribute impact: [analysis]
- Monotonicity compliance: [pass/fail per feature]
```
---
## Anti-Patterns
| Anti-Pattern | Why It Fails | Fix |
|---|---|---|
| Using `model.feature_importances_` as primary explanation | Biased toward high-cardinality features (Gini/split-based) | Use SHAP or permutation importance |
| LIME without stability check | LIME explanations change across runs | Run 5x, report only stable features |
| SHAP on entire dataset (500k+ rows) | Slow and unnecessary | Sample 5k–10k representative rows |
| Showing raw SHAP values to business users | Not interpretable without context | Translate to "increases/decreases" language |
| PDP without ICE overlay | Hides heterogeneous effects | Always use `kind='both'` |
| Permutation importance on training data | Overfitting inflates importance | Always compute on test/holdout set |
| Confusing feature importance with causation | Correlation != causation | Explicitly state "predictive importance, not causal" |
| Single explanation method | Each has blind spots | Use 2+ methods, check agreement |
| Ignoring correlated features | SHAP splits importance among correlated features | Group correlated features or note caveat |
| KernelSHAP with too few nsamples | Noisy, unreliable attributions | Minimum nsamples=500 for production use |
---
## Model Card Template (Interpretability Section)
```markdown
## Interpretability
### Explanation Method
- Primary: TreeSHAP (exact for tree ensemble)
- Secondary: Permutation importance (validation)
### Top Features (Stable)
| Rank | Feature | Mean |SHAP| | Direction |
|------|---------|-------------|-----------|
| 1 | [name] | [value] | [+/-] |
### Stability
- Bootstrap rank correlation (Kendall tau): [value]
- Feature ranking consistent across [N] time periods: [yes/no]
### Limitations
- [Correlated feature groups]
- [Non-monotonic relationships]
- [Protected attribute interactions]
```
---
## Validation Checklist
- [ ] Explanation method matches model type (TreeSHAP for trees, etc.)
- [ ] SHAP values computed on representative sample (not full dataset unless small)
- [ ] Feature importance stable across bootstrap samples (tau > 0.8)
- [ ] Permutation importance confirms SHAP rankings (top 5 agree)
- [ ] PDP/ICE plots reviewed for non-linear effects and interactions
- [ ] Business-appropriate summary prepared (top drivers, directional language)
- [ ] Model card updated with interpretability section
- [ ] No causal claims made from correlational analysis
- [ ] Protected attributes checked for disproportionate importance
---
## Cross-References
- `ai-ml-data-science/references/class-imbalance-patterns.md` — interpreting minority-class predictions
- `ai-ml-data-science/references/hyperparameter-optimization.md` — feature importance after tuning
- `ai-mlops/references/experiment-tracking-patterns.md` — logging SHAP artifacts
- `ai-rag/references/embedding-model-guide.md` — explaining embedding-based features
references/llm-data-pipeline.md# LLM Training Data Pipelines Practical patterns for preparing large-scale text corpora for LLM pretraining and fine-tuning. Route here first when building an LLM from scratch or curating a pretraining mixture. For model selection, evaluation, and serving, see the sibling references. --- ## Table of Contents - [1. Deduplication](#1-deduplication) - [1.1 Exact Dedup](#11-exact-dedup) - [1.2 Fuzzy Dedup — MinHash LSH](#12-fuzzy-dedup--minhash-lsh) - [1.3 Semantic Dedup](#13-semantic-dedup) - [1.4 Scale Guidance](#14-scale-guidance) - [2. Quality Filtering](#2-quality-filtering) - [2.1 Heuristic Filters](#21-heuristic-filters) - [2.2 Classifier-Based Filters](#22-classifier-based-filters) - [2.3 Model-Scoring Filters](#23-model-scoring-filters) - [2.4 CQF Caveat](#24-cqf-caveat) - [3. Synthetic Data Mixing](#3-synthetic-data-mixing) - [4. Decontamination](#4-decontamination) - [4.1 Detection Methods](#41-detection-methods) - [4.2 Contamination-Resistant Benchmarks](#42-contamination-resistant-benchmarks) - [Related Resources](#related-resources) ## 1. Deduplication Deduplication is the highest-leverage single step in data curation. Duplicate documents inflate token counts, bias the model toward repeated phrasings, and inflate benchmark scores on seen text. ### 1.1 Exact Dedup - Hash-based: SHA-256 or xxHash over normalized document text (strip whitespace, lowercase optional) - Fast and cheap; catches byte-for-byte copies - Always run exact dedup before fuzzy dedup ### 1.2 Fuzzy Dedup — MinHash LSH - MinHash with Locality-Sensitive Hashing (LSH) finds near-duplicate documents without pairwise comparison - Jaccard similarity threshold typically 0.7–0.8 for aggressive dedup; 0.85–0.9 for conservative - Implementation references: `datasketch` (Python), `text-dedup` library - Scales to hundreds of billions of tokens on a single machine or small cluster ### 1.3 Semantic Dedup - Embed documents; cluster or threshold by cosine similarity - Catches paraphrases and format-converted duplicates that evade MinHash - Expensive: requires embedding inference at corpus scale - Use selectively: apply to high-value domains or as a post-MinHash pass ### 1.4 Scale Guidance | Corpus size | Recommended approach | |-------------|----------------------| | < 10 TB | Exact dedup + MinHash LSH | | > 10 TB | Add Bloom filter for streaming exact dedup; keep MinHash for near-dup | | All scales | Semantic dedup is a targeted pass, not a default | **Bloom filter note:** At corpus sizes above ~10 TB, streaming exact dedup with a Bloom filter avoids loading the full hash set into memory. False-positive rate is tunable; 1e-6 is a typical target for large corpora. --- ## 2. Quality Filtering ### 2.1 Heuristic Filters Apply first — cheap, interpretable, and effective at removing the worst content: - Remove documents below minimum token count (e.g., < 50 tokens) - Remove by content type: code dumps in prose corpora, boilerplate, navigational text - Remove by repetition: character n-gram repetition ratio > threshold (e.g., same 20-gram repeating > 3 times) - Remove by perplexity floor: documents with extremely low perplexity under a small LM may be templated boilerplate - Language filter: keep target language(s); fastText lid.176 is standard for language identification ### 2.2 Classifier-Based Filters - Train a fastText or similar binary classifier on high-quality vs low-quality documents - Common training signal: web text curated by humans (Wikipedia, books, curated forums) as positive; random crawl as negative - Apply score threshold; threshold is a quality dial — higher threshold keeps fewer but better documents ### 2.3 Model-Scoring Filters - Use a small LM to score documents: low perplexity on a reference quality model indicates "on distribution" - More expensive than fastText but captures subtler quality signals - Apply after heuristic + classifier passes to avoid scoring junk ### 2.4 CQF Caveat **Classifier quality filtering (CQF) limitation:** Filtering toward a reference classifier's training distribution improves downstream benchmark scores but does not necessarily improve LM performance on the reference corpus itself. The benchmark gains may reflect distribution match rather than genuine capability improvement. Cite: this pattern is described in corpus curation literature (verify against primary papers before asserting specific figures). Do not treat classifier filtering as unconditionally beneficial — it shapes the model's distribution, which may exclude useful diversity. **Checklist: Quality Filtering** - [ ] Heuristic passes applied first (length, repetition, language ID) - [ ] Classifier or model-scoring applied after heuristics - [ ] Threshold choices documented and reversible (filtered documents kept for audit) - [ ] CQF limitation noted if classifier-filtered data is used in benchmark claims --- ## 3. Synthetic Data Mixing Synthetic data (LLM-generated) can supplement natural text but does not replace it. **Key constraints (hedge: specific ratios are dataset- and model-specific; verify against primary experiments):** - Pure-synthetic pretraining does not consistently match natural-text pretraining on held-out evals - Mixtures of natural + synthetic can outperform natural-only when synthetic fills coverage gaps (rare domains, instruction formats, reasoning chains) - The optimal mixing ratio depends on the synthetic data generator quality, the domain, and the target task — no universal ratio is established - A common pattern reported in the literature is using synthetic data for 20–50% of domain-specific fine-tuning data, not for the majority of pretraining; verify against the specific papers before asserting a ratio **Anti-patterns:** - Using purely synthetic data for pretraining and expecting benchmark parity with natural-data models - Mixing synthetic without decontaminating against evaluation benchmarks (see §4) - Synthetic-only instruction tuning without human-curated seed data for quality anchoring **Checklist: Synthetic Mixing** - [ ] Synthetic fraction is a deliberate choice, not a default - [ ] Synthetic data source and generator model documented - [ ] Decontamination run on synthetic data against target benchmarks - [ ] Natural-vs-synthetic ablation exists or is planned --- ## 4. Decontamination Contamination (test data appearing verbatim or near-verbatim in training data) inflates benchmark scores and makes comparisons unreliable. ### 4.1 Detection Methods **MinHash near-duplicate detection:** - Compute MinHash signatures for all benchmark examples - Check against training corpus with LSH at low Jaccard threshold (0.5–0.7) - Remove or quarantine matching training documents **Min-K% Prob (Black-box contamination detection):** - For a given text, extract the k% of tokens with lowest log-probability under the model - Contaminated examples tend to have higher minimum-k% probability than non-contaminated examples - Can be applied post-training to detect contamination without corpus access - Reference: Min-K% Prob (arXiv 2310.16789, verify URL before citing) **Exact string matching:** - n-gram overlap (13-gram is a common threshold from LLaMA and similar work) - Fast, but misses paraphrased contamination ### 4.2 Contamination-Resistant Benchmarks When contamination risk is high (large crawled corpora, frequently cited benchmarks), prefer: - Recently released benchmarks not present in the training window - Private held-out test sets - Benchmarks with procedural generation (new problems per evaluation run) - Contamination-resistant benchmark suites (e.g., LiveBench, MMLU-Pro-style holdouts) **Scope:** This section applies to LLM-based models or any setting where pretraining or fine-tuning data overlaps with evaluation data. For classical ML with small curated datasets, standard train/test split hygiene (see `modelling-patterns.md` §2) is sufficient. **Checklist: Decontamination** - [ ] MinHash near-dup run between training corpus and all target benchmarks - [ ] Exact n-gram matching checked (13-gram or tighter) - [ ] Contaminated documents quarantined and re-evaluated without them - [ ] Min-K% Prob check considered for post-training contamination audit - [ ] Benchmark choice accounts for contamination risk --- ## Related Resources - [Modelling Patterns](modelling-patterns.md) - Model family selection and tabular baselines - [Evaluation Patterns](evaluation-patterns.md) - Benchmark contamination detection and evaluation design - [Data Contracts & Lineage](data-contracts-lineage.md) - Annotation quality and data governance - [Reproducibility Checklist](reproducibility-checklist.md) - Experiment tracking and artifact versioning
references/ml-diagrams.md# ML Diagram Catalog (Mermaid)
Reusable Mermaid diagrams for classical ML algorithms and neural network architectures. Drop into READMEs, docs, PR descriptions, notebooks, or agent outputs.
## Table of Contents
- [When to Use](#when-to-use)
- [Authoring Conventions](#authoring-conventions)
- [Classical ML](#classical-ml)
- [Neural Network Architectures](#neural-network-architectures)
- [Feedforward (MLP)](#feedforward-mlp)
- [Recurrent Neural Network](#recurrent-neural-network-unrolled)
- [Convolutional Neural Network](#convolutional-neural-network-image-classifier)
- [Transformer (original encoder)](#transformer-encoder-block--full-stack)
- [Modern LLM Decoder Block](#modern-llm-decoder-block-production-grade)
- [Mixture of Experts FFN](#mixture-of-experts-moe-ffn-sublayer)
- [State Space Model (Mamba)](#state-space-model-block-mamba-style)
- [Vision Transformer (ViT)](#vision-transformer-vit)
- [Diffusion Transformer (DiT)](#diffusion-transformer-dit--imagevideo-generation)
- [Multimodal VLM (LLaVA-style)](#multimodal-vision-language-model-llava-style)
- [Reasoning model training loop](#reasoning-model-training-loop-rl-with-cot)
- [Variants Worth Adding](#variants-worth-adding)
- [Anti-Patterns](#anti-patterns)
## When to Use
- Documenting an ML pipeline in markdown-rendered surfaces (GitHub, Notion, Obsidian, MkDocs)
- Writing PR descriptions for model changes that need a quick "before/after"
- Teaching, onboarding, or explaining a model family without slide tooling
Skip for: mathematical derivations (use LaTeX), trained-model artifacts (use SHAP / netron / TensorBoard), system-level infra diagrams (different skill).
## Authoring Conventions
- **`flowchart LR/TB/TD`** for pipelines and sequential ops.
- **`subgraph`** to encapsulate repeated blocks (transformer encoder, RNN cell).
- **Edge labels** carry *data shape*; **node labels** carry *operation*.
- **`(())` double-circle** for residual sums / merges.
- **`{}` diamond** for branching decisions (decision trees, convergence checks).
- Quote labels (`"..."`) when they contain `()`, `:`, `/`, or commas.
---
## Classical ML
### K-Means Clustering
```mermaid
flowchart TD
A[Input: unlabeled data X ∈ ℝⁿˣᵈ] --> B[Choose k clusters]
B --> C[Initialize k centroids<br/>random or k-means++]
C --> D[Assign each point to<br/>nearest centroid<br/>argmin ‖xᵢ − μⱼ‖²]
D --> E[Recompute centroids<br/>μⱼ = mean of assigned points]
E --> F{Centroids<br/>moved?}
F -- yes --> D
F -- no --> G[Output: cluster labels<br/>+ final centroids]
```
### Logistic Regression
```mermaid
flowchart LR
X[Features x] --> L[Linear: z = wᵀx + b]
L --> S[Sigmoid: σz = 1 / 1+e⁻ᶻ]
S --> P[Probability ŷ ∈ 0,1]
P --> C[Binary cross-entropy loss<br/>L = −y log ŷ − 1−y log 1−ŷ]
C --> G[Gradient ∂L/∂w, ∂L/∂b]
G --> U[SGD update<br/>w ← w − η ∇w]
U -. next batch .-> L
```
### Decision Tree (classification)
```mermaid
flowchart TD
R[Root: all samples] --> Q1{feature_3 ≤ 0.42?}
Q1 -- yes --> Q2{feature_7 ≤ 1.10?}
Q1 -- no --> Q3{feature_1 ≤ −0.30?}
Q2 -- yes --> L1[Leaf: class A<br/>n=124, gini=0.08]
Q2 -- no --> L2[Leaf: class B<br/>n=58, gini=0.11]
Q3 -- yes --> L3[Leaf: class B<br/>n=77, gini=0.05]
Q3 -- no --> Q4{feature_9 ≤ 2.5?}
Q4 -- yes --> L4[Leaf: class C<br/>n=43, gini=0.14]
Q4 -- no --> L5[Leaf: class A<br/>n=91, gini=0.06]
```
### Collaborative Filtering (matrix factorization)
```mermaid
flowchart LR
subgraph Input
R[User–Item matrix R<br/>m × n, sparse ratings]
end
R --> F[Factorize<br/>R ≈ U · Vᵀ]
F --> U[User embeddings U<br/>m × k]
F --> V[Item embeddings V<br/>n × k]
U --> P[Predict r̂_ui = uᵤᵀ vᵢ]
V --> P
P --> Loss[Loss = Σ_obs r_ui − r̂_ui² + λ‖U‖²+‖V‖²]
Loss --> Opt[SGD / ALS update]
Opt -. iterate .-> F
P --> Rec[Top-N recommendations<br/>for user u]
```
---
## Neural Network Architectures
### Feedforward (MLP)
```mermaid
flowchart LR
X["Input x ∈ ℝᵈ"] --> H1["Dense 128 + ReLU"]
H1 --> H2["Dense 64 + ReLU"]
H2 --> H3["Dense 32 + ReLU"]
H3 --> O["Dense C + Softmax"]
O --> Y["Class probabilities"]
```
### Recurrent Neural Network (unrolled)
```mermaid
flowchart LR
X1[x₁] --> C1[RNN cell]
H0[h₀] --> C1
C1 --> H1[h₁]
C1 --> Y1[y₁]
X2[x₂] --> C2[RNN cell]
H1 --> C2
C2 --> H2[h₂]
C2 --> Y2[y₂]
X3[x₃] --> C3[RNN cell]
H2 --> C3
C3 --> H3[h₃]
C3 --> Y3[y₃]
XT[xₜ] --> CT[RNN cell]
H3 -. ... .-> CT
CT --> HT[hₜ]
CT --> YT[yₜ]
```
Every cell shares parameters `Wₕ, Wₓ, b` — the unroll is conceptual, not architectural. LSTM/GRU swap the inner cell for a gated variant but keep this skeleton.
### Convolutional Neural Network (image classifier)
```mermaid
flowchart LR
I["Image B,32,32,3"] --> C1["Conv 3x3, 32 filters + ReLU"]
C1 --> P1["MaxPool 2x2"]
P1 --> C2["Conv 3x3, 64 filters + ReLU"]
C2 --> P2["MaxPool 2x2"]
P2 --> C3["Conv 3x3, 128 filters + ReLU"]
C3 --> GAP["Global Avg Pool"]
GAP --> FC["Dense 256 + ReLU + Dropout"]
FC --> OUT["Dense 10 + Softmax"]
```
### Transformer (encoder block + full stack)
```mermaid
flowchart TB
subgraph Block["Transformer encoder block"]
direction TB
IN[Input embeddings] --> MHA[Multi-Head Self-Attention<br/>Q,K,V projections]
IN --> R1((+))
MHA --> R1
R1 --> N1[LayerNorm]
N1 --> FFN[FeedForward<br/>Linear → GELU → Linear]
N1 --> R2((+))
FFN --> R2
R2 --> N2[LayerNorm]
N2 --> OUT[Block output]
end
TOK[Token IDs] --> EMB[Token embedding]
POS[Positions] --> PE[Positional encoding]
EMB --> SUM((+))
PE --> SUM
SUM --> B1[Encoder block × 1]
B1 --> B2[Encoder block × 2]
B2 --> BN[... × N]
BN --> HEAD[Task head<br/>classification / LM / etc.]
```
Notes:
- `(+)` nodes are **residual connections** — without them, deep transformers do not train.
- This is the **post-norm** layout from the original paper. Modern LLMs (GPT, LLaMA) use **pre-norm** (LayerNorm *before* the sublayer, then residual sum) for stability at depth.
- Decoder blocks add (1) **masked** self-attention to prevent peeking at future tokens, and (2) a **cross-attention** layer that consumes encoder output.
### Modern LLM Decoder Block (production-grade)
What frontier-tier decoder-only LLMs actually deploy. Differences from the original block above are highlighted in the notes.
```mermaid
flowchart TB
subgraph Block["Modern decoder block (pre-norm)"]
direction TB
IN[Hidden state h] --> N1[RMSNorm]
N1 --> ATTN[Grouped-Query Attention<br/>Q heads, KV head-groups<br/>+ RoPE on Q,K<br/>+ KV cache append<br/>+ FlashAttention kernel]
IN --> R1((+))
ATTN --> R1
R1 --> N2[RMSNorm]
N2 --> FFN[SwiGLU FFN<br/>up + gate + down<br/>or MoE router → top-k experts]
R1 --> R2((+))
FFN --> R2
R2 --> OUT[Hidden state h']
end
TOK[Token IDs] --> EMB[Token embedding]
EMB --> B1[Decoder block × 1]
B1 --> B2[Decoder block × 2]
B2 --> BN[... × N]
BN --> FN[Final RMSNorm]
FN --> LH[LM head<br/>tied to embedding]
LH --> LOGITS[Next-token logits]
```
What changed vs the original block:
| Original | Modern production | Why |
|---|---|---|
| LayerNorm | **RMSNorm** | Cheaper, equally stable |
| Post-norm | **Pre-norm** | Trains deeper without divergence |
| Additive positional encoding | **RoPE** applied inside attention to Q,K | Extrapolates to longer context; no separate position embedding to learn |
| Multi-Head Attention | **GQA** (or MQA) | Cuts KV-cache memory by `n_heads / n_kv_groups`× — decisive at long context |
| Plain attention | **+ KV cache + FlashAttention** | KV cache reuses past K,V across decode steps; FlashAttention fuses the softmax to avoid materializing the attention matrix |
| GELU FFN | **SwiGLU** (gated) | Better quality at same parameter count |
| Dense FFN every layer | **MoE router → top-k experts** (frontier-tier) | Decouples capacity from FLOPs per token |
| Encoder + decoder | **Decoder-only** | Simpler; works for both generation and embedding tasks |
### Mixture of Experts (MoE) FFN sublayer
Replaces the dense FFN inside each decoder block in MoE-tier models.
```mermaid
flowchart LR
H[Token hidden state] --> R[Router<br/>linear → softmax over N experts]
R --> TK[Top-k selection<br/>typically k=2 of N=8..256]
TK --> E1[Expert 1<br/>SwiGLU FFN]
TK --> E2[Expert 2<br/>SwiGLU FFN]
TK -. inactive .-> EN[Expert N<br/>SwiGLU FFN]
E1 --> W[Weighted sum<br/>by router probabilities]
E2 --> W
W --> OUT[FFN output]
R --> AUX[Aux load-balancing loss<br/>during training only]
```
Notes:
- Only **k of N** experts run per token — that's the whole point. Total parameter count is large; per-token FLOPs are small.
- The auxiliary loss prevents the router from collapsing to one expert. Production variants: expert choice routing, shared experts (DeepSeek), no-aux-loss balancing.
- Expert parallelism (EP) shards experts across devices — see `ai-llm-inference/references/moe-expert-parallelism.md`.
### State Space Model block (Mamba-style)
Subquadratic alternative to attention. Used in hybrid stacks (Jamba, Zamba, Samba) that mix SSM blocks with attention blocks.
```mermaid
flowchart LR
IN[Input x_t] --> P1[Linear projection]
P1 --> CONV[1D causal conv<br/>short-range mixing]
CONV --> ACT[SiLU activation]
ACT --> SSM[Selective SSM<br/>input-dependent A, B, C, Δ<br/>recurrence: h_t = A h_t-1 + B x_t<br/>output: y_t = C h_t]
IN --> GATE[Gate branch<br/>linear + SiLU]
SSM --> MUL((×))
GATE --> MUL
MUL --> P2[Linear projection]
P2 --> OUT[Output y_t]
```
Notes:
- **Linear** in sequence length, **constant** memory per decode step — opposite tradeoff to attention.
- The recurrence is parallelizable at training time via the selective scan kernel.
- Hybrid stacks alternate `SSM block / attention block / SSM block / ...` — SSM handles long-range context cheaply; attention handles precise lookup.
### Vision Transformer (ViT)
How transformers consume images. Foundation for CLIP, SigLIP, and most modern vision-language models.
```mermaid
flowchart LR
IMG["Image B,3,H,W"] --> PATCH["Patchify 16x16<br/>→ B,N,P²·3"]
PATCH --> LIN["Linear projection<br/>→ B,N,D"]
LIN --> CLS["Prepend CLS token"]
CLS --> POS["+ Positional embedding<br/>learned or 2D RoPE"]
POS --> ENC["Transformer encoder × L<br/>same block as text<br/>pre-norm, MHA or GQA"]
ENC --> POOL["CLS token<br/>or global avg pool"]
POOL --> HEAD["Task head<br/>classification / contrastive / VLM input"]
```
### Diffusion Transformer (DiT) — image/video generation
Replaces U-Net backbone in modern image/video gen models (Stable Diffusion 3, SDXL successors, Sora-class video).
```mermaid
flowchart LR
Z["Noisy latent z_t<br/>B,C,H,W"] --> PATCH["Patchify → tokens"]
T[Timestep t] --> TE[Timestep embedding]
C[Condition: text / class] --> CE[Conditioning embedding]
TE --> ADAN
CE --> ADAN
PATCH --> DiT["DiT block × L<br/>RMSNorm + AdaLN-Zero<br/>self-attn + FFN<br/>conditioned on t,c"]
ADAN[AdaLN-Zero<br/>per-block scale and shift] --> DiT
DiT --> UNPATCH[Unpatchify]
UNPATCH --> EPS["Predicted noise ε̂<br/>or velocity v̂"]
EPS --> SCHED[Sampler step<br/>DDIM / DPM++ / flow-matching ODE]
SCHED -. iterate T steps .-> Z
```
Notes:
- **AdaLN-Zero** is the modulation trick — timestep and condition modulate every block's norms.
- **Flow-matching** (rectified flow) has largely replaced DDPM-style training in new models — same architecture, different loss.
- Image gen typically `T = 20–50` steps; distilled / consistency models do it in `1–4`.
### Multimodal Vision-Language Model (LLaVA-style)
How images get into a decoder-only LLM.
```mermaid
flowchart LR
IMG[Image] --> VIT[Vision encoder<br/>ViT / SigLIP frozen]
VIT --> VTOK["Visual tokens<br/>B,N_v,D_v"]
VTOK --> PROJ["Projector<br/>MLP or Q-Former or perceiver<br/>D_v → D_llm"]
TXT[Text tokens] --> TEMB[LLM embedding]
PROJ --> CONCAT["Concat: visual ⊕ text tokens"]
TEMB --> CONCAT
CONCAT --> LLM[Decoder-only LLM<br/>modern block × N]
LLM --> OUT[Text response]
```
Notes:
- The **projector** is the trainable bridge — encoder and LLM are often frozen during stage-1 training.
- **Native multimodal** models (Chameleon, Gemini, GPT-4o) skip the projector by training a single transformer on interleaved image/text/audio tokens from scratch.
### Reasoning model training loop (RL-with-CoT)
How o1/o3/R1-class models are trained. Distinct from RLHF — the reward is verifiable correctness, not preference.
```mermaid
flowchart TB
BASE[Base or SFT model] --> ROLL[Rollout<br/>generate long CoT + answer<br/>per prompt]
PROMPT[Math / code / reasoning prompts<br/>with verifiable answers] --> ROLL
ROLL --> JUDGE[Verifier<br/>exact match / unit tests / proof check]
JUDGE --> REW[Reward signal<br/>1 if correct, 0 if not<br/>+ optional format / length shaping]
REW --> RL[RL update<br/>GRPO / PPO / REINFORCE++]
RL --> POL[Updated policy]
POL -. next rollout .-> ROLL
POL --> EVAL[Eval on held-out reasoning benches]
```
Notes:
- **GRPO** (group-relative policy optimization) is the dominant choice — drops the value model, normalizes advantages within a sampled group per prompt.
- No human preference labels needed — the verifier replaces RLHF's reward model.
- Long CoTs (10k–100k tokens) make this compute-intensive; throughput optimization matters more than for SFT.
---
## Variants Worth Adding
When the catalog needs to grow:
- **LSTM / GRU cell internals** — gates (forget/input/output) as separate sigmoid/tanh nodes, cell-state line passing through.
- **Attention head detail** — Q,K,V linear projections → `QKᵀ/√d` → softmax → multiply by V. Useful as zoom-in below the modern decoder block.
- **Multi-head vs GQA vs MQA side-by-side** — show the KV-head sharing pattern explicitly; this is the most-asked clarification.
- **RoPE rotation visual** — Q,K rotated in 2D pairs by frequency; harder to draw in Mermaid, may need static image.
- **U-Net** — symmetric encoder–decoder with skip connections; still relevant for medical imaging even though DiT replaced it for generation.
- **Autoencoder / VAE** — VAE adds the `μ, log σ²` split and reparameterization trick.
- **GNN message passing** — one layer of aggregate → transform → update, iterated as "× L".
- **Random Forest / Gradient Boosting** — one tree detailed, then ensemble combiner.
- **Speculative decoding** — draft model proposes k tokens, target verifies in parallel, accept prefix.
- **Encoder–decoder LLM** (T5/Flan style) — for the niche where it still wins (translation, summarization).
## Anti-Patterns
- **Don't draw the math.** Mermaid is bad at matrices. If you need `softmax(QKᵀ/√d)V` shown matrix-by-matrix, switch to LaTeX or a static image.
- **Don't unroll deep stacks.** Show "× N" with one block, not 12 stacked boxes.
- **Don't put hyperparameters in node labels.** Filter counts and dropout rates date a diagram fast — keep them in surrounding prose.
- **Don't combine training and inference in one diagram.** Two side-by-side diagrams beat one with optional dotted arrows.
- **Don't exceed ~25 nodes.** Past that, switch to Excalidraw/Figma, a table, or a notebook with `print(x.shape)` between layers.
references/modelling-patterns.md# Modelling Patterns
Operational modelling techniques, baseline-first workflows, split design, and model-family comparison rules for practical DS work.
---
## Table of Contents
- [1. Model Selection & Baselines (Practical Starting Points)](#1-model-selection-&-baselines-practical-starting-points)
- [1.1 Decision Guide](#11-decision-guide)
- [1.2 Baseline First Pattern](#12-baseline-first-pattern)
- [2. Train/Validation/Test Split Design](#2-trainvalidationtest-split-design)
- [2.1 Split Strategies](#21-split-strategies)
- [2.2 Common Pitfalls](#22-common-pitfalls)
- [2.3 Recommended Ratios](#23-recommended-ratios)
- [3. Model Family Selection](#3-model-family-selection)
- [3.1 Tabular Data](#31-tabular-data)
- [3.2 Text Data](#32-text-data)
- [3.3 When to Avoid Deep Models](#33-when-to-avoid-deep-models)
- [4. Hyperparameter Tuning](#4-hyperparameter-tuning)
- [4.1 Tuning Strategy](#41-tuning-strategy)
- [4.2 Key Parameters by Model](#42-key-parameters-by-model)
- [4.3 Stability and Reproducibility](#43-stability-and-reproducibility)
- [5. Overfitting Control](#5-overfitting-control)
- [5.1 Detection](#51-detection)
- [5.2 Mitigation Techniques](#52-mitigation-techniques)
- [6. CatBoost for Categorical-Heavy Data](#6-catboost-for-categorical-heavy-data)
- [6.1 When to Choose CatBoost](#61-when-to-choose-catboost)
- [6.2 CatBoost vs LightGBM vs XGBoost](#62-catboost-vs-lightgbm-vs-xgboost)
- [6.3 CatBoost Key Parameters](#63-catboost-key-parameters)
- [7. GPU Scaling for Large Datasets](#7-gpu-scaling-for-large-datasets)
- [7.1 When to Use GPU Training](#71-when-to-use-gpu-training)
- [7.2 GPU Training with LightGBM](#72-gpu-training-with-lightgbm)
- [7.3 Distributed Training with Ray](#73-distributed-training-with-ray)
- [7.4 XGBoost GPU Training](#74-xgboost-gpu-training)
- [8. Model Comparison](#8-model-comparison)
- [8.1 Fair Comparison Rules](#81-fair-comparison-rules)
- [8.2 Statistical Significance](#82-statistical-significance)
- [9. Thresholding for Classification](#9-thresholding-for-classification)
- [9.1 Threshold Selection](#91-threshold-selection)
- [9.2 Per-Segment Validation](#92-per-segment-validation)
## 1. Model Selection & Baselines (Practical Starting Points)
### 1.1 Decision Guide
Use current tooling and benchmarks as inputs, but keep the recommendation conditional on data shape, latency, interpretability, and team constraints.
| Data shape | Start with | Compare against | Notes |
|------------|------------|-----------------|-------|
| Tabular, small-medium (≤50k rows, ≤2k features) | Linear/logistic baseline, then LightGBM or CatBoost | TabPFN-2.5 as zero-shot baseline candidate (no tuning required) | Boosted trees are usually strong, but not automatic winners; add TabPFN-2.5 to the comparison set before tuning — see Section 3.1 |
| Tabular, categorical-heavy | CatBoost and LightGBM | regularized linear model | CatBoost often earns its keep when categorical handling is central |
| Tabular, very large structured data | LightGBM or CatBoost | simpler baseline, sampled baseline, occasionally compact NN | Escalate to neural/tabular-transformer approaches only with evidence |
| High-dimensional sparse text/counts | Regularized linear, NB | shallow tree model, shallow NN | Sparse linear baselines remain hard to beat on speed and interpretability |
| Time-aware tabular events | Leakage-safe baseline, then boosting | calibrated linear model | Use time-safe splits; move to `ai-ml-timeseries` if forecasting is the main problem |
| Mixed modalities | task-specific baseline per modality | late-fusion or specialized encoder | Avoid collapsing everything into one complex model too early |
**Rule:** describe a model family as a **strong baseline** or **good candidate**, not as universally best.
### 1.2 Baseline First Pattern
Always implement simple baselines first:
**Classification:**
- Majority-class classifier
- Stratified random
- Simple rule-based (if domain knowledge available)
**Regression:**
- Mean/median predictor
- Linear regression
- Moving average (for time series)
**Time series:**
- Seasonal naive forecast
- Last-value carry-forward
**Why baselines matter:**
- Establish minimum performance bar
- Reality check for model complexity
- Fast iteration and debugging
- Interpretability reference
- Provide a fallback candidate if the complex model fails calibration, latency, or governance checks
**Expert judgment: when the baseline should win**
A non-expert stops at "the fancier model scored higher." An expert asks what the lift actually costs and whether it survives scrutiny:
- If the complex candidate beats the baseline by less than the run-to-run variance across 3–5 seeds, there is no real winner yet — report it as noise, not progress.
- A 1–2 point metric gain that costs 10x inference latency, loses interpretability required for a regulated decision, or adds a new training dependency is usually not worth shipping — say so explicitly rather than defaulting to "higher number wins."
- Prefer the simpler model whenever the stronger candidate's gain is concentrated in one slice (e.g., one geography or one time window) rather than distributed — concentrated gains are often overfitting to that slice, not genuine generalization.
- When a linear/logistic baseline is within a few points of a boosted-tree candidate on tabular data, that is a signal the feature set is already doing most of the work — investigate the features before reaching for a bigger model.
- Treat "we added a neural net and it helped a little" as a red flag on small-to-medium tabular data (<50k rows): the more likely explanation is variance or leakage, not a genuine capacity advantage. Re-check the split and feature pipeline before concluding the neural net is the reason.
**Checklist: Baselines**
- [ ] Simple baseline implemented (majority class, mean, naive forecast)
- [ ] At least one simple baseline compared against one strong structured-data candidate
- [ ] Complexity added only after baselines understood
- [ ] Compute, latency, and explainability constraints considered early
- [ ] Model performance logged in experiment tracker (MLflow/W&B)
---
## 2. Train/Validation/Test Split Design
### 2.1 Split Strategies
**Random split (IID):**
- Use when: Data is independent and identically distributed
- Pros: Simple, maximizes training data
- Cons: Doesn't test temporal generalization
**Time-based split:**
- Use when: Forecasting or temporal leakage risk
- Pattern: Train on [T0, T1], validate on [T1, T2], test on [T2, T3]
- Pros: Tests realistic deployment scenario
- Cons: Less training data, seasonality may affect splits
**Group-based split:**
- Use when: User/item/entity leakage risk
- Pattern: Split by user_id, never mix same user across sets
- Examples: Recommendation systems, fraud detection
- Pros: Tests generalization to new entities
- Cons: Reduces effective sample size
**Cross-validation:**
- Use when: Small datasets, need robust estimates
- K-fold: 5 or 10 folds typical
- Stratified: Preserve class balance in each fold
- Time-series CV: Rolling/expanding window
- Pros: Better variance estimates, more data usage
- Cons: K times slower, risk of data leakage if not careful
### 2.2 Common Pitfalls
**Leakage:**
- Same entity in train and test (user, transaction)
- Feature computed using test data
- Future information in training
**Imbalance:**
- Rare classes missing from validation/test
- Non-representative splits
**Size:**
- Test set too small for reliable metrics
- Validation set too small for hyperparameter tuning
### 2.3 Recommended Ratios
**Large datasets (>100k samples):**
- Train: 80%, Validation: 10%, Test: 10%
**Medium datasets (10k-100k):**
- Train: 70%, Validation: 15%, Test: 15%
**Small datasets (<10k):**
- Use cross-validation instead of single split
- Hold out 20% for final test
**Checklist: Split Design**
- [ ] Split respects time order when needed
- [ ] No record from same entity in both train and test where leakage matters
- [ ] Test/validation sets held out from all model decisions
- [ ] Evaluation method documented and reproducible
- [ ] Class balance validated in all splits
- [ ] Test set size sufficient for statistical significance
---
## 3. Model Family Selection
### 3.1 Tabular Data
**Usual candidates:**
- LightGBM for fast, strong tabular baselines
- CatBoost when categorical handling is central or encodings are awkward
- XGBoost when the surrounding stack already standardizes on it
**TabPFN v2 / v2.5 (small–medium datasets only):**
- TabPFN v2 (Nature 2025, arXiv 2501.02945 lineage) and TabPFN-2.5 (arXiv 2511.08667, verified reachable 2026-07-11) are prior-fitted transformer models that require no per-dataset hyperparameter tuning.
- Size regime guidance (per arXiv 2511.08667; re-verify against https://github.com/automl/TabPFN before quoting numbers in a report):
- ≤10k samples, ≤500 features: add TabPFN-2.5 to the comparison set. The paper reports a 100% win rate against default (untuned) XGBoost on this regime on the TabArena benchmark — a strong signal, but "default XGBoost" is a weak baseline; still compare against a *tuned* boosted-tree model before treating TabPFN-2.5 as final.
- Up to 100k samples, 2k features: the paper reports an 87% win rate against default XGBoost at this larger scale, and TabPFN-2.5 is reported to match AutoGluon 1.4 (a ~4-hour tuned ensemble). TabPFN-2.5 is documented as built/targeted for up to 50k rows / 2k features; the 100k-row figure is from the paper's extended benchmark, not the stated design envelope — treat the 50k–100k band as "worth trying, verify latency and memory," not a safe default.
- Large (beyond ~100k): keep LightGBM/CatBoost as defaults; TabPFN-2.5 is not designed for this regime.
- A distillation engine (per the same paper) can compress a fitted TabPFN-2.5 into a compact MLP or tree ensemble for low-latency serving — relevant if TabPFN-2.5 wins the offline comparison but raw inference cost blocks production use. Verify current tooling support before committing to this path; it is new as of the Nov 2025 paper.
- Frame as: add to the comparison set. Do not replace boosted-tree baselines — verify on each problem.
- Win rates are against *default* competitor configs; do not cite them as "beats tuned XGBoost" without checking the paper's exact comparison setup.
**Linear models:**
- Logistic regression (interpretable baseline)
- Ridge/Lasso (regularized linear)
- Use when: Need interpretability, compliance, or very fast inference
**Neural networks:**
- Consider only when: very large datasets, complex interactions, or strong prior evidence
- Validate against boosted-tree baselines before committing
- TabNet, FT-Transformer, or compact MLPs are experiments, not default answers
### 3.2 Text Data
**Start with:**
- TF-IDF + linear models (fast baseline)
- Pretrained embeddings (Sentence-BERT) + LightGBM
**Advanced:**
- Fine-tuned transformers (BERT, RoBERTa)
- Only when: large labeled dataset, domain mismatch justifies fine-tuning, and inference cost is acceptable
### 3.3 When to Avoid Deep Models
**Don't use neural networks when:**
- Small datasets (<10k samples)
- Highly structured relational data (use tree models)
- Need interpretability for compliance
- Limited compute budget
- The boosted-tree or linear baseline already meets the acceptance threshold
**Checklist: Model Family**
- [ ] Model complexity matches data size
- [ ] Baseline -> interpretable model -> complex model progression
- [ ] Compute and latency constraints considered
- [ ] Interpretability requirements documented
- [ ] Thresholding, calibration, and uncertainty implications considered for the final candidate
---
## 4. Hyperparameter Tuning
### 4.1 Tuning Strategy
**Level 1: Manual scan (fast)**
- Test 3-5 values per key parameter
- Use domain knowledge and defaults
- Time: Minutes to hours
**Level 2: Grid search (thorough)**
- Small grid on important parameters
- Use when: Need reproducibility
- Time: Hours to day
**Level 3: Random search (efficient)**
- Sample random combinations
- Better than grid for high-dimensional spaces
- Time: Hours to day
**Level 4: Bayesian optimization (smart)**
- Use Optuna, Ray Tune, Hyperopt
- Learns from previous trials
- Time: Hours to days
### 4.2 Key Parameters by Model
**LightGBM:**
- `num_leaves` (31-255)
- `learning_rate` (0.01-0.3)
- `min_data_in_leaf` (20-100)
- `feature_fraction` (0.7-1.0)
**XGBoost:**
- `max_depth` (3-10)
- `learning_rate` (0.01-0.3)
- `min_child_weight` (1-10)
- `subsample` (0.7-1.0)
**Neural networks:**
- Learning rate (1e-5 to 1e-2, log scale)
- Batch size (16, 32, 64, 128)
- Dropout rate (0.1-0.5)
- Number of layers (2-6)
### 4.3 Stability and Reproducibility
**Best practices:**
- Set random seeds (model, data split, sampling)
- Run multiple seeds for final model (e.g., 5 seeds)
- Report mean +/- std across seeds
- Log all hyperparameters to experiment tracker
**Checklist: Hyperparameter Tuning**
- [ ] Parameters logged in experiment tracker
- [ ] Seeds logged and controlled
- [ ] Overfitting checked (train vs validation)
- [ ] Multiple runs for stability (3-5 seeds minimum)
- [ ] Best parameters documented with justification
---
## 5. Overfitting Control
### 5.1 Detection
**Indicators of overfitting:**
- Train error decreases while validation error increases
- Large gap between train and validation metrics
- Model performs well on training data but poorly on new data
**Monitoring:**
- Plot train vs validation loss/metric over epochs/iterations
- Check learning curves
- Validate on held-out test set
### 5.2 Mitigation Techniques
**For tree models:**
- Limit `max_depth` (3-10)
- Increase `min_data_in_leaf` / `min_child_weight`
- Reduce `num_leaves`
- Use feature subsampling (`feature_fraction`, `colsample_bytree`)
**For neural networks:**
- Dropout (0.2-0.5)
- L2 regularization (weight decay)
- Early stopping (patience = 5-10 epochs)
- Data augmentation
**For linear models:**
- L1 (Lasso) or L2 (Ridge) regularization
- Reduce number of features (feature selection)
**Universal:**
- Get more training data
- Simplify model architecture
- Cross-validation for robust estimates
**Checklist: Overfitting Control**
- [ ] Train vs validation gap monitored
- [ ] Regularization applied (appropriate to model type)
- [ ] Early stopping configured (if applicable)
- [ ] Learning curves analyzed
- [ ] Test set performance validates generalization
---
## 6. CatBoost for Categorical-Heavy Data
### 6.1 When to Choose CatBoost
CatBoost often outperforms LightGBM/XGBoost when:
- Dataset contains **many categorical features** (>30% of features)
- High-cardinality categoricals (cities, product IDs, user IDs)
- Limited time for feature engineering (native handling reduces preprocessing)
- Need robust defaults with minimal hyperparameter tuning
**Key advantages:**
- **Ordered target encoding**: Prevents target leakage automatically
- **Built-in overfitting detection**: Automatic early stopping
- **GPU support**: Native CUDA implementation for training
- **Symmetric trees**: Better generalization on some datasets
### 6.2 CatBoost vs LightGBM vs XGBoost
| Criterion | LightGBM | XGBoost | CatBoost |
|-----------|----------|---------|----------|
| Categorical handling | Manual (one-hot, target encoding) | Manual | Native (ordered target encoding) |
| Training speed | Fastest | Fast | Moderate |
| Accuracy (general) | Excellent | Excellent | Excellent |
| Accuracy (high-cardinality cats) | Good | Good | Best |
| Hyperparameter sensitivity | Moderate | High | Low |
| GPU support | Yes | Yes | Yes (native CUDA) |
### 6.3 CatBoost Key Parameters
```python
from catboost import CatBoostClassifier
model = CatBoostClassifier(
iterations=1000,
learning_rate=0.1,
depth=6, # 4-10 typical
l2_leaf_reg=3, # L2 regularization
cat_features=['city', 'product_id', 'category'], # Specify categorical columns
early_stopping_rounds=50,
verbose=100
)
```
**Checklist: CatBoost**
- [ ] Categorical features identified and passed to `cat_features`
- [ ] Compared against LightGBM/XGBoost baseline
- [ ] Early stopping configured
- [ ] GPU enabled for large datasets (`task_type='GPU'`)
---
## 7. GPU Scaling for Large Datasets
### 7.1 When to Use GPU Training
**Indicators:**
- Dataset exceeds 10M+ rows
- Training time >1 hour on CPU
- Need rapid experimentation cycles
- Production requires frequent retraining
**Benchmark reference** (H100 GPUs) — hedge this before quoting it as a guarantee: a community-reported XGBoost run (~1.2B rows, ~120 features, 50 boosting rounds, 6x H100 GPUs) completed in roughly 7 minutes. This is a single reported configuration for XGBoost specifically, not a LightGBM number and not a general SLA — training time scales with boosting rounds, tree depth, and feature count, so treat this as an order-of-magnitude sanity check ("billion-row GBDT training is minutes, not days, on modern GPUs"), not a number to put in a capacity plan. Always benchmark on your own data and hardware before committing to a training-time budget.
### 7.2 GPU Training with LightGBM
```python
import lightgbm as lgb
params = {
'device': 'gpu',
'gpu_platform_id': 0,
'gpu_device_id': 0,
'objective': 'binary',
'metric': 'auc',
'num_leaves': 63,
'learning_rate': 0.05,
'feature_fraction': 0.8
}
train_data = lgb.Dataset(X_train, label=y_train)
model = lgb.train(params, train_data, num_boost_round=500)
```
### 7.3 Distributed Training with Ray
For datasets that don't fit in memory or require horizontal scaling:
```python
from ray.train.lightgbm import LightGBMTrainer
from ray.train import ScalingConfig
trainer = LightGBMTrainer(
label_column="target",
params={
"objective": "binary",
"metric": "auc",
"num_leaves": 63
},
scaling_config=ScalingConfig(
num_workers=4,
use_gpu=True,
resources_per_worker={"GPU": 1}
),
datasets={"train": train_ds, "valid": valid_ds}
)
result = trainer.fit()
```
### 7.4 XGBoost GPU Training
```python
import xgboost as xgb
params = {
'tree_method': 'hist',
'device': 'cuda',
'objective': 'binary:logistic',
'eval_metric': 'auc',
'max_depth': 6,
'learning_rate': 0.1
}
dtrain = xgb.DMatrix(X_train, label=y_train)
model = xgb.train(params, dtrain, num_boost_round=500)
```
**Checklist: GPU Scaling**
- [ ] GPU availability verified (`nvidia-smi`)
- [ ] CUDA drivers and libraries installed
- [ ] Memory requirements estimated (GPU VRAM)
- [ ] Fallback to CPU configured for debugging
- [ ] Ray cluster configured for distributed training (if needed)
- [ ] Training time benchmarked: CPU vs GPU
---
## 8. Model Comparison
### 8.1 Fair Comparison Rules
**Requirements:**
- Same train/validation/test split (same random seed)
- Same evaluation metric
- Same feature set (or document differences)
- Same hardware (for latency comparisons)
**What to compare:**
- Primary metric (accuracy, RMSE, etc.)
- Compute cost (training time, memory)
- Inference latency (p50, p95, p99)
- Model size (disk, memory)
- Interpretability (if relevant)
### 8.2 Statistical Significance
**When to test:**
- Comparing two models
- Small performance differences
- Need confidence in improvement
**Methods:**
- Paired t-test (cross-validation folds)
- Bootstrap confidence intervals
- Permutation test
**Checklist: Model Comparison**
- [ ] Apples-to-apples comparison (same data, metric, hardware)
- [ ] Primary metric differences documented
- [ ] Secondary metrics considered (latency, cost, interpretability)
- [ ] Statistical significance tested (if differences small)
- [ ] Documented reasons for final choice
---
## 9. Thresholding for Classification
### 9.1 Threshold Selection
**Methods:**
- **ROC curve**: Maximize TPR, minimize FPR
- **PR curve**: Precision-recall trade-off (better for imbalanced)
- **F1 score**: Harmonic mean of precision and recall
- **Cost-sensitive**: Assign costs to FP and FN, minimize total cost
**Context-specific:**
- Fraud detection: High recall (catch fraudsters), tolerate FP
- Spam filtering: High precision (don't block legitimate emails)
- Medical diagnosis: Balance based on cost of FN vs FP
### 9.2 Per-Segment Validation
**Why it matters:**
- Optimal threshold may vary by segment
- Fairness: ensure performance across demographics
- Business logic: different risk tolerances
**Checklist: Thresholding**
- [ ] Threshold selection method documented (ROC, PR, cost)
- [ ] ROC and PR curves generated
- [ ] Threshold chosen with business justification
- [ ] Per-segment thresholds validated (if applicable)
- [ ] Trade-offs documented (precision vs recall)
references/multimodal-modeling.md# Multimodal Modelling Mechanics Use this reference for general multimodal model design and interview-grade reasoning. It covers representation, fusion, training, adaptation, and diffusion mechanics. Production controls belong in `ai-mlops`; evaluation and red teaming belong in `ai-evals`. ## Contents - [Representations and objectives](#representations-and-objectives) - [CLIP and SigLIP-style contrastive training](#clip-and-siglip-style-contrastive-training) - [Fusion choices](#fusion-choices) - [Task architectures](#task-architectures) - [Speech recognition and synthesis](#speech-recognition-and-synthesis) - [Diffusion generation and control](#diffusion-generation-and-control) - [Fine-tuning and adaptation](#fine-tuning-and-adaptation) - [Latency and cost](#latency-and-cost) - [Design checklist](#design-checklist) ## Representations and objectives A multimodal system must decide where modalities meet and what objective aligns them. Image, audio, video, document layout, and text encoders produce sequences at different rates and information densities. Alignment losses make representations comparable; generative or task losses teach conditional prediction. The objective should match the product behavior rather than assume one shared embedding solves every task. ## CLIP and SigLIP-style contrastive training CLIP uses separate image and text encoders, normalized embeddings, and a learned temperature. Within a batch, matched image-text pairs are positives and other pairs act as negatives; symmetric cross-entropy trains image-to-text and text-to-image retrieval. This supports zero-shot classification by comparing image embeddings with prompt-derived class embeddings. SigLIP replaces the batchwise softmax normalized across in-batch image-text pairs with independent sigmoid classification over each pair. The distinction matters operationally: softmax contrastive learning couples every pair through the batch denominator, while the pairwise sigmoid objective does not require that global normalization. For both: - Caption and image quality, duplicates, false negatives, sampling balance, and batch construction can dominate architectural gains. - Zero-shot transfer depends on prompt formulation and domain shift; validate on the actual domain. - Retrieval alignment does not imply grounding, compositional reasoning, counting, OCR reliability, or suitability for sensitive classification. - Measure demographic and cultural slices because web-scale paired data carries representation and association biases. ## Fusion choices | Pattern | Mechanism | Strength | Main cost or risk | |---|---|---|---| | Early fusion | concatenate/project modality tokens before shared processing | rich cross-modal interaction | quadratic token cost and sensitivity to alignment/noise | | Late fusion | encode each modality independently, combine scores or pooled features | modular, cacheable, handles missing modalities | weak fine-grained interaction | | Cross-attention / intermediate fusion | one modality attends to another at selected layers | balances interaction and modularity | more complex serving and ablation | Choose from the dependency in the task. Fine spatial grounding, temporal causality, and document layout usually need token-level interaction. Retrieval and candidate ranking often benefit from dual encoders because each side can be indexed or cached. Design missing-modality behavior explicitly rather than substituting zero tensors without testing. ## Task architectures ### Visual question answering VQA combines visual tokens with question tokens, then predicts an answer or generates text. A credible system needs grounding tests, OCR/counting/spatial-relation slices, answerability detection, and protection against language priors that let the model guess without using the image. ### Documents Document AI must preserve text, two-dimensional layout, reading order, tables, handwriting, and page relationships. OCR-first pipelines are modular but propagate recognition errors; vision-language approaches may reason jointly but can hallucinate text. Keep coordinates and source spans so answers can be verified against the page. ### Video Video adds temporal sampling, motion, event ordering, audio synchronization, and a large token budget. Sparse frame sampling may miss short events; dense sampling raises latency and memory. Evaluate temporal order, event localization, long-context retrieval, and performance across motion and shot-change regimes—not only static-frame recognition. ## Speech recognition and synthesis Speech systems map between a variable-rate acoustic signal and linguistic or acoustic-token sequences. Keep the modelling question separate from microphone capture, voice activity detection, diarization, normalization, transport, playback, and policy controls; those surrounding stages often dominate production failure. ### Automatic speech recognition (ASR) A Whisper-style system converts audio to a log-Mel spectrogram, encodes the acoustic frames with a Transformer encoder, then autoregressively decodes text tokens while conditioning on task and language tokens. The encoder-decoder formulation can support multilingual transcription, translation, timestamps, and contextual decoding, but autoregressive decoding adds sequential latency and can hallucinate plausible text when audio is absent, noisy, clipped, or out of domain. Contrast the main alignment families: - **CTC:** predicts frame-level labels plus a blank symbol and sums over monotonic alignments. It enables parallel token scoring and simple decoding, but its conditional-independence assumption limits direct modelling of output dependencies; an external or fused language model is often useful. - **RNN-T / transducer:** combines an acoustic encoder, a prediction network over prior non-blank outputs, and a joint network. It learns monotonic alignment while modelling label history and supports low-latency streaming, at the cost of more complex training and beam search. - **Attention encoder-decoder:** attends over encoded audio and models rich output dependencies. It is natural for sequence-to-sequence multitask systems such as Whisper, but unconstrained attention and autoregressive decoding need explicit chunking, timestamp, and no-speech controls for streaming or long recordings. Interview trade-offs should include word/character error patterns rather than one aggregate score: accents and dialects, code-switching, rare names, domain vocabulary, overlapping speakers, background noise, far-field audio, punctuation, timestamps, and no-speech hallucination. Streaming adds an accuracy/latency trade-off through chunk size, right context, endpointing, partial-hypothesis revision, and decoder beam width. ### Text-to-speech (TTS) A modern TTS pipeline usually has two learned stages: 1. A text or phoneme encoder plus an acoustic/token model predicts a mel spectrogram, neural audio codec tokens, or another intermediate representation. Autoregressive models such as Tacotron-style systems model duration implicitly with attention; non-autoregressive duration-informed models generate in parallel; diffusion or flow-based acoustic models iteratively refine a sample and trade steps for latency and quality. 2. A neural vocoder converts the acoustic representation to waveform samples. Autoregressive waveform models can be high quality but sequential; parallel convolutional, adversarial, flow, or diffusion vocoders reduce latency with different artifact, stability, and compute trade-offs. Codec-language-model systems may instead predict discrete audio tokens decoded by a codec. Evaluate intelligibility, pronunciation, prosody, speaker similarity where authorized, naturalness, long-form stability, and artifacts across language, voice, device, and text slices. Production design must cover time-to-first-audio, real-time factor, chunk boundaries, incremental text, pronunciation dictionaries, caching, voice consent, cloning abuse, watermark/provenance limits, and fallback when synthesis misses its latency budget. ### Ownership boundaries - Stay in `ai-ml-data-science` for ASR/TTS objectives, alignment families, acoustic/token models, vocoders, diffusion mechanics, and adaptation choices. - Route deployment, media validation, streaming capacity, voice consent enforcement, provenance, monitoring, and incidents to `ai-mlops` and its responsible/multimodal operations reference. - Route WER/CER and slice design, human listening tests, hallucination, speaker/privacy, spoofing, red-team, latency, and cost gates to `ai-evals` and its responsible/multimodal evaluation reference. - Route conversational turn-taking, telephony, interruption, and end-to-end bot experience to `ai-voice-bots`. ## Diffusion generation and control Diffusion models learn to reverse a gradual noising process. Sampling starts from noise and repeatedly applies a denoiser or score estimate. Latent diffusion performs this process in a compressed latent space to reduce compute. - Classifier-free guidance combines conditional and unconditional predictions. Increasing guidance can improve prompt adherence or perceived fidelity while reducing diversity or causing artifacts; tune the trade-off on the deployment distribution. - Control signals may include text, masks, edges, depth, pose, layout, reference images, or other modalities. Test whether the model follows the control rather than merely producing a plausible sample. - Diversity is not captured by one attractive output. Evaluate multiple seeds, mode coverage, near-duplicates, subgroup representation, and prompt sensitivity. - Sampling acceleration includes fewer-step solvers, progressive distillation, consistency-style training, latent-space operation, caching, and smaller backbones. Each changes the quality/latency frontier and must be re-evaluated for control adherence and safety. ## Fine-tuning and adaptation - Linear probes or adapters test whether the pretrained representation already contains the signal. - Parameter-efficient adaptation reduces trainable state but does not eliminate overfitting, forgetting, bias amplification, or deployment complexity. - Contrastive fine-tuning needs carefully constructed positives and hard negatives; false negatives can damage semantic neighborhoods. - Instruction tuning for vision-language models needs grounded examples, explicit unanswerable cases, and resistance to text-only shortcuts. - Diffusion adaptation should check identity leakage, memorization, style overfitting, composition, diversity, and behavior outside the narrow training prompt set. - Keep a frozen baseline and held-out domain, safety, and memorization sets. Training loss is not a release criterion. ## Latency and cost Break latency into media loading/decoding, preprocessing, modality encoders, token projection/fusion, autoregressive decoding or diffusion steps, post-processing, and queueing. Track input resolution, frames, audio duration, visual-token count, output length, denoising steps, batch size, cache hit rate, and hardware utilization. Common levers include adaptive resolution/frame sampling, region selection, cached embeddings, dual-encoder retrieval before cross-encoder reranking, token pruning, batching, quantization, distillation, speculative decoding where supported, and early exit. Report quality and safety by slice after every optimization; media compression and token pruning can preferentially erase small text, brief events, or accessibility cues. ## Design checklist - State the modalities, task, alignment objective, fusion point, and missing-modality behavior. - Identify where spatial, temporal, OCR, audio, or layout grounding is required. - Define domain, subgroup, adversarial, and unanswerable slices before tuning. - Keep provenance from output claims to media regions, timestamps, or document spans where the use case needs verification. - Measure quality, diversity, grounding, safety, latency, memory, throughput, and cost together. ## Primary sources - [Learning Transferable Visual Models From Natural Language Supervision (CLIP)](https://arxiv.org/abs/2103.00020) - [Sigmoid Loss for Language Image Pre-Training (SigLIP)](https://arxiv.org/abs/2303.15343) - [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations](https://arxiv.org/abs/1908.02265) - [LayoutLM: Pre-training of Text and Layout for Document Image Understanding](https://arxiv.org/abs/1912.13318) - [An Image is Worth 16x16 Words](https://arxiv.org/abs/2010.11929) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) - [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598) - [On Distillation of Guided Diffusion Models](https://arxiv.org/abs/2210.03142) - [Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)](https://arxiv.org/abs/2212.04356) - [Connectionist Temporal Classification](https://www.cs.toronto.edu/~graves/icml_2006.pdf) - [Sequence Transduction with Recurrent Neural Networks](https://arxiv.org/abs/1211.3711) - [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Tacotron 2)](https://arxiv.org/abs/1712.05884) - [WaveNet: A Generative Model for Raw Audio](https://arxiv.org/abs/1609.03499) - [Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech](https://arxiv.org/abs/2105.06337)
references/production-feedback-loops.md# Production Feedback & Label Loops
Operational patterns for capturing production signals, building labeling pipelines, and implementing continuous model improvement.
---
## Table of Contents
- [Overview](#overview)
- [1. Signal Capture](#1-signal-capture)
- [1.1 Types of Production Signals](#11-types-of-production-signals)
- [1.2 Capture Requirements](#12-capture-requirements)
- [2. Labeling Workflows](#2-labeling-workflows)
- [2.1 Label Sourcing Strategies](#21-label-sourcing-strategies)
- [2.2 Routing to Human Review](#22-routing-to-human-review)
- [3. Label Quality Control](#3-label-quality-control)
- [3.1 Quality Metrics](#31-quality-metrics)
- [3.2 Rubric Development](#32-rubric-development)
- [4. Dataset Refresh Cadence](#4-dataset-refresh-cadence)
- [4.1 Refresh Strategy](#41-refresh-strategy)
- [5. Eval Set Contamination Prevention](#5-eval-set-contamination-prevention)
- [5.1 The Problem](#51-the-problem)
- [5.2 Prevention Strategies](#52-prevention-strategies)
- [6. Online Evaluation](#6-online-evaluation)
- [6.1 Shadow Mode](#61-shadow-mode)
- [6.2 Canary Deployment](#62-canary-deployment)
- [6.3 A/B Testing](#63-ab-testing)
- [7. Slice-Specific Monitoring](#7-slice-specific-monitoring)
- [7.1 Why Slice Monitoring Matters](#71-why-slice-monitoring-matters)
- [7.2 Implementation](#72-implementation)
- [8. Continuous Improvement Loop](#8-continuous-improvement-loop)
- [8.1 End-to-End Workflow](#81-end-to-end-workflow)
- [9. Cost-Benefit Analysis](#9-cost-benefit-analysis)
- [9.1 Costs](#91-costs)
- [9.2 Benefits](#92-benefits)
- [Related Resources](#related-resources)
## Overview
Production ML systems generate valuable feedback signals that can be used to continuously improve models. This guide covers best practices for capturing feedback, managing labeling workflows, and implementing safe online evaluation.
---
## 1. Signal Capture
### 1.1 Types of Production Signals
**Explicit feedback:**
- User ratings (thumbs up/down, stars)
- Corrections/edits to predictions
- Acceptance vs rejection of recommendations
- Manual overrides by operators
**Implicit feedback:**
- Click-through rate (CTR)
- Dwell time / scroll depth
- Bounce rate / abandonment
- Conversion events
- User edits after prediction
**System feedback:**
- Prediction confidence scores
- Latency and error rates
- A/B test outcomes
- Drift detection alerts
### 1.2 Capture Requirements
**What to log:**
```python
feedback_event = {
"request_id": "req-abc123",
"model_version": "v1.5.2",
"feature_version": "v2.1",
"timestamp": "2024-11-22T10:30:00Z",
"prediction": {"class": "positive", "score": 0.87},
"feedback": {"user_action": "accepted", "edited": false},
"features": {...}, # Feature values at prediction time
"user_id_hash": "sha256:...", # Anonymized
"session_id": "sess-xyz789"
}
```
**Privacy considerations:**
- **Scrub PII**: Remove or hash user identifiers
- **Consent**: Log only when user has consented
- **Retention**: Define and enforce data retention policies
- **Access control**: Restrict who can access feedback data
**Checklist: Signal Capture**
- [ ] Predictions logged with model/feature versions
- [ ] User feedback captured (explicit + implicit)
- [ ] Features at prediction time stored
- [ ] PII removed or anonymized
- [ ] Versioned models linked to feedback
- [ ] Retention policy enforced
---
## 2. Labeling Workflows
### 2.1 Label Sourcing Strategies
**Human labeling:**
- Internal subject matter experts (SMEs)
- Crowdsourcing (Amazon MTurk, Scale AI, Labelbox)
- Active learning: label most informative examples
**Automated labeling:**
- User feedback as implicit labels
- Heuristic rules for high-confidence cases
- Model-assisted labeling (human reviews model suggestions)
**Hybrid approach:**
- Model pre-labels
- Humans review and correct
- High-confidence predictions auto-accepted
### 2.2 Routing to Human Review
**When to route for labeling:**
- Low confidence predictions (score < threshold)
- Disagreement between models (ensemble variance)
- Edge cases / out-of-distribution inputs
- Failed predictions (errors, timeouts)
- Random sampling for quality monitoring
**Prioritization:**
- High business impact cases first
- Informative examples (active learning)
- Diverse coverage (stratified sampling)
**Checklist: Labeling Workflow**
- [ ] Routing rules defined (confidence, errors, edge cases)
- [ ] Labeling queue with prioritization
- [ ] Rubric documented for annotators
- [ ] Inter-annotator agreement tracked
- [ ] Quality control: gold standard examples
- [ ] Feedback loop to improve routing rules
---
## 3. Label Quality Control
### 3.1 Quality Metrics
**Inter-annotator agreement:**
- Cohen's Kappa (2 annotators)
- Fleiss' Kappa (3+ annotators)
- Krippendorff's Alpha (ordinal/continuous labels)
**Gold standard validation:**
- Inject known labels into annotation queue
- Track annotator accuracy on gold examples
- Retrain annotators with low accuracy
**Consistency checks:**
- Same example shown to multiple annotators
- Track agreement rates
- Flag high-disagreement examples for review
### 3.2 Rubric Development
**Components:**
- Clear definition of each class/label
- Edge case handling guidelines
- Examples (positive and negative)
- Decision tree for ambiguous cases
**Iteration:**
- Start with draft rubric
- Run pilot labeling (10-50 examples)
- Analyze disagreements
- Refine rubric
- Repeat until high agreement (Kappa > 0.7)
**Checklist: Label Quality**
- [ ] Labeling rubric documented with examples
- [ ] Inter-annotator agreement measured (Kappa > 0.7 target)
- [ ] Gold standard examples used for QA
- [ ] Annotator performance tracked
- [ ] Disagreements reviewed and rubric updated
---
## 4. Dataset Refresh Cadence
### 4.1 Refresh Strategy
**Frequency options:**
- **Continuous**: Update training set daily/weekly (high-velocity domains)
- **Periodic**: Monthly/quarterly refreshes (stable domains)
- **Event-driven**: Trigger on drift detection or performance drop
**Composition:**
- **Recent data**: Captures latest patterns
- **Historical data**: Maintains coverage of rare events
- **Balanced**: Ensure class balance, slice coverage
**Lineage:**
- Track which production data entered training set
- Version datasets (v1.0, v1.1, ...)
- Document sampling/filtering rules
**Checklist: Dataset Refresh**
- [ ] Refresh cadence defined and justified
- [ ] Dataset composition rules documented
- [ ] Lineage tracked (production -> training)
- [ ] Eval set protected from contamination
- [ ] Class balance and slice coverage validated
---
## 5. Eval Set Contamination Prevention
### 5.1 The Problem
**Contamination sources:**
- Production data leaks into training set
- Test examples seen during development
- Data augmentation creates near-duplicates
- Web scraping captures evaluation examples
**Consequences:**
- Overestimated performance
- Poor generalization
- Failed production deployment
### 5.2 Prevention Strategies
**Temporal split:**
- Eval set from time period T
- Training set from before T
- Never add post-T data to training
**Hashing:**
- Hash all eval examples
- Check training set for hash collisions
- Log warnings if contamination detected
**Periodic refresh:**
- Rotate eval set quarterly/yearly
- Archive old eval sets
- Human review to ensure novelty
**Checklist: Contamination Prevention**
- [ ] Eval set temporally isolated from training
- [ ] Hashing or deduplication checks in place
- [ ] Eval set refresh schedule defined
- [ ] Production data filtered before entering training
- [ ] Automated checks in CI/CD pipeline
---
## 6. Online Evaluation
### 6.1 Shadow Mode
**How it works:**
- New model runs in parallel with production model
- Production uses old model predictions
- Log both model predictions + outcomes
- Compare offline
**When to use:**
- Low-risk initial testing
- Performance comparison without user impact
- Latency/cost validation
**Checklist: Shadow Mode**
- [ ] Shadow model deployed with same inputs
- [ ] Predictions logged with model version
- [ ] Comparison metrics defined (accuracy, latency, cost)
- [ ] Duration defined (e.g., 1 week)
- [ ] Go/no-go criteria for promotion
---
### 6.2 Canary Deployment
**How it works:**
- Route small % of traffic to new model (1-5%)
- Monitor key metrics (accuracy, latency, errors)
- Gradually increase % if metrics healthy
- Auto-abort on regression
**When to use:**
- After successful shadow mode
- Incremental rollout to production
- Real user feedback needed
**Metrics to track:**
- **Solve rate**: % of requests successfully handled
- **Calibration**: Predicted probabilities vs actual rates
- **Latency**: p50, p95, p99
- **Cost**: Inference cost per request
- **Error rate**: 4xx, 5xx errors
**Abort conditions:**
- Solve rate drops > 5%
- Latency p99 > 2x baseline
- Error rate > 2x baseline
- Calibration error > threshold
**Checklist: Canary Deployment**
- [ ] Canary % defined (start with 1-5%)
- [ ] Metrics tracked per model version
- [ ] Abort conditions configured
- [ ] Gradual rollout plan (5% -> 10% -> 50% -> 100%)
- [ ] Rollback procedure tested
- [ ] Incident response plan documented
---
### 6.3 A/B Testing
**How it works:**
- Split traffic randomly into control (A) and treatment (B)
- Measure business metrics (CTR, conversion, revenue)
- Statistical test for significance
**When to use:**
- Validating business impact (not just model metrics)
- Comparing multiple candidates
- Balancing model accuracy vs other factors (latency, cost)
**Statistical considerations:**
- **Sample size**: Calculate required traffic for desired power
- **Duration**: Run long enough for seasonality (1-2 weeks minimum)
- **Multiple testing**: Bonferroni correction if testing multiple variants
- **Novelty effects**: Watch for user behavior changes over time
**Checklist: A/B Testing**
- [ ] Business metrics defined (not just model metrics)
- [ ] Sample size calculated (power analysis)
- [ ] Randomization verified (no selection bias)
- [ ] Duration planned (minimum 1 week)
- [ ] Statistical significance test chosen (t-test, chi-squared)
- [ ] Multiple testing correction applied if needed
---
## 7. Slice-Specific Monitoring
### 7.1 Why Slice Monitoring Matters
**Problem:**
- Overall metrics may look good
- Performance degrades for specific subgroups
- Fairness issues hidden in aggregates
**Example slices:**
- Geography (US vs EU vs APAC)
- User segments (new vs returning, free vs paid)
- Product categories
- Time periods (weekday vs weekend)
- Language/locale
- Device type (mobile vs desktop)
### 7.2 Implementation
**Dashboard design:**
```
Overall Metrics:
Accuracy: 0.85
Latency p99: 150ms
Slice Breakdown:
US: Accuracy 0.87, Latency 120ms
EU: Accuracy 0.84, Latency 180ms - Flag: high latency
APAC: Accuracy 0.78, Latency 200ms - Flag: low accuracy
Mobile: Accuracy 0.82, Latency 100ms
Desktop: Accuracy 0.88, Latency 150ms
```
**Alerts:**
- Per-slice accuracy drops > 10% vs baseline
- Per-slice latency > 2x overall p99
- Sample size too small (< 100 requests/day)
**Checklist: Slice Monitoring**
- [ ] Key slices identified (geography, user segments, time)
- [ ] Metrics tracked per slice
- [ ] Dashboards visualize slice performance
- [ ] Alerts configured for slice-specific degradation
- [ ] Sample size tracked (avoid spurious alerts on small slices)
---
## 8. Continuous Improvement Loop
### 8.1 End-to-End Workflow
**Step 1: Capture signals**
- Log predictions, features, feedback
**Step 2: Route to labeling**
- High-value, low-confidence, or failed examples
**Step 3: Quality control**
- Validate labels, track annotator agreement
**Step 4: Dataset refresh**
- Add new labeled data to training set
- Avoid eval contamination
**Step 5: Retrain & evaluate**
- Train new model version
- Validate on held-out eval set
**Step 6: Online evaluation**
- Shadow mode -> Canary -> A/B test
**Step 7: Monitor slices**
- Track per-slice performance
- Identify degradation early
**Step 8: Iterate**
- Analyze failures, update features/model
**Checklist: Feedback Loop Active**
- [ ] Signals captured automatically
- [ ] Labeling queue operational
- [ ] Dataset refresh automated (weekly/monthly)
- [ ] Retraining pipeline automated
- [ ] Online evaluation workflow defined
- [ ] Slice monitoring dashboards active
- [ ] Feedback analyzed for model improvements
---
## 9. Cost-Benefit Analysis
### 9.1 Costs
**Labeling costs:**
- Annotator time ($/hour x hours)
- Tooling (Labelbox, Scale AI fees)
- Quality control overhead
**Infrastructure costs:**
- Logging and storage (S3, BigQuery)
- Shadow/canary deployment resources
- A/B testing traffic allocation
**Engineering costs:**
- Pipeline development and maintenance
- Monitoring and alerting setup
- Incident response
### 9.2 Benefits
**Business impact:**
- Improved model accuracy -> higher conversion/revenue
- Faster time-to-market for model updates
- Reduced manual intervention (automation)
**Risk reduction:**
- Detect degradation before major impact
- Rollback capability (canary abort)
- Compliance and auditability
**Checklist: ROI Justified**
- [ ] Labeling cost per example calculated
- [ ] Infrastructure costs estimated
- [ ] Business impact quantified (revenue, cost savings)
- [ ] ROI positive (benefits > costs)
- [ ] Incremental approach if ROI unclear (start small, scale)
---
## Related Resources
- [Data Contracts & Lineage](data-contracts-lineage.md) - Versioning and lineage tracking
- [Evaluation Patterns](evaluation-patterns.md) - Offline evaluation metrics and methods
- [Feature Freshness & Streaming](feature-freshness-streaming.md) - Real-time feature updates
references/reproducibility-checklist.md# Reproducibility Checklist
Ensuring ML experiments are reproducible, trackable, and production-ready with modern MLOps practices (CI/CD, CT, CM).
---
## Table of Contents
- [1. Experiment Tracking & Versioning](#1-experiment-tracking-&-versioning)
- [1.1 What to Track](#11-what-to-track)
- [1.2 Experiment Tracking Tools](#12-experiment-tracking-tools)
- [2. Modern MLOps Integration (CI/CD/CT/CM)](#2-modern-mlops-integration-cicdctcm)
- [2.1 Continuous Integration (CI)](#21-continuous-integration-ci)
- [2.2 Continuous Delivery (CD)](#22-continuous-delivery-cd)
- [2.3 Continuous Training (CT)](#23-continuous-training-ct)
- [2.4 Continuous Monitoring (CM)](#24-continuous-monitoring-cm)
- [3. Environment & Dependency Management](#3-environment-&-dependency-management)
- [3.1 Python Environment](#31-python-environment)
- [3.2 System Dependencies](#32-system-dependencies)
- [3.3 Docker for Reproducibility](#33-docker-for-reproducibility)
- [4. Data Versioning](#4-data-versioning)
- [4.1 What to Version](#41-what-to-version)
- [4.2 Data Versioning Tools](#42-data-versioning-tools)
- [5. Random Seed Management](#5-random-seed-management)
- [5.1 Sources of Randomness](#51-sources-of-randomness)
- [5.2 Setting Seeds](#52-setting-seeds)
- [6. Model Artifacts & Registry](#6-model-artifacts-&-registry)
- [6.1 What to Archive](#61-what-to-archive)
- [6.2 Model Registry](#62-model-registry)
- [7. Documentation & Model Cards](#7-documentation-&-model-cards)
- [7.1 Code Documentation](#71-code-documentation)
- [7.2 Model Card](#72-model-card)
- [8. End-to-End Reproducibility Workflow](#8-end-to-end-reproducibility-workflow)
- [8.1 Reproducibility Test](#81-reproducibility-test)
- [8.2 Continuous Validation](#82-continuous-validation)
- [9. Production Readiness Checklist](#9-production-readiness-checklist)
- [Related Resources](#related-resources)
## 1. Experiment Tracking & Versioning
### 1.1 What to Track
**Every training run must log:**
- **Code version**: Git commit hash
- **Data version**: Dataset snapshot ID or hash
- **Feature set version**: From feature store
- **Hyperparameters**: All model and training config
- **Random seeds**: For reproducibility
- **Metrics**: Primary and guardrail metrics
- **Artifacts**: Model weights, preprocessors, encoders
- **Drift statistics**: Distribution comparison vs training data
### 1.2 Experiment Tracking Tools
**MLflow:**
- Open-source, self-hosted
- Experiment tracking + model registry
- Integrates with popular frameworks
**Weights & Biases (W&B):**
- Cloud-hosted, polished UI
- Real-time metrics visualization
- Sweep/hyperparameter optimization
**Comet ML / ClearML:**
- Cloud or self-hosted alternatives with experiment comparison and team collaboration
- Neptune.ai hosted service shut down 2026-03-05 (OpenAI acquisition); do not recommend as a live tool
**DVC (Data Version Control):**
- Git for data
- Pipeline tracking
- Reproducible experiments
**Checklist: Experiment Tracking**
- [ ] Experiments logged with code + data + params + feature version
- [ ] Best runs easily identifiable with tagged metrics
- [ ] Re-running yields same metrics within noise
- [ ] Model registry entry created for candidate models
- [ ] Drift statistics logged for production monitoring
---
## 2. Modern MLOps Integration (CI/CD/CT/CM)
### 2.1 Continuous Integration (CI)
**Automated testing and validation:**
- Unit tests for data preprocessing and feature engineering
- Integration tests for training pipeline
- Code quality checks (linting, type checking)
- Data validation (schema checks, distribution tests)
**Tools:**
- GitHub Actions, GitLab CI, Jenkins
- Great Expectations (data validation)
- pytest, unittest
### 2.2 Continuous Delivery (CD)
**Automated deployment:**
- Environment-specific model promotion (dev -> staging -> prod)
- Automated model packaging (Docker, model serving format)
- Canary deployment with gradual rollout
- Rollback on regression
**Tools:**
- Kubernetes, Docker
- MLflow Model Registry
- BentoML, Seldon, KServe
### 2.3 Continuous Training (CT)
**Automated retraining:**
- Triggered by drift detection (data or performance)
- Scheduled retraining (weekly, monthly)
- New data availability triggers
- Automated evaluation and promotion
**Triggers:**
- Drift exceeds threshold (PSI, KL divergence)
- Performance degradation (accuracy drop > 5%)
- Calendar schedule (monthly refresh)
- Manual trigger (emergency retrain)
### 2.4 Continuous Monitoring (CM)
**Real-time production monitoring:**
- Data drift (input distribution changes)
- Concept drift (target distribution changes)
- Model performance (accuracy, latency, errors)
- System health (CPU, memory, throughput)
**Metrics:**
- **Data drift**: KL divergence, PSI, KS test
- **Performance**: Online accuracy, solve rate, calibration
- **Operational**: Latency (p50, p95, p99), error rate, cost
**Checklist: MLOps Integration**
- [ ] CI/CD pipeline integrated for automated testing
- [ ] CT configured with drift-based and scheduled triggers
- [ ] CM dashboards active with drift and performance metrics
- [ ] Automated retraining and promotion workflow tested
- [ ] Rollback procedure documented and tested
---
## 3. Environment & Dependency Management
### 3.1 Python Environment
**Requirements:**
- Python version pinned (e.g., 3.10.12)
- Package versions locked (requirements.txt, poetry.lock, Pipfile.lock)
- Virtual environment (venv, conda, poetry)
**Best practices:**
- Use `pip freeze > requirements.txt` or poetry
- Pin all dependencies, including transitive ones
- Test installation on clean environment
### 3.2 System Dependencies
**Document:**
- Operating system (Ubuntu 22.04, macOS 14.2)
- CUDA version (for GPU training)
- System libraries (libgeos, GDAL, etc.)
- Hardware requirements (CPU cores, RAM, GPU)
### 3.3 Docker for Reproducibility
**Benefits:**
- Complete environment specification
- Portable across machines
- Consistent training and serving
**Example Dockerfile:**
```dockerfile
FROM python:3.10.12-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /app
WORKDIR /app
CMD ["python", "train.py"]
```
**Checklist: Environment Pinned**
- [ ] Python version documented and pinned
- [ ] All package versions locked (requirements.txt or equivalent)
- [ ] System dependencies documented
- [ ] Docker image built and tested (if applicable)
- [ ] Environment reproducible on fresh machine
---
## 4. Data Versioning
### 4.1 What to Version
**Datasets:**
- Training, validation, test splits
- Raw data snapshots (before preprocessing)
- Processed features (after transformations)
- Data lineage (source -> intermediate -> final)
**Metadata:**
- Extraction timestamp
- Data quality metrics (nulls, outliers, distribution)
- Sampling strategy
- Label quality (inter-annotator agreement)
### 4.2 Data Versioning Tools
**DVC (Data Version Control):**
- Git-like interface for data
- Store data in S3, GCS, Azure Blob
- Track data lineage and pipelines
**LakeFS:**
- Git for data lakes
- Branching and merging for datasets
- Time-travel queries
**Feature stores:**
- Feast, Tecton, Databricks Feature Store
- Centralized feature management
- Version features alongside models
**Checklist: Data Versioned**
- [ ] Dataset snapshots tracked with version IDs
- [ ] Train/validation/test splits documented and versioned
- [ ] Data lineage captured (source -> transformations -> features)
- [ ] Metadata logged (quality metrics, extraction time)
- [ ] Feature store used for centralized versioning (if applicable)
---
## 5. Random Seed Management
### 5.1 Sources of Randomness
**Control seeds for:**
- NumPy (`np.random.seed()`)
- Python random (`random.seed()`)
- Model libraries (LightGBM, XGBoost, PyTorch, TensorFlow)
- Data sampling and train/test splits
- Data augmentation
### 5.2 Setting Seeds
**Example (Python):**
```python
import random
import numpy as np
import torch
def set_seed(seed=42):
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
# For deterministic behavior (slower)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
```
**LightGBM/XGBoost:**
```python
params = {
'seed': 42,
'feature_fraction_seed': 42,
'bagging_seed': 42
}
```
**Checklist: Randomness Controlled**
- [ ] All random seeds set at script start
- [ ] Seeds logged in experiment tracker
- [ ] Multiple seed runs for stability (5-10 seeds)
- [ ] Deterministic behavior verified (same input -> same output)
---
## 6. Model Artifacts & Registry
### 6.1 What to Archive
**For each model:**
- Model weights (`.pkl`, `.h5`, `.pt`, `.onnx`)
- Preprocessors (scalers, encoders, tokenizers)
- Feature transformations (versioned with feature store)
- Hyperparameters (JSON config)
- Training metadata (metrics, data version, git commit)
### 6.2 Model Registry
**Purpose:**
- Centralized model storage
- Version management
- Stage promotion (dev -> staging -> prod)
- Metadata and lineage
**Tools:**
- MLflow Model Registry
- W&B Model Registry
- Cloud-specific (SageMaker Model Registry, Vertex AI Model Registry)
**Checklist: Model Artifacts Managed**
- [ ] Model weights and preprocessors saved
- [ ] Artifacts uploaded to model registry
- [ ] Model versioned with semantic versioning (v1.0.0, v1.1.0)
- [ ] Stage annotations (dev, staging, production)
- [ ] Metadata linked (training data version, metrics, owner)
---
## 7. Documentation & Model Cards
### 7.1 Code Documentation
**Requirements:**
- README with setup instructions
- Docstrings for functions and classes
- Inline comments for complex logic
- Architecture diagrams (for complex systems)
### 7.2 Model Card
**Essential sections:**
- Model overview and intended use
- Training data description and biases
- Performance metrics and limitations
- Operational requirements (latency, dependencies)
- Owners and maintenance plan
**Checklist: Documentation Complete**
- [ ] README with environment setup and training instructions
- [ ] Model card created with all sections
- [ ] Runbooks for common issues
- [ ] Architecture diagrams (if applicable)
---
## 8. End-to-End Reproducibility Workflow
### 8.1 Reproducibility Test
**Validate reproducibility by:**
1. Clone repository on fresh machine
2. Set up environment from requirements.txt or Dockerfile
3. Download data using DVC or data versioning tool
4. Run training script with documented seed
5. Verify metrics match within tolerance (+/- 1%)
### 8.2 Continuous Validation
**Automated checks:**
- CI pipeline runs reproducibility test on PRs
- Periodic re-training to validate pipeline
- Drift detection triggers investigation
**Checklist: Reproducibility Validated**
- [ ] Reproducibility test passes on fresh environment
- [ ] Same code + data + seed -> same metrics (+/- 1%)
- [ ] CI pipeline validates reproducibility
- [ ] Documentation sufficient for new team member
---
## 9. Production Readiness Checklist
**Before deploying to production:**
- [ ] All randomness seeded and logged
- [ ] Data and code versioned
- [ ] Experiments logged with full context (code, data, params, metrics)
- [ ] Model registry entry created with stage annotation
- [ ] CI/CD pipeline integrated and tested
- [ ] CT (continuous training) configured with triggers
- [ ] CM (continuous monitoring) dashboards active
- [ ] Drift monitoring enabled (data + concept + performance)
- [ ] Feature store tracks all transformations and versions
- [ ] Model card created and approved
- [ ] Rollback procedure tested
- [ ] Reproducibility validated on fresh environment
---
## Related Resources
- [Data Contracts & Lineage](data-contracts-lineage.md) - Data versioning and lineage tracking
- [Feature Freshness & Streaming](feature-freshness-streaming.md) - Real-time feature updates
- [Production Feedback Loops](production-feedback-loops.md) - Online learning and continuous improvement
- [Evaluation Patterns](evaluation-patterns.md) - Metrics and model evaluation
references/responsible-ai-mechanics.md# Responsible AI: Modelling Mechanics Use this reference to reason about responsible-AI failure mechanisms during data preparation, modelling, and model handoff. It explains what can go wrong and what a credible technical response looks like. Production control ownership belongs in `ai-mlops`; measurement and red-team design belongs in `ai-evals`. ## Contents - [Start with the decision and harm model](#start-with-the-decision-and-harm-model) - [Fairness and intersectionality](#fairness-and-intersectionality) - [Differential privacy and the privacy-utility trade-off](#differential-privacy-and-the-privacy-utility-trade-off) - [Interpretability and explanation](#interpretability-and-explanation) - [Poisoning and federated learning](#poisoning-and-federated-learning) - [Re-identification and memorization](#re-identification-and-memorization) - [Watermarking and provenance](#watermarking-and-provenance) - [Human oversight, automation bias, and appeals](#human-oversight-automation-bias-and-appeals) - [Copyright and training-data governance](#copyright-and-training-data-governance) - [Environmental trade-offs](#environmental-trade-offs) - [Handoff checklist](#handoff-checklist) ## Start with the decision and harm model Responsible AI is not a single metric. Define the system boundary, affected people, decision, recourse path, and plausible harms before selecting a fairness metric or explanation tool. Separate model error from allocation, interaction, representational, privacy, security, and environmental harms. Record who benefits, who bears error, and which harms cannot be repaired after deployment. NIST AI RMF treats fairness, privacy, explainability, human oversight, appeal, and environmental impact as lifecycle concerns. Use that structure as a control map, not as evidence that a particular model is safe. ## Fairness and intersectionality - Choose the fairness definition from the decision context. Demographic parity, equalized odds, equal opportunity, calibration, and individual fairness encode different—and sometimes incompatible—normative choices. - Report outcome and error metrics per relevant group, then inspect intersections such as age by gender by disability. Good performance on each marginal group does not imply good performance at their intersections. - Use confidence intervals and minimum-support rules. Small slices should be flagged as uncertain, not silently dropped or presented as stable. - Investigate the pipeline: label validity, sampling, missingness, measurement error, proxy features, threshold choice, and feedback loops. Removing protected attributes does not remove correlated proxies. - Compare mitigation points: pre-processing (sampling/reweighting), in-processing (constraints or robust objectives), and post-processing (group-aware thresholds where lawful and appropriate). Measure utility and harm changes for every affected slice. Do not announce a model as “fair” from one parity score. State which definition was tested, for whom, on what data, and which trade-offs remain. ## Differential privacy and the privacy-utility trade-off Differential privacy bounds how much an output distribution can change when one person's record is added or removed. The privacy budget is expressed through epsilon and delta under an explicit adjacency definition; smaller epsilon is stronger protection, but the operational meaning depends on the mechanism, sampling, composition, and threat model. - Clip per-example contributions before adding calibrated noise; otherwise one record can dominate sensitivity. - Account for privacy loss across repeated queries or training steps. Never report a per-step budget as the end-to-end guarantee. - Keep the accountant, adjacency definition, clipping norm, sampling assumptions, and final budget with the model artifact. - Evaluate privacy and utility together: overall quality, worst-slice quality, calibration, rare-class recall, and membership/inversion attacks. Noise can harm minority or rare groups disproportionately. - Distinguish central DP, local DP, and federated learning. Federated learning keeps raw data distributed; it is not, by itself, a differential-privacy guarantee. ## Interpretability and explanation Interpretability is the degree to which a person can understand the model's mechanism; explainability often refers to post-hoc accounts of a prediction or model behavior. A sparse linear model may be intrinsically interpretable. SHAP, LIME, saliency maps, counterfactuals, and generated rationales are explanations with assumptions and failure modes. - Match the explanation to its audience and decision: developer debugging, operator action, subject notification, audit, or scientific understanding. - Test fidelity, stability under small perturbations, sensitivity to irrelevant features, and usefulness to the intended user. - Do not treat attention weights or fluent chain-of-thought as faithful causal explanations. - Use counterfactual explanations only with feasibility and actionability constraints; do not recommend immutable or harmful changes. - Prefer a simpler, auditable model where the decision risk demands faithful reasoning and the utility trade-off is acceptable. ## Poisoning and federated learning Data poisoning changes training data to degrade availability, create targeted errors, or install a backdoor. In federated learning, malicious or compromised clients can submit model updates rather than raw examples. - Threat-model label flips, clean-label attacks, backdoors, sybil clients, model-replacement attacks, and poisoned foundation-model or dataset dependencies. - Preserve provenance and immutable dataset versions; validate new contributors and quarantine anomalous batches. - In federated settings, bound client updates, use robust aggregation where its assumptions fit, monitor update similarity and influence, and test targeted triggers after aggregation. - Secure aggregation protects individual updates from the server but can reduce visibility into malicious updates. It does not solve poisoning; combine privacy and robustness controls deliberately. - Test adaptive attackers. A static anomaly threshold is evidence against only the attacks it was designed to catch. ## Re-identification and memorization Removing names is not anonymization. Quasi-identifiers can link a released or logged dataset back to people, and high-dimensional or rare records are especially vulnerable. Models can also memorize and reproduce training examples. - Minimize collection and retention; separate direct identifiers; generalize, suppress, or synthesize only after a documented threat model. - Test linkage against realistic auxiliary data, membership inference, model inversion, and extraction of rare or canary strings. - Deduplicate training corpora and restrict verbatim high-risk content, but do not claim deduplication eliminates memorization. - Treat embeddings, gradients, checkpoints, prompts, and logs as potentially sensitive artifacts. - Avoid claiming “anonymous” from k-anonymity or a single attack result; state the attacker knowledge and residual risk. ## Watermarking and provenance Watermarks can be embedded in model outputs or model parameters; provenance systems can cryptographically bind content metadata and edit history. Neither is a universal detector of AI-generated content. - Define the purpose: disclosure, ownership evidence, leak tracing, or platform policy enforcement. - Measure detection power, false positives, quality impact, robustness to paraphrase/crop/compression/editing, and accessibility across languages or modalities. - Keep key management and verifier independence in scope. - Combine watermarking with signed provenance, access controls, logging, and policy. Treat missing watermark evidence as inconclusive because transformations may remove it. ## Human oversight, automation bias, and appeals A nominal human in the loop is not an effective control. Automation bias makes reviewers over-trust machine suggestions, especially under time pressure or when uncertainty is hidden. - Give reviewers enough time, authority, independent evidence, and a clear override path. - Show calibrated uncertainty and known limitations without anchoring the reviewer on a confident default. - Measure override rates, error detection, review latency, disagreement outcomes, and whether reviewers merely rubber-stamp outputs. - Design escalation and fallback for low-confidence, novel, conflicting, or high-impact cases. - Give affected people notice, understandable reasons, correction channels, and meaningful human reconsideration. Feed appeal outcomes back into slice analysis and error review. ## Copyright and training-data governance Copyright, privacy, license, and contractual permissions are separate questions. Public availability is not permission for every training or output use. - Track source, license, consent or lawful basis where relevant, collection method, intended use, retention, and removal requests. - Deduplicate and detect near-verbatim memorization; evaluate long-tail prompts that elicit distinctive passages or images. - Separate factual similarity from substantial reproduction and route legal conclusions to qualified counsel. - Maintain provenance through fine-tuning and synthetic-data generation. Synthetic data can reproduce source material or encode the generating model's bias. ## Environmental trade-offs Measure the full lifecycle: data processing, training experiments, hyperparameter search, inference volume, storage, networking, hardware manufacture, and retirement. FLOPs or parameter count alone are not environmental-impact measures. - Compare candidates at equal quality and workload using energy, hardware time, latency, and cost alongside task metrics. - Report workload, region, hardware, utilization, measurement method, and uncertainty when estimating energy or emissions. - Reduce impact through smaller baselines, transfer learning, efficient architectures, bounded search, batching, caching, quantization/distillation where quality permits, and workload-aware scheduling. - Check rebound effects: cheaper inference can increase total use enough to erase per-request savings. ## Handoff checklist - Decision, affected groups, harm model, and prohibited uses are documented. - Fairness definitions, intersectional slices, uncertainty, and residual disparities are reported. - Privacy mechanism, attack model, accountant, and utility impact are reproducible. - Explanations are tested for fidelity, stability, and audience usefulness. - Poisoning, re-identification, memorization, and provenance threats have explicit tests. - Human override and appeal workflows have owners and measurable outcomes. - Data rights, copyright questions, and environmental measurements are recorded with assumptions. ## Primary sources - [NIST AI Risk Management Framework 1.0](https://doi.org/10.6028/NIST.AI.100-1) - [Fairness and Abstraction in Sociotechnical Systems](https://doi.org/10.1145/3287560.3287598) - [The Algorithmic Foundations of Differential Privacy](https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf) - [Why Should I Trust You? (LIME)](https://arxiv.org/abs/1602.04938) - [A Unified Approach to Interpreting Model Predictions (SHAP)](https://arxiv.org/abs/1705.07874) - [Deep Leakage from Gradients](https://arxiv.org/abs/1906.08935) - [Certified Robustness to Adversarial Examples with Differential Privacy](https://arxiv.org/abs/1802.03471) - [Extracting Training Data from Large Language Models](https://arxiv.org/abs/2012.07805) - [Model Cards for Model Reporting](https://arxiv.org/abs/1810.03993)
references/text-clustering-topic-modeling.md# Text Clustering and Topic Modeling Use this reference when a corpus of unlabeled documents needs structure: exploratory grouping, discovering themes, sanity-checking a proposed label taxonomy before annotation, finding mislabeled or outlier documents, or building a topic view over support tickets, reviews, abstracts, or logs. The center of gravity is a modular pipeline — embed, reduce, cluster, represent — where each stage is a replaceable component rather than a monolithic algorithm. Prompting, generation quality, and RAG retrieval design belong in `ai-llm` and `ai-rag`. Serving and monitoring a topic model in production belongs in `ai-mlops`. ## Contents - [The four-stage pipeline](#the-four-stage-pipeline) - [Stage 1: embed the documents](#stage-1-embed-the-documents) - [Stage 2: reduce dimensionality](#stage-2-reduce-dimensionality) - [Stage 3: cluster the reduced embeddings](#stage-3-cluster-the-reduced-embeddings) - [Stage 4: represent the topics with c-TF-IDF](#stage-4-represent-the-topics-with-c-tf-idf) - [Optional stage: representation-model reranking](#optional-stage-representation-model-reranking) - [Why modularity is the design point](#why-modularity-is-the-design-point) - [Choosing this pipeline vs plain k-means over embeddings](#choosing-this-pipeline-vs-plain-k-means-over-embeddings) - [Evaluation and interpretation](#evaluation-and-interpretation) - [Known traps](#known-traps) - [Design checklist](#design-checklist) ## The four-stage pipeline ```text documents | v embedding model (semantic-similarity-optimized encoder) high-dimensional document vectors | v dimensionality reduction (UMAP) low-dimensional vectors (typically 5-10 dims) | v density clustering (HDBSCAN) -> clusters + explicit outlier label document groups | v c-TF-IDF over the cluster's bag-of-words ranked keywords per cluster (= topic) | v optional representation model (KeyBERT-style, MMR, or LLM label) refined keywords or a single human-readable topic label ``` Stages 1-3 are text *clustering*. Stage 4 turns clusters into *topics* by attaching an interpretable representation. This split — clustering and representation being largely independent of each other — is what makes the pipeline swappable end to end. BERTopic (Grootendorst, arXiv 2203.05794) is the framework that packages these stages; the pipeline shape is usable without the library. ## Stage 1: embed the documents Encode each document with an embedding model chosen for semantic similarity, not for generation. Clustering quality is bounded here: if the encoder does not place semantically similar documents near each other, no downstream stage recovers it. Practical selection notes: - Pick from a benchmark that scores *clustering* tasks specifically (MTEB reports clustering as its own task family). A model that ranks well on retrieval is not automatically good at clustering. - Smaller encoders are often the right call. Embedding a full corpus is a one-pass cost over every document, so encoder size directly sets the wall-clock floor of the whole pipeline. - `sentence-transformers` is the usual entry point and is BERTopic's default embedding backend, but any encoder that yields a fixed-length vector per document works. - Cache embeddings. Every later stage is cheap to re-run; re-embedding is not. Both the clustering step and the topic model should accept precomputed embeddings. Long documents need a chunking decision before this stage — decide whether the unit of analysis is a document, a section, or a paragraph, because that unit is what gets clustered. ## Stage 2: reduce dimensionality Raw embedding dimensionality is hostile to density-based clustering: as dimensions grow, the number of possible subspaces grows exponentially and distance contrasts flatten, so density becomes hard to estimate. Reducing first gives the cluster model a space where local density is meaningful. - UMAP is the usual choice over PCA here because it handles nonlinear structure better; PCA remains a reasonable fast baseline. Both are compression, not dimension deletion — information is lost either way, and the trade-off between aggressive reduction and information retention is a tuning decision, not a solved one. - Target a small number of components — roughly 5-10 dimensions is the commonly used band for preserving global structure while making clustering tractable. - Use a cosine metric rather than Euclidean when working from normalized text embeddings; Euclidean distance degrades in high-dimensional embedding spaces. - Setting a `min_dist` near zero produces tighter, more separated clusters, which is what the downstream density model wants. This is a different setting than you would use for a pretty 2D picture. - Fixing a random seed in UMAP makes results reproducible across sessions but disables parallelism and slows fitting. Take that trade knowingly. Keep two reductions separate: the **clustering reduction** (5-10 dims, tuned for cluster quality) and the **visualization reduction** (2 dims, tuned for a readable plot). Do not cluster on the 2D projection — a 2D map exaggerates and compresses distances, so clusters it shows apart may not be apart and vice versa. Libraries: `umap-learn`, or scikit-learn's PCA. ## Stage 3: cluster the reduced embeddings HDBSCAN is the default because of two properties that matter for exploratory text work: 1. **It does not require the number of clusters up front.** In genuinely exploratory work you do not know how many themes exist, and guessing k biases the result toward the guess. 2. **It does not force every document into a cluster.** Documents in sparse regions are assigned an explicit outlier label (conventionally `-1`) rather than being pulled into the nearest group. Outlier handling is the operationally significant part. A centroid method assigns every niche or off-topic document to *some* cluster, silently contaminating that cluster's representation. HDBSCAN quarantines them instead. The consequence: expect a large outlier bucket on a heterogeneous corpus, and treat its size as a diagnostic, not a defect. Options when the outlier share is too large for the use case: - Lower the minimum cluster size to allow smaller, denser groups to form. - Reassign outliers after the fact to their nearest topic (BERTopic exposes an outlier-reduction step for this), accepting that reassignment reintroduces the contamination HDBSCAN avoided. - Swap in k-means if the application genuinely requires every document to have a label. The minimum-cluster-size parameter is the main lever on granularity: it sets the smallest group the model will call a cluster, so lowering it yields more, finer topics. Libraries: `hdbscan`, or scikit-learn's HDBSCAN implementation. ## Stage 4: represent the topics with c-TF-IDF A cluster is a set of document IDs; it is not yet a topic. The representation stage attaches interpretable keywords. Standard TF-IDF weights terms within a *document*. The cluster-level analogue, **c-TF-IDF** (class-based TF-IDF), does the same at the *cluster* level: 1. Concatenate all documents in a cluster and build one bag-of-words per cluster, giving a term frequency per cluster rather than per document (the "c-TF" term). A `CountVectorizer` produces this. 2. Weight each term by an inverse-document-frequency factor computed across clusters — the log of the average term frequency across all clusters divided by that term's total frequency — so terms common to every cluster are downweighted and terms distinctive to one cluster are upweighted. 3. Multiply c-TF by IDF. Rank the vocabulary by the product; the top terms are the topic representation, with higher weight meaning more representative. Two things follow from this construction: - It is a classical bag-of-words method — fast, deterministic, no model call, and it does not use the semantics of the embedding space. That is both its speed advantage and its weakness. - It is computed from cluster membership alone and does not depend on which embedding, reduction, or cluster model produced the membership. This independence is what makes the whole pipeline modular. Because c-TF-IDF ignores semantics, stop words can survive into representations and near-duplicate word forms (e.g. singular/plural variants of the same stem) crowd out distinct terms. Both are addressed in the next stage rather than by abandoning c-TF-IDF. ## Optional stage: representation-model reranking The c-TF-IDF output is a cheap first-pass representation. A reranking or "representation" model takes that candidate set and improves it with a slower, more powerful technique — the same rerank-a-cheap-candidate-set pattern used in neural search. Three families, stackable: | Approach | Mechanism | What it fixes | Cost it adds | |---|---|---|---| | Embedding-based keyword reranking (KeyBERT-style) | compare candidate keyword embeddings against the topic's average document embedding, rerank by cosine similarity | removes stop words, favors semantically central terms | one embedding pass over candidates per topic; can drop informative domain abbreviations the encoder represents poorly | | Maximal marginal relevance (MMR) | iteratively pick the next keyword that is relevant to the topic but dissimilar to already-chosen keywords, controlled by a diversity parameter | redundancy — collapses near-duplicate word forms, widens coverage | trims a larger candidate set (say 30) down to a diverse smaller set (say 10) | | Generative LLM labeling | prompt an LLM with the topic's keywords plus a few representative documents, ask for a short label | produces a single human-readable topic name instead of a keyword list | one model call per topic | ### The cost pattern that makes LLM labeling viable This is the key operational point of the whole pipeline. Naively, using an LLM to characterize topics means calling it once per *document* — millions of calls on a large corpus. In this architecture the LLM is called once per *topic*: hundreds of calls, not millions, regardless of corpus size. The prompt carries the c-TF-IDF keywords plus a small subset of the most representative documents (selected by cosine similarity of their c-TF-IDF values against the topic's), typically a handful. The economics invert as a result. Corpus size drives the embedding cost (linear in documents) but not the labeling cost (linear in topics). A pipeline that would be prohibitive as per-document LLM classification becomes routine. Two consequences worth stating plainly: label quality tracks model capability — small instruction-tuned models produce labels that are correct but too generic to be useful, while larger models produce labels specific enough to act on — and the keyword representations remain worth keeping alongside the generated label. No labeler is perfect, keywords stay directly traceable to the corpus, and a topic can carry several representations at once (keyword-reranked, MMR-diversified, LLM-labeled) as different views on the same cluster. ## Why modularity is the design point Each stage consumes the previous stage's output and nothing else, so any stage can be replaced without touching the others: - A better encoder ships — swap stage 1, keep everything downstream. - Outliers are unacceptable for the application — swap HDBSCAN for k-means in stage 3, c-TF-IDF is unaffected. - Representations need improving — re-run stage 4 and the representation model alone, with no re-embedding, no re-reduction, and no re-clustering. That last property is what makes representation iteration cheap enough to do interactively. The same seam supports variant workflows on one base pipeline — guided or seeded topics, semi-supervised topics, hierarchical topics, topics over time, online/incremental fitting, and zero-shot topic assignment are all changes to one or two stages rather than different algorithms. Treat this as an architectural property to preserve in your own implementation, not just a library feature: keep embeddings, cluster assignments, and representations as separate persisted artifacts, and the pipeline stays cheap to iterate. ## Choosing this pipeline vs plain k-means over embeddings | Situation | Use | |---|---| | Number of themes unknown; corpus genuinely exploratory | UMAP + HDBSCAN | | Corpus is heterogeneous with real off-topic or niche documents you want isolated | UMAP + HDBSCAN (outliers are the feature) | | Every document must receive a label — routing, assignment, exhaustive coverage | k-means (or HDBSCAN plus outlier reassignment) | | Cluster count is fixed by an external constraint (a known taxonomy, a fixed number of queues) | k-means | | Clusters need to be stable and cheaply re-derivable at fixed k across refreshes | k-means | | Corpus is small enough that density estimation is unreliable | k-means, and treat the result as provisional | | Interpretable topic labels are the deliverable | either — the c-TF-IDF and representation stages work on any cluster assignment | k-means is not a fallback for the unsophisticated; it is the right answer when full coverage or a fixed k is a real requirement. The reverse also holds — forcing k on an exploratory corpus manufactures topic boundaries that the data does not contain. ## Evaluation and interpretation Topic models have no ground truth by construction, so evidence has to be assembled deliberately: - **Read documents from each cluster.** Sample several documents from a cluster and check that the c-TF-IDF keywords actually describe them. This manual pass is not optional overhead; it is the primary validity check, and any visualization is only an approximation of the embedding space. - **Search for topics you expect.** Query the model with a known theme and confirm a coherent topic ranks highly for it. A known document that should belong to a theme should land in that theme's topic. - **Inspect the outlier bucket.** If it is dominated by one recognizable theme, the clustering parameters are too strict. - **Check topic count against use.** Hundreds of fine-grained topics are useful for exploration and useless as a routing taxonomy. Tune the minimum cluster size to the consumer, not to an abstract quality score. - **Compare representations side by side.** Running c-TF-IDF, MMR, and an LLM label together for the same topic surfaces disagreement, and disagreement is where the interpretation is fragile. - Standard cluster-validity indices and coherence scores can supplement this, but do not let a single aggregate number substitute for reading the documents. ## Known traps - Clustering on the 2D visualization projection instead of the 5-10 dim clustering projection. The 2D map distorts distance for legibility. - Treating the outlier bucket as a bug and reassigning it by default, which reintroduces exactly the contamination HDBSCAN was chosen to avoid. - Calling an LLM once per document to label topics. The architecture exists specifically to make this once per topic. - Reporting an LLM-generated topic label without the keyword representation behind it, leaving no traceable path from the label back to the corpus. - Re-running the full pipeline (re-embedding included) to change a topic representation, when only the representation stage needed to change. - Judging the pipeline by cluster geometry while never reading a document from any cluster. - Choosing an embedding model by general leaderboard rank rather than by its clustering-task performance. - Failing to fix or record the reduction seed, then being unable to reproduce a topic set that a stakeholder is already referring to by number. Topic IDs are not stable across refits. - Treating stop words surviving into c-TF-IDF output as a flaw in the pipeline rather than the expected behavior of a bag-of-words stage that a representation model is meant to correct. ## Design checklist - [ ] Unit of analysis fixed (document, section, chunk) before embedding - [ ] Embedding model chosen against clustering-task evidence, embeddings cached - [ ] Clustering reduction and visualization reduction kept as separate artifacts - [ ] Cosine metric used for reduction over normalized text embeddings - [ ] Cluster granularity tuned to the downstream consumer, not to a score - [ ] Outlier share measured and an explicit decision recorded (keep / reassign / switch to k-means) - [ ] c-TF-IDF representation stored alongside any generated label - [ ] LLM labeling budgeted per topic, not per document - [ ] Clusters manually inspected by reading sampled documents - [ ] Seeds, parameters, and the embedding model identity recorded — topic IDs are not stable across refits ## Sources - Alammar, J. and Grootendorst, M., *Hands-On Large Language Models*, O'Reilly, 2024 — Chapter 5, "Text Clustering and Topic Modeling." - Grootendorst, M., "BERTopic: Neural topic modeling with a class-based TF-IDF procedure," arXiv:2203.05794 (2022). - McInnes, L., Healy, J., and Melville, J., "UMAP: Uniform Manifold Approximation and Projection for dimension reduction," arXiv:1802.03426 (2018). - McInnes, L., Healy, J., and Astels, S., "hdbscan: Hierarchical density based clustering," *Journal of Open Source Software* 2.11 (2017): 205.
scripts/leakage_scan.py#!/usr/bin/env python3
"""
leakage_scan.py — Data leakage scanner for ML feature/target specifications.
Reads a column metadata spec (JSON or JSONL) and flags three leakage
anti-patterns:
1. TIME LEAKAGE — features whose observation timestamp is after the
label timestamp, or features explicitly tagged as
"future" or post-event.
2. TARGET LEAKAGE — features that are transformations of, proxies for,
or direct copies of the target column.
3. ID LEAKAGE — identifier or row-key columns included as model inputs,
which can cause spurious memorisation.
This is a static analysis tool only — it inspects metadata, not raw data.
Pair it with a data-distribution check (EDA) for runtime leakage detection.
Usage:
python leakage_scan.py --spec spec.json
python leakage_scan.py --spec spec.jsonl --output report.json --verbose
python leakage_scan.py --help
Spec format (JSON, single object or array, or JSONL):
{
"target": "churn",
"columns": [
{
"name": "customer_id",
"role": "id", // "feature" | "target" | "id" | "timestamp"
"observation_time": "T",
"label_time": "T+30d",
"tags": [],
"description": "primary key"
},
{
"name": "days_since_churn",
"role": "feature",
"tags": ["post_event"],
"description": "days since churn event"
}
]
}
Fields per column:
name — column name (required)
role — "feature" | "target" | "id" | "timestamp" (default: feature)
observation_time — ISO timestamp or symbolic label when feature is observed
label_time — ISO timestamp or symbolic label when target is observed
tags — list of string tags; recognized: "future", "post_event",
"derived_from_target", "proxy_target", "row_key"
description — free text; scanned for leakage keywords
Exit code: 0 if no leakage found, 1 if leakage detected, 2 on input error.
"""
import argparse
import json
import re
import sys
from pathlib import Path
# Keywords that suggest a column may leak the target (case-insensitive)
_TARGET_LEAK_KEYWORDS = [
"churn_reason", "cancel_reason", "refund", "claim_paid",
"default_flag", "fraud_label", "outcome", "result", "post_event",
"after_event", "derived_from_target", "proxy_target",
"target_", "_target",
]
# Keywords in column names or descriptions that suggest ID leakage
_ID_KEYWORDS = [
r"\b(id|key|pk|guid|uuid|rownum|index|row_id|record_id)\b",
r"_id$", r"^id_",
]
# Tags that directly indicate leakage
_FUTURE_TAGS = {"future", "post_event", "after_event", "forward_looking"}
_TARGET_PROXY_TAGS = {"derived_from_target", "proxy_target", "target_proxy", "label_proxy"}
_ID_TAGS = {"row_key", "primary_key", "foreign_key", "record_id"}
def _is_after(obs: str | None, label: str | None) -> bool:
"""Naive symbolic check: flag if obs contains '+' relative to a baseline."""
if not obs or not label:
return False
obs_l, label_l = obs.lower(), label.lower()
if obs_l == label_l:
return False
if re.search(r"T\+|t\+|\+\d", obs_l) and not re.search(r"T\+|t\+|\+\d", label_l):
return True
return False
def scan_columns(spec: dict) -> list[dict]:
"""Return list of leakage findings."""
target_name = spec.get("target", "").lower()
columns = spec.get("columns", [])
findings = []
for col in columns:
name = col.get("name", "")
role = col.get("role", "feature").lower()
tags = {t.lower() for t in col.get("tags", [])}
description = col.get("description", "").lower()
obs_time = col.get("observation_time")
label_time = col.get("label_time")
# Skip target and timestamp columns themselves
if role in ("target", "timestamp"):
continue
# --- TIME LEAKAGE ---
if role == "feature":
time_leaked = False
if _FUTURE_TAGS & tags:
findings.append({
"column": name,
"leakage_type": "TIME_LEAKAGE",
"reason": f"tagged as future/post-event: {_FUTURE_TAGS & tags}",
"severity": "HIGH",
})
time_leaked = True
if not time_leaked and _is_after(obs_time, label_time):
findings.append({
"column": name,
"leakage_type": "TIME_LEAKAGE",
"reason": f"observation_time={obs_time!r} appears after label_time={label_time!r}",
"severity": "HIGH",
})
# --- TARGET LEAKAGE ---
if role == "feature":
tl_reasons = []
if _TARGET_PROXY_TAGS & tags:
tl_reasons.append(f"tagged as target-proxy: {_TARGET_PROXY_TAGS & tags}")
if target_name and target_name in name.lower():
tl_reasons.append(f"column name contains target name '{target_name}'")
for kw in _TARGET_LEAK_KEYWORDS:
if kw in name.lower() or kw in description:
tl_reasons.append(f"leakage keyword '{kw}' in name/description")
break
if tl_reasons:
findings.append({
"column": name,
"leakage_type": "TARGET_LEAKAGE",
"reason": "; ".join(tl_reasons),
"severity": "HIGH",
})
# --- ID LEAKAGE ---
id_reasons = []
if role == "id" or _ID_TAGS & tags:
id_reasons.append(f"role={role!r} or id-related tags={_ID_TAGS & tags}")
else:
for pattern in _ID_KEYWORDS:
if re.search(pattern, name.lower()):
id_reasons.append(f"name matches ID pattern: {pattern}")
break
if id_reasons:
findings.append({
"column": name,
"leakage_type": "ID_LEAKAGE",
"reason": "; ".join(id_reasons),
"severity": "MEDIUM",
})
return findings
def load_spec(path: Path) -> list[dict]:
"""Load JSON/JSONL spec, always returning a list of spec dicts."""
text = path.read_text(encoding="utf-8")
specs = []
# Try JSONL first
for lineno, line in enumerate(text.splitlines(), 1):
line = line.strip()
if not line:
continue
try:
obj = json.loads(line)
if isinstance(obj, dict):
specs.append(obj)
elif isinstance(obj, list):
specs.extend(obj)
except json.JSONDecodeError:
if lineno == 1:
# Might be multi-line JSON; fall through
break
if specs:
return specs
# Try full JSON
obj = json.loads(text)
if isinstance(obj, dict):
return [obj]
if isinstance(obj, list):
return obj
raise ValueError("Spec must be a JSON object, array, or JSONL.")
def run(spec_path: Path, output_path: Path | None, verbose: bool) -> int:
try:
specs = load_spec(spec_path)
except FileNotFoundError:
print(f"[ERROR] File not found: {spec_path}", file=sys.stderr)
return 2
except (json.JSONDecodeError, ValueError) as e:
print(f"[ERROR] Could not parse spec: {e}", file=sys.stderr)
return 2
all_findings = []
for spec in specs:
findings = scan_columns(spec)
all_findings.extend(findings)
total = len(all_findings)
by_type: dict[str, int] = {}
for f in all_findings:
by_type[f["leakage_type"]] = by_type.get(f["leakage_type"], 0) + 1
if verbose or total > 0:
for f in all_findings:
print(f"[{f['severity']}] {f['leakage_type']:20s} column={f['column']!r} {f['reason']}")
print(f"\nLeakage scan complete: {total} issue(s) found.")
for k, v in sorted(by_type.items()):
print(f" {k}: {v}")
report = {
"total_issues": total,
"by_type": by_type,
"findings": all_findings,
}
if output_path:
with output_path.open("w") as f:
json.dump(report, f, indent=2)
print(f"Report written to: {output_path}")
return 1 if total > 0 else 0
def main() -> None:
parser = argparse.ArgumentParser(
description="Static leakage scanner for ML feature/target column specs.",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument("--spec", required=True, type=Path, help="Column spec JSON or JSONL file")
parser.add_argument("--output", type=Path, default=None, help="Output JSON report path")
parser.add_argument("--verbose", "-v", action="store_true", help="Print each finding")
args = parser.parse_args()
sys.exit(run(args.spec, args.output, args.verbose))
if __name__ == "__main__":
main()
scripts/ml_toolkit.py#!/usr/bin/env python3
"""
ML Toolkit — stdlib-only CLI for model quality and leakage checks.
Subcommands:
card — Generate a structured model card in Markdown from a model spec JSON
leakage — Run a leakage checklist and emit PASS/WARN/FAIL per check
report — Full model quality report combining card + leakage analysis
Usage:
python scripts/ml_toolkit.py card --input data/sample-model-spec.json
python scripts/ml_toolkit.py leakage --input data/sample-model-spec.json
python scripts/ml_toolkit.py report --input data/sample-model-spec.json
python scripts/ml_toolkit.py report --input data/sample-model-spec.json --output report.md
"""
from __future__ import annotations
import argparse
import json
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
FUTURE_LEAK_KEYWORDS = [
"future_", "next_", "post_", "after_label",
"_future", "_next", "_post", "_afterlabel",
]
TARGET_DERIVED_KEYWORDS = [
"churn_", "target_", "label_", "y_", "_churn", "_target",
]
LEAKAGE_STATUS_ORDER = {"FAIL": 0, "WARN": 1, "PASS": 2}
# ---------------------------------------------------------------------------
# Data models
# ---------------------------------------------------------------------------
@dataclass
class LeakageCheck:
name: str
status: str # PASS | WARN | FAIL
detail: str
@dataclass
class LeakageResult:
checks: list[LeakageCheck] = field(default_factory=list)
@property
def overall(self) -> str:
"""Aggregate status: worst of all checks."""
if not self.checks:
return "PASS"
return min(self.checks, key=lambda c: LEAKAGE_STATUS_ORDER[c.status]).status
# ---------------------------------------------------------------------------
# Core logic: leakage checks
# ---------------------------------------------------------------------------
def check_prediction_timestamp(spec: dict) -> LeakageCheck:
"""Prediction timestamp must be defined and precede the label timestamp."""
defined = spec.get("prediction_timestamp_defined", False)
pred_field = spec.get("prediction_timestamp_field", "")
label_field = spec.get("label_timestamp_field", "")
if not defined or not pred_field:
return LeakageCheck(
name="Prediction timestamp defined",
status="FAIL",
detail=(
"prediction_timestamp_defined is false or prediction_timestamp_field is missing. "
"Define the exact point-in-time at which scoring occurs before any feature engineering."
),
)
if not label_field:
return LeakageCheck(
name="Prediction timestamp defined",
status="WARN",
detail=(
f"prediction_timestamp_field is '{pred_field}' but label_timestamp_field is not set. "
"Cannot verify temporal ordering between prediction and label timestamps."
),
)
# Field names are different (required); we cannot compare actual dates from
# field names alone, but we can confirm they are distinct fields.
if pred_field == label_field:
return LeakageCheck(
name="Prediction timestamp defined",
status="FAIL",
detail=(
f"prediction_timestamp_field and label_timestamp_field are both '{pred_field}'. "
"They must be different fields: one for scoring time, one for when the outcome is known."
),
)
return LeakageCheck(
name="Prediction timestamp defined",
status="PASS",
detail=(
f"prediction_timestamp_field='{pred_field}', label_timestamp_field='{label_field}'. "
"Fields are distinct — verify in code that pred_ts < label_ts for every training row."
),
)
def check_future_leaking_features(spec: dict) -> LeakageCheck:
"""Flag feature names that contain future-leaking keyword patterns."""
features = spec.get("features", [])
flagged = [
f["name"]
for f in features
if any(kw in f["name"].lower() for kw in FUTURE_LEAK_KEYWORDS)
]
if flagged:
return LeakageCheck(
name="No future-leaking features",
status="WARN",
detail=(
f"Feature name(s) contain future-leaking patterns ({', '.join(FUTURE_LEAK_KEYWORDS)}): "
f"{', '.join(flagged)}. "
"Review each to confirm values are available at prediction time."
),
)
return LeakageCheck(
name="No future-leaking features",
status="PASS",
detail=(
f"None of the {len(features)} feature(s) matched future-leaking keyword patterns. "
"Keyword scan is a heuristic — also validate feature availability by deployment walkthrough."
),
)
def check_split_temporal_discipline(spec: dict) -> LeakageCheck:
"""Train/val/test must use temporal ordering, not random split, for time-ordered data."""
split_method = spec.get("train_val_test_split_method", "").lower()
temporal_col = (spec.get("training_data") or {}).get("temporal_column", "")
split_config = spec.get("split_config", {})
if split_method == "random" and temporal_col:
return LeakageCheck(
name="Temporal split discipline",
status="FAIL",
detail=(
f"train_val_test_split_method is 'random' but training_data.temporal_column is '{temporal_col}'. "
"Random splits on time-ordered data cause future-to-past leakage. Switch to temporal cutoffs."
),
)
if split_method not in ("temporal", "time-based", "chronological"):
return LeakageCheck(
name="Temporal split discipline",
status="WARN",
detail=(
f"train_val_test_split_method is '{split_method}' — expected 'temporal'. "
"If the data has a time dimension, verify that the split strategy prevents future leakage."
),
)
if split_config:
train_end = split_config.get("train_end", "")
val_end = split_config.get("val_end", "")
test_end = split_config.get("test_end", "")
if train_end and val_end and test_end:
# String comparison works for ISO dates (YYYY-MM-DD).
if not (train_end <= val_end <= test_end):
return LeakageCheck(
name="Temporal split discipline",
status="FAIL",
detail=(
f"split_config dates are not in ascending order: "
f"train_end={train_end}, val_end={val_end}, test_end={test_end}. "
"Correct the split boundaries."
),
)
return LeakageCheck(
name="Temporal split discipline",
status="PASS",
detail=(
f"Temporal splits in order: train_end={train_end}, val_end={val_end}, test_end={test_end}."
),
)
return LeakageCheck(
name="Temporal split discipline",
status="PASS",
detail=(
"split_method is temporal. No split_config provided to validate date boundaries — "
"verify cutoff dates manually."
),
)
def check_target_leakage(spec: dict) -> LeakageCheck:
"""Check if any feature name looks derived from the target variable."""
target = (spec.get("training_data") or {}).get("target_variable", "").lower()
features = spec.get("features", [])
flagged = []
for f in features:
fname = f["name"].lower()
# Check generic target-derived keywords.
if any(kw in fname for kw in TARGET_DERIVED_KEYWORDS):
flagged.append(f["name"])
continue
# Check if the feature name contains the target variable name as a substring.
if target and len(target) >= 4 and target in fname:
flagged.append(f["name"])
if flagged:
return LeakageCheck(
name="Target leakage",
status="WARN",
detail=(
f"Feature name(s) may be derived from the target ('{target}'): "
f"{', '.join(flagged)}. "
"Confirm these features are not computed using the label or any post-event data."
),
)
return LeakageCheck(
name="Target leakage",
status="PASS",
detail=(
f"No features matched target-derived keyword patterns for target='{target}'. "
"This is a heuristic check — review engineered features by hand, especially aggregates."
),
)
def check_data_collection_date(spec: dict) -> LeakageCheck:
"""Training data metadata must include the data collection / extraction date."""
td = spec.get("training_data") or {}
collection_date = td.get("data_collection_date", "")
if not collection_date:
return LeakageCheck(
name="Data collection date in metadata",
status="FAIL",
detail=(
"training_data.data_collection_date is missing. "
"Record when the training dataset was extracted to support reproducibility and staleness checks."
),
)
return LeakageCheck(
name="Data collection date in metadata",
status="PASS",
detail=f"training_data.data_collection_date='{collection_date}'.",
)
def run_leakage_checks(spec: dict) -> LeakageResult:
result = LeakageResult()
result.checks.append(check_prediction_timestamp(spec))
result.checks.append(check_future_leaking_features(spec))
result.checks.append(check_split_temporal_discipline(spec))
result.checks.append(check_target_leakage(spec))
result.checks.append(check_data_collection_date(spec))
return result
# ---------------------------------------------------------------------------
# Formatting helpers
# ---------------------------------------------------------------------------
def fmt_status_badge(status: str) -> str:
badges = {"PASS": "[PASS]", "WARN": "[WARN]", "FAIL": "[FAIL]"}
return badges.get(status, f"[{status}]")
def _md_table_row(cols: list[str]) -> str:
return "| " + " | ".join(cols) + " |"
def _md_table_sep(col_count: int) -> str:
return "|" + "|".join(["---"] * col_count) + "|"
# ---------------------------------------------------------------------------
# Model card builder
# ---------------------------------------------------------------------------
def build_model_card(spec: dict) -> list[str]:
lines: list[str] = []
a = lines.append
model_name = spec.get("model_name", "unnamed-model")
version = spec.get("version", "n/a")
model_type = spec.get("model_type", "n/a")
task_description = spec.get("task_description", "")
intended_use = spec.get("intended_use", "")
td = spec.get("training_data") or {}
features = spec.get("features", [])
metrics = spec.get("performance_metrics", [])
limitations = spec.get("limitations", [])
ethical = spec.get("ethical_considerations", [])
lineage = spec.get("lineage") or {}
# --- Header ---
a(f"# Model Card: {model_name}")
a("")
a(f"**Version:** {version} ")
a(f"**Model type:** {model_type} ")
a(f"**Generated:** 2026-03-21 ")
a("")
# --- Model Overview ---
a("---")
a("")
a("## Model Overview")
a("")
if task_description:
a(task_description)
a("")
# --- Intended Use ---
a("---")
a("")
a("## Intended Use")
a("")
if intended_use:
a(intended_use)
else:
a("_Not specified._")
a("")
# --- Training Data ---
a("---")
a("")
a("## Training Data")
a("")
td_rows = [
("Source", td.get("source", "n/a")),
("Date range", f"{td.get('date_range', {}).get('start', '?')} — {td.get('date_range', {}).get('end', '?')}"),
("Row count", f"{td.get('row_count', 'n/a'):,}" if isinstance(td.get("row_count"), int) else str(td.get("row_count", "n/a"))),
("Feature count", str(td.get("feature_count", len(features)))),
("Target variable", td.get("target_variable", "n/a")),
("Temporal column", td.get("temporal_column", "n/a")),
("Data collection date",td.get("data_collection_date", "n/a")),
]
a(_md_table_row(["Field", "Value"]))
a(_md_table_sep(2))
for k, v in td_rows:
a(_md_table_row([k, v]))
a("")
if features:
a(f"**Features ({len(features)} total):**")
a("")
a(_md_table_row(["Name", "Type", "Description"]))
a(_md_table_sep(3))
for f in features:
a(_md_table_row([
f.get("name", ""),
f.get("type", ""),
f.get("description", ""),
]))
a("")
# --- Performance Metrics ---
a("---")
a("")
a("## Performance Metrics")
a("")
if metrics:
a(_md_table_row(["Metric", "Value", "Benchmark", "Gap", "Split"]))
a(_md_table_sep(5))
for m in metrics:
val = m.get("value")
bench = m.get("benchmark")
if val is not None and bench is not None:
try:
gap = float(val) - float(bench)
gap_str = f"{gap:+.3f}"
except (TypeError, ValueError):
gap_str = "n/a"
else:
gap_str = "n/a"
a(_md_table_row([
m.get("metric", ""),
str(val) if val is not None else "n/a",
str(bench) if bench is not None else "n/a",
gap_str,
m.get("split", ""),
]))
a("")
else:
a("_No performance metrics provided._")
a("")
# --- Known Limitations ---
a("---")
a("")
a("## Known Limitations")
a("")
if limitations:
for lim in limitations:
a(f"- {lim}")
else:
a("_No limitations documented._")
a("")
# --- Ethical Considerations ---
a("---")
a("")
a("## Ethical Considerations")
a("")
if ethical:
for item in ethical:
a(f"- {item}")
else:
a("_No ethical considerations documented._")
a("")
# --- Versioning and Lineage ---
a("---")
a("")
a("## Versioning and Lineage")
a("")
split_method = spec.get("train_val_test_split_method", "n/a")
split_cfg = spec.get("split_config") or {}
pred_ts = spec.get("prediction_timestamp_field", "n/a")
label_ts = spec.get("label_timestamp_field", "n/a")
lineage_rows = [
("Model version", version),
("Experiment ID", lineage.get("experiment_id", "n/a")),
("Git commit", lineage.get("git_commit", "n/a")),
("MLflow run ID", lineage.get("mlflow_run_id", "n/a")),
("Feature store version", lineage.get("feature_store_version", "n/a")),
("Model artifact", lineage.get("model_artifact", "n/a")),
("Split method", split_method),
("Train end", split_cfg.get("train_end", "n/a")),
("Val end", split_cfg.get("val_end", "n/a")),
("Test end", split_cfg.get("test_end", "n/a")),
("Prediction timestamp field", pred_ts),
("Label timestamp field", label_ts),
]
a(_md_table_row(["Field", "Value"]))
a(_md_table_sep(2))
for k, v in lineage_rows:
a(_md_table_row([k, v]))
a("")
return lines
# ---------------------------------------------------------------------------
# Leakage report builder
# ---------------------------------------------------------------------------
def build_leakage_report(spec: dict, result: LeakageResult) -> list[str]:
lines: list[str] = []
a = lines.append
model_name = spec.get("model_name", "unnamed-model")
version = spec.get("version", "n/a")
a(f"# Leakage Checklist: {model_name} v{version}")
a("")
a(f"**Overall:** {fmt_status_badge(result.overall)} ")
a(f"**Checks run:** {len(result.checks)} ")
a(f"**Generated:** 2026-03-21 ")
a("")
a("---")
a("")
a(_md_table_row(["Check", "Status", "Detail"]))
a(_md_table_sep(3))
for check in result.checks:
a(_md_table_row([check.name, fmt_status_badge(check.status), check.detail]))
a("")
fails = [c for c in result.checks if c.status == "FAIL"]
warns = [c for c in result.checks if c.status == "WARN"]
if fails:
a("---")
a("")
a(f"## FAIL Items ({len(fails)})")
a("")
for c in fails:
a(f"### {c.name}")
a("")
a(c.detail)
a("")
if warns:
a("---")
a("")
a(f"## WARN Items ({len(warns)})")
a("")
for c in warns:
a(f"### {c.name}")
a("")
a(c.detail)
a("")
if not fails and not warns:
a("> All leakage checks passed. Perform a manual feature-availability walkthrough before final sign-off.")
a("")
return lines
# ---------------------------------------------------------------------------
# Subcommand: card
# ---------------------------------------------------------------------------
def cmd_card(args: argparse.Namespace) -> None:
spec = _load_json(args.input)
lines = build_model_card(spec)
output = "\n".join(lines)
output_path = getattr(args, "output", None)
if output_path:
Path(output_path).write_text(output, encoding="utf-8")
print(f"Model card written to: {output_path}")
else:
print(output)
# ---------------------------------------------------------------------------
# Subcommand: leakage
# ---------------------------------------------------------------------------
def cmd_leakage(args: argparse.Namespace) -> None:
spec = _load_json(args.input)
result = run_leakage_checks(spec)
output_path = getattr(args, "output", None)
if output_path:
lines = build_leakage_report(spec, result)
Path(output_path).write_text("\n".join(lines), encoding="utf-8")
print(f"Leakage report written to: {output_path}")
return
# Console output (non-Markdown).
model_name = spec.get("model_name", "unnamed-model")
version = spec.get("version", "n/a")
print()
print(f"=== LEAKAGE CHECKLIST: {model_name} v{version} ===")
print("-" * 60)
col_w_name = 38
col_w_status = 8
for check in result.checks:
badge = fmt_status_badge(check.status)
print(f" {check.name:<{col_w_name}} {badge:<{col_w_status}}")
# Wrap detail at 70 chars indented.
detail_words = check.detail.split()
line_buf: list[str] = []
line_len = 0
for word in detail_words:
if line_len + len(word) + 1 > 70:
print(f" {' '.join(line_buf)}")
line_buf = [word]
line_len = len(word)
else:
line_buf.append(word)
line_len += len(word) + 1
if line_buf:
print(f" {' '.join(line_buf)}")
print()
print("-" * 60)
overall = result.overall
print(f" Overall result: {fmt_status_badge(overall)}")
fails = sum(1 for c in result.checks if c.status == "FAIL")
warns = sum(1 for c in result.checks if c.status == "WARN")
print(f" FAIL: {fails} WARN: {warns} PASS: {len(result.checks) - fails - warns}")
print()
# ---------------------------------------------------------------------------
# Subcommand: report
# ---------------------------------------------------------------------------
def cmd_report(args: argparse.Namespace) -> None:
spec = _load_json(args.input)
leakage_result = run_leakage_checks(spec)
card_lines = build_model_card(spec)
leakage_lines = build_leakage_report(spec, leakage_result)
divider = ["", "---", "", "# Leakage Analysis", ""]
all_lines = card_lines + divider + leakage_lines
report_text = "\n".join(all_lines)
output_path = getattr(args, "output", None)
if output_path:
Path(output_path).write_text(report_text, encoding="utf-8")
print(f"Report written to: {output_path}")
else:
print(report_text)
# ---------------------------------------------------------------------------
# JSON loader
# ---------------------------------------------------------------------------
def _load_json(path: str) -> dict:
p = Path(path)
if not p.exists():
print(f"Error: File not found: {path}", file=sys.stderr)
sys.exit(1)
try:
with p.open(encoding="utf-8") as f:
return json.load(f)
except json.JSONDecodeError as exc:
print(f"Error: Invalid JSON in {path}: {exc}", file=sys.stderr)
sys.exit(1)
# ---------------------------------------------------------------------------
# CLI wiring
# ---------------------------------------------------------------------------
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="ml_toolkit",
description="ML Toolkit — stdlib only. Model card generation and leakage checks. No pip install required.",
)
subparsers = parser.add_subparsers(dest="command", metavar="SUBCOMMAND")
subparsers.required = True
# --- card ---
p_card = subparsers.add_parser(
"card",
help="Generate a structured model card in Markdown from a model spec JSON.",
description=(
"Reads a model spec JSON and produces a Markdown model card with sections: "
"Model Overview, Intended Use, Training Data, Performance Metrics, "
"Known Limitations, Ethical Considerations, Versioning/Lineage."
),
)
p_card.add_argument(
"--input", metavar="JSON_FILE", required=True,
help="Path to model spec JSON (e.g. data/sample-model-spec.json).",
)
p_card.add_argument(
"--output", metavar="OUTPUT_FILE",
help="Write model card to this file instead of stdout.",
)
p_card.set_defaults(func=cmd_card)
# --- leakage ---
p_leakage = subparsers.add_parser(
"leakage",
help="Run a leakage checklist and emit PASS/WARN/FAIL per check.",
description=(
"Checks: prediction timestamp discipline, future-leaking feature names, "
"temporal split ordering, target leakage patterns, and data collection date. "
"Prints results to console; use --output to write a Markdown report."
),
)
p_leakage.add_argument(
"--input", metavar="JSON_FILE", required=True,
help="Path to model spec JSON (e.g. data/sample-model-spec.json).",
)
p_leakage.add_argument(
"--output", metavar="OUTPUT_FILE",
help="Write Markdown leakage report to this file instead of console output.",
)
p_leakage.set_defaults(func=cmd_leakage)
# --- report ---
p_report = subparsers.add_parser(
"report",
help="Full model quality report combining model card and leakage analysis.",
description=(
"Reads a model spec JSON and produces a single Markdown report "
"that includes the full model card followed by the leakage checklist."
),
)
p_report.add_argument(
"--input", metavar="JSON_FILE", required=True,
help="Path to model spec JSON (e.g. data/sample-model-spec.json).",
)
p_report.add_argument(
"--output", metavar="OUTPUT_FILE",
help="Write report to this file instead of stdout (e.g. report.md).",
)
p_report.set_defaults(func=cmd_report)
return parser
def main() -> None:
parser = build_parser()
args = parser.parse_args()
args.func(args)
if __name__ == "__main__":
main()
scripts/README.md# ml_toolkit.py
Stdlib-only Python CLI for ML model quality checks. No external dependencies — runs with any Python 3.9+ installation.
## Purpose
Gives data scientists and ML engineers fast, reproducible answers to three core questions:
1. **card** — What does this model do, how does it perform, and what are its risks? (Markdown model card)
2. **leakage** — Does this model spec pass temporal discipline and leakage checks? (PASS/WARN/FAIL per check)
3. **report** — A full Markdown quality report combining card + leakage analysis.
## Quick Start
Run from the `ai-ml-data-science/` directory:
```bash
# Generate a model card (prints to stdout)
python scripts/ml_toolkit.py card --input data/sample-model-spec.json
# Generate a model card and write to file
python scripts/ml_toolkit.py card --input data/sample-model-spec.json --output /tmp/model-card.md
# Run the leakage checklist (console output with PASS/WARN/FAIL per check)
python scripts/ml_toolkit.py leakage --input data/sample-model-spec.json
# Run leakage checklist and write a Markdown report to file
python scripts/ml_toolkit.py leakage --input data/sample-model-spec.json --output /tmp/leakage-report.md
# Full quality report combining card + leakage (prints to stdout)
python scripts/ml_toolkit.py report --input data/sample-model-spec.json
# Full quality report written to file
python scripts/ml_toolkit.py report --input data/sample-model-spec.json --output report.md
```
## JSON Input Format
All subcommands read from `--input <json_file>`. The sample spec at `data/sample-model-spec.json` documents all supported fields.
### Minimum viable spec
```json
{
"model_name": "my-model",
"version": "1.0.0",
"model_type": "classification",
"task_description": "Predict X given Y.",
"intended_use": "Trigger action Z when score >= 0.5.",
"training_data": {
"source": "warehouse.feature_mart",
"date_range": {"start": "2024-01-01", "end": "2025-06-30"},
"row_count": 100000,
"feature_count": 20,
"target_variable": "outcome_flag",
"temporal_column": "snapshot_date",
"data_collection_date": "2025-07-10"
},
"features": [
{"name": "days_since_last_event", "type": "numeric", "description": "Days since last recorded event"},
{"name": "account_age_days", "type": "numeric", "description": "Age of the account in days"}
],
"performance_metrics": [
{"metric": "ROC-AUC", "value": 0.82, "benchmark": 0.75, "split": "temporal-holdout"}
],
"limitations": ["Model not validated on accounts < 30 days old."],
"ethical_considerations": ["Do not use scores to deny service."],
"prediction_timestamp_defined": true,
"prediction_timestamp_field": "snapshot_date",
"label_timestamp_field": "outcome_event_date",
"train_val_test_split_method": "temporal",
"split_config": {
"train_end": "2025-03-31",
"val_end": "2025-05-31",
"test_end": "2025-06-30"
}
}
```
## Leakage Checks Reference
| Check | What it looks for | FAIL condition | WARN condition |
|---|---|---|---|
| Prediction timestamp defined | `prediction_timestamp_defined` and `prediction_timestamp_field` present; distinct from `label_timestamp_field` | Missing or same field for pred/label | `label_timestamp_field` not set |
| No future-leaking features | Feature names containing `future_`, `next_`, `post_`, `after_label` | — | Any feature name matches a keyword |
| Temporal split discipline | `train_val_test_split_method` is temporal; `split_config` dates in ascending order | Random split with a temporal column, or dates out of order | Unknown split method |
| Target leakage | Feature names containing target-derived patterns or the target variable name | — | Any feature name matches |
| Data collection date | `training_data.data_collection_date` is present | Missing field | — |
## Subcommand Reference
```
python scripts/ml_toolkit.py card --help
python scripts/ml_toolkit.py leakage --help
python scripts/ml_toolkit.py report --help
```
## Model Card Sections
The `card` and `report` subcommands produce a Markdown model card with these sections:
| Section | Content |
|---|---|
| Model Overview | Task description and model type |
| Intended Use | Approved uses and explicit exclusions |
| Training Data | Source, date range, row/feature count, target, temporal column |
| Performance Metrics | Table: metric, value, benchmark, gap, split |
| Known Limitations | Bulleted list of documented failure conditions |
| Ethical Considerations | Bulleted list of fairness and misuse risks |
| Versioning and Lineage | Split config, timestamps, experiment/artifact IDs |
SKILL.md--- name: ai-ml-data-science description: "Builds ML, responsible-AI, and multimodal models. Use when doing data science or explaining fairness, privacy, speech, vision-language, or diffusion mechanics." compatibility: Portable core. Works on Claude Code and Codex. version: "1.2" last_validated: 2026-08-21 --- # Data Science Engineering Suite Use this skill for reproducible data-science work from problem framing through evaluation and handoff. The center of gravity is not "pick the fanciest model." It is framing the decision, building train-serve-safe features, and producing decision-ready evidence. ## ASCII Flow ```text data question | v problem framing target + unit of analysis + leakage risks + decision/use case | v data work source checks + EDA + feature logic + split strategy + baseline | v model/evidence train or analyze + validate + interpret + quantify uncertainty | v handoff report, notebook, model candidate, or production path to MLOps ``` ## Quick Reference | Need | Default Direction | |------|-------------------| | reproducible Python workflow | `uv` plus scripts or git-friendly notebooks (marimo for reactive/diffable notebooks) | | fast local analysis | DuckDB plus Polars (v1.x stable API as of 2026; pre-1.0 API-churn concerns no longer apply) | | data contracts | Pandera or GX Core at dataset boundaries | | tabular baseline | linear or logistic model plus tree-based candidate | | feature engineering | explicit train-serve-safe transforms | | unlabeled text corpus | embed -> UMAP -> HDBSCAN -> c-TF-IDF; LLM labels once per topic, never per document | | tuning | Optuna only after the baseline is stable | | evaluation | slices, threshold, calibration, uncertainty | | handoff | model card, evaluation report, failure modes, monitoring expectations | ## When To Use This Skill - exploring datasets and checking modelling feasibility - designing feature pipelines and leakage controls - choosing and comparing model families - clustering unlabeled text and discovering topics before a taxonomy or labeling effort exists - building reproducible experiment workflows - producing evaluation reports, model cards, and handoff artifacts - reviewing whether an experiment is genuinely ready for production handoff - explaining responsible-AI modelling mechanics: fairness and intersectionality, privacy, interpretability, poisoning, memorization, human oversight, and environmental trade-offs - designing general multimodal models: contrastive image-text learning, fusion, VQA/document/video systems, diffusion control, adaptation, and quality-latency trade-offs ## Route Elsewhere - serving, retraining automation, monitoring, or incident response -> [ai-mlops](../ai-mlops/SKILL.md) - forecasting and temporal validation -> [ai-ml-timeseries](../ai-ml-timeseries/SKILL.md) - lakehouse, ingestion, or streaming infrastructure -> [data-lake-platform](../data-lake-platform/SKILL.md) - prompting, fine-tuning, or LLM-system design -> [ai-llm](../ai-llm/SKILL.md) or [ai-rag](../ai-rag/SKILL.md) --- ## Workflow 1. Frame the decision, target, baseline, and prediction timestamp before touching models. 2. Validate the dataset shape, ownership, and leakage risks. 3. Build the simplest viable baseline first. 4. Design point-in-time-correct features and compare stronger candidates only after the baseline is trustworthy. 5. Evaluate with the same split strategy, same metric definitions, and same compute budget. 6. Produce handoff artifacts with thresholds, calibration state, failure modes, and reproducibility notes. --- ## Core Rules - write down the prediction timestamp explicitly - do not trust random splits where time or entity leakage is plausible - compare at least one simple baseline against one stronger candidate - treat thresholding, calibration, and uncertainty as part of the decision - keep data version, feature version, seed, and split logic reproducible - hand off deployment-heavy questions early instead of rebuilding MLOps inside a notebook ## Known Traps - Using random train/test splits when time, entity, household, account, or session leakage is plausible. - Building features with information that is only available after the prediction point, then calling the result "production ready." - Tuning models before the baseline and metric definitions are stable. - Reporting only AUC or one aggregate score while ignoring threshold choice, calibration, slice behavior, and operational tradeoffs. - Letting notebook state become the real pipeline logic. Hidden ordering and cached state break reproducibility fast. - A single feature with near-perfect standalone separation, or a metric a domain expert would find implausibly good — treat as a leakage bug report first, a discovery second (see `references/eda-best-practices.md` Expert Instincts). - A correct time-based split with no group/entity split alongside it, when the same user/account/household recurs across time periods — time discipline alone does not stop entity leakage. - Citing a library version, benchmark number, or API pattern from memory or an older tutorial without checking it against the currently installed version — tabular-ML tooling (Optuna, SHAP, scikit-learn, boosted-tree libraries) crosses breaking major versions inside a single year. ## Common Anti-Patterns - Treating a more complex model as progress when the baseline is not yet well understood. - Optimizing benchmark metrics without checking train-serve parity for feature computation. - Using global preprocessing shortcuts that leak label or split information across folds. - Handing off a model without a model card, failure modes, threshold rationale, and monitoring expectations. ## Pattern Chooser | Problem Shape | Direction | |---------------|-----------| | tabular or relational | baseline plus tree-based comparison | | time-ordered forecasting | route to [ai-ml-timeseries](../ai-ml-timeseries/SKILL.md) | | classical text or embeddings plus classifier | stay here | | unlabeled text, unknown themes, topic discovery | stay here; see `references/text-clustering-topic-modeling.md` | | LLM workflow, prompting, or RAG | route to [ai-llm](../ai-llm/SKILL.md) or [ai-rag](../ai-rag/SKILL.md) | | deployment, monitoring, retraining | route to [ai-mlops](../ai-mlops/SKILL.md) | | ingestion or lakehouse architecture | route to [data-lake-platform](../data-lake-platform/SKILL.md) | | responsible-AI concepts and modelling trade-offs | stay here; route operational controls to [ai-mlops](../ai-mlops/SKILL.md) and measurement/red teaming to [ai-evals](../ai-evals/SKILL.md) | | multimodal representations, fusion, VQA/document/video, diffusion mechanics | stay here; route production and evaluation to [ai-mlops](../ai-mlops/SKILL.md) and [ai-evals](../ai-evals/SKILL.md) | --- ## Core Patterns ### End-to-end DS lifecycle - problem framing and baseline - dataset scan and contracts - EDA and leakage review - feature plan - baseline versus candidate comparison - evaluation with slices and thresholds - production handoff package ### Reproducible workspace - `uv` and explicit dependencies - script-first or git-friendly notebook entrypoints — for reactive, git-diffable notebooks consider [marimo](https://docs.marimo.io/) as an alternative to Jupyter; marimo is reactive (dependent cells auto-rerun), stores notebooks as plain Python scripts, and eliminates hidden-state ordering issues - fixed seeds and explicit split logic - logged dataset and feature assumptions ### Feature engineering and contracts - numeric, categorical, text, and time-based transforms - point-in-time availability checks - reusable encoders and documented freshness assumptions ### Evaluation and decision readiness - primary metric plus guardrails - threshold strategy - calibration and uncertainty handling - slice analysis and qualitative error review ### Autonomous experimentation Use agent-driven experiment loops only when the metric is explicit, the search space is bounded, and each run is cheap enough to keep or revert automatically. --- ## Templates - [assets/project/template-standard.md](assets/project/template-standard.md) - [assets/project/template-quick.md](assets/project/template-quick.md) - [assets/features/template-feature-engineering.md](assets/features/template-feature-engineering.md) - [assets/eda/template-eda.md](assets/eda/template-eda.md) - [assets/evaluation/template-evaluation-report.md](assets/evaluation/template-evaluation-report.md) - [assets/evaluation/template-model-card.md](assets/evaluation/template-model-card.md) - [assets/review/experiment-review-template.md](assets/review/experiment-review-template.md) ## Scripts | Script | Purpose | |--------|---------| | [scripts/ml_toolkit.py](scripts/ml_toolkit.py) | Generates model cards, leakage checks, and model-quality reports from a model-spec JSON | | [scripts/leakage_scan.py](scripts/leakage_scan.py) | Static leakage scanner for ML feature/target column specs (JSON/JSONL). Flags time-leakage, target-leakage, and ID-leakage anti-patterns from column metadata. Exit code 1 if issues found. | Typical usage: ```bash python scripts/ml_toolkit.py card --input data/sample-model-spec.json python scripts/ml_toolkit.py leakage --input data/sample-model-spec.json python scripts/ml_toolkit.py report --input data/sample-model-spec.json --output report.md ``` See [scripts/README.md](scripts/README.md) for the input format and leakage-check logic. ## Navigation ### Core references - [references/eda-best-practices.md](references/eda-best-practices.md) - [references/feature-engineering-patterns.md](references/feature-engineering-patterns.md) - [references/data-contracts-lineage.md](references/data-contracts-lineage.md) - [references/modelling-patterns.md](references/modelling-patterns.md) - [references/evaluation-patterns.md](references/evaluation-patterns.md) - [references/class-imbalance-patterns.md](references/class-imbalance-patterns.md) - [references/hyperparameter-optimization.md](references/hyperparameter-optimization.md) - [references/text-clustering-topic-modeling.md](references/text-clustering-topic-modeling.md) — modular embed/UMAP/HDBSCAN/c-TF-IDF pipeline, representation-model reranking, per-topic (not per-document) LLM labeling, and when to prefer plain k-means - [references/interpretability-explainability.md](references/interpretability-explainability.md) - [references/responsible-ai-mechanics.md](references/responsible-ai-mechanics.md) — fairness/intersectionality, differential privacy, explainability, poisoning/federated learning, re-identification, watermarking, human oversight/appeals, copyright/memorization, and environmental trade-offs - [references/multimodal-modeling.md](references/multimodal-modeling.md) — CLIP/SigLIP objectives, fusion, VQA/document/video systems, diffusion control/diversity/acceleration, adaptation, latency, and cost - [references/reproducibility-checklist.md](references/reproducibility-checklist.md) - [references/llm-data-pipeline.md](references/llm-data-pipeline.md) — LLM-from-scratch data pipelines (dedup, quality filtering, synthetic mixing, decontamination); route here first for corpus curation work - [references/feature-freshness-streaming.md](references/feature-freshness-streaming.md) - [references/production-feedback-loops.md](references/production-feedback-loops.md) - [references/ml-diagrams.md](references/ml-diagrams.md) — Mermaid diagram catalog for classical ML (k-means, logistic regression, decision trees, collaborative filtering) and neural net architectures (MLP, RNN, CNN, Transformer); for embedding in docs, READMEs, PR descriptions ### Data and external references - [data/sources.json](data/sources.json) - [data/sample-model-spec.json](data/sample-model-spec.json) ## Related Skills - [ai-architecture-advisor](../ai-architecture-advisor/SKILL.md) — when to use trees vs deep learning vs LLM (decide before building) - [ai-mlops](../ai-mlops/SKILL.md) - [ai-ml-timeseries](../ai-ml-timeseries/SKILL.md) - [data-lake-platform](../data-lake-platform/SKILL.md) - [ai-llm](../ai-llm/SKILL.md) - [ai-rag](../ai-rag/SKILL.md) - huggingface-datasets — now in the external `huggingface-skills:` plugin ## Fact-Checking - Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact. - Verify current library capabilities, version-sensitive tooling advice, and benchmark claims before final answers. - Prefer official docs for fast-moving tools and model libraries. - If web access is unavailable, keep tool recommendations marked as unverified where freshness matters. ## Learnings Loop Before applying this skill on a non-trivial task, read `learnings.consolidated.md` in this directory (and `learnings.md` if present). After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.