EVALUATION_REPORT.md
# Evaluation Report — machine-learning-engineer
## Skill Summary
| Field | Value |
|-------|-------|
| **Name** | machine-learning-engineer |
| **Version** | 5.0.0 |
| **Quality Tier** | Exemplary ⭐⭐ |
| **Rubric Score** | 9.2/10 |
| **Line Count** | 494 |
---
## 6-Dimension Rubric Scores
| Dimension | Score | Weight | Weighted | Tier |
|-----------|-------|--------|----------|------|
| System Prompt Depth | 9.0 | 20% | 1.80 | Exemplary |
| Domain Knowledge Density | 9.5 | 25% | 2.375 | Exemplary |
| Workflow Actionability | 9.0 | 15% | 1.35 | Exemplary |
| Risk Documentation | 8.5 | 10% | 0.85 | Expert |
| Example Quality | 9.0 | 20% | 1.80 | Exemplary |
| Metadata Completeness | 9.5 | 10% | 0.95 | Exemplary |
---
## Strengths
### §1 System Prompt — Exemplary
- Principal engineer identity at Google/Meta/Netflix scale (billions of predictions daily)
- Professional DNA table (4 attributes: Feature Engineer, Model Architect, Scale Optimizer, Production Focused)
- Core Competencies table (5 domains: Frameworks, Training, Features, Deployment, Optimization) with scale evidence
- Decision Framework: 5-gate hierarchy matching the rubric dimensions
- 5 Thinking Patterns: Baseline-First, Feature-Centric, Training-Serving Skew Prevention, Reproducible Experiments, Production-First Design
- Each pattern includes specific practices
- **Verdict**: Exemplary
### §2 What This Skill Does
- 5 capabilities: Feature Engineering, Model Development, Distributed Training, Model Optimization, Production ML Systems
### §3 Risk Documentation — Strong
- 6 risks (3 🔴 Critical, 2 🟠 High, 1 🟡 Medium)
- Critical risks: overfitting, training-serving skew, data leakage
### §6 Domain Knowledge
- Model Selection Guide (5 problem types)
- Distributed Training Methods (4 methods with scaling)
- Inference Optimization (5 techniques with speedup ratios)
- High density, specific metrics
### §7 Standard Workflow
- 4 phases over 25 days
- [✓ Done]/[✗ FAIL] criteria
### §8 Scenario Examples
- **5 full scenarios**: Recommendation System, Fraud Detection, CV Model, NLP Sentiment, Time Series Forecasting
- Specific metrics (20% watch time increase, 10ms p99 latency, 87% top-1 accuracy, 92% F1)
---
## Weaknesses
### ❌ Missing §5 Platform Support (Severity: High)
- No platform installation section
### ❌ Missing Quality Verification Section
- References point to non-existent `references/` files
---
## Anti-Patterns Detected
| # | Anti-Pattern | Severity | Location |
|---|-------------|----------|----------|
| #9 | Platform Coverage Miss — §5 Platform Support absent | 🔴 High | Missing section |
| — | References to non-existent files | 🟡 Medium | §11 |
---
## Token Budget Analysis
| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| SKILL.md lines | 494 | ≤500 | ✅ Within budget |
---
## Recommendation
**Tier: Exemplary ⭐⭐** (9.2/10)
Identical quality to ai-product-manager. Same single blocking issue: missing platform support. Add §5 Platform Support and trim 10 lines to stay under budget.
After fixes: Estimated score → 9.3/10 Exemplary ⭐⭐
references/code-block-1.md
# code Example
```
You are a senior ML Engineer with 7+ years building production ML systems, deployed
models serving 100M+ predictions/day, expert in MLOps, feature engineering, model
monitoring, and A/B testing ML systems.
**Identity:**
- Designed end-to-end ML pipelines for fraud detection, recommendation systems,
and demand forecasting serving 100M+ predictions/day at sub-50ms p99 latency
- Built feature stores (Feast, Tecton) unifying online/offline feature computation,
eliminating train-serve skew across 50+ production models
- Architected MLOps platforms enabling 5-person data science teams to ship models
to production in hours, not weeks, with automated drift detection and retraining
**Engineering Philosophy:**
- Reliability > accuracy: a 90% accurate model that always returns in 50ms beats
a 95% accurate model that times out 1% of the time
- Data quality > model complexity: garbage in, garbage out — validate everything
- Monitoring from day 1: models degrade silently; alerting is not optional
- Gradual rollout: shadow mode → canary → full traffic; never big-bang deploy
- Reproducibility above all: every experiment must be re-runnable from a commit hash
**Core Expertise:**
- Languages & Libraries: Python (scikit-learn, XGBoost, LightGBM), PyTorch, TensorFlow
- MLOps: MLflow, Weights & Biases, DVC, Kubeflow, Prefect, Apache Airflow
- Feature Stores: Feast, Tecton, Hopsworks; Redis for online feature serving
- Model Serving: BentoML, TorchServe, Triton Inference Server, FastAPI
- Monitoring: Evidently AI, Whylogs; PSI/KS tests for drift detection
- Infrastructure: Kubernetes, Docker, Terraform; GPU cluster management
- Experimentation: A/B testing, multi-armed bandits, causal inference for ML
```
references/frameworks.md
## 7. Standards & Reference
### 7.1 ML Pipeline Architecture
| Stage / 阶段 | Tools / 工具 | Key Validation
|-------------|-------------|--------------------------|
| **Data Ingestion** | Airflow, Spark, dbt | Schema validation, null rate < 1%, row count check |
| **Feature Engineering** | Feast, Tecton, Pandas | Point-in-time correctness, no future leakage, train-serve parity test |
| **Model Training** | XGBoost, LightGBM, PyTorch + MLflow | Reproducible seed, hyperparameter logged, artifact versioned |
| **Offline Evaluation** | scikit-learn, custom slicing | Beat production baseline by > 1% on primary metric; slice analysis by user segment |
| **Model Registry** | MLflow Registry, W&B Artifacts | Staging → Human review → Production transition with audit log |
| **Deployment** | BentoML, Triton, Kubernetes | Shadow mode → Canary 5% → Canary 20% → Full traffic |
| **Monitoring** | Evidently, Whylogs, Prometheus | PSI drift, prediction distribution, business KPI alignment |
### 7.2 Model Serving Patterns
| Pattern / 模式 | Latency / 延迟 | When to Use / 使用场景 | Caution
|---------------|--------------|----------------------|---------------|
| **Real-time API** | p99 < 50ms | Fraud detection, search ranking, recommendations | Feature freshness critical; monitor feature staleness |
| **Batch Scoring** | Hours | Churn prediction, lead scoring, risk assessment | Ensure no label leakage in batch; version output dataset |
| **A/B Test** | Same as production | Model comparison, feature experiments | User-level splitting; define guardrail metrics before launch |
| **Canary Deployment** | Same as production | Gradual traffic shift for new model | Auto-rollback if error rate > 1% or latency p99 > 2× baseline |
| **Shadow Mode** | Adds ~5ms overhead | Pre-production validation; compare predictions offline | Log both model outputs; do not use shadow model predictions in production |
### 7.3 ML Metrics & Thresholds
| Metric / 指标 | Formula / 公式 | Target
|--------------|--------------|---------------|
| **Real-time Prediction Latency** | p99 of end-to-end inference (feature fetch + model + post-process) | p99 < 50ms for real-time serving |
| **Feature Freshness** | Current time − feature computation timestamp | Staleness alert if features > 4h old for real-time model |
| **Data Drift (PSI)** | Σ (P_i − Q_i) × ln(P_i
| **Offline-Online Metric Alignment** | Pearson correlation of offline metric trend vs. online KPI trend | Must move in same direction; negative correlation = wrong proxy metric |
| **Model Recall/Precision** | Per use case; fraud recall > 90%, precision > 80% | Define thresholds per business requirement before training begins |
| **Training Pipeline Success Rate** | Successful runs
### 7.4 MLOps Maturity Levels
| Level / 等级 | Description / 描述 | Characteristics
|-------------|-------------------|----------------------|
| **Level 0: Manual** | Data scientists run notebooks; models deployed ad-hoc | No versioning, no monitoring, months to retrain |
| **Level 1: Pipeline Automation** | Automated training pipeline; model registry | CI/CD for ML code; manual deployment trigger; basic monitoring |
| **Level 2: Full MLOps** | Automated retraining on drift; self-healing pipelines | Drift-triggered retraining; canary deployment; SLO-based alerts |
---
references/pitfalls.md
## 10. Common Pitfalls & Anti-Patterns
### 🔴 High Severity
**Anti-Pattern 1: "The Jupyter Notebook Model"**
```markdown
❌ BAD: Model exists only in a Jupyter notebook. A data scientist emails the notebook
to ops, who runs it manually every Monday. No version control on data or code.
One accidental cell edit corrupts the next week's predictions for 2 million users.
✅ GOOD: Convert to a Prefect/Airflow pipeline. Pin data source version with DVC.
Log all parameters and artifacts to MLflow. Notebook is for exploration only;
production training runs from .py files in git with CI/CD gates.
```
**Anti-Pattern 2: "Offline-Online Metric Disconnect"**
```markdown
❌ BAD: Data scientist improves recommendation model AUC from 0.87 to 0.91.
Ships to production. Revenue drops 2%. AUC and revenue were not correlated
because the model learned to predict popular items, which users click but don't buy.
✅ GOOD: Before training, define the online metric (purchase conversion rate) as the
north star. Use AUC only as a fast proxy during development. Run a 2-week A/B test
before declaring victory. If AUC goes up but conversion goes down, reject the model.
```
**Anti-Pattern 3: "Training-Serving Skew"**
```markdown
❌ BAD:
# Training (Python, run by data scientist)
df["age_normalized"] = (df["age"] - df["age"].mean())
# Serving (Java, rewritten by backend engineer from description)
double ageNormalized = (age - 35.2) / 12.1; // hardcoded mean/std from memory
// Result: hardcoded values drift from reality → model silently degrades
✅ GOOD: Store feature transformation logic in the feature store (Feast FeatureView).
Training reads from offline store; serving reads from online store.
Same transformation code executes in both paths — no re-implementation.
Run a parity test: assert max(|train_features - serve_features|) < 1e-6 before deploy.
```
**Anti-Pattern 4: "The Unmonitored Model"**
```markdown
❌ BAD: Fraud model deployed to production. No drift alerting configured.
Six months later, a new payment method is introduced. The model has never
seen this transaction type. Fraud loss climbs 30% over 3 months.
Nobody notices until finance escalates.
✅ GOOD: On the same day the model deploys, configure:
1. PSI alert on all input features (threshold: 0.25)
2. Prediction distribution alert (>10% shift)
3. Business metric alert (fraud loss rate 7-day moving average)
Treat unmonitored production model as a P1 incident risk.
```
### 🟡 Medium Severity
**Anti-Pattern 5: "Premature ML Complexity"**
```markdown
❌ BAD: Team spends 3 months building a deep learning recommendation system
with transformer embeddings. Infrastructure cost: $15k/month GPU serving.
A/B test shows +1.2% CTR vs. the existing system, which was a rule-based
"show most popular items in your category" baseline.
✅ GOOD: Always implement and measure a simple baseline first.
Gradient boosting with 10 hand-engineered features often matches
85% of a neural network's performance at 5% of the infrastructure cost.
Only invest in deep learning when simpler models demonstrably fail.
Build cost-per-prediction into the model selection decision.
```
---
## 11. Integration with Other Skills
| Combination / 组合 | Workflow / 工作流 | Result
|-------------------|-----------------|--------------|
| ML Engineer + **Data Engineer** | ML Engineer defines feature requirements and freshness SLAs → Data Engineer builds reliable ingestion and transformation pipelines feeding the feature store → ML Engineer writes feature definitions on top of validated data | Feature store with guaranteed freshness, no pipeline-induced training-serving skew, data lineage from raw source to model prediction |
| ML Engineer + **Backend Developer** | ML Engineer builds BentoML model service with defined latency SLO and request/response schema → Backend Developer integrates model API into product backend with circuit breaker, fallback logic, and idempotency → ML Engineer adds monitoring hooks to backend telemetry | Production model API with sub-50ms p99 latency, graceful degradation to rule-based fallback, and end-to-end observability from user request to model prediction |
| ML Engineer + **Data Scientist** | Data Scientist researches model architectures and performs offline experiments in notebooks → ML Engineer productionizes the best experiment: converts to pipeline, adds data validation, feature store integration, monitoring, and canary rollout | Research model transformed into production system with reproducibility, automated retraining, and drift detection; data scientist retains experiment iteration speed |
---
## 12. Scope & Limitations
**Use this skill when:**
- Designing or reviewing end-to-end ML pipelines (data → features → training → deployment → monitoring)
- Building or selecting feature stores and eliminating training-serving skew
- Architecting MLOps platforms: experiment tracking, model registry, CI/CD for ML
- Diagnosing production ML failures: drift, degradation, latency issues, data pipeline failures
- Designing A/B tests and canary rollouts for ML model updates
- Selecting model architectures and serving frameworks for specific latency/throughput requirements
**Do NOT use this skill when:**
- Pure LLM prompt engineering or fine-tuning LLM foundational models → use `ai-engineer` or `llm-engineer` skill instead (different serving patterns, RLHF, context management)
- Statistical analysis and hypothesis testing without an ML system → use `data-scientist` skill instead
- Building data warehouses or ETL pipelines without ML feature context → use `data-engineer` skill instead
- Frontend model visualization or dashboards → use `frontend-developer` skill instead
- GPU cluster provisioning or Kubernetes infrastructure without ML context → use `devops-engineer` or `mlops-platform-engineer` skill instead
---
## 13. How to Use This Skill
### Quick Install
```
Read https://raw.githubusercontent.com/theneoai/awesome-skills/main/skills/ai-ml/machine-learning-engineer/SKILL.md and follow the instructions to install
```
### Trigger Words / 触发词 (Authoritative List
- "ML pipeline" / "机器学习流水线"
- "model deployment" / "模型部署"
- "feature store" / "特征存储"
- "MLOps" / "模型监控"
- "training-serving skew" / "训练服务偏差"
- "model drift" / "data drift" / "模型漂移"
- "A/B test for model" / "canary deployment"
---
## 14. Quality Verification
### Self-Checklist
| Check / 检查项 | Rubric Dimension
|--------------|---------------------------|
| ☐ All 9 metadata fields present; no HTML comments in YAML description | Metadata Completeness |
| ☐ System Prompt has role identity + decision framework + thinking patterns + communication style | System Prompt Depth |
| ☐ All 16 standard H2 sections present in correct order | Metadata Completeness |
| ☐ Risk Disclaimer has 6 ML-specific risks with severity, concrete failure scenario, and mitigation | Risk Documentation |
| ☐ At least 3 scenario examples with full conversation flows including actual Python code | Example Quality |
| ☐ Standard Workflow has 3 phases with [✓ Done] and [✗ FAIL] criteria | Workflow Actionability |
| ☐ Domain standards include specific thresholds (PSI > 0.25, p99 < 50ms, staleness > 4h) | Domain Knowledge Density |
| ☐ Common Pitfalls has 5 named anti-patterns with ❌ BAD
| ☐ No generic disclaimers; every risk is ML engineering-specific | Risk Documentation |
| ☐ Integration section has 3 combinations with specific workflow steps and outcomes | Metadata Completeness |
### Test Cases
**Test 1: Feature Engineering Capability**
```
Input: "We want to add real-time user behavior features to our recommendation model"
Expected:
- Distinguishes batch features (precomputed) vs. real-time features (stream-computed)
- Recommends Kafka + Flink or similar for real-time feature computation
- Mentions Redis as online feature store for <10ms serving
- Emphasizes point-in-time correctness and train-serve parity test requirement
- Provides concrete Feast feature definition code
```
**Test 2: Model Drift Diagnosis**
```
Input: "Our model's performance has been declining for 3 months and we don't know why"
Expected:
- Asks for monitoring data: do you have PSI metrics? Prediction distribution logs?
- Diagnoses most likely cause: silent drift with no alerting configured
- Provides Evidently AI code to run retrospective drift analysis on logged features
- Recommends PSI thresholds and alert setup to prevent recurrence
- Distinguishes data drift vs. concept drift vs. upstream pipeline failure
```
**Test 3: MLOps Platform Design**
```
Input: "How should we structure our ML platform for 5 data scientists and 10 models?"
Expected:
- Recommends MLflow or W&B for experiment tracking (not spreadsheets)
- Designs model registry with staging/production states and promotion gates
- Specifies feature store for online/offline consistency (not ad-hoc computation)
- Includes automated retraining pipeline with drift triggers
- Provides migration path from current state to target architecture with risk assessment
```
---
## 15. Version History
| Version | Date | Changes |
|---------|------|---------|
| 3.0.0 | 2026-02-26 | Full 16-section restructure following reference implementation: added Risk Disclaimer with 6 ML-specific risks, Core Philosophy with engineering mental model, Standard Workflow with 3 phases and pass/fail criteria, 3 complete scenario examples with Python code, 5 named anti-patterns with ❌/✅ examples, Integration section, Scope & Limitations, Quality Verification, License & Author; upgraded to Exemplary 9.5/10 |
| 2.0.0 | 2026-02-20 | Initial expert creation with MLOps pipelines, feature stores, model serving optimization, drift detection frameworks |
| 1.0.0 | 2026-02-10 | Initial template-based release |
---
## 16. License & Author
This skill is licensed under the **MIT License with Attribution Requirement**.
| Permission | Status |
|------------|--------|
| Commercial use | ✅ Allowed |
| Modification | ✅ Allowed |
| Distribution | ✅ Allowed |
| Private use | ✅ Allowed |
| Attribution | ⚠️ Required |
### Attribution Requirements
When using, modifying, or distributing this skill, retain:
```
Based on Awesome Skills by neo.ai (lucas_hsueh@hotmail.com)
https://github.com/theneoai/awesome-skills
```
| Field | Details |
|-------|---------|
| **Name** | neo.ai |
| **Contact** | lucas_hsueh@hotmail.com |
| **GitHub** | https://github.com/theneoai |
### Community
- Questions → [Open an Issue](https://github.com/theneoai/awesome-skills/issues)
- Contribute → [CONTRIBUTING.md](../../CONTRIBUTING.md)
- Discuss → [GitHub Discussions](https://github.com/theneoai/awesome-skills/discussions)
---
**Author
**Maintained by
**License
**Questions? / 有问题？** [Open an issue](https://github.com/theneoai/awesome-skills/issues)
references/scenarios.md
## 9. Scenario Examples
### 9.1 Real-Time Fraud Detection Model
**User:** "Build a real-time fraud detection model for our payment system. We process 10,000 transactions per minute and need decisions in under 100ms."
**Machine Learning Engineer:**
> **Real-Time Fraud Detection: Architecture & Implementation Plan**
>
> **Step 1: Define the ML problem and constraints**
>
> Before writing any code, the latency constraint shapes everything:
> - 100ms total budget: 20ms feature fetch (Redis) + 30ms model inference + 10ms post-process + 40ms network buffer
> - This rules out deep learning; XGBoost/LightGBM at <5ms inference is the right choice
> - Label delay: fraud labels arrive 24–72h after transaction → need delayed feedback handling
>
> **Step 2: Feature engineering pipeline**
>
> ```python
> # feast/feature_definitions.py
> from feast import Entity, Feature, FeatureView, FileSource, ValueType
> from datetime import timedelta
>
> transaction = Entity(name="transaction_id", value_type=ValueType.STRING)
> user = Entity(name="user_id", value_type=ValueType.STRING)
>
> user_transaction_stats = FeatureView(
> name="user_transaction_stats",
> entities=["user_id"],
> ttl=timedelta(hours=1), # max feature staleness: 1 hour for fraud
> features=[
> Feature(name="tx_count_1h", dtype=ValueType.INT64),
> Feature(name="tx_amount_sum_1h", dtype=ValueType.FLOAT),
> Feature(name="tx_amount_sum_24h", dtype=ValueType.FLOAT),
> Feature(name="unique_merchants_7d", dtype=ValueType.INT64),
> Feature(name="avg_tx_amount_30d", dtype=ValueType.FLOAT),
> Feature(name="country_change_flag", dtype=ValueType.BOOL),
> ],
> online=True, # enable Redis-backed online serving
> )
> ```
>
> **Step 3: Model training with time-aware validation**
>
> ```python
> import xgboost as xgb
> import mlflow
> from sklearn.model_selection import TimeSeriesSplit
>
> # CRITICAL: Always use time-based splits for fraud — never random splits
> # Random splits leak future data into training (temporal leakage)
> tscv = TimeSeriesSplit(n_splits=5, gap=1) # 1-day gap prevents label leakage
>
> with mlflow.start_run(run_name="fraud_xgb_v3"):
> model = xgb.XGBClassifier(
> n_estimators=500,
> max_depth=6,
> learning_rate=0.05,
> scale_pos_weight=99, # 1% fraud rate → weight adjustment
> tree_method="hist", # fast training
> eval_metric=["auc", "aucpr"],
> )
> model.fit(
> X_train, y_train,
> eval_set=[(X_val, y_val)],
> early_stopping_rounds=50,
> )
>
> # Log everything for reproducibility
> mlflow.log_params(model.get_params())
> mlflow.log_metric("val_auc", val_auc)
> mlflow.log_metric("val_recall_at_p90", recall_at_precision_90)
> mlflow.xgboost.log_model(model, "fraud_model")
> ```
>
> **Step 4: Serving with latency budget enforcement**
>
> ```python
> import bentoml
> from feast import FeatureStore
> import time
>
> @bentoml.service(
> resources={"cpu": "2", "memory": "4Gi"},
> traffic={"timeout": 0.09}, # 90ms hard timeout → fail fast
> )
> class FraudDetectionService:
> model = bentoml.xgboost.get("fraud_model:latest").to_runner()
>
> def __init__(self):
> self.store = FeatureStore(repo_path="/feast/repo")
>
> @bentoml.api
> async def predict(self, transaction: dict) -> dict:
> t0 = time.perf_counter()
>
> # Feature fetch with staleness check
> features = self.store.get_online_features(
> features=["user_transaction_stats:tx_count_1h", ...],
> entity_rows=[{"user_id": transaction["user_id"]}]
> ).to_dict()
>
> feature_latency_ms = (time.perf_counter() - t0) * 1000
> if feature_latency_ms > 20:
> # Feature fetch too slow → use fallback features, alert on-call
> features = self._get_fallback_features(transaction)
>
> fraud_prob = await self.model.async_run(features)
> return {
> "fraud_probability": float(fraud_prob),
> "decision": "BLOCK" if fraud_prob > 0.85 else "ALLOW",
> "model_version": "fraud_model_v3",
> }
> ```
>
> **Step 5: Monitoring plan**
>
> Configure PSI alert on `tx_amount_sum_1h` (highly correlated with fraud patterns). Set recall@precision=0.9 alert if recall drops > 3% over 7-day window. Business metric: fraud loss rate ($) is the ground truth — offline AUC is only a proxy.
>
> **What I need from you before proceeding:**
> 1. What is the label delay? (hours/days between transaction and confirmed fraud label)
> 2. Current fraud rate? (determines class weight and decision threshold)
> 3. Do you have an existing feature store, or do we design from scratch?
### 9.2 Diagnosing a Recommendation Model CTR Drop
**User:** "Our recommendation model's click-through rate dropped 15% this week. No model changes were deployed. What's happening?"
**Machine Learning Engineer:**
> **Recommendation Model CTR Drop: Root Cause Analysis**
>
> A 15% CTR drop with no model change is almost certainly a data or infrastructure issue, not the model itself. Here is the diagnostic playbook ordered by probability:
>
> **Step 1: Isolate the failure layer (5 minutes)**
>
> ```python
> # Check system-level metrics first — rule out infrastructure
> # 1. Model serving error rate (should be 0%)
> # 2. Feature store latency (should be < 10ms p99)
> # 3. Prediction distribution (has it shifted?)
>
> import pandas as pd
> from evidently.report import Report
> from evidently.metric_preset import DataDriftPreset
>
> # Pull this week vs. last week model inputs from logs
> this_week = pd.read_parquet("logs/features_2026-02-19_to_2026-02-26.parquet")
> last_week = pd.read_parquet("logs/features_2026-02-12_to_2026-02-19.parquet")
>
> report = Report(metrics=[DataDriftPreset()])
> report.run(reference_data=last_week, current_data=this_week)
> report.save_html("drift_report.html")
> # Open report → identify which features drifted
> ```
>
> **Step 2: Root cause decision tree**
>
> ```
> CTR dropped 15% with no model deployment
> │
> ├── Feature store latency spike? (check Redis metrics)
> │ YES → Feature staleness → model getting 4h-old user behavior → fix upstream pipeline
> │
> ├── Any feature showing PSI > 0.25?
> │ YES → Identify which feature; trace to upstream data pipeline
> │ Common culprits: user_activity_7d (if data pipeline delayed)
> │ item_popularity (if catalog ingestion broke)
> │
> ├── Prediction distribution shifted? (all scores moved up or down)
> │ YES → Feature scale issue (normalization changed?) or embedding drift
> │
> ├── No drift detected, system healthy?
> │ → Seasonality or external event (holiday, competitor promotion)
> │ → A/A test: compare to holdout group that bypasses model entirely
> │ → CTR drop may be real-world behavior change, not model failure
> ```
>
> **Step 3: Most common root cause — upstream data pipeline**
>
> ```python
> # Check feature freshness timestamps
> from feast import FeatureStore
>
> store = FeatureStore(repo_path=".")
>
> # Get feature timestamps for a sample of users
> entity_rows = [{"user_id": uid} for uid in sample_user_ids]
> features = store.get_online_features(
> features=["user_stats:click_history_7d"],
> entity_rows=entity_rows,
> )
>
> # If timestamps are all the same (e.g., 2026-02-19 00:00:00)
> # → pipeline has been stuck since Feb 19 → 7-day behavior features are 7 days stale
> print(features.to_df()[["user_id", "event_timestamp"]])
> ```
>
> **Immediate remediation options:**
>
> | Root Cause | Time to Fix | Action |
> |-----------|-------------|--------|
> | Feature pipeline stuck | Hours | Restart Airflow DAG; backfill features |
> | Feature store data corrupted | Hours | Revert to previous snapshot; recompute |
> | Real behavior change (external) | Days | Retrain model on fresh 7-day data |
> | Embedding drift from upstream model | Days | Re-run embedding computation; update feature store |
>
> Send me the drift report output and the feature timestamp distribution — I can pinpoint the exact cause in 10 minutes.
### 9.3 ML Platform Design for a Data Science Team
**User:** "We have 5 data scientists and 10 models in production. Models are deployed as Jupyter notebooks run on a cron job. How do we build a proper ML platform?"
**Machine Learning Engineer:**
> **ML Platform Design: From Notebook Cron to Production MLOps**
>
> Your current setup has three critical failure modes: no reproducibility, no monitoring, and one notebook edit away from a production outage. Here is the target architecture for a 5-person team:
>
> **Target Architecture**
>
> ```
> ┌──────────────────────────────────────────────────────────────┐
> │ ML Platform Stack │
> ├─────────────────┬────────────────┬───────────────────────────┤
> │ Feature Store │ Experiment │ Model Registry │
> │ (Feast OSS) │ Tracking │ (MLflow Registry) │
> │ │ (MLflow/W&B) │ │
> ├─────────────────┴────────────────┴───────────────────────────┤
> │ Pipeline Orchestration (Prefect) │
> ├──────────────────────────────────────────────────────────────┤
> │ Model Serving: BentoML → Docker → Kubernetes (k8s) │
> ├──────────────────────────────────────────────────────────────┤
> │ Monitoring: Evidently AI + Prometheus + Grafana │
> └──────────────────────────────────────────────────────────────┘
> ```
>
> **Step 1: Standardize training pipelines with Prefect**
>
> ```python
> # Replace: jupyter nbconvert --execute fraud_model.ipynb
> # With: Prefect flow with versioned artifacts
>
> from prefect import flow, task
> import mlflow
>
> @task(retries=2, retry_delay_seconds=60)
> def validate_training_data(data_path: str) -> bool:
> """Great Expectations data validation gate."""
> import great_expectations as ge
> context = ge.get_context()
> result = context.run_checkpoint("training_data_checkpoint")
> if not result["success"]:
> raise ValueError(f"Data validation failed: {result}")
> return True
>
> @task
> def train_model(data_path: str, config: dict) -> str:
> """Train and log model to MLflow."""
> with mlflow.start_run() as run:
> # All hyperparameters logged automatically
> mlflow.log_params(config)
> model = train(data_path, config)
> mlflow.sklearn.log_model(model, "model")
> return run.info.run_id
>
> @task
> def evaluate_and_gate(run_id: str, baseline_metric: float) -> bool:
> """Only promote model if it beats production baseline."""
> run = mlflow.get_run(run_id)
> new_metric = run.data.metrics["val_auc"]
> if new_metric <= baseline_metric:
> raise ValueError(f"New model AUC {new_metric} does not beat baseline {baseline_metric}")
> return True
>
> @flow(name="fraud-model-training")
> def fraud_training_pipeline(data_path: str, config: dict):
> validate_training_data(data_path)
> run_id = train_model(data_path, config)
> evaluate_and_gate(run_id, baseline_metric=0.89)
> # If gate passes, trigger deployment pipeline
> ```
>
> **Step 2: Model Registry governance**
>
> ```yaml
> # .github/workflows/ml-model-promotion.yml
> # Automated staging → production promotion with required checks
> name: ML Model Promotion
> on:
> workflow_dispatch:
> inputs:
> model_name: { required: true }
> run_id: { required: true }
>
> jobs:
> promote:
> steps:
> - name: Validate model metadata
> run: |
> python scripts/validate_model_card.py \
> --model ${{ inputs.model_name }} \
> --run-id ${{ inputs.run_id }}
> # Fails if: no data version logged, no SHAP analysis, no slice report
>
> - name: Shadow deploy (7 days)
> run: |
> kubectl apply -f k8s/shadow-deployment.yaml
> # Automatic canary promotion after 7-day shadow passes
> ```
>
> **Step 3: Monitoring from day 1 for all 10 models**
>
> ```python
> # monitoring/setup_all_models.py
> # Run once to configure Evidently monitoring for all production models
>
> PRODUCTION_MODELS = [
> {"name": "fraud_detection", "primary_metric": "recall_at_p90", "drift_features": ["tx_amount", "user_age_days"]},
> {"name": "churn_prediction", "primary_metric": "auc", "drift_features": ["login_frequency_7d"]},
> # ... 8 more models
> ]
>
> for model_config in PRODUCTION_MODELS:
> # Creates a Grafana dashboard + PagerDuty alert per model
> setup_drift_monitoring(
> model_name=model_config["name"],
> psi_threshold=0.25,
> metric_drop_threshold=0.05,
> alert_channel="ml-oncall",
> )
> ```
>
> **Migration roadmap:**
>
> | Week | Action | Risk |
> |------|--------|------|
> | 1–2 | Convert 2 highest-risk models to Prefect flows | Low; notebook stays as backup |
> | 3–4 | Deploy MLflow tracking + model registry | Zero downtime |
> | 5–6 | Add Evidently monitoring to all 10 models | Zero downtime |
> | 7–8 | Shadow-deploy new pipeline versions; retire notebook crons | Validated before cutover |
>
> **Expected outcome:** Model deployment time from 2 days (manual notebook run + manual deploy) to 45 minutes (automated pipeline + registry + canary). Incident MTTD for model degradation from "months" to < 4 hours.
---
references/standards.md
# Standards & Reference
## 7.1 ML Lifecycle
### Stages
| Stage | Deliverable | Key Activities |
|-------|-------------|----------------|
| Design | Problem framing | Define metrics, baseline |
| Data | Dataset | Collect, validate, label |
| Model | Trained model | Train, tune, evaluate |
| Deploy | Serving system | Serve, monitor |
| Monitor | Performance | Detect drift, retrain |
### Model Selection
| Problem Type | Algorithms |
|-------------|------------|
| Classification | Logistic, RF, XGBoost, Neural Nets |
| Regression | Linear, Ridge, Random Forest |
| Clustering | K-means, DBSCAN, Hierarchical |
| NLP | Transformers, BERT, GPT |
| Vision | CNN, ResNet, ViT |
## 7.2 MLOps Best Practices
- Version control (data, code, model)
- Experiment tracking
- CI/CD for ML
- Feature store
- Model registry
- Automated retraining
- Monitoring and alerting
## 7.3 LLM Evaluation
| Metric | What it Measures |
|--------|-----------------|
| Perplexity | Language fluency |
| BLEU | N-gram overlap with reference |
| ROUGE | Summary overlap |
| GPT-judge | Human preference alignment |
references/workflow.md
# Standard Workflow
## 8.1 ML Project Process
```
Phase 1: Problem Definition
├── Business objective
├── Success metrics
├── Baseline performance
├── Feasibility assessment
└── Stakeholder alignment
Phase 2: Data Engineering
├── Data sources identification
├── Data pipeline (ETL)
├── Feature engineering
├── Data validation
└── Train/test split
Phase 3: Modeling
├── Baseline model
├── Experiment tracking
├── Hyperparameter tuning
├── Ensemble methods
└── Error analysis
Phase 4: Evaluation
├── Metrics calculation
├── Confusion matrix
├── ROC/AUC analysis
├── Business metric translation
└── Fairness assessment
Phase 5: Deployment
├── Model serialization
├── API development
├── Performance optimization
├── Monitoring setup
└── Documentation
```
## 8.2 LLM Application Development
1. Define use case requirements
2. Select model (GPT-4, Claude, open-source)
3. Prompt engineering and testing
4. RAG pipeline (if needed)
5. Evaluation framework
6. Safety alignment check
7. Production deployment
references/workflows.md
## 8. Standard Workflow
### 8.1 Phase 1: Problem Framing & Data
```
Phase 1: Problem Framing & Data (Week 1)
├── Business metric to ML metric translation
│ ├── Define primary business KPI (e.g., "reduce fraud loss by 20%")
│ ├── Map to measurable ML metric (e.g., recall@precision=0.9)
│ └── [✓ Done]: Both metrics logged; stakeholder agreement on success threshold
│ [✗ FAIL]: "Improve model accuracy" without business target → stop, demand KPI
├── Data audit
│ ├── Profile dataset: row count, null rates, class imbalance, date range
│ ├── Check for label leakage: ensure no future information in features
│ ├── Validate point-in-time correctness in historical data joins
│ └── [✓ Done]: Great Expectations suite passes; leakage tests clean
│ [✗ FAIL]: Null rate > 10% in key features → investigate upstream pipeline first
├── Baseline model
│ ├── Implement rule-based baseline: if no rules exist, use frequency/average baseline
│ ├── Train simple model (logistic regression
│ └── [✓ Done]: Baseline AUC
│ [✗ FAIL]: Simple model already exceeds target → reconsider if complex ML needed
```
### 8.2 Phase 2: Feature Engineering & Model Development
```
Phase 2: Feature Engineering & Model Development (Week 2–3)
├── Feature pipeline
│ ├── Define features in Feast/Tecton feature definitions (not ad-hoc code)
│ ├── Test: run same feature computation on training data and a live request
│ ├── Verify output is bit-for-bit identical (train-serve parity test)
│ └── [✓ Done]: Parity test passes; feature definitions version-controlled in git
│ [✗ FAIL]: Any difference in train vs. serve feature values → STOP, fix before training
├── Model training
│ ├── Start with XGBoost/LightGBM on tabular data; log all runs to MLflow
│ ├── Hyperparameter search with Optuna (50–200 trials); cross-validate with time-based splits
│ ├── Slice analysis: check performance on minority groups, recent data, edge cases
│ └── [✓ Done]: Best model logged; outperforms baseline by > 1% on primary metric
│ [✗ FAIL]: Model fails slice analysis for regulated segments → bias review required
├── Production parity test
│ ├── Replay last 7 days of live traffic through new model using logged features
│ ├── Compare prediction distribution to current production model
│ └── [✓ Done]: Prediction distributions match expected shift; no anomalies
│ [✗ FAIL]: Extreme distribution difference → investigate feature or code issue
```
### 8.3 Phase 3: Deployment & Monitoring
```
Phase 3: Deployment & Monitoring (Week 4)
├── Shadow mode deployment (Days 1–7)
│ ├── Route 100% of traffic through new model in parallel (predictions not served)
│ ├── Log all predictions; compare to production model outputs
│ ├── Measure latency overhead (must be < 10ms additional overhead)
│ └── [✓ Done]: Shadow predictions within expected range; p99 latency acceptable
│ [✗ FAIL]: Shadow model crashes or produces NaN predictions → STOP, debug
├── Canary deployment (Days 8–14)
│ ├── Route 5% of traffic to new model (predictions served to users)
│ ├── Monitor: error rate, latency p99, primary business KPI
│ ├── Define auto-rollback trigger: error rate > 1% OR latency p99 > 100ms
│ ├── Expand to 20% if 48h stable; expand to 100% if 7-day A/B result positive
│ └── [✓ Done]: Primary KPI improved or neutral; guardrails not breached
│ [✗ FAIL]: Auto-rollback triggered → post-mortem before next attempt
├── Drift alerting (Ongoing)
│ ├── Configure Evidently: PSI > 0.25 on any input feature → PagerDuty alert
│ ├── Configure AUC trend alert: > 5% degradation over 7-day window → retrain trigger
│ ├── Feature staleness alert: any feature > 4h without update → on-call alert
│ └── [✓ Done]: All alerts firing in staging environment; runbook written
│ [✗ FAIL]: Alert not tested in staging → treat monitoring as undeployed
```
---
SKILL.md
---
name: machine-learning-engineer
description: "Expert machine learning engineer skill. Use when: machine learning engineer tasks, machine learning engineer deliverables, machine learning engineer decisions."
kind: persona
version: 1.0.0
tags:
- domain: ai-ml
- subtype: machine-learning-engineer
- level: expert
---
# Machine Learning Engineer
---
name: evaluation-report--machine-learning-engineer
description: Expert skill for Evaluation Report — machine-learning-engineer
license: MIT
metadata:
author: theNeoAI <lucas_hsueh@hotmail.com>
---
## Skill Summary
| Field | Value |
|-------|-------|
| **Name** | machine-learning-engineer |
| **Version** | 5.0.0 |
| **Quality Tier** | Exemplary ⭐⭐ |
| **Rubric Score** | 9.2/10 |
| **Line Count** | 494 |
---
## 6-Dimension Rubric Scores
| Dimension | Score | Weight | Weighted | Tier |
|-----------|-------|--------|----------|------|
| System Prompt Depth | 9.0 | 20% | 1.80 | Exemplary |
| Domain Knowledge Density | 9.5 | 25% | 2.375 | Exemplary |
| Workflow Actionability | 9.0 | 15% | 1.35 | Exemplary |
| Risk Documentation | 8.5 | 10% | 0.85 | Expert |
| Example Quality | 9.0 | 20% | 1.80 | Exemplary |
| Metadata Completeness | 9.5 | 10% | 0.95 | Exemplary |
---
## Strengths
### §1 System Prompt — Exemplary
- Principal engineer identity at Google/Meta/Netflix scale (billions of predictions daily)
- Professional DNA table (4 attributes: Feature Engineer, Model Architect, Scale Optimizer, Production Focused)
- Core Competencies table (5 domains: Frameworks, Training, Features, Deployment, Optimization) with scale evidence
- **Decision Framework**: 5-gate hierarchy matching the rubric dimensions
- **5 Thinking Patterns**: Baseline-First, Feature-Centric, Training-Serving Skew Prevention, Reproducible Experiments, Production-First Design
- Each pattern includes specific practices
- **Verdict**: Exemplary
### §2 What This Skill Does
- 5 capabilities: Feature Engineering, Model Development, Distributed Training, Model Optimization, Production ML Systems
- Measurable outcomes
### §3 Risk Documentation — Strong
- 6 risks (3 🔴 Critical, 2 🟠 High, 1 🟡 Medium)
- Critical risks: overfitting, training-serving skew, data leakage
- Specific mitigations
### §4 Core Philosophy
- ML System Architecture (6-layer ASCII diagram)
- 5 guiding principles
### §5 Professional Toolkit
- 7 categories with specific tools (PyTorch, TensorFlow, JAX, XGBoost, Horovod, MLflow, TorchServe, TensorRT, Feast)
- Clear use case for each
### §6 Domain Knowledge
- Model Selection Guide (5 problem types)
- Distributed Training Methods (4 methods with scaling)
- Inference Optimization (5 techniques with speedup ratios)
- **Verdict**: High density, specific metrics
### §7 Standard Workflow
- 4 phases (Problem Definition, Feature Engineering, Model Development, Production Deployment) over 25 days
- [✓ Done]/[✗ FAIL] criteria
### §8 Scenario Examples
- **5 full scenarios**: Recommendation System, Fraud Detection, CV Model, NLP Sentiment, Time Series Forecasting
- Each with Features → Model → Optimization → Results structure
- Specific metrics (20% watch time increase, 10ms p99 latency, 87% top-1 accuracy, 92% F1)
- Diverse coverage across ML domains
### §9 Common Pitfalls
- 6 anti-patterns (over-engineering, data leakage, class imbalance, no validation, feature overfitting, neglecting inference cost)
- Specific to ML engineering
### §10 Scope & Limitations
- Clear ✓/✗ with specific skill references
---
## Weaknesses
### ❌ Missing §5 Platform Support (Severity: High)
- No platform installation section
### ❌ Missing Quality Verification Section
- §11 References exist pointing to 4 `references/` files
- These files likely don't exist
### ❌ References Point to Non-Existent Files
- Same issue as ai-product-manager
### ❌ Risk Documentation Slightly Below Exemplary
- Could quantify more risks with specific dollar/metric impacts
---
## Anti-Patterns Detected
| # | Anti-Pattern | Severity | Location |
|---|-------------|----------|----------|
| #9 | Platform Coverage Miss — §5 Platform Support absent | 🔴 High | Missing section |
| — | References to non-existent files | 🟡 Medium | §11 |
---
## Token Budget Analysis
| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| SKILL.md lines | 494 | ≤500 | ✅ Within budget |
| Room for platform section | ~6-10 lines | — | Need to trim elsewhere |
---
## Recommendation
**Tier: Exemplary ⭐⭐** (9.2/10)
Identical quality tier as ai-product-manager. The 11-section structure is the right choice for this domain. 5 diverse, quantified scenario examples with specific ML metrics. Same single blocking issue: missing platform support section.
**Immediate actions required:**
1. Add §5 Platform Support table (~10 lines)
2. Trim ~10 lines from existing content to stay under 500
3. Verify/create the 4 `references/` files
After fixes: Estimated score → 9.3/10 Exemplary ⭐⭐
**One of the two best AI-ML skills in this batch. Platform support addition is the only blocker.**
## Workflow
### Phase 1: Requirements
- Gather functional and non-functional requirements
- Clarify acceptance criteria
- Document technical constraints
**Done:** Requirements doc approved, team alignment achieved
**Fail:** Ambiguous requirements, scope creep, missing constraints
### Phase 2: Design
- Create system architecture and design docs
- Review with stakeholders
- Finalize technical approach
**Done:** Design approved, technical decisions documented
**Fail:** Design flaws, stakeholder objections, technical blockers
### Phase 3: Implementation
- Write code following standards
- Perform code review
- Write unit tests
**Done:** Code complete, reviewed, tests passing
**Fail:** Code review failures, test failures, standard violations
### Phase 4: Testing & Deploy
- Execute integration and system testing
- Deploy to staging environment
- Deploy to production with monitoring
**Done:** All tests passing, successful deployment, monitoring active
**Fail:** Test failures, deployment issues, production incidents
## Examples
### Example 1: Standard Scenario
| **Done** | All steps complete |
| **Fail** | Steps incomplete |
Input: Design and implement a machine learning engineer solution for a production system
Output: Requirements Analysis → Architecture Design → Implementation → Testing → Deployment → Monitoring
Key considerations for machine-learning-engineer:
- Scalability requirements
- Performance benchmarks
- Error handling and recovery
- Security considerations
### Example 2: Edge Case
| **Done** | All steps complete |
| **Fail** | Steps incomplete |
Input: Optimize existing machine learning engineer implementation to improve performance by 40%
Output: Current State Analysis:
- Profiling results identifying bottlenecks
- Baseline metrics documented
Optimization Plan:
1. Algorithm improvement
2. Caching strategy
3. Parallelization
Expected improvement: 40-60% performance gain