references/preprocessing.md
---
author: Joseph OBrien
status: unpublished
updated: '2025-12-23'
version: 1.0.1
tag: skill
type: reference
parent: machine-learning
---
# Data Preprocessing Patterns
## Missing Value Strategies
### Numerical Features
```python
# Simple imputation
df['col'].fillna(df['col'].median(), inplace=True)
# Model-based imputation
from sklearn.impute import KNNImputer
imputer = KNNImputer(n_neighbors=5)
df_imputed = imputer.fit_transform(df)
```
### Categorical Features
```python
# Mode imputation
df['col'].fillna(df['col'].mode()[0], inplace=True)
# New category for missing
df['col'].fillna('MISSING', inplace=True)
```
## Feature Scaling
```python
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
# Standard scaling (mean=0, std=1) - use for most models
scaler = StandardScaler()
# MinMax scaling [0,1] - use for neural networks
scaler = MinMaxScaler()
# Robust scaling - use when outliers present
scaler = RobustScaler()
X_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
```
## Categorical Encoding
```python
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
import category_encoders as ce
# One-hot encoding (low cardinality)
encoder = OneHotEncoder(sparse=False, handle_unknown='ignore')
# Target encoding (high cardinality)
encoder = ce.TargetEncoder(cols=['category_col'])
# Frequency encoding
df['col_freq'] = df['col'].map(df['col'].value_counts(normalize=True))
```
## Feature Engineering Examples
### Temporal Features
```python
df['hour'] = df['timestamp'].dt.hour
df['day_of_week'] = df['timestamp'].dt.dayofweek
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int)
df['month'] = df['timestamp'].dt.month
# Cyclical encoding
df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)
```
### Numerical Transformations
```python
# Log transform (right-skewed data)
df['col_log'] = np.log1p(df['col'])
# Box-Cox transform
from scipy.stats import boxcox
df['col_bc'], lambda_param = boxcox(df['col'] + 1)
# Polynomial features
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)
```
### Interaction Features
```python
# Manual interactions
df['feature_interaction'] = df['feat1'] * df['feat2']
df['feature_ratio'] = df['feat1'] / (df['feat2'] + 1e-8)
# Automated interactions
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, interaction_only=True)
```
## Outlier Handling
```python
# IQR method
Q1 = df['col'].quantile(0.25)
Q3 = df['col'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
df_clean = df[(df['col'] >= lower_bound) & (df['col'] <= upper_bound)]
# Clipping
df['col_clipped'] = df['col'].clip(lower=lower_bound, upper=upper_bound)
# Z-score method
from scipy import stats
z_scores = np.abs(stats.zscore(df['col']))
df_clean = df[z_scores < 3]
```
## Data Leakage Prevention
**Common Leakage Sources:**
1. Future information in features
2. Target information in features
3. Train-test contamination during preprocessing
**Prevention:**
```python
# Always fit on train, transform on test
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Use pipelines
from sklearn.pipeline import Pipeline
pipeline = Pipeline([
('scaler', StandardScaler()),
('model', LogisticRegression())
])
pipeline.fit(X_train, y_train)
```
SKILL.md
---
name: machine-learning
description: Machine learning development patterns, model training, evaluation, and
deployment. Use when building ML pipelines, training models, feature engineering,
model evaluation, or deploying ML systems to production.
author: Joseph OBrien
status: unpublished
updated: '2025-12-23'
version: 1.0.1
tag: skill
type: skill
---
# Machine Learning
Comprehensive machine learning skill covering the full ML lifecycle from experimentation to production deployment.
## When to Use This Skill
- Building machine learning pipelines
- Feature engineering and data preprocessing
- Model training, evaluation, and selection
- Hyperparameter tuning and optimization
- Model deployment and serving
- ML experiment tracking and versioning
- Production ML monitoring and maintenance
## ML Development Lifecycle
### 1. Problem Definition
**Classification Types:**
- Binary classification (spam/not spam)
- Multi-class classification (image categories)
- Multi-label classification (document tags)
- Regression (price prediction)
- Clustering (customer segmentation)
- Ranking (search results)
- Anomaly detection (fraud detection)
**Success Metrics by Problem Type:**
| Problem Type | Primary Metrics | Secondary Metrics |
|--------------|-----------------|-------------------|
| Binary Classification | AUC-ROC, F1 | Precision, Recall, PR-AUC |
| Multi-class | Macro F1, Accuracy | Per-class metrics |
| Regression | RMSE, MAE | R², MAPE |
| Ranking | NDCG, MAP | MRR |
| Clustering | Silhouette, Calinski-Harabasz | Davies-Bouldin |
### 2. Data Preparation
**Data Quality Checks:**
- Missing value analysis and imputation strategies
- Outlier detection and handling
- Data type validation
- Distribution analysis
- Target leakage detection
**Feature Engineering Patterns:**
- Numerical: scaling, binning, log transforms, polynomial features
- Categorical: one-hot, target encoding, frequency encoding, embeddings
- Temporal: lag features, rolling statistics, cyclical encoding
- Text: TF-IDF, word embeddings, transformer embeddings
- Geospatial: distance features, clustering, grid encoding
**Train/Test Split Strategies:**
- Random split (standard)
- Stratified split (imbalanced classes)
- Time-based split (temporal data)
- Group split (prevent data leakage)
- K-fold cross-validation
### 3. Model Selection
**Algorithm Selection Guide:**
| Data Size | Problem | Recommended Models |
|-----------|---------|-------------------|
| Small (<10K) | Classification | Logistic Regression, SVM, Random Forest |
| Small (<10K) | Regression | Linear Regression, Ridge, SVR |
| Medium (10K-1M) | Classification | XGBoost, LightGBM, Neural Networks |
| Medium (10K-1M) | Regression | XGBoost, LightGBM, Neural Networks |
| Large (>1M) | Any | Deep Learning, Distributed training |
| Tabular | Any | Gradient Boosting (XGBoost, LightGBM, CatBoost) |
| Images | Classification | CNN, ResNet, EfficientNet, Vision Transformers |
| Text | NLP | Transformers (BERT, RoBERTa, GPT) |
| Sequential | Time Series | LSTM, Transformer, Prophet |
### 4. Model Training
**Hyperparameter Tuning:**
- Grid Search: exhaustive, good for small spaces
- Random Search: efficient, good for large spaces
- Bayesian Optimization: smart exploration (Optuna, Hyperopt)
- Early stopping: prevent overfitting
**Common Hyperparameters:**
| Model | Key Parameters |
|-------|---------------|
| XGBoost | learning_rate, max_depth, n_estimators, subsample |
| LightGBM | num_leaves, learning_rate, n_estimators, feature_fraction |
| Random Forest | n_estimators, max_depth, min_samples_split |
| Neural Networks | learning_rate, batch_size, layers, dropout |
### 5. Model Evaluation
**Evaluation Best Practices:**
- Always use held-out test set for final evaluation
- Use cross-validation during development
- Check for overfitting (train vs validation gap)
- Evaluate on multiple metrics
- Analyze errors qualitatively
**Handling Imbalanced Data:**
- Resampling: SMOTE, undersampling
- Class weights: weighted loss functions
- Threshold tuning: optimize decision threshold
- Evaluation: use PR-AUC over ROC-AUC
### 6. Production Deployment
**Model Serving Patterns:**
- REST API (Flask, FastAPI, TF Serving)
- Batch inference (scheduled jobs)
- Streaming (real-time predictions)
- Edge deployment (mobile, IoT)
**Production Considerations:**
- Latency requirements (p50, p95, p99)
- Throughput (requests per second)
- Model size and memory footprint
- Fallback strategies
- A/B testing framework
### 7. Monitoring & Maintenance
**What to Monitor:**
- Prediction latency
- Input feature distributions (data drift)
- Prediction distributions (concept drift)
- Model performance metrics
- Error rates and types
**Retraining Triggers:**
- Performance degradation below threshold
- Significant data drift detected
- Scheduled retraining (daily, weekly)
- New training data available
## MLOps Best Practices
### Experiment Tracking
Track for every experiment:
- Code version (git commit)
- Data version (hash or version ID)
- Hyperparameters
- Metrics (train, validation, test)
- Model artifacts
- Environment (packages, versions)
### Model Versioning
```
models/
├── model_v1.0.0/
│ ├── model.pkl
│ ├── metadata.json
│ ├── requirements.txt
│ └── metrics.json
├── model_v1.1.0/
└── model_v2.0.0/
```
### CI/CD for ML
1. **Continuous Integration:**
- Data validation tests
- Model training tests
- Performance regression tests
2. **Continuous Deployment:**
- Staging environment validation
- Shadow mode testing
- Gradual rollout (canary)
- Automatic rollback
## Reference Files
For detailed patterns and code examples, load reference files as needed:
- **`references/preprocessing.md`** - Data preprocessing patterns and feature engineering techniques
- **`references/model_patterns.md`** - Model architecture patterns and implementation examples
- **`references/evaluation.md`** - Comprehensive evaluation strategies and metrics
## Integration with Other Skills
- **performance** - For optimizing inference latency
- **testing** - For ML-specific testing patterns
- **database-optimization** - For feature store queries
- **debugging** - For model debugging and error analysis