batch-inference-jobs/non-template/SKILL.md
---
name: batch-inference-non-template
description: "Batch inference on media files (images, audio, video) using InputSpec for explicit file-to-bytes conversion. Use for Whisper, ViT, and custom image/audio models."
parent_skill: batch-inference-jobs
---
# Batch Inference Jobs: Media Files
Run batch inference on images, audio, video, and other binary files stored in Snowflake stages. This approach uses `InputSpec` with `column_handling` to explicitly convert stage file paths to raw bytes.
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
---
## When to Use
- Model expects raw bytes as input (not chat messages)
- Using models like Whisper, ViT, ResNet, CLIP, custom image/audio models
- You need explicit control over the file-to-bytes conversion
- Model was logged with a signature expecting binary input columns
## The Problem: Stage Paths vs Model Signatures
Your input DataFrame contains **string paths** pointing to files on a stage:
```
┌─────────────────────────────────────────┐
│ IMAGE_PATH (string) │
├─────────────────────────────────────────┤
│ @MY_DB.MY_SCHEMA.IMAGES/cat.jpg │
│ @MY_DB.MY_SCHEMA.IMAGES/dog.jpg │
└─────────────────────────────────────────┘
```
But models expect **actual file contents** (raw bytes, tensors, etc.)—not path strings.
## The Solution: InputSpec with column_handling
The `InputSpec` with `column_handling` tells batch inference how to convert each column from stage paths to the format the model expects:
```python
from snowflake.ml.model.batch import InputSpec, InputFormat, FileEncoding
input_spec = InputSpec(
column_handling={
"IMAGE_PATH": {
"input_format": InputFormat.FULL_STAGE_PATH,
"convert_to": FileEncoding.RAW_BYTES,
}
}
)
```
**What happens at runtime:**
```
Stage Path String → InputSpec Conversion → Model Input
"@DB.SCHEMA.STAGE/cat.jpg" → Read file from stage → b'\xff\xd8\xff\xe0...' (raw bytes)
```
## Step 1: Check Model Signature
**⚠️ CRITICAL:** Before configuring `column_handling`, always check the model's function signature to understand what input columns and types the model expects.
```sql
-- First verify the model exists
SHOW MODELS LIKE '<MODEL_NAME>' IN SCHEMA <DATABASE>.<SCHEMA>;
-- Then get version/signature details (only run after confirming model exists above — errors if model not found)
SHOW VERSIONS IN MODEL <DATABASE>.<SCHEMA>.<MODEL_NAME>;
```
Look at the `signatures` section in the `model_spec` output. Pay attention to:
1. **Input column names** - Must match your DataFrame column names OR be mapped via `column_handling`
2. **Input types** - Determines what `FileEncoding` to use:
- `BYTES` → Use `FileEncoding.RAW_BYTES`
**Example signature analysis:**
```yaml
signatures:
predict:
inputs:
- name: image
type: BYTES # Expects raw bytes
- name: prompt
type: STRING # Regular string column
outputs:
- name: text
type: STRING
```
**Column name mapping:**
If your DataFrame column name differs from the model's expected input name, the `column_handling` key should match your **DataFrame column name**, and the conversion will feed into the model's expected input:
```python
# DataFrame has "IMAGE_PATH", model expects "image" input
# First rename column to match model signature
input_df = input_df.with_column_renamed("IMAGE_PATH", "IMAGE")
# Use DataFrame column name in column_handling
input_spec = InputSpec(
column_handling={
"IMAGE": { # Matches the renamed DataFrame column and model input name
"input_format": InputFormat.FULL_STAGE_PATH,
"convert_to": FileEncoding.RAW_BYTES,
}
}
)
```
**Checking for additional parameters:**
Also check if the signature has `params`:
```yaml
signatures:
predict:
inputs:
- name: audio
type: BYTES
params:
- name: language
type: STRING
- name: task
type: STRING
```
If `params` is not empty, pass those via `InputSpec(params={...})`:
```python
input_spec = InputSpec(
column_handling={
"AUDIO": {
"input_format": InputFormat.FULL_STAGE_PATH,
"convert_to": FileEncoding.RAW_BYTES,
}
},
params={"language": "en", "task": "transcribe"}
)
```
## FileEncoding Options
| FileEncoding | Output Type | Use For |
|--------------|-------------|---------|
| `RAW_BYTES` | `bytes` | Most models (images, audio, binary files) |
| `BASE64` | `str` (base64 encoded) | Models expecting base64 strings |
| `BASE64_DATA_URL` | `str` (data URL format) | Models expecting `data:image/jpeg;base64,...` |
## Complete Example: Image Classification
```python
from snowflake.snowpark import Session
from snowflake.ml.registry import Registry
from snowflake.ml.model.batch import (
JobSpec, OutputSpec, SaveMode,
InputSpec, InputFormat, FileEncoding,
)
from snowflake.ml.utils.stage_file import list_stage_files
session = Session.builder.config("connection_name", "<CONNECTION>").create()
reg = Registry(session=session)
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
# List image files from stage
input_df = list_stage_files(
session,
"@<DATABASE>.<SCHEMA>.<STAGE>/images",
pattern=".*\\.jpg", # for jpg files, adjust as needed
column_name="IMAGES"
)
output_location = "@<DATABASE>.<SCHEMA>.<STAGE>/output/"
job = mv.run_batch(
X=input_df,
compute_pool="<COMPUTE_POOL>",
output_spec=OutputSpec(
stage_location=output_location,
mode=SaveMode.OVERWRITE,
),
input_spec=InputSpec(
column_handling={
"IMAGES": {
"input_format": InputFormat.FULL_STAGE_PATH,
"convert_to": FileEncoding.RAW_BYTES,
}
}
),
)
job.wait()
# Read results
results_df = session.read.option("pattern", ".*\\.parquet").parquet(output_location)
results_df.show()
```
## Complete Example: Audio Transcription (Whisper)
```python
from snowflake.ml.model.batch import (
OutputSpec, SaveMode,
InputSpec, InputFormat, FileEncoding,
)
from snowflake.ml.utils.stage_file import list_stage_files
# List audio files from stage
input_df = list_stage_files(
session,
"@<DATABASE>.<SCHEMA>.<STAGE>/audio",
pattern=".*\\.wav",
column_name="AUDIO"
)
output_location = "@<DATABASE>.<SCHEMA>.<STAGE>/output/"
job = mv.run_batch(
X=input_df,
compute_pool="<GPU_COMPUTE_POOL>",
output_spec=OutputSpec(
stage_location=output_location,
mode=SaveMode.OVERWRITE,
),
input_spec=InputSpec(
column_handling={
"AUDIO": {
"input_format": InputFormat.FULL_STAGE_PATH,
"convert_to": FileEncoding.RAW_BYTES,
}
}
),
)
job.wait()
# Read transcription results
results_df = session.read.option("pattern", ".*\\.parquet").parquet(output_location)
results_df.show()
# Output includes "outputs" column with {"text": "transcribed text..."}
```
## Multiple File Columns
If your model takes multiple file inputs (e.g., image + audio), configure each column:
```python
input_spec = InputSpec(
column_handling={
"IMAGE_PATH": {
"input_format": InputFormat.FULL_STAGE_PATH,
"convert_to": FileEncoding.RAW_BYTES,
},
"AUDIO_PATH": {
"input_format": InputFormat.FULL_STAGE_PATH,
"convert_to": FileEncoding.RAW_BYTES,
}
}
)
```
## Mixed Columns (Files + Tabular)
You can combine file columns with regular tabular columns. Only specify file columns in `column_handling`—other columns pass through unchanged:
```python
# DataFrame with both file paths and metadata
input_df = session.create_dataframe([
["@STAGE/img1.jpg", "outdoor", 0.8],
["@STAGE/img2.jpg", "indoor", 0.6],
], schema=["IMAGE_PATH", "CATEGORY", "CONFIDENCE"])
# Only configure the file column
input_spec = InputSpec(
column_handling={
"IMAGE_PATH": {
"input_format": InputFormat.FULL_STAGE_PATH,
"convert_to": FileEncoding.RAW_BYTES,
}
# CATEGORY and CONFIDENCE pass through as-is
}
)
```
## Verifying Your Setup
Before running a large batch job, test with a small sample:
```python
# Test with 2-3 files first
test_df = input_df.limit(3)
test_job = mv.run_batch(
X=test_df,
compute_pool="<COMPUTE_POOL>",
output_spec=OutputSpec(
stage_location="@<DATABASE>.<SCHEMA>.<STAGE>/test/",
mode=SaveMode.OVERWRITE,
),
input_spec=input_spec,
)
test_job.wait()
# Check results
results = session.read.option("pattern", ".*\\.parquet").parquet("@<DATABASE>.<SCHEMA>.<STAGE>/test/")
results.show()
```
## Common Mistakes
| Mistake | Error | Fix |
|---------|-------|-----|
| Missing `@` prefix | `File not found` | Use `@DB.SCHEMA.STAGE/file.ext` |
| Column name mismatch | `KeyError` or silent failure | Ensure `column_handling` key matches DataFrame column exactly |
| Wrong InputFormat | `Invalid path format` | Use `FULL_STAGE_PATH` for complete paths |
| Forgetting InputSpec | Model receives string instead of bytes | Always include `input_spec` for file columns |
## Troubleshooting
**Job fails with "file not found"**
- Verify the stage path is correct
- Ensure paths include the `@` prefix
**Model receives wrong data type**
- Check that `column_handling` column name matches your DataFrame column exactly
- Verify the model expects `RAW_BYTES` vs `BASE64`
**Empty or incorrect results**
- Test with a single file first
- Check model's expected input signature
## Stopping Points
- ✋ After test job completes, verify results before running full batch
## Output
- Batch inference job completed
- Parquet files with predictions (original input columns + model output columns)
batch-inference-jobs/SKILL.md
---
name: batch-inference-jobs
description: "Run batch inference on models in Snowflake Model Registry. Covers BOTH approaches: (1) Native SQL batch using run() on warehouses for SQL pipelines/dbt, and (2) Job-based batch using run_batch() on SPCS compute pools for large-scale/unstructured data. Triggers: batch inference, bulk predictions, run_batch, run(), offline scoring, score dataset, batch predictions on table, image inference, audio transcription, multimodal."
parent_skill: machine-learning
---
# Batch Inference Jobs
Run inference on registered models for batch workloads. Snowflake offers **two batch inference approaches**:
| Approach | API | Compute | Best For |
|----------|-----|---------|----------|
| **Native SQL Batch** | `mv.run()` | Virtual Warehouse | SQL pipelines, dbt, Dynamic Tables, Snowpark |
| **Job-based Batch** | `mv.run_batch()` | SPCS Compute Pool | Large-scale processing, unstructured data (images/audio/video) |
> **Documentation**: [Model Inference in Snowflake](https://docs.snowflake.com/en/developer-guide/snowflake-ml/inference/inference-overview)
---
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
## Step 0: Choose Inference Approach
**MANDATORY:** Before proceeding, ask the user which batch inference approach they need:
```
For batch inference, there are two approaches:
1. **Warehouse-based** (`mv.run()`) - Runs on virtual warehouses
- Best for: SQL pipelines, dbt models, Dynamic Tables, Snowpark DataFrames
- Simpler setup, no compute pool required
- Ideal for tabular data and lightweight models
2. **SPCS Job-based** (`mv.run_batch()`) - Runs on SPCS compute pools
- Best for: Large-scale processing, GPU models, unstructured data (images/audio/video)
- Requires compute pool setup
- Supports parallel replicas for high throughput
Which approach do you need?
```
**⚠️ STOP**: Wait for user response.
**Routing based on response:**
- **Warehouse-based** → Refer user to the **model-registry** skill which covers `mv.run()` inference in detail. Do NOT continue with this skill.
- **SPCS Job-based** → Continue to Step 1 below.
---
## Job-based Batch Inference (`run_batch()`)
Run large-scale inference jobs on SPCS compute pools. Best for unstructured data (images/audio/video), GPU models, large-scale backfills.
> Requires `snowflake-ml-python>=1.28.0`.
**For unstructured data** (images, audio, video, multimodal LLMs):
→ Load `template/SKILL.md`
## Prerequisites
- `snowflake-ml-python>=1.28.0`
- Model registered in Snowflake Model Registry
- Compute pool (CPU or GPU depending on model)
- Stage for output files (with SSE encryption)
## Limitations
- For multi-modal use cases, encryption is only supported on the server side
- Partitioned models are not supported
## Workflow
### Step 1: Identify Model and Version
**Ask user:**
```
To run batch inference, I need:
1. **Model name**: What model do you want to use? (from Model Registry)
2. **Database/Schema**: Where is the model registered?
```
**⚠️ STOP**: Wait for user response.
**After user responds, verify model exists:**
If multiple versions exist, ask user which version to use. Otherwise, use the latest.
**Get available functions:**
```python
mv.show_functions()
```
Note the function names (e.g., `predict`, `encode`, `__call__`). If the model has **multiple functions**, you'll need to specify which one to use in JobSpec. If the model has only **one function**, you can omit `function_name` from JobSpec.
### Step 2: Identify Input Data
**Ask user:**
```
What data do you want to run inference on?
1. **Snowflake table** - Tabular data (e.g., MY_DB.SCHEMA.INPUT_TABLE)
2. **Inline data** - Small dataset to create as DataFrame
3. **Unstructured data (non-template)** - Images/audio/video for models expecting raw bytes
- Use with: Whisper, ViT, ResNet, YOLO, custom image/audio models
- Best for: Focused tasks like image classification, audio transcription, object detection
4. **Unstructured data (template/LLM)** - Multimodal LLMs with OpenAI chat format
- Use with: Qwen-VL, LLaVA, MedGemma, other vision-language LLMs
- Best for: Image captioning, visual Q&A, multimodal reasoning
```
**⚠️ STOP**: Wait for user response.
**Routing based on response:**
- **Option 3 (non-template)** → Load `non-template/SKILL.md`
- **Option 4 (template/LLM)** → Load `template/SKILL.md`
**For Snowflake table:**
```python
input_df = session.table("<DATABASE>.<SCHEMA>.<TABLE_NAME>")
```
**For inline data:**
```python
input_df = session.create_dataframe([
(5.1, 3.5, 1.4, 0.2),
(4.9, 3.0, 1.4, 0.2),
], schema=["feature_1", "feature_2", "feature_3", "feature_4"])
```
### Step 3: Configure Output Stage
Batch inference writes results as Parquet files to a Snowflake stage. The user must provide an output stage location.
**Ask user:**
```
Where should I write the inference results?
Provide a stage location (e.g., @MY_DB.MY_SCHEMA.OUTPUT_STAGE/results/)
```
**⚠️ STOP**: Wait for user response.
**If user doesn't have a stage, create one:**
**⚠️ IMPORTANT**: The stage **must** use `SNOWFLAKE_SSE` encryption (server-side encryption). Client-side encryption is not supported for batch inference output.
**Output location format:**
```
@<DATABASE>.<SCHEMA>.<STAGE_NAME>/<optional_path>/
```
Examples:
- `@MY_DB.ML_SCHEMA.INFERENCE_STAGE/predictions/`
- `@MY_DB.ML_SCHEMA.OUTPUT_STAGE/batch_2024_01/`
### Step 4: Configure Compute Pool
**Query available compute pools:**
You can view available compute pool families at `https://docs.snowflake.com/en/sql-reference/sql/create-compute-pools` if needed.
**Recommend compute pool based on model type:**
**Ask user to confirm or create compute pool:**
```
Based on your model, I recommend:
- **Compute Pool**: <`POOL_NAME`> (<INSTANCE_FAMILY>)
Do you want to use this pool, or specify a different one?
```
**If user needs a new compute pool:**
Offer to create a new compute pool with appropriate instance family (CPU vs GPU)
### Step 5: Configure Job Parameters
**Configure JobSpec for scaling:**
```python
from snowflake.ml.model.batch import JobSpec
# Basic (single replica, model has only one function)
job_spec = JobSpec()
# Basic (single replica, model has multiple functions - must specify which one)
job_spec = JobSpec(function_name="<FUNCTION_NAME>")
# Scaled (multiple replicas)
job_spec = JobSpec(
function_name="<FUNCTION_NAME>", # Optional if model has only one function
replicas=2, # Number of replicas / instances
num_workers=2, # Workers per replica
)
```
> **Note**: `function_name` is only required when the model has multiple functions. If the model has a single function, it will be used automatically.
### Step 6: Present Configuration Summary
**⚠️ MANDATORY CHECKPOINT**: Before submitting, present summary:
```
I will submit a batch inference job with these settings:
- **Model**: <DATABASE>.<SCHEMA>.<MODEL_NAME> (version: <VERSION>)
- **Function**: <FUNCTION_NAME or "default (only one function)">
- **Input**: <INPUT_SOURCE> (<ROW_COUNT> rows)
- **Compute Pool**: <POOL_NAME>
- **Output**: @<DATABASE>.<SCHEMA>.<STAGE>/output/
- **Replicas**: <N>
Ready to submit? (Yes/No)
```
**⚠️ STOP**: Wait for explicit user approval.
### Step 7: Generate and Execute Batch Inference Code
Set up the session following your loaded environment guide, then generate the batch inference code.
**Template: Basic Tabular Inference**
```python
from snowflake.ml.registry import Registry
from snowflake.ml.model.batch import JobSpec, OutputSpec, SaveMode
# Session setup per environment guide
# e.g., create_snowpark_session() or get_active_session()
session = <SESSION_SETUP>
session.use_database("<DATABASE>")
session.use_schema("<SCHEMA>")
reg = Registry(session=session)
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
input_df = session.table("<INPUT_TABLE>")
output_location = "@<DATABASE>.<SCHEMA>.<STAGE>/output/"
job = mv.run_batch(
X=input_df,
compute_pool="<COMPUTE_POOL>",
output_spec=OutputSpec(
stage_location=output_location,
mode=SaveMode.OVERWRITE,
),
job_spec=JobSpec(), # Omit function_name if model has only one function
)
print(f"Job submitted. Waiting for completion...")
job.wait()
print(f"Job completed with status: {job.status}")
```
**Template: Scaled Inference with Multiple Replicas**
```python
job = mv.run_batch(
X=input_df,
compute_pool="<COMPUTE_POOL>",
output_spec=OutputSpec(
stage_location=output_location,
mode=SaveMode.OVERWRITE,
),
job_spec=JobSpec(
function_name="<FUNCTION_NAME>", # Optional if model has only one function
replicas=<N>,
num_workers=2,
),
)
```
### Step 8: Retrieve and Present Results
**After job completes, show output location:**
```sql
LS @<DATABASE>.<SCHEMA>.<STAGE>/output/;
```
**Read results as DataFrame:**
```python
results_df = session.read.option("pattern", ".*\\.parquet").parquet(output_location)
results_df.show(10)
```
**Save results to table (optional):**
```python
output_table = "<OUTPUT_TABLE_NAME>"
results_df.write.mode("overwrite").save_as_table(output_table)
print(f"Results saved to {output_table}")
```
**Present to user:**
```
Batch inference completed!
- **Status**: DONE
- **Output Location**: @<DATABASE>.<SCHEMA>.<STAGE>/output/
- **Files**: <N> parquet files
Would you like me to:
1. Show sample results
2. Save results to a table
3. Clean up resources
```
## Common Use Cases
### Classification/Regression (sklearn, xgboost, lightgbm)
```python
# Input: DataFrame with feature columns matching model signature
input_df = session.table("MY_DB.MY_SCHEMA.FEATURES_TABLE")
job = mv.run_batch(
X=input_df,
compute_pool="CPU_POOL",
output_spec=OutputSpec(stage_location=output_location, mode=SaveMode.OVERWRITE),
job_spec=JobSpec(),
)
```
### Text Embeddings (SentenceTransformer)
```python
# Input: DataFrame with text column
input_df = session.create_dataframe([
("The quick brown fox",),
("Snowflake is great",),
], schema=["input_feature_0"])
job = mv.run_batch(
X=input_df,
compute_pool="CPU_POOL",
output_spec=OutputSpec(stage_location=output_location, mode=SaveMode.OVERWRITE),
job_spec=JobSpec(function_name="encode"), # SentenceTransformer uses encode
)
```
## Reading Output
Batch inference writes results as **Parquet files** to the specified output stage location.
### Handling Partial Output
A job can fail midway, leaving partial data. Batch inference writes a `_SUCCESS` sentinel file upon completion.
**Best practices:**
- Only read output after `_SUCCESS` file exists
- Use an empty output directory
- Use `SaveMode.ERROR` to fail if directory not empty (safer for production)
```python
# Safe production pattern
output_spec=OutputSpec(
stage_location=output_location,
mode=SaveMode.ERROR, # Fail if output exists (prevents overwriting)
)
```
| SaveMode | Behavior |
|----------|----------|
| `OVERWRITE` | Replace existing output |
| `ERROR` | Fail if output directory not empty |
### Output Structure
The output contains:
- **All original input columns** - Your input data is preserved
- **Prediction column(s)** - Model outputs appended with names like `output_feature_0`, `predictions`, etc.
The exact output column name depends on the model's signature. Common patterns:
| Model Type | Output Column | Format |
|------------|---------------|--------|
| XGBoost/sklearn classifiers | `output_feature_0` | Integer (class label) |
| XGBoost/sklearn regressors | `output_feature_0` | Float (predicted value) |
| SentenceTransformer | `output_feature_0` | Array of floats (embedding vector) |
### Reading Results
```python
# List output files
session.sql(f"LS {output_location}").show()
# Read all parquet files
results_df = session.read.option("pattern", ".*\\.parquet").parquet(output_location)
results_df.show()
# Save to table for easier access
results_df.write.mode("overwrite").save_as_table("PREDICTION_RESULTS")
```
## Troubleshooting
### Job Management
```python
from snowflake.ml.jobs import list_jobs, delete_job, get_job
# View logs to troubleshoot
job.get_logs()
# Cancel a running job
job.cancel()
# List all jobs
list_jobs().show()
# Get handle to existing job by name
job = get_job("my_db.my_schema.job_name")
# Delete a job
delete_job(job)
```
> **Note**: The `result()` function from ML Job APIs is **not supported** for Batch Inference Jobs.
### Job Status
Check job status programmatically:
```python
print(f"Status: {job.status}")
print(f"Job ID: {job.id}")
```
### Common Issues
| Issue | Cause | Solution |
|-------|-------|----------|
| `Model not found` | Wrong model name or schema | Verify with `SHOW MODELS IN SCHEMA` |
| `Compute pool not ready` | Pool is starting/suspended | Wait or run `ALTER COMPUTE POOL ... RESUME` |
| `Permission denied` | Missing grants | Grant usage on compute pool and stage |
| `Column mismatch` | Input doesn't match model signature | Check `mv.show_functions()` for expected inputs |
### Checking Model Signature
```python
# View model functions and their signatures
mv.show_functions()
```
## Stopping Points
- ✋ Step 0: After asking warehouse vs SPCS approach
- ✋ Step 1: After asking for model name/database
- ✋ Step 2: After asking for input data source
- ✋ Step 3: After asking for output stage location
batch-inference-jobs/template/SKILL.md
---
name: batch-inference-template
description: "Batch inference with multimodal LLMs using OpenAI chat message format. Stage paths in messages are auto-resolved. Use for Qwen-VL, TinyLlama, and other vLLM-based chat models."
parent_skill: batch-inference-jobs
---
# Batch Inference Jobs: Chat Models
Run batch inference on multimodal LLMs using OpenAI-style chat messages. This approach uses a `MESSAGES` column with structured conversation data, and stage paths embedded in messages are automatically resolved.
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
---
## When to Use
- Model uses OpenAI chat completion signature
- Using vLLM-based multimodal LLMs (Qwen-VL, LLaVA, TinyLlama, MedGemma)
- Input is structured as conversation messages
- You want automatic resolution of stage paths in message content
## Key Advantage: Auto Stage Path Resolution
When using the OpenAI message format, stage paths inside `image_url.url`, `video_url.url`, or `input_audio.data` fields are **automatically detected and resolved**—no `column_handling` configuration needed.
## Step 1: Check Model Signature
**⚠️ CRITICAL:** Before building the input DataFrame, always check the model's function signature to determine how parameters should be passed.
```sql
-- First verify the model exists
SHOW MODELS LIKE '<MODEL_NAME>' IN SCHEMA <DATABASE>.<SCHEMA>;
-- Then get version/signature details (only run after confirming model exists above — errors if model not found)
SHOW VERSIONS IN MODEL <DATABASE>.<SCHEMA>.<MODEL_NAME>;
```
Look at the `signatures` section in the `model_spec` output. Pay attention to:
1. **`inputs`** - These are DataFrame columns (e.g., `messages`, `temperature`, `max_completion_tokens`)
2. **`params`** - These are passed via `InputSpec(params={...})`
**Example signature analysis:**
```yaml
signatures:
__call__:
inputs:
- name: messages
...
- name: temperature
type: DOUBLE
- name: max_completion_tokens
type: INT64
...
params: [] # Empty = no InputSpec params
```
**Decision logic:**
| Signature | How to Pass Parameters |
|-----------|----------------------|
| Parameters in `inputs`, `params: []` empty | Add as DataFrame columns |
| Parameters in `params` list | Use `InputSpec(params={...})` |
## Message Format
Messages follow the OpenAI chat completion format:
```python
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
]
```
For multimodal content, use the structured content format:
```python
messages = [
{"role": "system", "content": [{"type": "text", "text": "You are an image analyzer."}]},
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image."},
{
"type": "image_url",
"image_url": {
"url": "@DB.SCHEMA.STAGE/image.jpg",
},
},
],
},
]
```
## Complete Example: Text-Only Chat
```python
import json
from snowflake.snowpark import Session
from snowflake.ml.registry import Registry
from snowflake.ml.model.batch import JobSpec, OutputSpec, SaveMode, InputSpec
session = Session.builder.config("connection_name", "<CONNECTION>").create()
reg = Registry(session=session)
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
# Create messages for text-only chat
messages_list = [
[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
],
[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing in simple terms."},
],
]
# Create DataFrame with MESSAGES column (JSON-encoded)
data = [json.dumps(m) for m in messages_list]
input_df = session.create_dataframe(data, schema=["MESSAGES"])
job = mv.run_batch(
X=input_df,
compute_pool="<GPU_COMPUTE_POOL>",
output_spec=OutputSpec(
stage_location="@<DATABASE>.<SCHEMA>.<STAGE>/output/",
mode=SaveMode.OVERWRITE,
),
input_spec=InputSpec(params={"temperature": 0.7, "max_completion_tokens": 256}),
job_spec=JobSpec(gpu_requests="1"),
)
job.wait()
```
## Complete Example: Multimodal Chat (Images)
For vision-language models, include image references in the messages.
```python
import json
from snowflake.ml.model.batch import JobSpec, OutputSpec, SaveMode, InputSpec
from snowflake.ml.model.inference_engine import InferenceEngine
# Stage path to your image
image_stage_path = "@<DATABASE>.<SCHEMA>.<STAGE>/images/cat.jpg"
messages_list = [
[
{"role": "system", "content": [{"type": "text", "text": "You are an expert image analyzer."}]},
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image in 20 words or less."},
{
"type": "image_url",
"image_url": {
"url": image_stage_path,
},
},
],
},
],
]
data = [json.dumps(m) for m in messages_list]
input_df = session.create_dataframe(data, schema=["MESSAGES"])
job = mv.run_batch(
X=input_df,
compute_pool="<GPU_COMPUTE_POOL>",
output_spec=OutputSpec(
stage_location="@<DATABASE>.<SCHEMA>.<STAGE>/output/",
mode=SaveMode.OVERWRITE,
),
input_spec=InputSpec(params={"temperature": 0.0}),
job_spec=JobSpec(gpu_requests="1"),
inference_engine_options={
"engine": InferenceEngine.VLLM,
"engine_args_override": [
"--max-model-len=7048",
"--gpu-memory-utilization=0.9",
]
}
)
job.wait()
```
## Processing Multiple Images
If the user provides image URLs or local images, upload them to a stage as the input for the batch inference jobs.
To process multiple images with the same prompt:
```python
import json
from snowflake.ml.utils.stage_file import list_stage_files
# Get list of image paths
image_files_df = list_stage_files(
session,
"@<DATABASE>.<SCHEMA>.<STAGE>/images",
pattern=".*\\.jpg", # for jpg files, adjust as needed
column_name="IMAGE_PATH"
)
image_paths = [row["IMAGE_PATH"] for row in image_files_df.collect()]
# Create messages for each image
messages_list = []
for image_path in image_paths:
messages = [
{"role": "system", "content": [{"type": "text", "text": "You are an expert image analyzer."}]},
{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image."},
{"type": "image_url", "image_url": {"url": image_path}},
],
},
]
messages_list.append(messages)
data = [json.dumps(m) for m in messages_list]
input_df = session.create_dataframe(data, schema=["MESSAGES"])
```
## Inference Parameters
Pass model inference parameters via `InputSpec.params`:
```python
input_spec = InputSpec(
params={
"temperature": 0.7,
"max_completion_tokens": 256,
"top_p": 0.9,
}
)
```
Common parameters for chat models:
- `temperature` - Sampling temperature (0.0 = deterministic)
- `max_completion_tokens` - Maximum tokens in response
- `top_p` - Nucleus sampling threshold
- `frequency_penalty` - Penalize repeated tokens
- `presence_penalty` - Penalize tokens already present
## vLLM Engine Options
For multimodal models, you may need to configure vLLM:
```python
from snowflake.ml.model.inference_engine import InferenceEngine
inference_engine_options = {
"engine": InferenceEngine.VLLM,
"engine_args_override": [
"--max-model-len=7048", # Context length
"--gpu-memory-utilization=0.9", # GPU memory usage
]
}
```
## Reading Output
Chat models return responses in OpenAI chat completion format:
```python
output_location = "@<DATABASE>.<SCHEMA>.<STAGE>/output/"
results_df = session.read.option("pattern", ".*\\.parquet").parquet(output_location)
results_df.show(1, max_width=200)
# Output columns: "MESSAGES", "id", "object", "created", "model", "choices", "usage"
#
# The "choices" column contains the assistant's response:
# [{"index": 0, "message": {"content": "...", "role": "assistant"}, "finish_reason": "stop"}]
```
To extract the response text:
```python
from snowflake.snowpark.functions import col, get
results_df.select(
col('"MESSAGES"'),
get(get(col('"choices"'), 0), "message")["content"].alias("response")
).show()
```
## Supported Content Types
The OpenAI message format supports:
| Content Type | Field | Example |
|--------------|-------|---------|
| Text | `type: "text"` | `{"type": "text", "text": "Hello"}` |
| Image | `type: "image_url"` | `{"type": "image_url", "image_url": {"url": "@STAGE/img.jpg"}}` |
| Video | `type: "video_url"` | `{"type": "video_url", "video_url": {"url": "@STAGE/vid.mp4"}}` |
| Audio | `type: "input_audio"` | `{"type": "input_audio", "input_audio": {"data": "@STAGE/audio.wav"}}` |
## Troubleshooting
**Model doesn't understand the image**
- Verify the model supports vision (e.g., Qwen-VL, LLaVA)
- Check that the stage path is correct and file exists
- Ensure the message format matches OpenAI's multimodal format
**Out of memory errors**
- Reduce `--max-model-len` in engine_args_override
- Increase `--gpu-memory-utilization` carefully
- Use a compute pool with more GPU memory
**Slow inference**
- Increase replicas in JobSpec for parallelism
- Use `JobSpec(replicas=N, gpu_requests="1")`
**JSON parsing errors**
- Ensure messages are properly JSON-encoded with `json.dumps()`
- Verify the message structure matches OpenAI format
## Output
- Batch inference job completed
- Parquet files with OpenAI chat completion format responses
- Output columns: `MESSAGES`, `id`, `object`, `created`, `model`, `choices`, `usage`
debug-inference/SKILL.md
---
name: debug-inference
description: "Debug model inference issues for both warehouse and SPCS. Covers dtype errors, nullable signature problems, service failures, OOM, container issues. Use when: inference error, mv.run fails, TypeError, np.radians error, service not starting, OOM, container crash."
parent_skill: machine-learning
---
# Debugging Model Inference
This skill helps diagnose and fix inference issues for models deployed via Snowflake Model Registry, whether running on warehouse or SPCS.
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
---
## Step 0: Triage - Determine Inference Type
**Ask user or detect from context:**
If the user mentions `service_name` parameter, REST API, or SPCS compute pool, it's SPCS inference. If they mention `MODEL()` SQL function or `mv.run()` without `service_name`, it's warehouse inference.
**If unclear, ask:**
```
Where is your inference running?
1. **Warehouse** - Using `mv.run()` without service_name, or SQL `MODEL()` calls
2. **SPCS Service** - Using `mv.run()` with service_name parameter, or REST API calls to a deployed service
```
**⚠️ STOP**: Wait for user response if inference type is not clear from context.
**Routing:**
- **If SPCS** → Go to [Step 1: Check Service Status & Logs](#step-1-spcs---check-service-status--logs)
- **If Warehouse** → Go to [Step 2: Check Model Signature](#step-2-check-model-signature) (Note: NumPy/nullable dtype issues are SPCS-specific)
---
## Step 1: SPCS - Check Service Status & Logs
Before diagnosing model-level issues, verify the service is healthy and retrieve logs to identify the error:
```sql
-- Check service status
DESCRIBE SERVICE <DATABASE>.<SCHEMA>.<SERVICE_NAME>;
-- Get detailed instance status
SELECT SYSTEM$GET_SERVICE_STATUS('<DATABASE>.<SCHEMA>.<SERVICE_NAME>');
-- Get recent logs from model-inference container
CALL SYSTEM$GET_SERVICE_LOGS('<DATABASE>.<SCHEMA>.<SERVICE_NAME>', 0, 'model-inference');
```
**Route based on errors found in logs:**
| Error Pattern in Logs | Issue | Go To |
|----------------------|-------|-------|
| `TypeError: loop of ufunc does not support argument 0 of type float` | Nullable dtype issue | [Issue A](#issue-a-numpy-ufunc-errors-with-nullable-dtypes) |
| `'float' object has no attribute 'radians'` | Nullable dtype issue | [Issue A](#issue-a-numpy-ufunc-errors-with-nullable-dtypes) |
| `OOMKilled`, memory errors | Out of memory | [Issue C](#issue-c-oom-and-memory-issues) |
| Container restart, crash | Container issues | [Issue D](#issue-d-container-logs-and-crashes) |
| Service status PENDING/STARTING | Service not ready | [Issue B](#issue-b-service-not-ready) |
---
## Step 2: Check Model Signature
Retrieve the model version and inspect function signatures for potential issues:
```python
from snowflake.ml.registry import Registry
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
# Inspect all functions and their signatures
for func in mv.show_functions():
print(f"\nFunction: {func['name']}")
print(f" Target Method: {func['target_method']}")
print(" Input Features:")
for feat in func['signature'].inputs:
print(f" {feat.name}: dtype={feat._dtype}, nullable={feat._nullable}")
print(" Output Features:")
for feat in func['signature'].outputs:
print(f" {feat.name}: dtype={feat._dtype}, nullable={feat._nullable}")
```
**What to look for:**
- Features with `nullable=True` on numeric types (DOUBLE, FLOAT, INT) can cause NumPy ufunc errors
- Missing or incorrect feature names
- Type mismatches between expected and actual input data
---
## Issue A: NumPy Ufunc Errors with Nullable Dtypes (SPCS Only)
**Note:** This issue only occurs with SPCS inference, not warehouse inference. The SPCS inference server handles dtype conversion differently.
### Symptoms
Errors like:
```
TypeError: loop of ufunc does not support argument 0 of type float which has no callable radians method
```
or:
```
AttributeError: 'float' object has no attribute 'radians'
```
### Root Cause
When you register a model using `sample_input_data` in `log_model()`, the signature inference defaults `nullable=True` for all features. This causes the inference server to use Pandas nullable extension dtypes (`pd.Float64Dtype()` instead of `np.float64`).
When model code calls `.values.T` on a DataFrame with nullable dtypes, it produces a `dtype=object` array containing Python `float` objects instead of a native NumPy array. NumPy ufuncs like `np.radians()`, `np.sin()`, `np.cos()` cannot operate on these object arrays.
**Example of the problem:**
```python
# With nullable=True (default), inference server does:
df = df.astype({"col": pd.Float64Dtype()}) # Nullable extension dtype
arr = df[["col"]].values.T # → dtype=object array with Python floats
# NumPy ufuncs fail on object arrays:
np.radians(arr) # TypeError!
```
### Diagnosis
Check if your model signature has `nullable=True` on numeric features:
```python
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
for func in mv.show_functions():
for feat in func['signature'].inputs:
if feat._nullable and feat._dtype.name in ['DOUBLE', 'FLOAT', 'INT64', 'INT32']:
print(f"WARNING: {feat.name} has nullable=True - may cause NumPy issues")
```
### Fix
**⚠️ MANDATORY CHECKPOINT**: Before re-registering the model, present to user:
```
I've identified the issue. The model signature has nullable=True on numeric features,
causing NumPy ufunc errors in SPCS inference.
Proposed fix:
- Re-register the model with an explicit signature (nullable=False on all numeric features)
- Affected features: [list features with nullable=True]
- New version name: <NEW_VERSION>
This will create a new model version. The existing version will not be modified.
Do you approve? (Yes/No/Modify)
```
**⚠️ STOP**: Wait for explicit user approval before proceeding.
Re-register the model with an explicit signature that sets `nullable=False`:
```python
from snowflake.ml.registry import Registry
from snowflake.ml.model.model_signature import FeatureSpec, DataType, ModelSignature
# Define explicit signature with nullable=False
input_features = [
FeatureSpec(name="FEATURE1", dtype=DataType.DOUBLE, nullable=False),
FeatureSpec(name="FEATURE2", dtype=DataType.DOUBLE, nullable=False),
# Add all your input features...
]
output_features = [
FeatureSpec(name="OUTPUT", dtype=DataType.DOUBLE, nullable=False),
]
predict_signature = ModelSignature(inputs=input_features, outputs=output_features)
# Register with explicit signature (not sample_input_data)
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
mv = reg.log_model(
model,
model_name="<MODEL_NAME>",
version_name="<NEW_VERSION>",
signatures={"predict": predict_signature}, # Explicit signature
conda_dependencies=["..."],
comment="Fixed nullable=False for NumPy compatibility"
)
```
---
## Issue B: Service Not Ready
### Symptoms
- Service status shows `PENDING` or `STARTING`
- Inference requests timeout
- `mv.run()` hangs or fails
### Diagnosis
```sql
-- Check service status
DESCRIBE SERVICE <SERVICE_NAME>;
-- Check detailed instance status
SELECT SYSTEM$GET_SERVICE_STATUS('<DATABASE>.<SCHEMA>.<SERVICE_NAME>');
```
Look for:
- `status: PENDING` - service waiting for resources
- `status: STARTING` - containers being pulled/started
- `status: FAILED` - deployment failed
### Common Causes and Fixes
| Cause | Fix |
|-------|-----|
| **Testing before service ready** | Wait for status `RUNNING` and all instances showing `"status":"READY"` in `SYSTEM$GET_SERVICE_STATUS()` |
| **Compute pool not ready** | Check with `SHOW COMPUTE POOLS` - resuming a suspended pool takes a few minutes |
| **Missing privileges** | Need `USAGE` on compute pool, `BIND SERVICE ENDPOINT` for HTTP endpoints |
| **Service fails to start** | Check logs: `CALL SYSTEM$GET_SERVICE_LOGS('<SERVICE_NAME>', 0, 'model-inference')` |
---
## Issue C: OOM and Memory Issues
### Diagnosis
Check platform metrics for memory usage patterns:
```sql
-- Memory and CPU usage (look for OOM patterns)
SELECT timestamp, metric_name, value, unit, container_name
FROM TABLE(<SERVICE_NAME>!SPCS_GET_METRICS())
WHERE metric_name IN ('container.memory.usage', 'container.cpu.usage', 'container.memory.max_usage')
ORDER BY timestamp DESC
LIMIT 50;
-- GPU metrics (if applicable)
SELECT timestamp, metric_name, value, container_name
FROM TABLE(<SERVICE_NAME>!SPCS_GET_METRICS())
WHERE metric_name LIKE '%gpu%'
ORDER BY timestamp DESC;
```
### Common OOM Fixes
| Symptom | Cause | Fix |
|---------|-------|-----|
| `container.memory.max_usage` near limit | Model too large | Use larger instance or reduce num_workers |
| Repeated container restarts | OOM kills | Check logs for "OOMKilled", reduce batch size |
| GPU memory errors in logs | Model doesn't fit | Reduce num_workers or use instance with more GPU memory |
---
## Issue D: Container Logs and Crashes
### Check Container Logs
**Model inference container:**
```sql
SELECT * FROM TABLE(<SERVICE_NAME>!SPCS_GET_LOGS())
WHERE container_name = 'model-inference'
ORDER BY timestamp DESC
LIMIT 100;
```
**Proxy container:**
```sql
SELECT * FROM TABLE(<SERVICE_NAME>!SPCS_GET_LOGS())
WHERE container_name = 'proxy'
ORDER BY timestamp DESC
LIMIT 100;
```
**Live logs (current container):**
```sql
CALL SYSTEM$GET_SERVICE_LOGS('<SERVICE_NAME>', 0, 'model-inference');
CALL SYSTEM$GET_SERVICE_LOGS('<SERVICE_NAME>', 0, 'proxy');
```
### Event Table (if service is terminated)
If the service is dead and logs are unavailable, query the event table directly:
```sql
-- Find your event table
SHOW PARAMETERS LIKE 'event_table' IN ACCOUNT;
-- Logs from event table
SELECT timestamp, value, resource_attributes
FROM <EVENT_TABLE>
WHERE resource_attributes:"snow.service.name" = '<SERVICE_NAME>'
AND record_type = 'LOG'
ORDER BY timestamp DESC
LIMIT 100;
-- Metrics from event table
SELECT timestamp, record:metric_name::string as metric, record:value as value
FROM <EVENT_TABLE>
WHERE resource_attributes:"snow.service.name" = '<SERVICE_NAME>'
AND record_type = 'METRIC'
ORDER BY timestamp DESC
LIMIT 100;
```
---
## Stopping Points
- ✋ Step 0: If inference type (warehouse vs SPCS) is unclear from context
- ✋ Issue A Fix: Before re-registering the model with a new signature — get explicit user approval
**Resume rule:** Upon user approval, proceed directly to the next step without re-asking.
---
## Quick Reference: Diagnostic Commands
| What to Check | Command |
|---------------|---------|
| Service status | `DESCRIBE SERVICE <SERVICE_NAME>` |
| Instance details | `SELECT SYSTEM$GET_SERVICE_STATUS('<SERVICE_NAME>')` |
| Container logs | `CALL SYSTEM$GET_SERVICE_LOGS('<SERVICE_NAME>', 0, 'model-inference')` |
| Platform metrics | `SELECT * FROM TABLE(<SERVICE_NAME>!SPCS_GET_METRICS())` |
| Model functions | `mv.show_functions()` |
| Model signature | `func['signature'].inputs` / `func['signature'].outputs` |
## Output
- Root cause identification for the inference failure
- Fix applied (re-registered model, service configuration change, etc.) or actionable remediation steps provided to user
distributed-training/dpf/SKILL.md
---
name: distributed-partition-function
description: "General-purpose distributed processing with DPF. Custom distributed workloads, multiple outputs per partition, ML and other processing."
parent_skill: distributed-training
path: machine-learning/distributed-training/dpf
---
# Distributed Partition Function (DPF)
General-purpose distributed processing framework. Use when you need more control than MMT provides.
## ⚠️ CRITICAL: Server-Side Execution Only
**DPF runs SERVER-SIDE on Snowflake compute pools, NOT in local/client environments.**
The `snowflake.ml.modeling.distributors` module is ONLY available inside:
- Snowflake ML Jobs (submitted through CLI/local development, running on Snowflake compute)
- Snowflake Notebooks with Container Runtime (for interactive work in Snowsight)
**If working from a local/client environment**, submit your code via ML Jobs. See `../../ml-jobs/SKILL.md` for submission methods.
## When to Load
Load this skill when:
- Training models but only outputting results (not persisting the model)
- Multiple models or artifacts per partition
- Custom serialization formats
- Non-ML distributed processing (ETL, analytics)
- User mentions: "DPF", "distributed partition function", "custom distributed processing"
## Workflow
### Step 1: Clarify Use Case
**Ask user:**
```
I'll help you set up distributed processing with DPF. Which mode?
1. **SQL Mode** - Partition by column values in a DataFrame
2. **Stage Mode** - Process files from a stage
Also confirm:
- What processing do you need per partition?
- What outputs should be saved?
- Output stage name?
```
**⚠️ STOP**: Wait for user response.
### Step 2: Define Processing Function
```python
def my_function(data_connector, context):
"""
Args:
data_connector: Access partition data via .to_pandas(), .to_torch_dataset(), etc.
context: Partition utilities
- context.partition_id: Current partition identifier
- context.upload_to_stage(obj, filename): Save artifacts
- context.download_from_stage(filename): Load artifacts
- context.with_session(func): Execute with Snowflake session
"""
df = data_connector.to_pandas()
# Process data
results = {"count": len(df), "partition": context.partition_id}
# Save outputs
context.upload_to_stage(results, "results.pkl")
```
### Step 3: Configure and Run
**SQL Mode (partition by column):**
```python
from snowflake.ml.modeling.distributors.distributed_partition_function.dpf import DPF
from snowflake.ml.modeling.distributors.distributed_partition_function.entities import (
ExecutionOptions, RunStatus
)
dpf = DPF(func=process_partition, stage_name="<OUTPUT_STAGE>")
dpf_run = dpf.run(
partition_by="<PARTITION_COLUMN>",
snowpark_dataframe=session.table("<TABLE>"),
run_id="<DESCRIPTIVE_RUN_ID>",
execution_options=ExecutionOptions(num_cpus_per_worker=1),
)
status = dpf_run.wait()
```
**Stage Mode (process files):**
```python
dpf = DPF(func=process_file, stage_name="<OUTPUT_STAGE>")
dpf_run = dpf.run_from_stage(
stage_location="@db.schema.input_stage/",
run_id="<RUN_ID>",
file_pattern="*.parquet",
)
status = dpf_run.wait()
```
**⚠️ STOP**: After run completes, verify with user:
```
DPF run complete.
Status: [SUCCESS/PARTIAL/FAILED]
Partitions processed: [N]
Would you like to:
1. Query results from stage
2. Check failed partitions
3. Run another processing job
```
### Step 4: Retrieve Results
```python
# Check progress
dpf_run.get_progress() # {"DONE": [...], "FAILED": [...]}
# Partition details
dpf_run.partition_details # Dict[str, SinglePartitionDetails]
# Restore completed run later
from snowflake.ml.modeling.distributors.distributed_partition_function.dpf_run import DPFRun
restored = DPFRun.restore_from("<RUN_ID>", "<STAGE_NAME>")
```
**Query Parquet results from stage:**
```python
session.sql("CREATE FILE FORMAT IF NOT EXISTS parquet_format TYPE = 'PARQUET'").collect()
results_df = session.sql(f"""
SELECT
$1:PARTITION_KEY::STRING AS PARTITION_KEY,
$1:VALUE::INTEGER AS VALUE
FROM @<STAGE>/<RUN_ID>/
(FILE_FORMAT => parquet_format, PATTERN => '.*\\.parquet')
""")
```
## Stopping Points
- ✋ **Step 1**: After clarifying use case (wait for user input)
- ✋ **Step 3**: After run completes (verify results)
## Output
- Artifacts saved to stage (per partition)
- Results queryable via SQL
- Run metadata for restoration
---
## ExecutionOptions Reference
```python
from snowflake.ml.modeling.distributors.distributed_partition_function.entities import ExecutionOptions
ExecutionOptions(
use_head_node=True, # Head node participates in execution (default True)
loading_wh=None, # Warehouse for data loading (see below)
num_cpus_per_worker=None, # CPUs per worker (None = auto)
num_gpus_per_worker=None, # GPUs per worker (None = auto)
max_retries=1, # Retry failed partitions
fail_fast=False, # Stop on first failure
)
```
**`loading_wh`**: In SQL mode, a virtual warehouse loads partition data. A larger warehouse makes a difference for large partitions. For many partitions or large tables, consider **Stage mode** (`run_from_stage`) instead -- workers read files directly from a Snowflake stage in parallel, bypassing the warehouse entirely.
**`num_cpus_per_worker`**: See `../references/compute-pool-sizing.md` for how this controls parallelism and memory per worker.
## Resource Sizing
See `../references/compute-pool-sizing.md` for instance families, node count sizing, and the `num_cpus_per_worker` tradeoff. For monitoring and troubleshooting, see `../references/monitoring-troubleshooting.md`.
---
## Monitoring DPF Jobs
For general job monitoring (status, logs, killing jobs) via Python and SQL, see `../references/monitoring-troubleshooting.md`.
**DPF-specific: Checking output on stage:**
```sql
ALTER STAGE <STAGE_NAME> REFRESH;
SELECT RELATIVE_PATH, SIZE FROM DIRECTORY(@<STAGE_NAME>)
WHERE RELATIVE_PATH LIKE '%<run_id>%'
ORDER BY RELATIVE_PATH;
```
> **DPF-specific:** The DPF framework writes per-partition `train.log` files to the output stage alongside your result files. These contain the most detailed per-partition application logs and errors -- check them for debugging failed partitions.
---
## Common Patterns
### Multiple Models per Partition
```python
def train_ensemble(data_connector, context):
df = data_connector.to_pandas()
X, y = df[["f1", "f2"]], df["target"]
from xgboost import XGBRegressor
from sklearn.ensemble import RandomForestRegressor
models = {
"xgboost": XGBRegressor().fit(X, y),
"rf": RandomForestRegressor().fit(X, y),
}
for name, model in models.items():
context.upload_to_stage(model, f"{name}.pkl")
```
### Results Only (No Model Persistence)
```python
def score_partition(data_connector, context):
df = data_connector.to_pandas()
from xgboost import XGBClassifier
model = XGBClassifier().fit(df[["f1", "f2"]], df["target"])
predictions = model.predict(df[["f1", "f2"]])
results_df = df.assign(prediction=predictions, partition=context.partition_id)
# Write to Snowflake table
context.with_session(lambda session:
session.create_dataframe(results_df)
.write.mode("append")
.save_as_table("PREDICTIONS")
)
```
### Write Parquet to Stage
```python
def process_partition(data_connector, context):
import pyarrow as pa
import pyarrow.parquet as pq
df = data_connector.to_pandas()
results = [{"PARTITION_KEY": context.partition_id, "VALUE": 123}]
context.upload_to_stage(
results,
"results.parquet",
write_function=lambda data, path: pq.write_table(
pa.Table.from_pylist(data), path
),
)
```
> **⚠️ Output filename must be a simple name (no paths).**
> The DPF framework organizes outputs into per-partition directories automatically. Use a flat filename like `"results.parquet"` or `"model.pkl"` -- never embed `context.partition_id` in the filename. The partition_id may contain slashes (e.g., `folder/file.parquet`) which creates nested directories that don't exist, causing `FileNotFoundError`.
---
## API Reference
### DPF
```python
from snowflake.ml.modeling.distributors.distributed_partition_function.dpf import DPF
dpf = DPF(func, stage_name)
```
- `func` (`Callable[[DataConnector, PartitionContext], None]`): Function executed per partition.
- `stage_name` (`str`): Output stage for run artifacts. Each run creates `@{stage_name}/{run_id}/`.
#### DPF.run()
```python
dpf_run = dpf.run(
partition_by: str,
snowpark_dataframe: snowpark.DataFrame,
run_id: str,
on_existing_artifacts: Literal["error", "overwrite"] = "error",
execution_options: Optional[ExecutionOptions] = None,
) -> DPFRun
```
- `partition_by`: Column name to partition by. Each unique value = one partition.
- `snowpark_dataframe`: DataFrame to partition. Must contain a single query with no post-actions.
- `run_id`: Unique identifier. Creates `@{stage_name}/{run_id}/` directory.
- `on_existing_artifacts`: `"error"` (default) raises if artifacts exist; `"overwrite"` replaces them.
- `execution_options`: See ExecutionOptions Reference above.
#### DPF.run_from_stage()
```python
dpf_run = dpf.run_from_stage(
stage_location: str,
run_id: str,
file_pattern: str = "*.parquet",
partition_ids: Optional[List[str]] = None,
on_existing_artifacts: Literal["error", "overwrite"] = "error",
execution_options: Optional[ExecutionOptions] = None,
) -> DPFRun
```
- `stage_location`: **Input** stage path (e.g., `"@my_db.my_schema.my_stage/data/"`). Each matching file becomes a partition.
- `file_pattern`: Glob to filter files (default `"*.parquet"`).
- `partition_ids`: Optional list of specific file paths (relative to `stage_location`) to process. When provided, `file_pattern` is ignored. Useful for rerunning failed partitions:
```python
failed = [p for p, d in run.partition_details.items() if d.status == "FAILED"]
dpf.run_from_stage(..., partition_ids=failed)
```
---
## Next Steps
- **Need simpler per-partition models** → Use `../mmt/SKILL.md` instead
- **Run partitioned inference through model registry** → Load `../../model-registry/partitioned-inference/SKILL.md`
distributed-training/estimators/SKILL.md
---
name: distributed-estimators
description: "Distributed model training with XGBEstimator, LightGBMEstimator, and PyTorchDistributor. Train one large model across multiple nodes/GPUs."
parent_skill: distributed-training
path: machine-learning/distributed-training/estimators
---
# Distributed Estimators
Train a single model across multiple nodes/GPUs using Snowflake's distributed trainers.
## When to Load
Load this skill when:
- User wants to train XGBoost, LightGBM, or PyTorch at scale
- Dataset is too large for single-node training
- User mentions: "distributed XGBoost", "XGBEstimator", "LightGBMEstimator", "PyTorchDistributor", "multi-node training", "multi-GPU training"
## Workflow
### Step 1: Confirm Training Setup
**Ask user:**
```
I'll help you set up distributed training. Please confirm:
1. Which framework? (XGBoost / LightGBM / PyTorch)
2. Training data table name?
3. Target/label column?
4. Feature columns? (or "all except target")
5. GPU or CPU training?
```
**In Snowsight notebooks**: Instead of asking about GPU/CPU, auto-detect by running:
```python
import torch
print(f"GPU available: {torch.cuda.is_available()}, count: {torch.cuda.device_count()}")
```
**⚠️ STOP**: Wait for user response before proceeding.
### Step 2: Set Up Data Connectors
```python
from snowflake.ml.data.data_connector import DataConnector
train_connector = DataConnector.from_dataframe(session.table('<TRAIN_TABLE>'))
eval_connector = DataConnector.from_dataframe(session.table('<EVAL_TABLE>')) # Optional
```
For PyTorch, use sharded connector:
```python
from snowflake.ml.data.sharded_data_connector import ShardedDataConnector
data_connector = ShardedDataConnector.from_dataframe(session.table('<TABLE>'))
```
### Step 3: Configure and Train
**If XGBoost:**
```python
from snowflake.ml.modeling.distributors.xgboost import XGBEstimator, XGBScalingConfig
label_col = '<TARGET_COLUMN>'
input_cols = [c for c in session.table('<TABLE>').columns if c != label_col]
params = {
'objective': 'reg:squarederror', # or 'binary:logistic', etc.
'max_depth': 6,
'learning_rate': 0.1
}
estimator = XGBEstimator(
params=params,
scaling_config=XGBScalingConfig(
num_workers=-1, # Auto-detect (default)
num_cpu_per_worker=-1, # Auto-detect (default)
use_gpu=None # None = auto-detect; set True for GPU
)
)
booster = estimator.fit(
dataset=train_connector,
input_cols=input_cols,
label_col=label_col,
eval_set=eval_connector,
verbose_eval=10
)
```
**If LightGBM:**
```python
from snowflake.ml.modeling.distributors.lightgbm import LightGBMEstimator, LightGBMScalingConfig
params = {
'objective': 'regression',
'metric': 'rmse',
'boosting_type': 'gbdt',
'num_leaves': 31,
'learning_rate': 0.05
}
estimator = LightGBMEstimator(
params=params,
scaling_config=LightGBMScalingConfig(
num_workers=-1, # Auto-detect (default)
num_cpu_per_worker=-1, # Auto-detect (default)
use_gpu=None # None = auto-detect; set True for GPU
)
)
booster = estimator.fit(
dataset=train_connector,
input_cols=input_cols,
label_col=label_col,
eval_set=eval_connector,
)
```
**If PyTorch:** See [PyTorch DDP](#pytorch-distributed-training-ddp) section below.
### Step 4: Verify Training Results
**⚠️ STOP**: After training completes, verify results with user:
```
Training complete. Here's what I found:
- Final eval metric: [value]
- Training mode: [single-node / distributed]
- Feature importance (top 5): [list]
Would you like to:
1. Register this model to the Model Registry
2. Inspect more model attributes
3. Retrain with different parameters
```
### Step 5: Access Model Attributes
Present relevant attributes based on user needs:
```python
# Get the trained booster
booster = estimator.get_booster()
# Get evaluation metrics history
eval_results = estimator.get_eval_results()
# Returns: {"train": {"rmse": [0.5, 0.4, ...]}, "eval": {"rmse": [0.6, 0.5, ...]}}
# Access final metric
final_metric = eval_results["eval"]["rmse"][-1]
```
**XGBoost-specific:**
```python
# Feature importance
importance = booster.get_score(importance_type='gain')
# Returns: {"feature1": 0.45, "feature2": 0.32, ...}
# Model config
config = booster.save_config() # JSON string
```
**LightGBM-specific:**
```python
# Feature importance
importance = booster.feature_importance(importance_type='gain')
feature_names = booster.feature_name()
importance_dict = dict(zip(feature_names, importance))
# Number of trees
num_trees = booster.num_trees()
# Model structure
model_dict = booster.dump_model()
```
## Stopping Points
- ✋ **Step 1**: After asking for training setup (wait for user input)
- ✋ **Step 4**: After training completes (verify results with user)
## Output
- Trained booster object ready for Model Registry
- Evaluation metrics and feature importance
- Model ready for `reg.log_model()` (see `../../model-registry/SKILL.md`)
---
## Scaling Configuration
See `../references/compute-pool-sizing.md` for guidance on sizing nodes and workers for your workload.
| Parameter | Default | Description |
|-----------|---------|-------------|
| `num_workers` | -1 | Worker processes (-1 = auto) |
| `num_cpu_per_worker` | -1 | CPUs per worker (-1 = auto) |
| `use_gpu` | None | Enable GPU training |
---
## PyTorch Distributed Training (DDP)
### Data Loading
```python
from snowflake.ml.data.sharded_data_connector import ShardedDataConnector
data_connector = ShardedDataConnector.from_dataframe(session.table("TRAINING_DATA"))
```
### Training Function
```python
import torch
import torch.nn as nn
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader
from snowflake.ml.modeling.distributors.pytorch import get_context
def train_func():
import torch.distributed as dist
context = get_context()
rank = context.get_rank()
local_rank = context.get_local_rank()
world_size = context.get_world_size()
dist.init_process_group(backend='gloo') # or 'nccl' for GPU
device = torch.device(f"cuda:{local_rank}" if torch.cuda.is_available() else "cpu")
# Initialize model with DDP
model = YourModel().to(device)
if world_size > 1:
model = DDP(model)
# Get data shard for this worker
dataset_map = context.get_dataset_map()
torch_dataset = dataset_map['train'].get_shard().to_torch_dataset(batch_size=1024)
dataloader = DataLoader(torch_dataset, batch_size=None)
# Training loop
for epoch in range(10):
for batch in dataloader:
# ... training logic ...
pass
# Save model (rank 0 only)
if rank == 0:
torch.save(model.module.state_dict(),
os.path.join(context.get_model_dir(), "model.pt"))
```
### Running PyTorch Training
```python
from snowflake.ml.modeling.distributors.pytorch import (
PyTorchDistributor, PyTorchScalingConfig, WorkerResourceConfig
)
pytorch_trainer = PyTorchDistributor(
train_func=train_func,
scaling_config=PyTorchScalingConfig(
num_nodes=2,
num_workers_per_node=1,
resource_requirements_per_worker=WorkerResourceConfig(num_cpus=0, num_gpus=1)
)
)
response = pytorch_trainer.run(dataset_map={'train': data_connector})
# Access results
model_dir = response.get_model_dir() # Stage path to saved model
metrics = response.get_metrics() # Metrics reported by rank 0
checkpoint = response.get_checkpoint_location() # Checkpoint location (if saved)
```
---
## Next Steps
After training:
- **Register model** → Load `../../model-registry/SKILL.md`
- **Create inference service** → Load `../../spcs-inference/SKILL.md`
- **Tune hyperparameters** → Return to router, select **Tuner** (`../tuner/SKILL.md`)
- **Compute pool sizing** → See `../references/compute-pool-sizing.md`
- **Monitoring & troubleshooting** → See `../references/monitoring-troubleshooting.md`
distributed-training/mmt/SKILL.md
---
name: many-model-training
description: "Train and run inference on one model per data partition using ManyModelTraining and ManyModelInference. Auto-serialization, get_model(), and distributed inference."
parent_skill: distributed-training
path: machine-learning/distributed-training/mmt
---
# Many Model Training & Inference (MMT/MMI)
Train separate ML models for each data partition in parallel, then run distributed inference using those models. MMT/MMI handles distributed orchestration, model serialization, and automatic model loading.
## When to Load
Load this skill when:
- User wants one model per partition (region, store, customer segment, etc.)
- User mentions: "many model", "train per partition", "ManyModelTraining", "model per region/store"
- User wants to run inference using MMT-trained models
- User mentions: "ManyModelInference", "inference per partition", "predict per region/store"
- User wants built-in `get_model()` convenience
**For Model Registry**: If user wants to register models in the Model Registry (versioning, SQL access), see `../../model-registry/partitioned-inference/SKILL.md` for `@partitioned_api` approach.
## Workflow
### Step 1: Confirm Setup
**Ask user:**
```
I'll help you train models per partition. Please confirm:
1. Training data table?
2. Partition column? (e.g., REGION, STORE_ID)
3. Target/label column?
4. Feature columns?
5. Model type? (XGBoost / LightGBM / sklearn / custom)
6. Stage for storing models?
```
**⚠️ STOP**: Wait for user response.
### Step 2: Define Training Function
```python
from snowflake.ml.modeling.distributors.many_model import ManyModelTraining
from snowflake.ml.modeling.distributors.distributed_partition_function.entities import RunStatus
def train_model(data_connector, context):
"""
Args:
data_connector: Access partition data via .to_pandas()
context: Partition info via context.partition_id
Returns:
Trained model (auto-serialized)
"""
df = data_connector.to_pandas()
print(f"Training for partition: {context.partition_id}")
from xgboost import XGBRegressor
model = XGBRegressor(n_estimators=100)
model.fit(df[['feature1', 'feature2']], df['target'])
return model # Auto-serialized to stage
```
### Step 3: Run Training
```python
trainer = ManyModelTraining(train_model, "<STAGE_NAME>")
training_run = trainer.run(
partition_by="<PARTITION_COLUMN>",
snowpark_dataframe=session.table("<TABLE>"),
run_id="<DESCRIPTIVE_RUN_ID>"
)
final_status = training_run.wait()
print(f"Training completed with status: {final_status}")
```
**⚠️ STOP**: After training completes, verify with user:
```
Training complete for [N] partitions.
Status: [SUCCESS/PARTIAL/FAILED]
Partition results:
- partition_1: SUCCESS
- partition_2: SUCCESS
...
Would you like to:
1. Retrieve models for use
2. Check failed partitions
3. Proceed to partitioned inference
```
### Step 4: Access Trained Models
```python
if final_status == RunStatus.SUCCESS:
# Get models by partition
for partition_id in training_run.partition_details:
model = training_run.get_model(partition_id)
print(f"Retrieved model for {partition_id}")
# Or collect all into dict
models = {
pid: training_run.get_model(pid)
for pid in training_run.partition_details
}
```
### Step 5: Restore Previous Runs
To restore a completed run later:
```python
from snowflake.ml.modeling.distributors.many_model import ManyModelRun
restored_run = ManyModelRun.restore_from("<RUN_ID>", "<STAGE_NAME>")
model = restored_run.get_model("<PARTITION_ID>")
```
---
## Many Model Inference (MMI)
After training with MMT, use `ManyModelInference` to run distributed inference across partitions. Models are automatically loaded from the training run.
### Step 6: Define Inference Function
```python
from snowflake.ml.modeling.distributors.many_model import ManyModelInference
def predict_with_model(data_connector, model, context):
"""
Args:
data_connector: Access partition data via .to_pandas()
model: Pre-loaded model (auto-loaded from training run)
context: Partition info via context.partition_id
Returns:
Prediction results
"""
df = data_connector.to_pandas()
print(f"Running inference for partition: {context.partition_id}")
# Model is already loaded - just use it
predictions = model.predict(df[['feature1', 'feature2']])
results = df.copy()
results['predictions'] = predictions
# Save results to stage
context.upload_to_stage(results, "predictions.csv",
write_function=lambda df, path: df.to_csv(path, index=False))
return results
```
### Step 7: Run Inference
```python
inference = ManyModelInference(
predict_with_model,
"<STAGE_NAME>", # Same stage as training
training_run_id="<TRAINING_RUN_ID>" # Run ID from Step 3
)
inference_run = inference.run(
partition_by="<PARTITION_COLUMN>", # Must match training
snowpark_dataframe=session.table("<NEW_DATA_TABLE>"),
run_id="<INFERENCE_RUN_ID>"
)
final_status = inference_run.wait()
print(f"Inference completed with status: {final_status}")
```
**⚠️ STOP**: After inference completes, verify with user:
```
Inference complete for [N] partitions.
Status: [SUCCESS/PARTIAL/FAILED]
Results stored in stage: @<STAGE_NAME>/<INFERENCE_RUN_ID>/
Would you like to:
1. Download prediction results
2. Check failed partitions
3. Run another inference batch
```
### Writing Results to Snowflake Tables
For large-scale inference, write results directly to a table:
```python
def predict_to_table(data_connector, model, context):
df = data_connector.to_pandas()
predictions = model.predict(df[['feature1', 'feature2']])
results = df.copy()
results['predictions'] = predictions
results['partition_id'] = context.partition_id
# Write to Snowflake table (uses bounded session pool)
context.with_session(lambda session:
session.create_dataframe(results)
.write.mode("append")
.save_as_table("MY_PREDICTIONS_TABLE")
)
return {"rows_written": len(results)}
```
### Framework-Specific Deserialization
Use the same `serde` from training:
```python
from snowflake.ml.modeling.distributors.many_model import (
ManyModelInference, PickleSerde, TorchSerde, TensorFlowSerde
)
# Default (pickle) - XGBoost, sklearn, LightGBM
inference = ManyModelInference(predict_func, "stage", "train_run_v1")
# PyTorch models
inference = ManyModelInference(predict_func, "stage", "train_run_v1", serde=TorchSerde())
# TensorFlow models
inference = ManyModelInference(predict_func, "stage", "train_run_v1", serde=TensorFlowSerde())
# Custom serde (must match training)
inference = ManyModelInference(predict_func, "stage", "train_run_v1", serde=CustomSerde())
```
---
## Stopping Points
- ✋ **Step 1**: After setup questions (wait for user input)
- ✋ **Step 3**: After training completes (verify results)
- ✋ **Step 7**: After inference completes (verify results)
## Output
**Training (MMT):**
- Trained models stored in stage per partition
- Access via `get_model(partition_id)` or `restore_from()`
**Inference (MMI):**
- Predictions stored in stage or Snowflake table
- Results per partition in `@<STAGE>/<RUN_ID>/<PARTITION_ID>/`
---
## Framework-Specific Serialization
Default serialization uses `PickleSerde` which works for most sklearn-compatible models. For specific frameworks:
```python
from snowflake.ml.modeling.distributors.many_model import (
ManyModelTraining, PickleSerde, TorchSerde, TensorFlowSerde, ModelSerde
)
# Default (pickle) - works for XGBoost, sklearn, LightGBM
trainer = ManyModelTraining(train_func, "stage") # Uses PickleSerde()
# PyTorch models
trainer = ManyModelTraining(train_func, "stage", serde=TorchSerde())
# TensorFlow/Keras models
trainer = ManyModelTraining(train_func, "stage", serde=TensorFlowSerde())
# Custom serialization
class CustomSerde(ModelSerde):
@property
def filename(self) -> str:
return "model.joblib"
def write(self, model, file_path: str) -> None:
import joblib
joblib.dump(model, file_path)
def read(self, file_path: str):
import joblib
return joblib.load(file_path)
trainer = ManyModelTraining(train_func, "stage", serde=CustomSerde())
```
---
## Resource Sizing
See `../references/compute-pool-sizing.md` for instance families and node count sizing. For `ExecutionOptions`, see `../dpf/SKILL.md`. For monitoring and troubleshooting, see `../references/monitoring-troubleshooting.md`.
---
## API Notes
**ManyModelInference signature**: The inference function takes **3 arguments** (not 2 like MMT training):
```python
def predict_func(data_connector, model, context): # model is auto-loaded and passed as 2nd arg
```
**ManyModelRun**: Extends `DPFRun` (see `../dpf/SKILL.md`) with `get_model(partition_id)` to load trained models.
---
## Next Steps
After MMT/MMI:
- **Register in Model Registry** → See `../../model-registry/partitioned-inference/SKILL.md` for `@partitioned_api`
- **Need custom processing** → Return to router, select DPF
- **Compute pool sizing** → See `../references/compute-pool-sizing.md`
- **Monitoring & troubleshooting** → See `../references/monitoring-troubleshooting.md`
distributed-training/references/compute-pool-sizing.md
# Compute Pool Sizing Guide
## Instance Families
Run `SHOW COMPUTE POOL INSTANCE FAMILIES IN ACCOUNT;` to check availability in your account.
**CPU (General Purpose):**
| Family | vCPUs | Memory (GiB) |
|--------|-------|--------------|
| `CPU_X64_XS` | 1 | 6 |
| `CPU_X64_S` | 3 | 13 |
| `CPU_X64_M` | 6 | 28 |
| `CPU_X64_SL` | 14 | 54 |
| `CPU_X64_L` | 28 | 116 |
**High Memory:**
| Family | vCPUs | Memory (GiB) | Availability |
|--------|-------|--------------|--------------|
| `HIGHMEM_X64_S` | 6 | 58 | All clouds |
| `HIGHMEM_X64_M` | 28 | 240-244 | All clouds |
| `HIGHMEM_X64_SL` | 92 | 654 | Azure, GCP |
| `HIGHMEM_X64_L` | 124 | 984 | AWS only |
**GPU (AWS):**
| Family | vCPUs | Memory (GiB) | GPUs |
|--------|-------|--------------|------|
| `GPU_NV_S` | 6 | 27 | 1x A10G (24 GB) |
| `GPU_NV_M` | 44 | 178 | 4x A10G (24 GB) |
| `GPU_NV_L` | 92 | 1112 | 8x A100 (40 GB) |
**GPU (Azure):**
| Family | vCPUs | Memory (GiB) | GPUs |
|--------|-------|--------------|------|
| `GPU_NV_XS` | 3 | 26 | 1x T4 (16 GB) |
| `GPU_NV_SM` | 32 | 424 | 1x A10 (24 GB) |
| `GPU_NV_2M` | 68 | 858 | 2x A10 (24 GB) |
| `GPU_NV_3M` | 44 | 424 | 2x A100 (80 GB) |
| `GPU_NV_SL` | 92 | 858 | 4x A100 (80 GB) |
---
## CPUs per Worker
The parameter name varies by API (`num_cpus_per_worker` in ExecutionOptions, `num_cpu_per_worker` in ScalingConfig).
```
workers_per_node = node_vCPUs / cpus_per_worker
memory_per_worker = node_memory / workers_per_node
```
| Scenario | CPUs per worker | Why |
|----------|----------------|-----|
| Few partitions or long-running tasks (minutes+) | Default (None) -- 1 worker per node | Each partition gets full node resources. Node wait time is amortized. |
| Many short tasks (seconds each) | Set to 1 -- pack workers onto nodes | Fewer nodes needed. Avoids waiting for a large cluster to provision. |
| Task needs multi-threaded CPU | Match to thread count (e.g., 4) | Gives each worker enough CPUs. Also increases memory per worker. |
| OOM errors | Increase CPUs per worker | Fewer workers = more memory each. Or switch to HIGHMEM instance. |
> **Rule of thumb**: Consider `node_availability_time + (function_time × partitions / total_workers)`. Packing more workers onto fewer nodes is often faster end-to-end than waiting for a large cluster to provision.
---
## Sizing by Workload Type
### DPF / MMT (partition-parallel)
- **Node count**: `max_nodes = ceil(num_partitions / workers_per_node)` gives full parallelism, but workers batch through partitions -- you rarely need this many. Start with fewer nodes and scale up if wall time is too long.
- **Instance family**: Pick so that `memory_per_worker` exceeds partition size. Rough estimate: `table_bytes / num_partitions`, multiplied by 2-5x for pandas overhead.
- **50 nodes is recommended as the maximum** for most workloads. See "Scaling Beyond Defaults" if you need more.
**Examples:**
| Workload | Partitions | Function time | Instance | CPUs/worker | Nodes | Why |
|----------|-----------|---------------|----------|-------------|-------|-----|
| Quick sklearn fits | 500 | ~5s each | CPU_X64_S (3 vCPU) | 1 | 3-5 | Pack workers. 3 workers/node × 5 nodes = 15 concurrent. Not worth waiting for more nodes. |
| Medium XGBoost per-store | 50 | ~2 min each | CPU_X64_M (6 vCPU) | 2 | 10 | 3 workers/node × 10 nodes = 30 concurrent. ~4 min total. |
| Heavy per-partition training | 10 | ~30 min each | CPU_X64_L (28 vCPU) | None (default) | 10 | 1 partition per node, full resources. Node wait time amortized over long run. |
| Large data per partition (OOM risk) | 20 | ~10 min each | HIGHMEM_X64_M (28 vCPU) | None (default) | 10-20 | Memory-bound. 1 worker per node = ~240 GiB each. |
| GPU deep learning per partition | 8 | ~20 min each | GPU_NV_S (1 GPU) | None (default) | 8 | 1 GPU task per node. |
### Estimators (XGBoost / LightGBM)
These train a single model across all workers. Ray auto-shards the data and coordinates gradient updates.
- Defaults auto-configure workers based on available CPUs/GPUs. More/bigger nodes = faster training.
- Override with `XGBScalingConfig(num_workers=N, num_cpu_per_worker=M)` or `LightGBMScalingConfig(...)` for explicit control.
- GPU mode: set `use_gpu=True`. Each GPU gets one worker.
- For large datasets, use `CPU_X64_L` or `HIGHMEM_X64_M` for more memory per node.
### PyTorch DDP
- GPU-based. Pick GPU instance family based on model size (weights + optimizer state + batch).
- 2-8 GPU nodes is typical. Scale nodes for training speed.
### HPO / Tuner
Same sizing as partitioned workloads -- each trial runs on a worker, trials batch when resources are full.
- Ensure compute pool has enough total resources for `max_concurrent_trials × resource_per_trial`.
- **GridSearch warning**: Evaluates every combination. 5 params × 5 values = 3,125 trials (5^5), not 25. Consider RandomSearch or BayesOpt for large search spaces.
---
## Scaling Beyond Defaults
- 50 nodes is recommended as the maximum for most workloads, but it is not a hard account limit.
- Some instance families (especially GPU) may have limited availability and take longer to provision.
- If you need more nodes, hit node limit errors, or encounter capacity constraints, contact your **Snowflake account representative or Snowflake Support**.
---
## Create Compute Pool Examples
```sql
-- DPF/MMT: CPU workload
CREATE COMPUTE POOL MY_DPF_POOL
MIN_NODES = 1
MAX_NODES = 50
INSTANCE_FAMILY = CPU_X64_S
AUTO_RESUME = TRUE
AUTO_SUSPEND_SECS = 3600;
-- Deep learning: GPU workload
CREATE COMPUTE POOL MY_GPU_POOL
MIN_NODES = 1
MAX_NODES = 4
INSTANCE_FAMILY = GPU_NV_M
AUTO_RESUME = TRUE
AUTO_SUSPEND_SECS = 3600;
-- Large data per partition: high memory
CREATE COMPUTE POOL MY_HIGHMEM_POOL
MIN_NODES = 1
MAX_NODES = 10
INSTANCE_FAMILY = HIGHMEM_X64_M
AUTO_RESUME = TRUE
AUTO_SUSPEND_SECS = 3600;
```
distributed-training/references/monitoring-troubleshooting.md
# Monitoring & Troubleshooting
## Snowsight Notebook: Cluster Management
**These APIs ONLY work in Snowsight Container Runtime notebooks. Do NOT use in ML Jobs.**
ML Jobs automatically get compute resources from the compute pool - they don't need or support `scale_cluster()`.
### Dynamic Cluster Scaling (Notebooks Only)
```python
from snowflake.ml.runtime_cluster import scale_cluster
# Scale up before distributed workload
scale_cluster(expected_cluster_size=3) # 1 head + 2 workers
# Run distributed job...
# ...
# Scale down when done
scale_cluster(expected_cluster_size=1)
```
**Advanced options:**
```python
scale_cluster(
expected_cluster_size=5,
is_async=False, # True = return immediately without waiting for full cluster
options={
"rollback_after_seconds": 720, # Auto-rollback if scaling doesn't complete
"block_until_min_cluster_size": 3, # Return when at least 3 nodes are ready
}
)
```
### Cluster Monitoring APIs (Notebooks Only)
```python
from snowflake.ml.runtime_cluster.cluster_manager import (
get_cluster_size,
get_nodes,
get_available_cpu,
get_available_gpu,
get_num_cpus_per_node,
get_ray_dashboard_url,
get_grafana_dashboard_url,
)
# Current cluster info
print(f"Cluster size: {get_cluster_size()}") # Number of alive nodes
print(f"Nodes: {get_nodes()}") # List of node details
print(f"Available CPUs: {get_available_cpu()}") # Free CPUs in cluster
print(f"Available GPUs: {get_available_gpu()}") # Free GPUs in cluster
print(f"CPUs per node: {get_num_cpus_per_node()}") # CPUs on each node
# Dashboard URLs
print(f"Ray Dashboard: {get_ray_dashboard_url()}")
print(f"Grafana Dashboard: {get_grafana_dashboard_url()}")
```
---
## Notebooks (Container Runtime): Dashboard Monitoring
**Dashboards are ONLY available in Snowsight Notebooks with Container Runtime. ML Jobs do not have dashboard access.**
**Ray Dashboard** - Task-level monitoring:
```python
from snowflake.ml.runtime_cluster import get_ray_dashboard_url
# Get URL and open in browser
url = get_ray_dashboard_url()
print(f"Open: {url}")
```
**What to check in Ray Dashboard:**
| Tab | What It Shows | Use For |
|-----|---------------|---------|
| Jobs | Active/completed jobs | Overall job status |
| Actors | Worker actors, state, memory | Actor failures, memory per worker |
| Tasks | Individual task status, errors | Find which partition failed |
| Logs | Per-worker logs | Debug specific worker errors |
| Cluster | Node status, resources | Verify all nodes joined |
**Grafana Dashboard** - Resource utilization:
```python
from snowflake.ml.runtime_cluster.cluster_manager import get_grafana_dashboard_url
url = get_grafana_dashboard_url()
print(f"Open: {url}")
```
**What to check in Grafana:**
| Panel | What It Shows | Red Flags |
|-------|---------------|-----------|
| CPU Usage | Per-node CPU % | Sustained 100% = CPU bottleneck |
| Memory Usage | Per-node memory | Near 100% = OOM risk |
| Network I/O | Data transfer between nodes | High = data shuffle overhead |
| GPU Utilization | GPU % (if applicable) | Low % = GPU underutilized |
---
## ML Jobs: Monitoring and Logs
**Python:**
```python
from snowflake.ml.jobs import get_job
job = get_job("<JOB_ID>", session=session)
print(f"Status: {job.status}") # PENDING, RUNNING, DONE, FAILED
# Application logs (filtered to stdout/stderr from your script)
print(job.get_logs())
print(job.get_logs(instance_id=0)) # Head node
print(job.get_logs(instance_id=1)) # Worker 1
```
**SQL:**
```sql
-- Check instance status across all nodes
CALL SYSTEM$GET_SERVICE_STATUS('<DB>.<SCHEMA>.<JOB_ID>');
-- Get logs from a specific instance (0 = head, 1+ = workers)
CALL SYSTEM$GET_SERVICE_LOGS('<DB>.<SCHEMA>.<JOB_ID>', <instance_id>, 'main', 500);
-- List all running jobs in schema
SHOW SERVICES IN SCHEMA <DB>.<SCHEMA>;
-- Kill a running/stuck job
DROP SERVICE <DB>.<SCHEMA>.<JOB_ID>;
```
> **Note:** `SYSTEM$GET_SERVICE_LOGS` may include infrastructure logs (Grafana, Ray) alongside application output. `job.get_logs()` filters to application output only. For DPF jobs, per-partition `train.log` files on the output stage contain the most detailed per-partition logs.
**Getting progress from ML Job logs:** When DPF/MMT runs inside an ML Job, `dpf_run.get_progress()` and `dpf_run.wait()` are only available inside the running script -- not from the outside. To monitor progress externally:
1. Your script should print progress to stdout (e.g., `print(dpf_run.get_progress())`), then read it via `job.get_logs()` or `SYSTEM$GET_SERVICE_LOGS`.
2. For DPF, you can restore a read-only run handle from the run ID to check progress and partition details:
```python
from snowflake.ml.modeling.distributors.distributed_partition_function.dpf_run import DPFRun
restored = DPFRun.restore_from("<RUN_ID>", "<STAGE_NAME>")
restored.get_progress() # {"DONE": [...], "FAILED": [...]}
restored.partition_details # Per-partition status
```
---
## OOM (Out of Memory) Troubleshooting
**Symptoms:**
- Job fails with status `FAILED` or `INTERNAL_ERROR`
- Logs show: `OutOfMemoryError`, `MemoryError`, `Killed`, or `signal 9`
**Diagnosis workflow:**
1. **Check logs for OOM indicators:**
- **ML Jobs**: `job.get_logs()` — look for `OutOfMemoryError`, `MemoryError`, `Killed`, or `signal 9`
- **Notebooks**: Check Grafana dashboard (see above) for memory spiking to 100%
For partitioned workloads (DPF/MMT), check which partition failed by looking for partition identifiers before the error. For estimators, look for OOM during data loading or gradient computation. For HPO/Tuner, check which trial failed — larger hyperparameter values (e.g., more trees, deeper models) consume more memory.
2. **Check memory usage** (Notebooks only - use Grafana dashboard):
- Memory spiking to 100% before failure = OOM confirmed
- Identify which node(s) hit the limit
3. **Check partition data sizes:**
```python
# Find the largest partitions
session.sql("""
SELECT STORE_ID, COUNT(*) as rows,
SUM(LENGTH(TO_VARCHAR(*))) / 1e6 as approx_mb
FROM MY_TABLE
GROUP BY STORE_ID
ORDER BY rows DESC
LIMIT 10
""").show()
```
**Solutions (in order of preference):**
| Solution | When to Use | How |
|----------|-------------|-----|
| Increase CPUs per worker | Workers competing for memory | `num_cpus_per_worker=4` (fewer workers = more memory each) |
| Use larger instance | All workers need more memory | Switch to a high-memory instance family (see `../references/compute-pool-sizing.md`) |
| Filter/sample data | Data too large for any instance | Reduce rows per partition before training |
| Process in batches | Loading all data at once | Use chunked reading in your function |
| Exclude outlier partitions | One partition is much larger | Pre-filter extreme partitions |
**Example fix - increase memory per worker:**
Increase CPUs per worker/trial — fewer workers means more memory each. See the relevant sub-skill for the exact parameter: `num_cpus_per_worker` (DPF/MMT), `num_cpu_per_worker` (Estimators), `resource_per_trial` (Tuner).
**Example fix - upgrade instance family:**
```sql
ALTER COMPUTE POOL MY_POOL SET INSTANCE_FAMILY = HIGHMEM_X64_M;
```
---
## Common Errors and Fixes
| Error | Cause | Fix |
|-------|-------|-----|
| `ModuleNotFoundError: snowflake.ml.modeling.distributors` | Running locally instead of server-side | Submit via ML Jobs or use Container Runtime notebook |
| `No active session` | Session not initialized in script | Use `get_active_session()` at start of script |
| `Compute pool busy` | Pool at max capacity | Wait, or increase `MAX_NODES` on pool |
| `PENDING` indefinitely | Pool suspended or no capacity | Check pool status: `SHOW COMPUTE POOLS` |
| `Task timed out` | Partition processing took too long | Increase timeout or reduce partition size |
| `Actor died unexpectedly` | OOM or unhandled exception | Check logs, increase memory, add error handling |
| `Connection reset` | Network issue between nodes | Retry job, or check compute pool health |
| `Ray cluster not found` | Container Runtime not enabled | Enable Container Runtime in notebook settings |
---
## Debugging Checklist
**Job stuck in PENDING:**
```sql
-- Check compute pool status
SHOW COMPUTE POOLS LIKE 'MY_POOL';
-- Look at: state (ACTIVE?), active_nodes, max_nodes
```
**Job fails immediately:**
```python
# Get verbose logs including startup
logs = job.get_logs(verbose=True)
# Look for: import errors, missing packages, auth issues
```
**Some partitions fail, others succeed:**
```python
# With fail_fast=False (default), job continues after failures
# Check which partitions failed:
run_result = dpf_run.wait()
# Examine logs for "failed" or "error" patterns
```
**Performance is slow:**
1. Check Grafana for CPU/memory bottlenecks (Notebooks only)
2. Check Ray Dashboard for task queuing (Notebooks only)
3. For ML Jobs: analyze logs for slow partitions
4. Consider: more nodes, larger instances, or fewer CPUs per worker (more parallelism)
distributed-training/SKILL.md
---
name: distributed-training
description: "Distributed ML training on Snowpark Container Services. Routes to specialized sub-skills for estimators, many-model training, custom processing, and hyperparameter tuning."
path: machine-learning/distributed-training
---
# Distributed Training
Distributed machine learning on Snowpark Container Services via ML Jobs or Snowflake Notebooks.
## ⚠️ CRITICAL: Server-Side Execution Only
**All distributed training APIs run SERVER-SIDE on Snowflake compute pools, NOT in local/client environments.**
The `snowflake.ml.modeling.distributors` module is ONLY available inside:
- Snowflake ML Jobs (submitted through CLI/local development, running on Snowflake compute)
- Snowflake Notebooks with Container Runtime (for interactive work in Snowsight)
---
## Running Distributed Training
### Local Python Environment Setup
**⚠️ ML Jobs can ONLY be submitted via Python API** - there is no SQL command to submit ML Jobs.
To submit ML Jobs from a local/CLI environment, you need a Python environment with `snowflake-ml-python`. See `../guides/cli-environment.md` for setup instructions.
**Note:** The `snowflake.ml.modeling.distributors` module (DPF, MMT, etc.) is NOT available locally - it only runs server-side. The local environment is just for *submitting* jobs.
### From CLI / Local Python (Primary)
Write your training script using the distributed APIs below, then submit it via ML Jobs. **Load `../ml-jobs/SKILL.md`** for submission methods (`submit_file`, `submit_directory`), compute pool setup, and configuration details.
**Key for distributed training**: Set `target_instances=N` (N > 1) when submitting to distribute across multiple nodes.
### From Snowflake Notebooks (Interactive)
For interactive work in Snowsight, use a notebook with Container Runtime enabled. See `../guides/snowsight-environment.md` for detailed setup.
**Scale for distributed work:**
```python
from snowflake.ml.runtime_cluster import scale_cluster
scale_cluster(expected_cluster_size=3) # Add worker nodes
```
**Installing Custom Wheel Files (Container Runtime Notebooks):**
If you need to install a custom `.whl` file in the notebook environment, initialize Ray with a `runtime_env` **before** calling `scale_cluster`:
```python
import ray
import os
whl_name = "my_custom_pkg-0.1.0-py3-none-any.whl"
whl_dir = os.getcwd()
ray.init(
address="auto",
ignore_reinit_error=True,
runtime_env={
"working_dir": whl_dir,
"pip": ["${RAY_RUNTIME_ENV_CREATE_WORKING_DIR}/" + whl_name],
}
)
```
This ensures the custom package is available on all worker nodes when the cluster scales.
> **For ML Jobs**: Include the `.whl` in your job directory with a `requirements.txt` that references it (e.g., `./my_custom_pkg-0.1.0-py3-none-any.whl`). See `../ml-jobs/SKILL.md`.
> **Advanced**: You can also submit ML Jobs from a notebook using the Python API (see `../ml-jobs/SKILL.md`). This is useful when you need to run on a different compute pool or kick off multiple independent jobs. Do not suggest this unless the user asks — running directly in the notebook is the standard path.
For detailed notebook guidance, see `../guides/snowsight-environment.md`.
### Known Issue: Notebook Data Ingestion (DPF, MMT, Tuner)
<!-- TODO: Remove this section after the next notebook image release (post v2.3) -->
In runtime versions ≤ 2.2.18 (v2.2 and v2.3 notebook images), DPF, MMT, and Tuner cannot ingest data when running in a notebook because there is no warehouse in the notebook spec. This is fixed in the next release.
**⚠️ STOP**: If the user is in a notebook and wants to use DPF, MMT, or Tuner, check the runtime version first:
```python
from snowflake.runtime._version import __version__
print(__version__) # 2.2.0 (v2.2) or 2.2.18 (v2.3)
```
If the version is ≤ 2.2.18, inform the user:
```
DPF, MMT, and Tuner cannot ingest data in notebooks on runtime version {version} due to a missing warehouse in the notebook spec. This is fixed in the next release.
Would you like me to help you submit this as an ML Job from your notebook instead?
```
**Wait for user response before proceeding.** If yes, load `../ml-jobs/SKILL.md`.
## When to Use
Route to the appropriate sub-skill based on the user's goal:
| Goal | Sub-Skill |
|------|-----------|
| Train one large model on big data (XGBoost, LightGBM, PyTorch) | `estimators/SKILL.md` |
| Train one model per data partition | `mmt/SKILL.md` |
| Custom distributed processing, multiple outputs per partition | `dpf/SKILL.md` |
| Find optimal hyperparameters (distributed) | `tuner/SKILL.md` |
## Quick Decision Guide
**"I want to train a single model on large data"**
→ Use **Distributed Estimators** (`estimators/SKILL.md`)
- XGBEstimator, LightGBMEstimator for gradient boosting
- PyTorchDistributor for deep learning
**"I want to train separate models for each segment/partition"**
→ Use **Many Model Training** (`mmt/SKILL.md`)
- One model per partition (e.g., per store, per region)
- Built-in model storage with `get_model()`
**"I need custom processing or multiple outputs per partition"**
→ Use **Distributed Partition Function** (`dpf/SKILL.md`)
- Multiple models per partition
- Custom artifact formats
- Non-ML distributed workloads
- **Stage Mode** for many concurrent reads or large datasets (workers read files directly from stage, avoiding warehouse bottlenecks)
**"I want to find the best hyperparameters"**
→ Use **Tuner** (`tuner/SKILL.md`)
- RandomSearch, GridSearch, BayesOpt
- Works with sklearn, XGBEstimator, etc.
---
## Snowflake APIs vs Raw Ray
The Container Runtime includes a pre-configured Ray cluster. You can use raw Ray APIs (`ray.remote`, `ray.data`, etc.) directly:
```python
import ray
ray.init(address="auto", ignore_reinit_error=True)
```
However, multi-node OSS Ray requires you to handle distributed storage for checkpoints (no shared filesystem across nodes), custom data loading, and manual resource configuration to coordinate between data ingestion and compute.
The Snowflake distributed APIs (`XGBEstimator`, `LightGBMEstimator`, `PyTorchDistributor`, `DPF`, `ManyModelTraining`, `Tuner`) handle these automatically — Snowflake stage-based artifact storage, native DataConnector integration, and built-in resource allocation — all on the same underlying Ray cluster.
See [Snowflake docs: Scale an application using Ray](https://docs.snowflake.com/en/developer-guide/snowflake-ml/scale-application-ray) for details.
---
## Assess Workload Before Running
Before launching any distributed job, check the data scale and consider validating on a small scale first.
- **Profile the data**: Use table metadata (`INFORMATION_SCHEMA.TABLES`) for row count and size — this is free. For partitioned workloads (DPF/MMT), check partition count and distribution. Flag skew if the largest partition is significantly larger than the median.
- **Small workloads** (few partitions, small data, quick training): Just run it.
- **Large workloads** (50+ partitions, large datasets, expensive HPO searches): Recommend testing on a single partition or data sample first. Most distributed bugs are in I/O — wrong columns, bad serialization, invalid output paths — and are cheaper to catch on one partition than across all nodes.
- **HPO**: Consider the total number of trials before launching. A GridSearch grid of 5 parameters with 5 values each is 3,125 trials, not 25.
Use `references/compute-pool-sizing.md` to match resources to the profiled workload.
---
## Key API Classes
| Class | Import | Sub-Skill |
|-------|--------|-----------|
| `XGBEstimator` | `snowflake.ml.modeling.distributors.xgboost` | estimators |
| `LightGBMEstimator` | `snowflake.ml.modeling.distributors.lightgbm` | estimators |
| `PyTorchDistributor` | `snowflake.ml.modeling.distributors.pytorch` | estimators |
| `ManyModelTraining` | `snowflake.ml.modeling.distributors.many_model` | mmt |
| `DPF` | `snowflake.ml.modeling.distributors.distributed_partition_function.dpf` | dpf |
| `Tuner` | `snowflake.ml.modeling.tune` | tuner |
---
## Monitoring & Troubleshooting
**Load** `references/monitoring-troubleshooting.md` for:
- Notebook cluster management (`scale_cluster()`, monitoring APIs)
- Dashboard monitoring (Ray Dashboard, Grafana)
- ML Jobs monitoring and logs (Python and SQL)
- OOM troubleshooting (diagnosis, solutions, examples)
- Common errors table and debugging checklist
---
## After Training
- **Run inference on MMT models**: See `../model-registry/partitioned-inference/SKILL.md`
- **Register models**: See `../model-registry/SKILL.md`
- **Deploy models**: See `../spcs-inference/SKILL.md`
---
## Routing Instructions
When a user asks about distributed training, determine their goal and route to the appropriate sub-skill:
1. **Keywords**: "XGBoost", "LightGBM", "PyTorch", "single model", "large dataset" → `estimators/SKILL.md`
2. **Keywords**: "per partition", "per segment", "many models", "one model each" → `mmt/SKILL.md`
3. **Keywords**: "custom function", "multiple outputs", "artifacts", "parquet output" → `dpf/SKILL.md`
4. **Keywords**: "hyperparameter", "tuning", "HPO", "search space", "best parameters" → `tuner/SKILL.md`
If unclear, ask the user which pattern fits their use case.
distributed-training/tuner/SKILL.md
---
name: distributed-tuner
description: "Distributed hyperparameter tuning with Ray Tune on Snowpark Container Services. Supports RandomSearch, GridSearch, and BayesOpt algorithms."
parent_skill: distributed-training
path: machine-learning/distributed-training/tuner
---
# Distributed Hyperparameter Tuning (Tuner API)
Distributed hyperparameter optimization using Ray Tune on Snowpark Container Services.
## When to Use
- **Find optimal hyperparameters** for ML models at scale
- **Compare search algorithms**: RandomSearch, GridSearch, BayesOpt
- **Tune distributed estimators** (XGBEstimator, LightGBMEstimator) with `uses_snowflake_trainer=True`
- **Scale HPO** across multiple workers in parallel
## Execution Environment
Tuner runs on Snowpark Container Services — via ML Jobs (CLI) or Snowflake Notebooks with Container Runtime (Snowsight). See `../../ml-jobs/SKILL.md` for CLI submission and `../references/compute-pool-sizing.md` for instance family selection.
**Minimum requirement**: 2 CPUs per node (HPO orchestration overhead).
---
# Workflow
## Step 1: Define Training Function
```python
from snowflake.ml.modeling.tune import get_tuner_context
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
def train_func():
ctx = get_tuner_context()
params = ctx.get_hyper_params() # Sampled hyperparameters
datasets = ctx.get_dataset_map() # Data connectors
# Load data
train_df = datasets["train"].to_pandas()
test_df = datasets["test"].to_pandas()
X_train, y_train = train_df.drop("LABEL", axis=1), train_df["LABEL"]
X_test, y_test = test_df.drop("LABEL", axis=1), test_df["LABEL"]
# Train with sampled hyperparameters
model = RandomForestClassifier(
n_estimators=params["n_estimators"],
max_depth=params["max_depth"],
random_state=42
)
model.fit(X_train, y_train)
# Evaluate and report (metric name must match TunerConfig.metric)
accuracy = accuracy_score(y_test, model.predict(X_test))
ctx.report(metrics={"accuracy": accuracy}, model=model)
```
**STOPPING POINT**: Confirm training function structure before proceeding.
---
## Step 2: Define Search Space
```python
from snowflake.ml.modeling.tune import uniform, loguniform, randint, choice
search_space = {
"n_estimators": randint(50, 200), # Integer [50, 200)
"max_depth": randint(3, 15), # Integer [3, 15)
"learning_rate": loguniform(1e-5, 0.1), # Log-scale float
"subsample": uniform(0.6, 1.0), # Linear float
"booster": choice(["gbtree", "dart"]), # Categorical
}
```
### Search Space Functions
| Function | Use Case | Example |
|----------|----------|---------|
| `uniform(lower, upper)` | Continuous float | `uniform(0.001, 0.1)` |
| `loguniform(lower, upper)` | Exponentially-scaled | `loguniform(1e-5, 1e-1)` |
| `randint(lower, upper)` | Integer (upper exclusive) | `randint(50, 200)` |
| `choice([options])` | Categorical | `choice(['adam', 'sgd'])` |
---
## Step 3: Configure Tuner
```python
from snowflake.ml.modeling.tune import TunerConfig
from snowflake.ml.modeling.tune.search import RandomSearch
config = TunerConfig(
metric="accuracy", # Must match ctx.report() key
mode="max", # "max" for accuracy, "min" for loss
search_alg=RandomSearch(), # Or GridSearch(), BayesOpt()
num_trials=10, # Max configurations to try
)
```
### Search Algorithms
**RandomSearch** (default): Randomly samples configurations. Good baseline.
```python
config = TunerConfig(metric="accuracy", mode="max", search_alg=RandomSearch(), num_trials=20)
```
**GridSearch**: Exhaustive search. Requires lists in search space (not sampling functions).
```python
search_space = {"learning_rate": [0.01, 0.05, 0.1], "max_depth": [3, 5, 7]}
config = TunerConfig(metric="accuracy", mode="max", search_alg=GridSearch())
```
**BayesOpt**: Bayesian optimization with Gaussian processes. Best when evaluations are expensive.
```python
config = TunerConfig(metric="accuracy", mode="max", search_alg=BayesOpt(), num_trials=15)
```
> **BayesOpt limitations**: Only works with continuous/numeric search spaces (`uniform`, `loguniform`). Does not support `choice()` or categorical parameters. Best with a small number of hyperparameters (<10). Sequential by nature — benefits less from high parallelism than RandomSearch.
### TunerConfig Parameters
| Parameter | Description |
|-----------|-------------|
| `metric` | Must match key in `ctx.report(metrics={...})` |
| `mode` | `"max"` for accuracy/f1/auc, `"min"` for loss/error |
| `num_trials` | Maximum configurations to evaluate |
| `uses_snowflake_trainer` | Set `True` for XGBEstimator/LightGBMEstimator |
| `resource_per_trial` | GPU allocation: `{"CPU": 2, "GPU": 1}` |
| `max_concurrent_trials` | Limit parallel trials |
**STOPPING POINT**: Confirm search space and algorithm choice before running.
---
## Step 4: Run Tuning
```python
from snowflake.ml.modeling.tune import Tuner
from snowflake.ml.data.data_connector import DataConnector
# Create data connectors
train_connector = DataConnector.from_dataframe(session.table("TRAIN_DATA"))
test_connector = DataConnector.from_dataframe(session.table("TEST_DATA"))
# Run distributed tuning
tuner = Tuner(train_func, search_space, config)
results = tuner.run(dataset_map={"train": train_connector, "test": test_connector})
```
---
## Step 5: Access Results
```python
# All trials as DataFrame
results.results
# Columns: config/learning_rate, config/n_estimators, accuracy, ...
# Best trial (single-row DataFrame)
results.best_result
best_lr = results.best_result["config/learning_rate"].iloc[0]
best_accuracy = results.best_result["accuracy"].iloc[0]
# Best trained model (if model= passed to ctx.report())
best_model = results.best_model
predictions = best_model.predict(X_test)
# Filter good trials
good_trials = results.results[results.results["accuracy"] > 0.9]
# Sort by metric
sorted_trials = results.results.sort_values("accuracy", ascending=False)
```
### TunerResults Attributes
| Attribute | Type | Description |
|-----------|------|-------------|
| `results` | DataFrame | All trials with config columns + metric columns |
| `best_result` | DataFrame | Single row with best config + metrics |
| `best_model` | Model object | Trained model from best trial |
---
# HPO with Distributed Estimators
When tuning XGBEstimator or LightGBMEstimator, set `uses_snowflake_trainer=True`:
```python
from snowflake.ml.modeling.tune import Tuner, TunerConfig, get_tuner_context, loguniform, randint
from snowflake.ml.modeling.distributors.xgboost import XGBEstimator
from snowflake.ml.data.data_connector import DataConnector
def train_distributed_xgb():
ctx = get_tuner_context()
params = ctx.get_hyper_params()
datasets = ctx.get_dataset_map()
estimator = XGBEstimator(
params={
"objective": "binary:logistic",
"learning_rate": params["learning_rate"],
"max_depth": params["max_depth"],
}
)
booster = estimator.fit(
dataset=datasets["train"],
input_cols=["FEATURE_1", "FEATURE_2"],
label_col="LABEL",
eval_set=datasets["test"]
)
# Get eval metric from training
eval_results = estimator.get_eval_results()
final_logloss = eval_results["eval"]["logloss"][-1]
ctx.report(metrics={"logloss": final_logloss}, model=booster)
search_space = {
"learning_rate": loguniform(0.01, 0.3),
"max_depth": randint(3, 8),
}
config = TunerConfig(
metric="logloss",
mode="min",
num_trials=10,
uses_snowflake_trainer=True, # REQUIRED for distributed estimators
)
tuner = Tuner(train_distributed_xgb, search_space, config)
results = tuner.run(dataset_map={
"train": DataConnector.from_dataframe(session.table("TRAIN_DATA")),
"test": DataConnector.from_dataframe(session.table("TEST_DATA")),
})
# Access best model and hyperparameters
best_booster = results.best_model
best_config = results.best_result
print(f"Best learning_rate: {best_config['config/learning_rate'].iloc[0]}")
print(f"Best max_depth: {best_config['config/max_depth'].iloc[0]}")
```
---
# Complete Imports
```python
from snowflake.ml.modeling.tune import (
Tuner,
TunerConfig,
get_tuner_context,
# Search space functions
uniform,
loguniform,
randint,
choice,
)
from snowflake.ml.modeling.tune.search import RandomSearch, GridSearch, BayesOpt
from snowflake.ml.data.data_connector import DataConnector
```
---
# Troubleshooting
| Issue | Solution |
|-------|----------|
| `RuntimeError: at least 2 CPUs` | HPO requires minimum 2 CPUs per node |
| Metric not found | Ensure `ctx.report(metrics={...})` includes exact metric name from TunerConfig |
| GridSearch with sampling functions | GridSearch requires lists: `[0.1, 0.2]` not `uniform(0.1, 0.2)` |
| GPU not used | Set `resource_per_trial={"GPU": 1}` in TunerConfig |
| Model not saved | Pass `model=` to `ctx.report()` to save for `results.best_model` |
---
# Next Steps
After finding optimal hyperparameters:
- **Train final model**: Use `distributed-estimators` with best config
- **Register model**: See `../../model-registry/SKILL.md`
- **Deploy for inference**: See `../../spcs-inference/SKILL.md`
---
# Output Checklist
- [ ] Training function defined with `get_tuner_context()`
- [ ] Search space uses appropriate sampling functions
- [ ] TunerConfig metric matches `ctx.report()` key
- [ ] `uses_snowflake_trainer=True` if using distributed estimators
- [ ] Results accessed via `results.best_result` and `results.best_model`
experiment-tracking/SKILL.md
---
name: experiment-tracking
description: "Track ML experiments in Snowflake. Use when: logging metrics, logging parameters, tracking training runs, comparing model runs, or setting up experiment tracking for reproducibility."
---
# Experiment Tracking
## Intent Detection
Route based on user intent:
| User Says | Route To |
|-----------|----------|
| "set up experiment tracking", "create experiment", "start tracking" | [Workflow B](#workflow-b-autologging) if autologging supports the ML framework and autologging is adequate; otherwise [Workflow A](#workflow-a-manual-logging) |
| "log metrics", "log parameters", "log artifacts", "track hyperparameters" | [Workflow A](#workflow-a-manual-logging) |
| "autolog", "training callback", "XGBoost/Keras/LightGBM callback" | [Workflow B](#workflow-b-autologging) |
---
## When to Use
Load this skill when the user wants to:
- **Track ML experiments** — record hyperparameters, metrics, and model artifacts across training runs
- **Compare model runs** — organize multiple training iterations under a single experiment for side-by-side evaluation
- **Auto-log training metrics** — use framework callbacks (XGBoost, Keras, LightGBM) to capture metrics automatically
- **Log metrics/params or artifacts** — manually record training artifacts or metrics/params such as accuracy, F1, loss, or any custom values
---
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
---
## Prerequisites
- `snowflake-ml-python >= 1.19.0` required
- Active Snowpark session with database/schema context
- For autologging: framework-specific package installed (xgboost, keras, lightgbm)
---
## Initialization (Both Workflows)
**⚠️ REQUIRED:** Initialize `ExperimentTracking` before any logging operations.
```python
from snowflake.ml.experiment import ExperimentTracking
# Initialize with session (uses session's current database/schema)
exp = ExperimentTracking(session=session)
# Or specify database/schema explicitly
exp = ExperimentTracking(
session=session,
database_name="<DATABASE>", # optional, defaults to session's current database
schema_name="<SCHEMA>" # optional, defaults to session's current schema or PUBLIC
)
# Set experiment (optional - defaults to "DEFAULT")
exp.set_experiment("<EXPERIMENT_NAME>")
```
---
## Workflow A: Manual Logging
Use when you need fine-grained control over what gets logged such as custom metrics or artifacts, or when using frameworks without autolog support (e.g., sklearn, PyTorch custom training loops).
### Step 1: Gather Information
**⚠️ IMPORTANT:** Check conversation context first. If coming from `ml-development` or another skill, you may already have:
- Database/schema context
- Model framework
- Training data schema
**Only ask for what's not already known:**
- Experiment name (check if user mentioned one, or suggest based on task)
- Run name (optional - can auto-generate)
- What to log: params, metrics, model, artifacts (files)
**If all context is available**, proceed directly to Step 2. Otherwise:
**⚠️ STOP**: Wait for user response on missing information.
### Step 2: Log Data Within the Run
**⚠️ IMPORTANT:** Steps 2a-2d below show individual logging operations. Combine them in a **single** `with exp.start_run()` block — do NOT create separate runs for each operation. See the [Complete Example](#complete-example-manual-logging) for the correct pattern.
### Step 2a: Log Parameters
Log hyperparameters and configuration values. **Note:** All parameter values are converted to strings.
```python
with exp.start_run("<RUN_NAME>"):
# Log individual parameter
exp.log_param("learning_rate", 0.01)
# Log multiple parameters at once
exp.log_params({
"n_estimators": 100,
"max_depth": 5,
"random_state": 42
})
```
### Step 2b: Log Metrics
Log evaluation metrics during or after training:
```python
with exp.start_run("<RUN_NAME>"):
# Log individual metric
exp.log_metric("accuracy", 0.95)
# Log multiple metrics at once
exp.log_metrics({
"accuracy": accuracy,
"f1_score": f1,
"precision": precision,
"recall": recall
})
```
**For training loops with multiple epochs**, use the `step` argument (defaults to 0) to track metrics over time:
```python
with exp.start_run("<RUN_NAME>"):
for epoch in range(num_epochs):
# ... training code ...
# Log metrics with step for epoch tracking
exp.log_metric("loss", train_loss, step=epoch)
exp.log_metrics({
"val_loss": val_loss,
"val_accuracy": val_acc
}, step=epoch)
```
### Step 2c: Log Model
Log the trained model to the run:
```python
from snowflake.ml.model.model_signature import infer_signature
sig = infer_signature(X_train, y_train)
with exp.start_run("<RUN_NAME>"):
# Train model
model.fit(X_train, y_train)
# Log model with signature
# NOTE: Use `signatures` (dict mapping method name to signature), NOT `model_signature`
exp.log_model(
model,
model_name="<MODEL_NAME>",
signatures={"predict": sig}
)
```
### Step 2d: Log Artifacts (Optional)
Log additional files or directories (plots, reports, configs):
```python
with exp.start_run("<RUN_NAME>"):
# Log a single file to root of run's artifact directory
exp.log_artifact("<LOCAL_FILE_PATH>")
# Log to a specific subdirectory within the run
exp.log_artifact("<LOCAL_FILE_PATH>", artifact_path="plots")
# Log an entire directory
exp.log_artifact("<LOCAL_DIRECTORY_PATH>", artifact_path="outputs")
```
**Retrieve artifacts later:**
```python
# List artifacts in a run
artifacts = exp.list_artifacts(run_name="<RUN_NAME>")
# Download artifacts to local directory
exp.download_artifacts(
run_name="<RUN_NAME>",
artifact_path="plots", # optional: specific subdir
target_path="./downloads" # optional: local destination
)
```
### Complete Example: Manual Logging
```python
from snowflake.ml.experiment import ExperimentTracking
from snowflake.ml.model.model_signature import infer_signature
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score
# Initialize
exp = ExperimentTracking(session=session)
exp.set_experiment("<EXPERIMENT_NAME>")
# Define hyperparameters
params = {"n_estimators": 100, "max_depth": 5, "random_state": 42}
# Train and log
with exp.start_run("<RUN_NAME>"):
# Log parameters
exp.log_params(params)
# Train model
model = RandomForestClassifier(**params)
model.fit(X_train, y_train)
# Evaluate
y_pred = model.predict(X_test)
# Log metrics
exp.log_metrics({
"accuracy": accuracy_score(y_test, y_pred),
"f1_score": f1_score(y_test, y_pred, average='weighted')
})
# Log model
sig = infer_signature(X_train, y_train)
exp.log_model(model, model_name="<MODEL_NAME>", signatures={"predict": sig})
```
---
## Workflow B: Autologging
Use when training with supported frameworks (XGBoost, Keras, LightGBM). Callbacks automatically log parameters, metrics per epoch/iteration, and the final model.
### Step 1: Identify Framework
| Framework | Callback Import |
|-----------|-----------------|
| XGBoost | `from snowflake.ml.experiment.callback.xgboost import SnowflakeXgboostCallback` |
| Keras | `from snowflake.ml.experiment.callback.keras import SnowflakeKerasCallback` |
| LightGBM | `from snowflake.ml.experiment.callback.lightgbm import SnowflakeLightgbmCallback` |
### Step 2: Gather Information
**⚠️ IMPORTANT:** Check conversation context first. If coming from `ml-development` or the user's code is visible, you may already know:
- Framework (from imports or model type)
- Model name
- Database/schema context
**Only ask for what's not already known:**
- Experiment name
- Run name
- Model name (for registry)
- Framework (if not evident from code/context)
**If all context is available**, proceed directly to Step 3. Otherwise:
**⚠️ STOP**: Wait for user response on missing information.
### Step 3: Create Callback
```python
from snowflake.ml.model.model_signature import infer_signature
# Infer signature from training data
sig = infer_signature(X_train, y_train)
# Create framework-specific callback
callback = <FRAMEWORK_CALLBACK>(
exp,
model_name="<MODEL_NAME>",
model_signature=sig
)
# Or customize what gets logged
callback = <FRAMEWORK_CALLBACK>(
exp,
model_name="<MODEL_NAME>",
model_signature=sig,
log_model=True, # default: True
log_metrics=True, # default: True
log_params=True, # default: True
log_every_n_epochs=1, # default: 1 (log every epoch)
version_name="v1" # optional: model version
)
```
### Step 4: Train with Callback
Pass the callback to the model's fit method:
```python
with exp.start_run("<RUN_NAME>"):
model.fit(X_train, y_train, callbacks=[callback])
```
### XGBoost Example
```python
from xgboost import XGBClassifier
from snowflake.ml.experiment.callback.xgboost import SnowflakeXgboostCallback
from snowflake.ml.model.model_signature import infer_signature
sig = infer_signature(X_train, y_train)
callback = SnowflakeXgboostCallback(exp, model_name="<MODEL_NAME>", model_signature=sig)
model = XGBClassifier(callbacks=[callback])
with exp.start_run("<RUN_NAME>"):
model.fit(X_train, y_train, eval_set=[(X_test, y_test)])
```
### Keras Example
```python
import keras
from snowflake.ml.experiment.callback.keras import SnowflakeKerasCallback
from snowflake.ml.model.model_signature import infer_signature
sig = infer_signature(X_train, y_train)
callback = SnowflakeKerasCallback(exp, model_name="<MODEL_NAME>", model_signature=sig)
model = keras.Sequential([keras.layers.Dense(64, activation='relu'), keras.layers.Dense(1)])
model.compile(optimizer='adam', loss='mse', metrics=['mae'])
with exp.start_run("<RUN_NAME>"):
model.fit(X_train, y_train, validation_split=0.2, callbacks=[callback])
```
### LightGBM Example
```python
from lightgbm import LGBMClassifier
from snowflake.ml.experiment.callback.lightgbm import SnowflakeLightgbmCallback
from snowflake.ml.model.model_signature import infer_signature
sig = infer_signature(X_train, y_train)
callback = SnowflakeLightgbmCallback(exp, model_name="<MODEL_NAME>", model_signature=sig)
model = LGBMClassifier()
with exp.start_run("<RUN_NAME>"):
model.fit(X_train, y_train, eval_set=[(X_test, y_test)], callbacks=[callback])
```
---
## API Reference
### ExperimentTracking Constructor
```python
exp = ExperimentTracking(
session,
database_name=None, # optional, uses session's current database
schema_name=None # optional, uses session's current schema or PUBLIC
)
```
**Note:** `ExperimentTracking` is a **singleton** — only one instance exists per session. Subsequent calls reuse the existing instance.
### ExperimentTracking Methods
| Method | Description | Parameters |
|--------|-------------|------------|
| `set_experiment(name, ...)` | Set active experiment (creates if not exists) | `name`: experiment identifier, `database_name`: optional, `schema_name`: optional |
| `start_run(name)` | Start or resume a run | `name`: run identifier (optional, auto-generated if None) |
| `end_run(name)` | End a run | `name`: run to end (optional, ends current run if None) |
| `log_param(key, value)` | Log single parameter | `key`: param name, `value`: any type (converted to string) |
| `log_params(params)` | Log multiple parameters | `params`: dict (values converted to string) |
| `log_metric(key, value, step)` | Log single metric | `key`: metric name, `value`: float, `step`: int (default 0) |
| `log_metrics(metrics, step)` | Log multiple metrics | `metrics`: dict of floats, `step`: int (default 0) |
| `log_model(model, ...)` | Log model to run | Wraps `Registry.log_model` — see model-registry skill |
| `log_artifact(path, artifact_path)` | Log file or directory | `path`: local path, `artifact_path`: destination dir (optional) |
| `list_artifacts(run_name, artifact_path)` | List artifacts in a run | `run_name`: run identifier, `artifact_path`: subdir (optional) |
| `download_artifacts(run_name, ...)` | Download artifacts locally | `run_name`: run identifier, `artifact_path`: optional, `target_path`: optional |
| `delete_experiment(name, ...)` | Delete experiment and all runs | `name`: experiment identifier, `database_name`: optional, `schema_name`: optional |
| `delete_run(name)` | Delete single run | `name`: run identifier |
**⚠️ Note:** `log_param`, `log_params`, `log_metric`, `log_metrics`, `log_model`, and `log_artifact` will **auto-start a new run** if no run is currently active.
### Callback Parameters (All Frameworks)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `experiment_tracking` | ExperimentTracking | — | Initialized experiment tracker (required) |
| `log_model` | bool | `True` | Whether to log the model |
| `log_metrics` | bool | `True` | Whether to log metrics |
| `log_params` | bool | `True` | Whether to log parameters |
| `log_every_n_epochs` | int | `1` | Log metrics every N epochs |
| `model_name` | str | `None` | Name for logged model (required if `log_model=True`) |
| `version_name` | str | `None` | Version name for the model |
| `model_signature` | ModelSignature | `None` | Input/output schema (recommended) |
---
## Managing Experiments
### View Existing Experiments and Runs
```sql
-- List all experiments in schema (returns empty list if none — safe)
SHOW EXPERIMENTS IN SCHEMA <DATABASE>.<SCHEMA>;
-- Or check for a specific experiment by name (returns empty list if not found — safe)
SHOW EXPERIMENTS LIKE '<EXPERIMENT_NAME>' IN SCHEMA <DATABASE>.<SCHEMA>;
-- List runs in an experiment (only run after confirming experiment exists above — errors if experiment not found)
SHOW RUNS IN EXPERIMENT <DATABASE>.<SCHEMA>.<EXPERIMENT>;
```
### End Run and View URLs
When a run ends (via `end_run()` or exiting the `with` block), URLs are printed:
```
🏃 View run <RUN_NAME> at: https://app.snowflake.com/...
🧪 View experiment at: https://app.snowflake.com/...
```
### Delete Experiment
```python
# Delete in current database/schema
exp.delete_experiment("<EXPERIMENT_NAME>")
# Delete in specific database/schema (both must be specified together)
exp.delete_experiment(
"<EXPERIMENT_NAME>",
database_name="<DATABASE>",
schema_name="<SCHEMA>"
)
```
### Delete Single Run
```python
exp.set_experiment("<EXPERIMENT_NAME>")
exp.delete_run("<RUN_NAME>")
```
---
## Stopping Points
- ✋ After Step 1 (Gather Information) if context is missing — wait for user response
---
## Output
- Experiment with logged runs in Snowflake
- URLs printed when run ends for quick access
- View in Snowsight: **AI & ML → Experiments**
feature-store/create/SKILL.md
---
name: feature-store-create
description: "Create Snowflake Feature Store, register entities, create feature views, temporal features, and aggregation API."
parent_skill: feature-store
path: machine-learning/feature-store/create
---
# Create Feature Store, Entities & Feature Views
## When to Load
Parent skill routes here for CREATE intent: "create feature store", "register entity", "create feature view", "set up feature store", "add features from a table", "temporal features", "aggregation API".
## Prerequisites
- `../references/api-reference.md` loaded (mandatory init from parent)
- User has confirmed database, schema, warehouse
- Snowpark session established
---
## Workflow
### Step 1: Create or Connect to Feature Store
**Ask user:**
```
Should I create a new feature store or connect to an existing one?
1. Create new (will create schema if it doesn't exist)
2. Connect to existing
```
**⚠️ STOP**: Wait for user response.
**If creating new:**
```python
from snowflake.ml.feature_store import FeatureStore, CreationMode
fs = FeatureStore(
session=session,
database="<DATABASE>",
name="<SCHEMA>",
default_warehouse="<WAREHOUSE>",
creation_mode=CreationMode.CREATE_IF_NOT_EXIST,
)
```
**If connecting to existing:**
```python
fs = FeatureStore(
session=session,
database="<DATABASE>",
name="<SCHEMA>",
default_warehouse="<WAREHOUSE>",
creation_mode=CreationMode.FAIL_IF_NOT_EXIST,
)
```
**Environment organization** (recommended starting point — schema-based):
```
ML_FEATURES_DB
├── DEV_FEATURE_STORE (dev experiments)
├── TEST_FEATURE_STORE (integration testing)
└── PROD_FEATURE_STORE (production serving)
```
**⚠️ STOP**: Confirm database/schema exist. Load `../references/design-guide.md` for full RBAC/org setup if creating new.
---
### Step 2: Explore Source Data
If adding features from a table:
1. **Identify** the source table
2. **Describe** the table:
```sql
DESCRIBE TABLE <database>.<schema>.<table>;
SELECT * FROM <table> LIMIT 5;
```
3. **Identify** the entity key (unique identifier column)
4. **Classify** columns using the Transformation Taxonomy (see `../references/design-guide.md`):
- Feature columns (MIT) → go into FeatureView
- Label columns → go into spine DataFrame
- Preprocessing candidates (MDT) → go into Model Registry Pipeline
5. **Check** if a `timestamp_col` is available for PIT correctness
6. **Flag temporal columns** — identify any date, timestamp, or YYYYMM-encoded numeric columns that could support temporal feature derivation (→ Step 6)
---
### Step 3: Register Entities
**Rules:**
- Join keys are immutable after registration — plan carefully
- Use **consistent key naming** everywhere (always `USER_ID`, never mix `CUSTOMER_ID`/`USR_ID`)
- Entity name is a **business object** (CUSTOMER, ORDER, PRODUCT, ROUTE)
**Ask user:**
```
What entities do your features describe? For each entity, I need:
- Entity name (e.g., CUSTOMER, PRODUCT, ORDER)
- Join key column(s) (e.g., CUSTOMER_ID, [ORDER_ID, LINE_ID])
- Optional description
Note: Join keys are immutable after registration — choose carefully.
```
**⚠️ STOP**: Wait for user response.
```python
from snowflake.ml.feature_store import Entity
entity = Entity(
name="<ENTITY_NAME>",
join_keys=["<KEY_COL>"],
desc="<description>"
)
fs.register_entity(entity)
```
**Verify:**
```python
fs.list_entities().show()
```
**Entity types:**
- **Simple**: Single join key (`USER_ID`)
- **Compound**: Multiple keys for M:N relationships (`[PRODUCT_ID, SUPPLIER_ID]`)
- **Hierarchical**: HOUSEHOLD → USER → SESSION (spine must include all FK levels)
---
### Step 4: Create Feature View
**Ask user:**
```
For your feature view, I need:
1. Feature view name
2. Which entity(ies) it relates to
3. Source table or query for feature logic
4. Is refreshing Snowflake-managed (auto-refresh) or externally managed (you manage refreshing)?
5. If managed: refresh frequency (e.g., "5 minutes", "1 hour", "1 day")
6. Does it include time-series features? If so, which column is the timestamp?
```
**⚠️ STOP**: Wait for user response.
**Build the feature DataFrame:**
```python
features_df = session.sql("""
SELECT
<KEY_COLUMN>,
<FEATURE_COLUMNS>
FROM <DATABASE>.<SCHEMA>.<SOURCE_TABLE>
WHERE <data_quality_filters>
""")
```
For common transformation patterns, load `../references/feature-patterns.md`.
**Create the FeatureView:**
```python
from snowflake.ml.feature_store import FeatureView
fv = FeatureView(
name="<ENTITY>_<DOMAIN>_FV",
entities=[entity],
feature_df=features_df,
timestamp_col="<TS_COL>",
refresh_freq="<FREQ>",
refresh_mode="INCREMENTAL",
desc="<description>",
)
```
**Attach feature descriptions:**
```python
fv = fv.attach_feature_desc({
"<FEATURE1>": "<description>",
"<FEATURE2>": "<description>",
})
```
**⚠️ MANDATORY CHECKPOINT**: Present the feature view configuration to user for approval before registering.
```
I will register this feature view:
- Name: <FV_NAME>
- Version: V01
- Entity: <ENTITY_NAME> (join keys: <KEYS>)
- Type: Managed / External
- Refresh: <FREQ> (<INCREMENTAL|FULL>)
- Features: <list of feature columns>
Approve? (Yes/No/Modify)
```
**Register:**
```python
registered_fv = fs.register_feature_view(
feature_view=fv,
version="V01",
block=True,
)
```
**Verify:**
```python
fs.list_feature_views().select("NAME", "VERSION", "SCHEDULING_STATE").show()
```
**Feature View types:**
| Type | `refresh_freq` | Backed By | Use Case |
|------|---------------|-----------|----------|
| Managed (DT) | `"1 day"`, `"1 hour"`, etc. | Dynamic Table | Pre-computed, auto-refresh |
| External | `None` | View | DBT-managed or external pipelines |
**Decide refresh_freq**: Match to source update frequency. Source updates daily → don't refresh every 5 minutes (wasteful).
**⚠️ STOP**: If registration fails with "Database not authorized", use `refresh_freq=None` for external feature view.
---
### Step 5: Aggregation API (Feature Class)
**Requires**: `snowflake-ml-python >= 1.21.0`
For declarative time-windowed aggregations with tiling:
```python
from snowflake.ml.feature_store import Feature
purchase_amount = Feature("PURCHASE_AMOUNT", "Amount of each purchase")
features = [
purchase_amount.sum(windows=["7d", "30d"]).alias("TOTAL_SPEND"),
purchase_amount.avg(windows=["7d", "30d"]).alias("AVG_SPEND"),
purchase_amount.count(windows=["7d", "30d"]).alias("PURCHASE_CNT"),
purchase_amount.std(windows=["30d"]).alias("SPEND_STD"),
]
fv = FeatureView(
name="USER_PURCHASE_FV",
entities=[user_entity],
feature_df=transactions_df,
feature_granularity="1 day",
features=features,
refresh_freq="1 day",
desc="User purchase aggregations over 7d and 30d windows",
)
```
**Tile size guidance:**
| Data Type | `feature_granularity` |
|-----------|----------------------|
| Clickstream | `"1 hour"` |
| Transactions | `"1 day"` |
| Weekly reports | `"1 week"` |
---
### Step 6: Temporal Feature Discovery
After registering the primary feature view, check if the source data supports derived temporal features. If it does, create a **separate** `<ENTITY>_TEMPORAL_FV` to keep feature views focused and composable.
#### When to Apply
Apply when source table contains **any** of:
- DATE or TIMESTAMP columns (e.g., `CREATED_AT`, `ORDER_DATE`, `EVENT_TS`)
- YYYYMM or YYYYMMDD encoded numeric columns (e.g., `FIRSTPAYMENTDATE = 202301`)
- Duration/term columns (e.g., `LOAN_TERM`, `CONTRACT_MONTHS`)
- Pairs of date columns that define a time span (e.g., start/end, origination/maturity)
If none exist, skip to Step 7.
#### Temporal Feature Catalog
| Category | Features | SQL Pattern | When to Use |
|----------|----------|-------------|-------------|
| **Calendar extraction** | `ORIG_MONTH`, `ORIG_QUARTER`, `ORIG_YEAR` | `MOD(YYYYMM_COL, 100)`, `CEIL(MOD()/3.0)::INT`, `FLOOR(YYYYMM_COL/100)` | Any date/YYYYMM column |
| **Seasonality flags** | `IS_WINTER_ORIG`, `IS_Q4_ORIG`, `IS_WEEKEND` | `CASE WHEN month IN (...) THEN 1 ELSE 0 END` | Seasonal patterns expected |
| **Duration / span** | `LOAN_DURATION_YEARS`, `CONTRACT_MONTHS` | `DATEDIFF(...)` on static date pairs | Two date columns define a span |
| **Cyclical encoding** | `MONTH_SIN`, `MONTH_COS`, `DOW_SIN`, `DOW_COS` | `SIN(2 * PI() * val / period)`, `COS(...)` | Month/day-of-week should wrap |
| **Epoch / vintage** | `DAYS_SINCE_EPOCH`, `ORIG_YEAR_BUCKET` | `DATEDIFF('day', '1970-01-01', date_col)`, `FLOOR(YEAR/5)*5` | Absolute time position matters |
| **Relative position** | `MONTH_IN_QUARTER`, `WEEK_IN_YEAR` | `MOD(month-1, 3)+1`, `WEEKOFYEAR(...)` | Position within cycle matters |
#### Leakage Guard
Before including a temporal feature, classify it:
| Classification | Safe? | Example | Action |
|---------------|-------|---------|--------|
| **Known at prediction time** | Yes | Origination month, loan term | Include |
| **Post-event / outcome-adjacent** | No | Months delinquent, months in repayment | Exclude — or include only with explicit user confirmation |
| **Requires CURRENT_TIMESTAMP** | No | Age of loan, days since origination | Compute at query/inference time (ODT) |
**⚠️ STOP**: Ask the user whether to include post-event columns. Explain the leakage risk.
#### Implementation Pattern
```python
temporal_df = session.sql("""
SELECT
<ENTITY_KEY>,
MOD(<YYYYMM_COL>, 100) AS ORIG_MONTH,
CEIL(MOD(<YYYYMM_COL>, 100) / 3.0)::INT AS ORIG_QUARTER,
FLOOR(<YYYYMM_COL> / 100) AS ORIG_YEAR,
(FLOOR(<END_YYYYMM> / 100) - FLOOR(<START_YYYYMM> / 100)) AS DURATION_YEARS,
CASE
WHEN MOD(<YYYYMM_COL>, 100) IN (1, 2, 3, 10, 11, 12) THEN 1
ELSE 0
END AS IS_WINTER_ORIG
FROM <DATABASE>.<SCHEMA>.<SOURCE_TABLE>
WHERE <same_data_quality_filters>
QUALIFY ROW_NUMBER() OVER (PARTITION BY <ENTITY_KEY> ORDER BY <tiebreaker>) = 1
""")
temporal_fv = FeatureView(
name="<ENTITY>_TEMPORAL_FV",
entities=[entity],
feature_df=temporal_df,
refresh_freq="1 day",
desc="Temporal features derived from <source_date_columns>",
)
temporal_fv = temporal_fv.attach_feature_desc({
"ORIG_MONTH": "Origination month (1-12)",
"ORIG_QUARTER": "Origination quarter (1-4)",
"ORIG_YEAR": "Origination year",
"DURATION_YEARS": "Duration in years between <start> and <end>",
"IS_WINTER_ORIG": "Winter origination flag (Oct-Mar = 1, Apr-Sep = 0)",
})
registered_temporal_fv = fs.register_feature_view(
feature_view=temporal_fv,
version="V01",
block=True,
)
```
All temporal features must be **deterministic and static** — derived only from source columns, not from `CURRENT_DATE()` or `CURRENT_TIMESTAMP()`. This ensures incremental refresh compatibility.
---
### Step 7: Verify Registration
```python
print(fs.list_entities().to_pandas())
print(fs.list_feature_views().to_pandas())
```
Check refresh mode is INCREMENTAL (not FULL). If FULL, check `../references/troubleshooting.md` for incremental refresh blockers.
---
### Step 8: Next Steps
**Ask user:**
```
Feature view registered successfully. Would you like to:
1. Create another feature view
2. Set up a feature pipeline (→ pipelines/SKILL.md)
3. Generate a training dataset (→ training/SKILL.md)
4. Audit/validate the feature view (→ monitor/SKILL.md)
5. Done for now
```
---
## Stopping Points
- ✋ Step 1: Before creating/connecting to feature store
- ✋ Step 3: Before registering each entity
- ✋ Step 4: Before registering feature view (mandatory approval)
- ✋ Step 6: Before including post-event temporal columns (leakage check)
- ✋ Step 8: Next action selection
## Output
- Initialized `fs` FeatureStore object
- Registered entities
- Registered feature view(s) with materialized data
- Temporal feature view (if applicable)
## Next Skill
- If user wants pipelines → **Load** `pipelines/SKILL.md`
- If user wants training data → **Load** `training/SKILL.md`
- If user wants to audit → **Load** `monitor/SKILL.md`
feature-store/lineage/SKILL.md
---
name: feature-store-lineage
description: "Feature lineage analysis (which models consume which features) and creating inference feature views from model signatures."
parent_skill: feature-store
path: machine-learning/feature-store/lineage
---
# Feature Lineage & Inference Feature Views
## When to Load
Parent skill routes here for LINEAGE intent: "feature lineage", "which models use", "model consumers", "impact analysis", "inference feature view", "model inference", "serve features", "inference FV", "create inference view", "batch inference features", "model input features".
## Prerequisites
- `../references/api-reference.md` loaded
- Feature store (`fs`) initialized with registered feature views
- Model Registry access for lineage and inference FV creation
---
## Part A: Feature Lineage
### Purpose
Answer: **"Which models consume features from a given feature view?"**
This enables impact analysis before modifying a feature view — you can determine which models would be affected.
### Step 1: Identify the Feature View to Analyze
**Ask user:**
```
Which feature view do you want to analyze for model consumers?
(I'll check all models in the registry that use features from this view)
```
**⚠️ STOP**: Wait for user response.
```python
fv = fs.get_feature_view("<FV_NAME>", "<VERSION>")
fv_features = [str(f) for f in fv.feature_descs] if fv.feature_descs else fv.feature_names
```
### Step 2: Scan Model Registry
Enumerate all models and check their input signatures against feature view columns:
```python
from snowflake.ml.registry import Registry
registry = Registry(session=session)
def get_model_input_features(registry, model_name, version_name):
"""Extract input feature names from a model version's signature."""
model = registry.get_model(model_name)
mv = model.version(version_name)
functions = mv.show_functions()
input_features = set()
for func_info in functions:
sig = func_info['signature']
for feat in sig.inputs:
input_features.add(feat.name.upper())
return input_features
# List all models
models_df = session.sql("SHOW MODELS IN SCHEMA <DATABASE>.<SCHEMA>").collect()
results = []
for row in models_df:
model_name = row['name']
versions_df = session.sql(f"SHOW VERSIONS IN MODEL {model_name}").collect()
for ver_row in versions_df:
version_name = ver_row['name']
try:
model_features = get_model_input_features(registry, model_name, version_name)
fv_feature_set = set(f.upper() for f in fv_features)
overlap = fv_feature_set & model_features
coverage = len(overlap) / len(model_features) if model_features else 0
if coverage > 0:
results.append({
'model': model_name,
'version': version_name,
'coverage': f"{coverage:.0%}",
'matched_features': len(overlap),
'total_model_features': len(model_features),
})
except Exception:
pass
# Display results
for r in results:
print(f" {r['model']} {r['version']}: {r['coverage']} coverage "
f"({r['matched_features']}/{r['total_model_features']} features)")
```
**Coverage interpretation:**
| Coverage | Meaning | Impact Level |
|----------|---------|-------------|
| 100% | Model fully depends on this FV | **Critical** — model will break |
| 50-99% | Significant consumer | **High** — model performance will degrade |
| 1-49% | Partial consumer | **Medium** — some features affected |
| 0% | Not a consumer | **None** |
### Step 3: Feature-Level Reverse Lookup
For a specific feature column, find all models that consume it:
```python
def feature_consumers(registry, session, feature_name, model_schema):
"""Find all models that consume a specific feature."""
models_df = session.sql(f"SHOW MODELS IN SCHEMA {model_schema}").collect()
consumers = []
for row in models_df:
model_name = row['name']
versions_df = session.sql(f"SHOW VERSIONS IN MODEL {model_name}").collect()
for ver_row in versions_df:
version_name = ver_row['name']
try:
model_features = get_model_input_features(registry, model_name, version_name)
if feature_name.upper() in model_features:
consumers.append(f"{model_name} {version_name}")
except Exception:
pass
return consumers
consumers = feature_consumers(registry, session, "CREDITSCORE", "<DATABASE>.<SCHEMA>")
print(f"Models consuming CREDITSCORE: {consumers}")
```
### Step 4: Lineage via Model Registry API
For models that were registered with feature store metadata, use the built-in lineage API:
```python
model = registry.get_model("<MODEL_NAME>")
mv = model.version("<VERSION>")
# Get upstream feature views
upstream_fvs = mv.lineage(direction='upstream', domain_filter={'feature_view'})
for fv_ref in upstream_fvs:
print(f"Upstream FV: {fv_ref}")
```
> **Note:** `lineage()` only works if the model was logged with feature store integration. For models registered without FS metadata, use the signature-based scan from Step 2.
---
## Part B: Create Inference Feature View from Model Signature
### Purpose
Given a trained model, create a feature view that provides exactly the features the model needs for inference — enabling automated batch or online inference pipelines.
### Step 5: Extract Model Input Features
```python
model = registry.get_model("<MODEL_NAME>")
mv = model.version("<VERSION>")
functions = mv.show_functions()
predict_func = next(f for f in functions if f['name'] == 'PREDICT')
input_features = [feat.name for feat in predict_func['signature'].inputs]
print(f"Model requires {len(input_features)} features: {input_features}")
```
### Step 6: Map Features to Source Feature Views
**Approach A: Lineage-based (preferred)**
```python
upstream_fvs = mv.lineage(direction='upstream', domain_filter={'feature_view'})
```
**Approach B: Name-matching fallback**
```python
all_fvs = fs.list_feature_views().to_pandas()
feature_to_fv = {}
for _, row in all_fvs.iterrows():
fv = fs.get_feature_view(row['NAME'], row['VERSION'])
for feat_name in fv.feature_names:
if feat_name.upper() in [f.upper() for f in input_features]:
feature_to_fv[feat_name.upper()] = (row['NAME'], row['VERSION'])
# Check coverage
mapped = set(feature_to_fv.keys())
unmapped = set(f.upper() for f in input_features) - mapped
print(f"Mapped: {len(mapped)}/{len(input_features)}")
if unmapped:
print(f"Unmapped (ODT candidates): {unmapped}")
```
**⚠️ STOP**: If unmapped features exist, ask user to classify them:
```
The following model input features were not found in any feature view:
<unmapped list>
These are likely:
1. On-Demand Transforms (ODT) — computed at inference time
2. Columns from a feature view not yet registered
3. Preprocessing outputs (MDT) — handled by the model pipeline
Please classify each unmapped feature.
```
### Step 7: Choose Inference Approach
**Ask user:**
```
How would you like to serve inference features?
1. Slice-based (recommended): Use fv.slice() + retrieve_feature_values()
- No new FV created
- Flexible, works for batch and online
- Features come directly from source FVs
2. Materialized: Create a dedicated inference FV backed by a Dynamic Table
- Joins source FV Dynamic Tables
- Single-table access for production serving
- Additional storage and compute cost
```
**⚠️ STOP**: Wait for user response.
### Step 8A: Slice-Based Inference (No New FV)
```python
# Get source feature views
fv1 = fs.get_feature_view("<SOURCE_FV_1>", "<VERSION>")
fv2 = fs.get_feature_view("<SOURCE_FV_2>", "<VERSION>")
# Build inference spine
inference_spine = session.create_dataframe(
entity_keys,
schema=["<ENTITY_KEY>"]
)
# Retrieve features from multiple FVs
inference_features = fs.retrieve_feature_values(
spine_df=inference_spine,
features=[
fv1.slice(["FEATURE_A", "FEATURE_B"]),
fv2.slice(["FEATURE_C", "FEATURE_D"]),
],
spine_timestamp_col="<TIMESTAMP>" if needed else None,
)
# Run model
predictions = mv.run(inference_features, function_name="predict")
predictions.show()
```
### Step 8B: Materialized Inference FV
```python
# Build SQL joining source FV Dynamic Tables
source_fv_1_dt = f"<DATABASE>.<SCHEMA>.\"<FV_1_NAME>$<VERSION>\""
source_fv_2_dt = f"<DATABASE>.<SCHEMA>.\"<FV_2_NAME>$<VERSION>\""
inference_sql = f"""
SELECT
a.<ENTITY_KEY>,
a.FEATURE_A, a.FEATURE_B,
b.FEATURE_C, b.FEATURE_D
FROM {source_fv_1_dt} a
INNER JOIN {source_fv_2_dt} b
ON a.<ENTITY_KEY> = b.<ENTITY_KEY>
"""
inference_df = session.sql(inference_sql)
inference_fv = FeatureView(
name="<MODEL>_INFERENCE_FV",
entities=[entity],
feature_df=inference_df,
refresh_freq="1 day",
desc=f"Inference features for <MODEL_NAME>. Sources: <FV_1>, <FV_2>",
)
inference_fv = inference_fv.attach_feature_desc({
"FEATURE_A": "From <FV_1>: <description>",
"FEATURE_B": "From <FV_1>: <description>",
"FEATURE_C": "From <FV_2>: <description>",
"FEATURE_D": "From <FV_2>: <description>",
})
```
**⚠️ MANDATORY CHECKPOINT**: Present the inference FV configuration before registering.
```
I will register this inference feature view:
- Name: <MODEL>_INFERENCE_FV V01
- Entity: <ENTITY_NAME>
- Sources: <list of source FVs>
- Features: <count> features covering <coverage>% of model signature
- Refresh: <FREQ>
- Unmapped (ODT): <list or "none">
Approve? (Yes/No/Modify)
```
```python
registered_inference_fv = fs.register_feature_view(
feature_view=inference_fv,
version="V01",
block=True,
)
```
> **RBAC Note:** The inference FV's Dynamic Table must be owned by the same role that owns the source DTs, or SELECT must be granted. Otherwise, DT refresh will fail with "not authorized".
### Step 9: Validate Inference FV
Run the inference FV checklist (I1–I6 from `monitor/SKILL.md`):
| # | Check | How to Verify |
|---|-------|---------------|
| I1 | Model signature coverage | Compare FV features against `mv.show_functions()` inputs |
| I2 | Source FV lineage documented | Check `desc` includes source FV names |
| I3 | ODT features identified | Unmapped features listed and classified |
| I4 | Preprocessing passthrough | If model has MDT, FV provides raw columns (not scaled/encoded) |
| I5 | Entity key alignment | Inference FV entity keys match all source FVs |
| I6 | Refresh frequency | Inference FV refresh ≥ fastest source FV |
```python
# I1: Signature coverage check
model_inputs = set(f.name.upper() for f in predict_func['signature'].inputs)
fv_features = set(f.upper() for f in registered_inference_fv.feature_names)
coverage = len(model_inputs & fv_features) / len(model_inputs)
print(f"I1 - Signature coverage: {coverage:.0%}")
assert coverage >= 0.9, f"Low coverage: {coverage:.0%}"
```
### Step 10: End-to-End Test
```python
# Build a small test spine
test_spine = session.sql(f"""
SELECT DISTINCT <ENTITY_KEY>
FROM <SOURCE_TABLE>
LIMIT 5
""")
# Retrieve inference features
test_features = fs.retrieve_feature_values(
spine_df=test_spine,
features=[registered_inference_fv],
)
# Run model prediction
test_predictions = mv.run(test_features, function_name="predict")
test_predictions.show()
```
---
## Stopping Points
- ✋ Step 1: Feature view selection for lineage analysis
- ✋ Step 6: Classification of unmapped features
- ✋ Step 7: Inference approach selection
- ✋ Step 8B: Before registering materialized inference FV (mandatory approval)
## Output
**Part A:**
- Model consumer report for the analyzed feature view
- Feature-level reverse lookup results
- Impact assessment for planned changes
**Part B:**
- Inference feature view (slice-based or materialized)
- Validation report (I1–I6)
- End-to-end test confirming predictions work
## Next Skill
- If user wants to enable online serving for inference → **Load** `online/SKILL.md`
- If user wants to audit/validate → **Load** `monitor/SKILL.md`
- If user wants batch inference pipeline → **Load** `pipelines/SKILL.md`
feature-store/migrate/SKILL.md
---
name: feature-store-migrate
description: "Migrate to Snowflake Feature Store from Feast, Tecton, or custom feature store platforms."
parent_skill: feature-store
path: machine-learning/feature-store/migrate
---
# Migration Guide
## When to Load
Parent skill routes here for MIGRATE intent: "migrate from Feast", "migrate from Tecton", "migration", "convert feature store", "move to Snowflake feature store".
## Prerequisites
- `../references/api-reference.md` loaded
- User has access to the source feature store configuration
---
## Concept Mapping
### Feast → Snowflake Feature Store
| Feast Concept | Snowflake Equivalent | Notes |
|---------------|---------------------|-------|
| Feature Store (repo) | `FeatureStore` (schema) | FS = a Snowflake schema |
| Entity | `Entity` | Same concept, register via `fs.register_entity()` |
| Feature View | `FeatureView` (managed) | With `refresh_freq` for auto-refresh |
| On-Demand Feature View | `FeatureView` (external) | With `refresh_freq=None` |
| Feature Service | `FeatureView.slice()` | Group features via slices |
| `get_historical_features()` | `fs.generate_dataset()` | Point-in-time retrieval |
| `get_online_features()` | `fs.read_feature_view(..., store_type=StoreType.ONLINE)` | Low-latency key lookup |
| Registry (SQLite/DB) | Snowflake metadata | Automatic, no external DB needed |
| Materialization job | Dynamic Table refresh | Automatic, no cron/Airflow needed |
| `feast apply` | `fs.register_entity()` / `fs.register_feature_view()` | Python API |
| `feast materialize` | Automatic via Dynamic Table | Set `refresh_freq` |
| `feature_store.yaml` | Python code (FeatureStore constructor) | No YAML config file |
### Tecton → Snowflake Feature Store
| Tecton Concept | Snowflake Equivalent | Notes |
|----------------|---------------------|-------|
| Workspace | `FeatureStore` (schema) | One schema per workspace |
| Entity | `Entity` | Same concept |
| Batch Feature View | `FeatureView` (managed) | Dynamic Table with `refresh_freq` |
| Stream Feature View | `FeatureView` (managed, short lag) | Use low `refresh_freq` (e.g., `"1 minute"`) |
| On-Demand Feature View | `FeatureView` (external, `refresh_freq=None`) | Or ODT at query time |
| Feature Service | `FeatureView.slice()` | Compose from multiple FVs |
| `get_features_for_events()` | `fs.generate_dataset()` | PIT retrieval |
| `get_online_features()` | `fs.read_feature_view(..., StoreType.ONLINE)` | Key-value lookup |
| Transformation (Python) | Snowpark DataFrame + SQL | In-warehouse computation |
| Materialization | Dynamic Table refresh | Automatic |
### Custom / In-House → Snowflake Feature Store
| Custom Concept | Snowflake Equivalent | Notes |
|----------------|---------------------|-------|
| Feature tables | Source tables → `FeatureView` | Register existing tables |
| ETL pipelines | `FeatureView` with `refresh_freq` | Or external FV if keeping existing ETL |
| Feature registry | Built-in (metadata on schema) | Automatic discovery and versioning |
| Training data joins | `fs.generate_dataset()` | PIT-correct by default |
| Serving layer | `OnlineConfig` + `StoreType.ONLINE` | Built-in low-latency serving |
---
## Migration Workflow
### Step 1: Inventory Source Feature Store
**Ask user:**
```
To plan the migration, I need to understand your current setup:
1. Which platform are you migrating from? (Feast / Tecton / Custom / Other)
2. How many entities do you have?
3. How many feature views/tables?
4. Do you use online serving?
5. Any custom transformations (UDFs, streaming)?
```
**⚠️ STOP**: Wait for user response.
---
### Step 2: Map Entities
For each entity in the source system, create the Snowflake equivalent:
```python
# Feast example:
# driver = Entity(name="driver", join_keys=["driver_id"])
# →
driver_entity = Entity(
name="DRIVER",
join_keys=["DRIVER_ID"],
desc="Driver entity (migrated from Feast)"
)
fs.register_entity(driver_entity)
```
**Naming conversion:**
- Feast/Tecton: `snake_case` → Snowflake: `SCREAMING_SNAKE_CASE`
- Ensure join key column names match the source data columns in Snowflake
---
### Step 3: Map Feature Views
For each source feature view:
1. **Identify the source data** in Snowflake (must be loaded first)
2. **Write the transformation** as a Snowpark DataFrame or SQL
3. **Choose pipeline type**: managed (Dynamic Table) or external
4. **Register** in Snowflake Feature Store
```python
# Feast example transformation → Snowpark SQL
feature_df = session.sql("""
SELECT
DRIVER_ID,
EVENT_TS,
CONV_RATE,
ACC_RATE,
AVG_DAILY_TRIPS
FROM RAW_DB.PUBLIC.DRIVER_STATS
""")
driver_fv = FeatureView(
name="DRIVER_STATS_FV",
entities=[driver_entity],
feature_df=feature_df,
timestamp_col="EVENT_TS",
refresh_freq="1 hour",
desc="Driver statistics (migrated from Feast driver_hourly_stats)",
)
fs.register_feature_view(driver_fv, version="V01", block=True)
```
**⚠️ MANDATORY CHECKPOINT**: For each feature view migration, present the configuration before registering.
---
### Step 4: Migrate Training Pipeline
Replace source platform's historical retrieval with Snowflake's:
```python
# Feast: store.get_historical_features(entity_df, features)
# →
dataset = fs.generate_dataset(
name="DRIVER_TRAINING",
spine_df=entity_df,
features=[driver_fv],
spine_timestamp_col="EVENT_TS",
spine_label_cols=["LABEL"],
version="V01",
)
```
---
### Step 5: Migrate Online Serving (if applicable)
Replace source platform's online retrieval:
```python
# Feast: store.get_online_features(features, entity_rows)
# →
from snowflake.ml.feature_store import OnlineConfig
config = OnlineConfig(enable=True, target_lag="15s")
fs.update_feature_view("DRIVER_STATS_FV", "V01", online_config=config)
# Read online
result = fs.read_feature_view(
"DRIVER_STATS_FV", "V01",
keys=[[driver_id]],
store_type=StoreType.ONLINE,
)
```
---
### Step 6: Validate Migration
**Dual-run validation** — run both old and new systems in parallel:
1. **Schema comparison**: Verify all features exist with correct types
2. **Value comparison**: Compare feature values for a sample of entities
3. **Training comparison**: Generate training dataset from both systems and compare metrics
4. **Latency comparison**: Measure online retrieval latency
```python
# Validate feature values match
old_features = fetch_from_old_system(entity_keys)
new_features = fs.read_feature_view("DRIVER_STATS_FV", "V01", keys=entity_keys)
# Compare column-by-column
for col in feature_columns:
old_vals = old_features[col]
new_vals = new_features[col]
assert np.allclose(old_vals, new_vals, rtol=1e-5), f"Mismatch in {col}"
```
**Consumer switchover process:**
1. Deploy new Snowflake-based feature pipeline alongside old system
2. Validate feature parity (schema, values, latency)
3. Switch consumers one at a time to new system
4. Monitor for 1-2 weeks
5. Decommission old system
---
## Important Considerations
- **Data must be in Snowflake** before migration. Load source data first.
- **Streaming features** (Kafka, Kinesis): Use Snowpipe Streaming to ingest, then register as managed FV with short `refresh_freq`.
- **Custom UDFs**: Rewrite in Snowpark Python UDFs or Snowflake SQL.
- **External orchestration** (Airflow, Dagster, dbt): Can still trigger `ALTER DYNAMIC TABLE REFRESH`, or let DT auto-refresh. For dbt, register the dbt-produced table as an external feature view (`refresh_freq=None`).
- **Feature parity**: Not all source features may translate 1:1. Document any gaps.
---
## Stopping Points
- ✋ Step 1: Source system inventory
- ✋ Step 3: Before registering each feature view
- ✋ Step 5: Before enabling online serving
- ✋ Step 6: After validation, before decommissioning old system
## Output
- Migrated entities and feature views in Snowflake
- Updated training pipeline using `generate_dataset`
- Online serving configured (if applicable)
- Validation report comparing old vs new system
## Next Skill
- If user wants to audit migrated features → **Load** `monitor/SKILL.md`
- If user wants lineage analysis → **Load** `lineage/SKILL.md`
feature-store/monitor/SKILL.md
---
name: feature-store-monitor
description: "Monitor, audit, validate, and promote Snowflake Feature Store: health checks, refresh history, freshness, suspend/resume, cost, validation checklist, environment promotion."
parent_skill: feature-store
path: machine-learning/feature-store/monitor
---
# Feature Store Operations, Monitoring & Validation
## When to Load
Parent skill routes here for MONITOR intent: "feature freshness", "refresh history", "pipeline health", "list feature views", "suspend feature view", "resume feature view", "feature store cost", "audit", "validate", "check feature store", "promote features", "DEV to PROD".
## Prerequisites
- `../references/api-reference.md` loaded
- Feature store (`fs`) initialized
---
## Workflow
### Step 1: Inventory Check
**List all feature store objects:**
```python
print("=== Entities ===")
fs.list_entities().show()
print("=== Feature Views ===")
fs.list_feature_views().select(
"NAME", "VERSION", "SCHEDULING_STATE", "DESC"
).show()
```
---
### Step 2: Health Check
**Check scheduling state for all feature views:**
```python
fv_status = fs.list_feature_views().select(
"NAME", "VERSION", "SCHEDULING_STATE"
)
fv_status.show()
```
| State | Meaning | Action |
|-------|---------|--------|
| `ACTIVE` | Refreshing on schedule | Healthy |
| `SUSPENDED` | Paused, not refreshing | Resume if needed |
| `DRAFT` | Not yet registered | Register to activate |
> **Note:** `fv.status` returns a `FeatureViewStatus` enum, not a string. Use `str(fv.status)` or `"ACTIVE" in str(fv.status)` for comparisons.
**Check refresh history for a specific feature view:**
```python
fv = fs.get_feature_view("<FV_NAME>", "<VERSION>")
fs.get_refresh_history(fv).show()
```
**SQL-based health check (deeper diagnostics):**
```sql
SELECT name, scheduling_state, last_completed_refresh_state,
target_lag_sec, target_lag_type, latest_data_timestamp
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLES())
WHERE schema_name = '<FEATURE_STORE_SCHEMA>'
ORDER BY name;
SELECT name, state, state_message, refresh_action,
refresh_start_time, refresh_end_time
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY(
NAME_PREFIX => '<DATABASE>.<SCHEMA>', ERROR_ONLY => TRUE
))
ORDER BY refresh_start_time DESC
LIMIT 10;
```
For more diagnostic queries → Load `../references/troubleshooting.md`.
---
### Step 3: Suspend / Resume Feature Views
**Suspend (stop refreshing):**
```python
fs.suspend_feature_view("<FV_NAME>", "<VERSION>")
```
**Resume (restart refreshing):**
```python
fs.resume_feature_view("<FV_NAME>", "<VERSION>")
```
> **Note:** If `resume_feature_view()` doesn't take effect, use SQL directly:
> ```sql
> ALTER DYNAMIC TABLE <DATABASE>.<SCHEMA>."<FV_NAME>$<VERSION>" RESUME;
> ```
**⚠️ STOP**: Confirm with user before suspending production feature views.
---
### Step 4: Read Feature Values
**Read current feature values (offline store):**
```python
df = fs.read_feature_view("<FV_NAME>", "<VERSION>")
df.show()
```
**Read specific keys:**
```python
df = fs.read_feature_view(
"<FV_NAME>", "<VERSION>",
keys=[["key_1"], ["key_2"]],
feature_names=["FEATURE_A", "FEATURE_B"],
)
df.show()
```
---
### Step 5: Cost Management
```sql
SELECT
name,
SUM(DATEDIFF('second', refresh_start_time, refresh_end_time)) / 3600.0 AS approx_compute_hours,
COUNT(*) AS refresh_count
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY())
WHERE schema_name = '<FEATURE_STORE_SCHEMA>'
GROUP BY name
ORDER BY approx_compute_hours DESC;
```
**Cost optimization strategies:**
- Increase `refresh_freq` for less time-sensitive features
- Use smaller warehouses for simple feature pipelines
- Dedicate warehouses per pipeline criticality tier
- Suspend unused or deprecated feature views
- Use `DOWNSTREAM` target lag for intermediate stages
---
### Step 6: Validate Against Best Practices
After registration or as a standalone audit, validate entity and feature view against this checklist. Report each item as **Pass**, **Warn**, or **Fail** with actionable recommendations.
#### Audit Entry Point
If the user wants to audit an existing feature view:
1. **Ask** for the feature store location (database and schema)
2. **Connect** to the feature store
3. **List** entities and feature views
4. **Ask** the user which entity and feature view to audit
5. **Retrieve** the feature view:
```python
fv = fs.get_feature_view(name="<FV_NAME>", version="<VERSION>")
print(fv.query)
print(fv.feature_descs)
print(fv.entities)
```
6. **Identify** the source table from the query SQL
7. **Run** the checklists below
#### Entity Checklist
| # | Check | Criteria |
|---|-------|----------|
| E1 | **Business naming** | Entity name is SCREAMING_SNAKE_CASE business object, not a table name |
| E2 | **Join key uniqueness** | Join keys uniquely identify the entity or grain is intentional |
| E3 | **Description provided** | `desc` parameter is non-empty and meaningful |
| E4 | **Consistent key naming** | Same key column name used everywhere |
#### Feature View Checklist
| # | Check | Criteria |
|---|-------|----------|
| F1 | **Description provided** | `desc` parameter is non-empty and meaningful |
| F2 | **All features documented** | Every feature column has an entry in `attach_feature_desc` |
| F3 | **Incremental refresh compatible** | Query avoids incremental blockers (see `../references/troubleshooting.md`) |
| F4 | **Change tracking enabled** | Source table has `CHANGE_TRACKING = TRUE` |
| F5 | **Label separation** | Prediction target columns NOT included as features |
| F6 | **Source table fully qualified** | SQL uses `DATABASE.SCHEMA.TABLE` format |
| F7 | **Data quality filters** | WHERE clause filters invalid/null rows |
| F8 | **Refresh frequency appropriate** | Matches source data update frequency |
| F9 | **Naming convention** | FV name follows `<ENTITY>_<DOMAIN>_FV` pattern; features use standard suffixes |
| F10 | **Version format** | Version uses zero-padded format (`V01`, `V02`) |
| F11 | **Timestamp column** | `timestamp_col` is set if PIT retrieval is needed |
| F12 | **No MDT in FeatureView** | No hard-coded scaling, encoding, or imputation |
#### Feature Engineering Checklist
| # | Check | Criteria |
|---|-------|----------|
| G1 | **Recency features** | Includes `MAX(timestamp)` for time-since-last |
| G2 | **Time-window aggregations** | Uses Aggregation API or tiled windows — or documents trade-off |
| G3 | **Variance/spread features** | Includes `STDDEV()` or range metrics (DECIMAL-cast) |
| G4 | **Feature view scope** | View has focused scope (<15 features). Split wide views using `.slice()` |
| G5 | **Temporal features** | If source has date/timestamp columns, temporal FV has been created — or reason documented |
| G6 | **Model consumers identified** | Feature lineage has been run (lineage/SKILL.md); impact understood |
| G7 | **Inference FV exists** | If model consumes this FV, inference pipeline exists — or reason documented |
#### Inference Feature View Checklist
| # | Check | Criteria |
|---|-------|----------|
| I1 | **Model signature coverage** | All non-ODT model input features present |
| I2 | **Source FV lineage documented** | Inference FV description records source FVs |
| I3 | **ODT features identified** | Unmapped features documented as ODT |
| I4 | **Preprocessing passthrough** | If model has MDT pipeline, FV provides raw columns |
| I5 | **Entity key alignment** | Inference FV entity keys match source FVs |
| I6 | **Refresh frequency** | Matches or exceeds fastest source FV |
**Present results as a table** with columns: Check, Status (Pass/Warn/Fail), Notes.
**⚠️ STOP**: If any item is **Fail**, recommend a specific fix. If any item is **Warn**, explain the trade-off.
---
### Step 7: Remediate Issues
After validation, **ask the user** if they'd like to fix issues found.
For each non-passing item, in priority order (Fail first, then Warn):
1. **Explain** the proposed change
2. **Ask** for approval
3. **Implement** after confirmation
4. **Verify** the fix
**Common remediation patterns** — see `../references/troubleshooting.md` for full list.
If the query changes, a new version must be registered:
```python
fs.register_feature_view(feature_view=updated_fv, version="V02", block=True)
```
After all fixes, **re-run Step 6** to confirm all checks pass.
**⚠️ STOP**: Never implement a fix without user approval.
---
### Step 8: Promote Features (DEV → PROD)
**Schema-based promotion** using zero-copy cloning:
```sql
CREATE OR REPLACE SCHEMA PROD_FEATURE_STORE CLONE DEV_FEATURE_STORE;
```
**Python-based promotion** (controlled recreation):
```python
dev_fs = FeatureStore(session, database, "DEV_FEATURE_STORE", warehouse, CreationMode.FAIL_IF_NOT_EXIST)
prod_fs = FeatureStore(session, database, "PROD_FEATURE_STORE", warehouse, CreationMode.CREATE_IF_NOT_EXIST)
fv = dev_fs.get_feature_view("USER_PURCHASE_FV", "V01")
# Recreate entity and FV in prod with same query and metadata
```
**Validation after promotion**: Schema match, row counts, sample value comparison.
**⚠️ MANDATORY CHECKPOINT**: Confirm target environment before promoting.
---
### Step 9: Update / Delete Feature Views and Entities
**Update description:**
```python
fs.update_feature_view(name="<FV_NAME>", version="<VERSION>", desc="Updated description")
```
**Delete a feature view:**
```python
fs.delete_feature_view("<FV_NAME>", "<VERSION>")
```
**Delete an entity** (must not be referenced by any feature views):
```python
fs.delete_entity("<ENTITY_NAME>")
```
**⚠️ STOP**: Confirm with user before deleting any objects.
---
## Stopping Points
- ✋ Before suspending production feature views
- ✋ Before deleting any objects
- ✋ Before implementing remediation fixes
- ✋ Before promoting to target environment
## Output
- Health status of all feature views
- Validation audit report with Pass/Warn/Fail
- Cost analysis and optimization recommendations
- Promoted feature views (if applicable)
## Next Skill
- If user wants to create new features → **Load** `create/SKILL.md`
- If user wants to adjust pipelines → **Load** `pipelines/SKILL.md`
- If user wants online serving → **Load** `online/SKILL.md`
- If user wants lineage analysis → **Load** `lineage/SKILL.md`
feature-store/online/SKILL.md
---
name: feature-store-online
description: "Enable and use Snowflake Feature Store online serving for low-latency feature retrieval, with production application patterns."
parent_skill: feature-store
path: machine-learning/feature-store/online
---
# Online Feature Serving
## When to Load
Parent skill routes here for ONLINE intent: "online features", "online serving", "low latency", "real-time features", "OnlineConfig", "StoreType.ONLINE", "production serving", "online feature table".
## Prerequisites
- `../references/api-reference.md` loaded
- Feature store (`fs`) initialized with registered feature views
- `snowflake-ml-python >= 1.18.0` for online features
- **Note:** The `online_config` API has been in private preview since `snowflake-ml-python 1.12.0`. Behavior and availability may change.
---
## Core Concepts
- **Online store**: Low-latency key-value store for serving features in production
- **Offline store**: Standard Dynamic Table / view for batch operations (training, batch inference)
- Both stores are kept in sync automatically — same feature definitions, no training/serving skew
- Online serving is enabled per feature view via `OnlineConfig` or `.with_online_store()`
- Online Feature Tables (OFTs) store **only latest values** (no history). Backed by Hybrid Tables.
---
## Workflow
### Step 1: Identify Feature Views for Online Serving
**Ask user:**
```
Which feature view(s) do you want to enable for online (low-latency) serving?
Note: Online serving adds infrastructure cost. Only enable for feature views
needed on the real-time inference path.
```
**⚠️ STOP**: Wait for user response.
**Retrieve the feature view:**
```python
fv = fs.get_feature_view("<FV_NAME>", "<VERSION>")
print(f"Online enabled: {fv.online}")
```
---
### Step 2: Enable Online Serving
There are two approaches:
**Approach A: OnlineConfig (update existing FV)**
```python
from snowflake.ml.feature_store import OnlineConfig
config = OnlineConfig(enable=True, target_lag="15s")
updated_fv = fs.update_feature_view(
name="<FV_NAME>",
version="<VERSION>",
online_config=config,
)
print(f"Online enabled: {updated_fv.online}")
```
**Approach B: with_online_store (at creation time)**
```python
fv = FeatureView(
name="<ENTITY>_REALTIME_FV",
entities=[entity],
feature_df=features_df,
refresh_freq="5 minutes",
desc="Low-latency features for real-time serving",
).with_online_store(enabled=True)
```
**⚠️ MANDATORY CHECKPOINT**: Confirm before enabling.
```
I will enable online serving for:
- Feature View: <FV_NAME> v<VERSION>
- Target lag: <lag> (how fresh online data should be)
This will create an online feature table with additional infrastructure cost.
Approve? (Yes/No/Modify)
```
> **⚠️ Provisioning delay:** After enabling, the online feature table needs time to provision and complete its initial refresh. This can take **30 seconds to several minutes**. Poll with a lightweight `read_feature_view(..., store_type=StoreType.ONLINE)` call in a retry loop.
**Target lag options and guidance:**
| Use Case | Target Lag | Refresh Freq |
|----------|-----------|--------------|
| Fraud detection | `"15s"` | `"1 minute"` |
| Recommendations | `"1m"` | `"5 minutes"` |
| Marketing | `"5m"` | `"15 minutes"` |
---
### Step 3: Read from Online Store
```python
from snowflake.ml.feature_store.feature_view import StoreType
result = fs.read_feature_view(
"<FV_NAME>", "<VERSION>",
keys=[[key_value_1], [key_value_2]],
feature_names=["FEATURE_A", "FEATURE_B"],
store_type=StoreType.ONLINE,
)
result.show()
```
**Key differences: ONLINE vs OFFLINE:**
| Aspect | OFFLINE (default) | ONLINE |
|--------|-------------------|--------|
| Latency | Seconds to minutes | Milliseconds |
| Use case | Training, batch inference | Real-time inference |
| Keys parameter | Optional | Recommended (key-based lookup) |
| Full table scan | Supported | Not recommended |
| Point-in-time | Supported via spine | Returns latest values only |
---
### Step 4: Production Application Pattern
**Flask/FastAPI serving example:**
```python
import os
from concurrent.futures import ThreadPoolExecutor
from snowflake.snowpark import Session
from snowflake.ml.feature_store import FeatureStore, CreationMode
from snowflake.ml.feature_store.feature_view import StoreType
session = Session.builder.configs(connection_params).create()
feature_store = FeatureStore(
session=session,
database="<DATABASE>",
name="<SCHEMA>",
default_warehouse="",
creation_mode=CreationMode.FAIL_IF_NOT_EXIST,
)
fv_1 = feature_store.get_feature_view("FV_FEATURES_1", "V01")
fv_2 = feature_store.get_feature_view("FV_FEATURES_2", "V01")
executor = ThreadPoolExecutor(max_workers=os.cpu_count() * 2)
def retrieve_features(fv, keys, feature_names):
return feature_store.read_feature_view(
fv, keys=[keys],
feature_names=feature_names,
store_type=StoreType.ONLINE,
).collect()
def predict(entity_key):
future_1 = executor.submit(retrieve_features, fv_1, [entity_key], ["F1", "F2"])
future_2 = executor.submit(retrieve_features, fv_2, [entity_key], ["F3", "F4"])
features_1 = future_1.result()
features_2 = future_2.result()
feature_vector = list(features_1[0][1:]) + list(features_2[0][1:])
prediction = model.predict(feature_vector)
return prediction
```
**Authentication for production:**
- Use Programmatic Access Tokens (PAT) or key-pair authentication
- Do not use password-based auth in production services
**Use batch lookups** (not individual calls) and a **dedicated warehouse** for OFT refresh.
---
### Step 5: Benchmarking & Performance
**Expected latency** for single-point lookup (~10 features):
| Percentile | Latency | Condition |
|-----------|---------|-----------|
| p50 | ~30ms | |
| p95 | ~50ms | |
| p99 | <100ms | <2000 QPS |
**If latency is high, check in this order:**
1. **API**: Are you using `read_feature_view(..., store_type=StoreType.ONLINE)`? Using `retrieve_feature_values()` (batch join API) is the most common benchmarking mistake.
2. **Config**: Is `online_config` set to `enable=True` on the feature view?
3. **Location**: Is your client running on an EC2 or SPCS instance in the **same region** as Snowflake? Local laptops, VPNs, or cross-region instances add 100ms+ of network jitter.
4. **Warm-up**: Has the warehouse been running lookups for at least 3-5 minutes? Hybrid Tables use a memory cache that must be primed.
**Hybrid Table warm-up:**
- Small datasets: ~3 minutes of queries before benchmarking
- Large datasets: 10-15 minutes before benchmarking
- Set `AUTO_SUSPEND` to 300-600 seconds to prevent cache loss. If the warehouse suspends, the next query reverts to ~500ms+ while data re-caches.
**Parallel feature retrieval:** If your application needs features from multiple feature views, use `ThreadPoolExecutor` (see Step 4 example) to fire requests in parallel. Sequential calls across 4 FVs = ~150-200ms; parallel = ~30-50ms (slowest call only).
---
### Step 6: Disable Online Serving
**⚠️ MANDATORY CHECKPOINT** — Disabling drops the online feature table.
```
I will disable online serving for:
- Feature View: <FV_NAME> v<VERSION>
This will DROP the online feature table. Any applications reading from
the online store for this feature view will stop working.
Approve? (Yes/No)
```
```python
disable_config = OnlineConfig(enable=False)
fs.update_feature_view(
name="<FV_NAME>",
version="<VERSION>",
online_config=disable_config,
)
```
---
## Stopping Points
- ✋ Step 1: Feature view selection for online serving
- ✋ Step 2: Before enabling online serving (mandatory approval — cost implications)
- ✋ Step 6: Before disabling online serving (mandatory approval — drops online table)
## Output
- Feature view(s) with online serving enabled
- Code pattern for production application integration
## Next Skill
- If user wants monitoring → **Load** `monitor/SKILL.md`
- If user needs production deployment guidance → **Load** `../../spcs-inference/SKILL.md`
- If user wants lineage/inference FV → **Load** `lineage/SKILL.md`
feature-store/pipelines/SKILL.md
---
name: feature-store-pipelines
description: "Build and manage Snowflake Feature Store pipelines — managed (Dynamic Table), external (dbt, custom), and inference."
parent_skill: feature-store
path: machine-learning/feature-store/pipelines
---
# Feature Pipelines
## When to Load
Parent skill routes here for PIPELINES intent: "feature pipeline", "refresh_freq", "managed feature view", "external feature view", "dbt features", "schedule features", "inference pipeline".
## Prerequisites
- `../references/api-reference.md` loaded
- Feature store (`fs`) and entities already created (via create/SKILL.md)
---
## Workflow
### Step 1: Determine Pipeline Type
**Ask user:**
```
What type of feature pipeline do you need?
1. Snowflake-managed — Automatic refresh via Dynamic Tables (recommended)
2. External — You maintain the feature table (e.g., via dbt, custom ETL)
3. Inference pipeline — Use a registered model to produce features
```
**⚠️ STOP**: Wait for user response.
---
### Step 2A: Snowflake-Managed Pipeline
A managed feature view uses a Dynamic Table that automatically refreshes from source data.
**Key decisions:**
| Decision | Options | Guidance |
|----------|---------|----------|
| Refresh frequency | Time delta or cron | Balance freshness vs cost. Minimum: 1 minute |
| Refresh mode | INCREMENTAL / FULL | INCREMENTAL preferred. Requires change tracking on sources |
| Clustering | Column list | Use join keys + frequently filtered columns |
**Change tracking requirement:**
```sql
SHOW TABLES LIKE '<SOURCE_TABLE>';
-- Look for change_tracking = 'ON'
ALTER TABLE <SOURCE_TABLE> SET CHANGE_TRACKING = TRUE;
```
If the user doesn't own the source table and can't enable change tracking, use `refresh_mode="FULL"`.
**Create the managed feature view:**
```python
feature_df = session.sql("""
SELECT <join_key>, <timestamp_col>, <feature_columns>
FROM <source_table>
-- feature transformation logic here
""")
fv = FeatureView(
name="<ENTITY>_<DOMAIN>_FV",
entities=[entity],
feature_df=feature_df,
timestamp_col="<TS_COL>",
refresh_freq="<FREQ>",
refresh_mode="INCREMENTAL",
desc="<description>",
)
```
**⚠️ MANDATORY CHECKPOINT** — Present the feature view configuration to user before registering:
```
I will register this managed feature view:
- Name: <FV_NAME> V01
- Source: <source_table>
- Refresh: <FREQ> (<INCREMENTAL|FULL>)
- Timestamp col: <TS_COL>
This creates a Dynamic Table with ongoing compute cost.
Approve? (Yes/No/Modify)
```
```python
registered_fv = fs.register_feature_view(fv, version="V01", block=True)
```
**Multi-stage pipelines:**
For complex transformations, chain multiple feature views where intermediate stages use `refresh_freq="DOWNSTREAM"`:
```python
clean_fv = FeatureView(
name="FV_CLEAN", entities=[entity], feature_df=clean_df,
refresh_freq="DOWNSTREAM",
)
agg_fv = FeatureView(
name="FV_AGGREGATED", entities=[entity], feature_df=agg_df,
refresh_freq="10 minutes",
timestamp_col="EVENT_TS",
)
```
---
### Step 2B: External Pipeline
For features maintained outside the Feature Store (e.g., dbt).
**The user is responsible for:**
1. Creating and maintaining the feature table
2. Populating it with fresh data
3. The feature view is a read-only registration
```python
feature_df = session.table("MY_DB.MY_SCHEMA.MY_FEATURE_TABLE").select(
"CUSTOMER_ID", "TX_DATETIME",
"TX_AMOUNT_1D", "TX_AMOUNT_7D", "TX_COUNT_30D"
)
external_fv = FeatureView(
name="<ENTITY>_<DOMAIN>_FV",
entities=[customer_entity],
feature_df=feature_df,
timestamp_col="TX_DATETIME",
refresh_freq=None,
desc="Features maintained by dbt pipeline",
)
fs.register_feature_view(external_fv, version="V01", block=True)
```
**dbt integration pattern:**
1. Build feature table in dbt: `dbt run --select ft_customer_transactions`
2. Register the dbt-produced table as an external feature view
3. dbt handles scheduling; feature store provides discovery, versioning, and retrieval
---
### Step 2C: Inference Pipeline
Use a registered model to produce prediction features as a new feature view.
```python
input_df = fs.read_feature_view("FV_FEATURES", "V01")
from snowflake.ml.registry import Registry
reg = Registry(session, database="MY_DB", schema="MODEL_REGISTRY")
mv = reg.get_model("MY_MODEL").version("V01")
inference_df = mv.run(input_df, function_name="predict")
inference_fv = FeatureView(
name="<MODEL>_INFERENCE_FV",
entities=[entity],
feature_df=inference_df,
refresh_freq="60 minutes",
desc="Ongoing inference from ML model",
)
fs.register_feature_view(inference_fv, version="V01", block=True)
```
For more advanced inference feature views mapped from model signatures → **Load** `lineage/SKILL.md`.
---
### Step 3: Verify Pipeline Health
```python
fs.list_feature_views().select("NAME", "VERSION", "SCHEDULING_STATE").show()
fs.get_refresh_history(registered_fv).show()
```
For deeper monitoring → **Load** `monitor/SKILL.md`
---
## Important Constraints
1. **Incremental DTs cannot depend on Full refresh DTs** — ensure upstream stages are also incremental
2. **Change tracking must stay enabled** on source tables for incremental refresh
3. **Avoid SELECT *** in feature DataFrames — use explicit column lists to prevent schema change failures
4. **Minimum refresh_freq is 1 minute**
5. **DOWNSTREAM target lag** only works for intermediate pipeline stages (not leaf nodes)
6. **Incremental refresh blockers** — see `../references/troubleshooting.md` for patterns that force FULL refresh
---
## Stopping Points
- ✋ Step 1: Pipeline type selection
- ✋ Before `register_feature_view` — present config for approval
- ✋ Step 3: After verification, offer next steps
## Output
- Registered feature view(s) backed by managed or external pipelines
- Verified refresh state
## Next Skill
- If user wants training data → **Load** `training/SKILL.md`
- If user wants online serving → **Load** `online/SKILL.md`
- If user wants monitoring → **Load** `monitor/SKILL.md`
- If user wants lineage/inference FV → **Load** `lineage/SKILL.md`
feature-store/references/api-reference.md
# Snowflake Feature Store — Python API Quick Reference
Package: `snowflake-ml-python` (>= 1.5.0, >= 1.18.0 for online features, >= 1.21.0 for Aggregation API)
```python
from snowflake.ml.feature_store import (
FeatureStore,
FeatureView,
Entity,
CreationMode,
OnlineConfig,
Feature,
)
from snowflake.ml.feature_store.feature_view import StoreType
```
---
## FeatureStore
### Constructor
```python
fs = FeatureStore(
session=session, # Snowpark Session (required)
database="MY_DB", # Database name (required, must exist)
name="MY_FEATURE_STORE", # Schema name (required)
default_warehouse="MY_WH", # Warehouse (required)
creation_mode=CreationMode.CREATE_IF_NOT_EXIST # or FAIL_IF_NOT_EXIST
)
```
- `CREATE_IF_NOT_EXIST`: Creates schema + tags if missing. Use for initial setup.
- `FAIL_IF_NOT_EXIST`: Connects to existing feature store. Use for subsequent connections.
### Entity Methods
| Method | Signature | Description |
|--------|-----------|-------------|
| `register_entity` | `(entity: Entity)` | Register entity in feature store |
| `get_entity` | `(name: str) → Entity` | Retrieve registered entity |
| `list_entities` | `() → DataFrame` | List all entities |
| `update_entity` | `(name: str, desc: str)` | Update entity description |
| `delete_entity` | `(name: str)` | Delete entity (fails if referenced by feature views) |
### Feature View Methods
| Method | Signature | Description |
|--------|-----------|-------------|
| `register_feature_view` | `(feature_view: FeatureView, version: str, block: bool=True, overwrite: bool=False) → FeatureView` | Materialize feature view |
| `get_feature_view` | `(name: str, version: str) → FeatureView` | Retrieve registered feature view |
| `list_feature_views` | `(entity_name=None, feature_view_name=None) → DataFrame` | List feature views |
| `update_feature_view` | `(name: str, version: str, desc=None, online_config=None, refresh_freq=None, warehouse=None) → FeatureView` | Update feature view properties |
| `delete_feature_view` | `(feature_view: FeatureView or str, version: str)` | Delete feature view |
| `suspend_feature_view` | `(feature_view or name: str, version: str)` | Suspend scheduling |
| `resume_feature_view` | `(feature_view or name: str, version: str)` | Resume scheduling |
| `read_feature_view` | `(feature_view, version=None, keys=None, feature_names=None, store_type=StoreType.OFFLINE) → DataFrame` | Read feature values |
| `get_refresh_history` | `(feature_view, version=None) → DataFrame` | Get refresh statistics |
### Dataset & Retrieval Methods
| Method | Signature | Description |
|--------|-----------|-------------|
| `generate_dataset` | `(name, spine_df, features, *, version=None, spine_timestamp_col=None, spine_label_cols=None, exclude_columns=None, include_feature_view_timestamp_col=False, desc="", output_type="dataset") → Dataset` | Generate training dataset with point-in-time joins |
| `generate_training_set` | `(spine_df, features, timestamp_col, spine_label_cols=None) → DataFrame` | Generate training set (returns DataFrame directly) |
| `retrieve_feature_values` | `(spine_df, features, spine_timestamp_col=None, exclude_columns=None, include_feature_view_timestamp_col=False) → DataFrame` | Enrich spine with features (for inference) |
| `load_feature_views_from_dataset` | `(ds: Dataset) → list[FeatureView]` | Get feature views used in a dataset |
### Other Methods
| Method | Signature | Description |
|--------|-----------|-------------|
| `update_default_warehouse` | `(warehouse_name: str)` | Change default warehouse |
---
## Entity
### Constructor
```python
entity = Entity(
name="CUSTOMER", # Entity name (required)
join_keys=["CUSTOMER_ID"], # List of join key column names (required)
desc="Customer entity" # Optional description
)
```
**Properties:** `name`, `join_keys`, `desc`, `owner`
**Notes:**
- Join keys are immutable after registration. To change, create a new entity.
- Entities referenced by feature views cannot be deleted.
- Composite entities use multiple join keys: `join_keys=["ORDER_ID", "LINE_ID"]`
---
## FeatureView
### Constructor
```python
fv = FeatureView(
name="MY_FEATURE_VIEW", # Name (required)
entities=[entity], # List of Entity objects (required)
feature_df=my_df, # Snowpark DataFrame with feature logic (required)
timestamp_col="EVENT_TS", # Timestamp column for temporal features (optional)
refresh_freq="5 minutes", # Refresh schedule (optional; None = external)
desc="Description", # Optional description
refresh_mode="INCREMENTAL", # "INCREMENTAL" or "FULL" (optional)
cluster_by=["COL1"], # Clustering columns (optional)
)
```
**Key parameters:**
- `refresh_freq`: Set to make it Snowflake-managed (Dynamic Table). Set `None` for external management (view). Accepts time deltas (`"5 minutes"`, `"1 hour"`) or cron (`"* * * * * America/Los_Angeles"`). Minimum: 1 minute.
- `refresh_mode`: `"INCREMENTAL"` (default, requires change tracking on sources) or `"FULL"`.
- `feature_df`: Must contain the join key columns from the associated entities.
- `timestamp_col`: Required for point-in-time correct retrieval.
### Key Methods
| Method | Description |
|--------|-------------|
| `attach_feature_desc(desc_dict)` | Attach descriptions to features: `{"COL": "description"}` |
| `slice(feature_names)` | Create a FeatureViewSlice with subset of features |
| `to_df()` | Convert feature view metadata to DataFrame |
| `feature_names` | List of feature column names |
| `feature_descs` | List of feature descriptions (SqlIdentifier objects — use `str()` to convert) |
| `status` | `DRAFT`, `ACTIVE`, `SUSPENDED` (FeatureViewStatus enum — use `str()` for comparison) |
| `version` | Version string |
| `query` | The underlying SQL query |
| `entities` | Linked entities list |
| `online` | Whether online serving is enabled |
### Online Store
```python
# Enable online store at creation time
fv = FeatureView(...).with_online_store(enabled=True)
# Or enable on existing feature view via OnlineConfig
from snowflake.ml.feature_store import OnlineConfig
config = OnlineConfig(enable=True, target_lag="15s")
fs.update_feature_view(name="MY_FV", version="v1", online_config=config)
```
---
## Feature (Aggregation API)
Requires `snowflake-ml-python >= 1.21.0`. Declarative time-windowed aggregations with tiling.
> **Limitation:** Time Window Aggregation API does not yet work with online feature store. Postgres support is in progress; not planned for hybrid table.
```python
from snowflake.ml.feature_store import Feature
amount = Feature("PURCHASE_AMOUNT", "Amount of each purchase")
features = [
amount.sum(windows=["7d", "30d"]).alias("TOTAL_SPEND"),
amount.avg(windows=["7d", "30d"]).alias("AVG_SPEND"),
amount.count(windows=["7d", "30d"]).alias("PURCHASE_CNT"),
amount.std(windows=["30d"]).alias("SPEND_STD"),
]
fv = FeatureView(
name="USER_PURCHASE_FV",
entities=[user_entity],
feature_df=transactions_df,
feature_granularity="1 day", # Tile size
features=features,
refresh_freq="1 day",
desc="User purchase aggregations",
)
```
**Available functions:** `.sum()`, `.count()`, `.avg()`, `.min()`, `.max()`, `.std()`, `.var()`, `.approx_count_distinct()`, `.last_n()`, `.first_n()`
**Tile size guidance:**
| Data Type | `feature_granularity` |
|-----------|----------------------|
| Clickstream | `"1 hour"` |
| Transactions | `"1 day"` |
| Weekly reports | `"1 week"` |
---
## OnlineConfig
```python
from snowflake.ml.feature_store import OnlineConfig
config = OnlineConfig(enable=True, target_lag="15s")
fs.update_feature_view(
name="MY_FV", version="v1",
online_config=config
)
```
**Target lag options:** `"15s"`, `"1m"`, `"5m"` — balance freshness vs cost.
---
## StoreType
```python
from snowflake.ml.feature_store.feature_view import StoreType
# Read from offline store (default)
fs.read_feature_view("MY_FV", "v1", store_type=StoreType.OFFLINE)
# Read from online store (low-latency)
fs.read_feature_view("MY_FV", "v1", keys=[[1], [2]], store_type=StoreType.ONLINE)
```
---
## CreationMode
| Mode | Behavior |
|------|----------|
| `CreationMode.CREATE_IF_NOT_EXIST` | Creates schema and tags if they don't exist |
| `CreationMode.FAIL_IF_NOT_EXIST` | Raises error if schema doesn't exist |
---
## Model Registry (for lineage and inference)
```python
from snowflake.ml.registry import Registry
registry = Registry(session=session)
model = registry.get_model("<MODEL_NAME>")
mv = model.version("<VERSION>")
mv.show_functions() # Get model signature
mv.lineage(direction='upstream', domain_filter={'feature_view'}) # Get source FVs
mv.run(inference_df, function_name="predict") # Run model
```
**`show_functions()` returns** `List[dict]` with keys: `name`, `target_method`, `signature` (ModelSignature with `.inputs` list of FeatureSpec with `.name`, `.dtype`).
feature-store/references/design-guide.md
# Feature Store Design & Organization Guide
Best practices for structuring, naming, versioning, securing, and promoting Snowflake Feature Stores.
---
## Transformation Taxonomy (MIT / MDT / ODT)
Classify every transformation before deciding where it belongs:
| Type | Full Name | Where | Reusable? | Examples |
|------|-----------|-------|-----------|----------|
| **MIT** | Model-Independent | FeatureView | Yes, across models | Aggregations, joins, derived columns |
| **MDT** | Model-Dependent | Model Registry (Pipeline) | No, tied to model | Scaling, encoding, imputation. **Fit on training data only** |
| **ODT** | On-Demand | Inference time | N/A | Time-since-last, distance, current weather |
**Anti-patterns:**
- MDT in FeatureView (hard-coded scaler params)
- MIT in training pipeline (per-model aggregation)
- ODT for stable features (30-day aggregates at request time)
---
## Naming Conventions
### Entities
- Use SCREAMING_SNAKE_CASE business domain nouns: `CUSTOMER`, `PRODUCT`, `ORDER`, `SESSION`
- Composite entities: `ORDER_LINE` (not `ORDER_AND_LINE_ITEM`)
### Feature Views
- Pattern: `<ENTITY>_<DOMAIN>_FV`
- Examples: `CUSTOMER_ORDER_FV`, `SESSION_ENGAGEMENT_FV`, `TAXI_ROUTE_TRIP_FV`
- Keep names descriptive but concise
### Feature Columns
- Use UPPER_SNAKE_CASE consistent with Snowflake conventions
- Include the aggregation and window in the name
| Suffix | Meaning | Example |
|--------|---------|---------|
| `_TS` | Timestamp | `ORDER_TS` |
| `_CNT` | Count | `ORDER_CNT` |
| `_DCNT` | Distinct count | `PRODUCT_VIEW_DCNT` |
| `_SUM` | Sum | `REVENUE_SUM` |
| `_AVG` | Average | `ORDER_VALUE_AVG` |
| `_AMT` | Amount (currency) | `TOTAL_AMT` |
| `IS_` | Boolean flag | `IS_CONVERTED` |
| `_<WINDOW>` | Aggregation window | `TOTAL_SPEND_7D` |
| `_STD` | Standard deviation | `FARE_STD` |
- Use `attach_feature_desc()` to add human-readable descriptions
### Versions
- Zero-padded sequential for lexicographic sort: `V01`, `V02`, ..., `V10`
- Also acceptable: `DEV_V01`, `1.0.0`, `20250115`
- Never reuse a version string for different logic — create a new version instead
- No built-in "latest version" alias — must sort version strings to find latest
---
## Schema Organization
### Single-Environment (Simple)
```
MY_DB/
└── FEATURE_STORE/ # Single feature store schema
├── Entities
├── Feature Views (Dynamic Tables)
└── Tags
```
### Multi-Environment (Recommended for Production)
```
ML_FEATURES_DB/
├── DEV_FEATURE_STORE/ # Development
├── TEST_FEATURE_STORE/ # Staging / QA
└── PROD_FEATURE_STORE/ # Production
```
Separate warehouses per environment for cost attribution:
- `DEV_WH` (XSMALL), `TEST_WH` (SMALL), `PROD_WH` (MEDIUM)
- `PROD_OFT_WH` (SMALL) — dedicated OFT refresh
### Hybrid Organization (Larger Orgs)
```
ML_FEATURES_DB/
├── SHARED_FEATURE_STORE/ (core cross-domain entities)
├── MARKETING_FEATURES/ (domain-specific)
├── FRAUD_FEATURES/ (domain-specific)
└── RECOMMENDATION_FEATURES/ (domain-specific)
```
---
## Environment Promotion (DEV → TEST → PROD)
### Zero-Copy Clone (Quick)
```sql
CREATE OR REPLACE SCHEMA PROD_FEATURE_STORE CLONE DEV_FEATURE_STORE;
```
### Python-Based Promotion (Controlled)
```python
dev_fs = FeatureStore(session, database, "DEV_FEATURE_STORE", warehouse, CreationMode.FAIL_IF_NOT_EXIST)
prod_fs = FeatureStore(session, database, "PROD_FEATURE_STORE", warehouse, CreationMode.CREATE_IF_NOT_EXIST)
fv = dev_fs.get_feature_view("USER_PURCHASE_FV", "V01")
# Recreate in prod with same query and metadata
```
Promotion means re-creating the FeatureView with identical logic but pointing to the target schema's source tables.
### Validation After Promotion
Schema match, row counts, sample value comparison.
### CI/CD Flow
Feature definitions as code → GitHub Actions → validate → test → deploy DEV → integration test → deploy PROD (approval gate required).
---
## Versioning Strategy
### When to Create a New Version
- Feature logic changes (different transformations)
- Source data schema changes
- Aggregation windows change
- New features added to the view
### When NOT to Create a New Version
- Data refreshes (handled automatically by Dynamic Tables)
- Bug fixes to source data (features auto-refresh)
### Deprecation Pattern
1. Create new version with updated logic
2. Update downstream consumers to reference new version
3. Suspend old version: `fs.suspend_feature_view("MY_FV", "V01")`
4. After confirmation period, delete: `fs.delete_feature_view("MY_FV", "V01")`
---
## Access Control (RBAC)
### Recommended Roles
| Role | Permissions | Users |
|------|------------|-------|
| `FS_ADMIN` | CREATE SCHEMA, manage entities and feature views | ML platform team |
| `FS_DEVELOPER` | CREATE DYNAMIC TABLE, register feature views | Data engineers, ML engineers |
| `FS_CONSUMER` | SELECT on feature views, generate datasets | Data scientists |
| `FS_READER` | SELECT on feature views (read-only) | Analysts, downstream services |
Hierarchy: FS_READER < FS_CONSUMER < FS_DEVELOPER < FS_ADMIN < SYSADMIN
### Grant Pattern
```sql
SET FS_DATABASE = 'ML_FEATURES_DB';
SET FS_SCHEMA = 'PROD_FEATURE_STORE';
SET FS_WAREHOUSE = 'PROD_WH';
CREATE ROLE IF NOT EXISTS FS_ADMIN;
CREATE ROLE IF NOT EXISTS FS_DEVELOPER;
CREATE ROLE IF NOT EXISTS FS_CONSUMER;
CREATE ROLE IF NOT EXISTS FS_READER;
GRANT ROLE FS_READER TO ROLE FS_CONSUMER;
GRANT ROLE FS_CONSUMER TO ROLE FS_DEVELOPER;
GRANT ROLE FS_DEVELOPER TO ROLE FS_ADMIN;
GRANT ROLE FS_ADMIN TO ROLE SYSADMIN;
-- Admin privileges
GRANT CREATE SCHEMA ON DATABASE IDENTIFIER($FS_DATABASE) TO ROLE FS_ADMIN;
GRANT USAGE ON DATABASE IDENTIFIER($FS_DATABASE) TO ROLE FS_ADMIN;
GRANT ALL ON SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_ADMIN;
-- Developer privileges
GRANT USAGE ON DATABASE IDENTIFIER($FS_DATABASE) TO ROLE FS_DEVELOPER;
GRANT ALL ON SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_DEVELOPER;
GRANT USAGE ON WAREHOUSE IDENTIFIER($FS_WAREHOUSE) TO ROLE FS_DEVELOPER;
GRANT CREATE DYNAMIC TABLE ON SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_DEVELOPER;
GRANT CREATE VIEW ON SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_DEVELOPER;
GRANT CREATE TAG ON SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_DEVELOPER;
-- Consumer privileges
GRANT USAGE ON DATABASE IDENTIFIER($FS_DATABASE) TO ROLE FS_CONSUMER;
GRANT USAGE ON SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_CONSUMER;
GRANT USAGE ON WAREHOUSE IDENTIFIER($FS_WAREHOUSE) TO ROLE FS_CONSUMER;
GRANT SELECT ON ALL DYNAMIC TABLES IN SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_CONSUMER;
GRANT SELECT ON FUTURE DYNAMIC TABLES IN SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_CONSUMER;
GRANT SELECT ON ALL VIEWS IN SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_CONSUMER;
GRANT SELECT ON FUTURE VIEWS IN SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_CONSUMER;
-- Reader privileges (read-only)
GRANT USAGE ON DATABASE IDENTIFIER($FS_DATABASE) TO ROLE FS_READER;
GRANT USAGE ON SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_READER;
GRANT SELECT ON ALL DYNAMIC TABLES IN SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_READER;
GRANT SELECT ON ALL VIEWS IN SCHEMA IDENTIFIER($FS_DATABASE || '.' || $FS_SCHEMA) TO ROLE FS_READER;
```
### Environment-Specific Roles (Optional)
```sql
CREATE ROLE IF NOT EXISTS FS_DEV_DEVELOPER;
CREATE ROLE IF NOT EXISTS FS_PROD_INFERENCE;
```
---
## Feature Categorization
| Category | Description | Example |
|----------|-------------|---------|
| **Raw** | Direct columns from source | `CUSTOMER_AGE`, `PRODUCT_PRICE` |
| **Derived** | Per-row transformations | `DAY_OF_WEEK`, `IS_WEEKEND`, `AMOUNT_TIER` |
| **Aggregated** | Group-by or window aggregations | `TX_SUM_7D`, `AVG_ORDER_VALUE_30D` |
| **Temporal** | Time-based patterns | `DAYS_SINCE_LAST_PURCHASE`, `IS_PEAK_HOUR`, `ORIG_MONTH` |
| **Cross-Entity** | Features joining multiple entities | `CUSTOMER_PRODUCT_AFFINITY_SCORE` |
---
## Performance Guidelines
- **Prefer incremental refresh** over full refresh for large datasets
- **Enable change tracking** on source tables before creating managed feature views
- **Use explicit column lists** in feature DataFrames (avoid `SELECT *`)
- **Cluster feature views** by frequently filtered columns
- **Set appropriate refresh_freq** — balance freshness needs against compute cost
- **Use DOWNSTREAM target lag** for intermediate pipeline stages
- **Dedicate warehouses** for large or critical feature pipelines
- **Cast float columns to DECIMAL** before aggregation to preserve incremental refresh
feature-store/references/feature-patterns.md
# Feature Transformation Patterns
Common patterns for defining features in Snowflake Feature Store using Snowpark Python and SQL.
---
## Per-Row Features
Functions applied to each row independently. One output row per input row.
```python
def compute_features(df: snowpark.DataFrame) -> snowpark.DataFrame:
df = df.fillna({"foo": 0})
df = df.with_column("zipcode", F.compute_zipcode(df["lat"], df["long"]))
return df
```
```sql
SELECT *, CASE WHEN amount > 1000 THEN 'high' ELSE 'low' END AS amount_tier
FROM source_table;
```
---
## Per-Group Features
Aggregate values within a group. One output row per group.
```python
def group_features(df: snowpark.DataFrame) -> snowpark.DataFrame:
return df.group_by("city").agg(
F.sum("rainfall").alias("total_rainfall"),
F.avg("temperature").alias("avg_temperature"),
F.count("*").alias("num_readings")
)
```
```sql
SELECT city,
SUM(rainfall) AS total_rainfall,
AVG(temperature) AS avg_temperature,
COUNT(*) AS num_readings
FROM weather_data
GROUP BY city;
```
---
## Row-Based Window Features
Aggregate over a fixed window of rows. One output row per window frame.
```python
from snowflake.snowpark import Window
def sum_past_3_transactions(df: snowpark.DataFrame) -> snowpark.DataFrame:
window = Window.partition_by("id").order_by("ts").rows_between(-2, Window.CURRENT_ROW)
return df.select(
"id", "ts",
F.sum("amount").over(window).alias("sum_past_3_transactions")
)
```
```sql
SELECT id, ts,
SUM(amount) OVER (
PARTITION BY id ORDER BY ts
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW
) AS sum_past_3_transactions
FROM transactions;
```
---
## Moving Aggregation Features
Compute moving statistics (sum, avg, min, max) within a specified window size. Uses the Snowpark analytics API.
```python
new_df = df.analytics.moving_agg(
aggs={"AMOUNT": ["SUM", "AVG"]},
window_sizes=[7, 30],
order_by=["TX_DATE"],
group_by=["CUSTOMER_ID"]
)
# Produces: AMOUNT_SUM_7, AMOUNT_SUM_30, AMOUNT_AVG_7, AMOUNT_AVG_30
```
---
## Cumulative Aggregation Features
Running totals from the start (or to the end) of a partition. One output row per input row.
```python
new_df = df.analytics.cumulative_agg(
aggs={"SALESAMOUNT": ["SUM", "MIN", "MAX"]},
order_by=["ORDERDATE"],
group_by=["PRODUCTKEY"],
is_forward=True
)
```
---
## Lag Features
Values from prior rows, offset by a specified number of rows. Useful for detecting trends.
```python
new_df = df.analytics.compute_lag(
cols=["AMOUNT"],
lags=[1, 7, 30],
order_by=["TX_DATE"],
group_by=["CUSTOMER_ID"]
)
# Produces: AMOUNT_LAG_1, AMOUNT_LAG_7, AMOUNT_LAG_30
```
---
## Lead Features
Values from subsequent rows.
```python
new_df = df.analytics.compute_lead(
cols=["AMOUNT"],
leads=[1, 7],
order_by=["TX_DATE"],
group_by=["CUSTOMER_ID"]
)
```
---
## Time-Based Window Aggregations (RANGE BETWEEN)
Aggregate over calendar time windows rather than row counts.
```sql
SELECT customer_id, tx_datetime, tx_amount,
SUM(tx_amount) OVER (
PARTITION BY customer_id ORDER BY tx_datetime
RANGE BETWEEN INTERVAL '1 DAY' PRECEDING AND CURRENT ROW
) AS tx_amount_1d,
SUM(tx_amount) OVER (
PARTITION BY customer_id ORDER BY tx_datetime
RANGE BETWEEN INTERVAL '7 DAYS' PRECEDING AND CURRENT ROW
) AS tx_amount_7d,
COUNT(*) OVER (
PARTITION BY customer_id ORDER BY tx_datetime
RANGE BETWEEN INTERVAL '30 DAYS' PRECEDING AND CURRENT ROW
) AS tx_count_30d
FROM transactions;
```
---
## Tile-Based Aggregation (Feature Store Native)
Use the Feature Store's built-in Aggregation API for efficient pre-computed tiles. Requires `snowflake-ml-python >= 1.24.0`.
```python
from snowflake.ml.feature_store import Feature
amount = Feature("PURCHASE_AMOUNT", "Amount of each purchase")
fv = FeatureView(
name="CUSTOMER_AGG_FEATURES",
entities=[customer_entity],
feature_df=transactions_df,
feature_granularity="1 day",
features=[
amount.sum(windows=["7d", "30d"]).alias("TOTAL_SPEND"),
amount.avg(windows=["7d", "30d"]).alias("AVG_SPEND"),
amount.count(windows=["7d", "30d"]).alias("PURCHASE_CNT"),
amount.std(windows=["30d"]).alias("SPEND_STD"),
],
timestamp_col="TX_DATETIME",
refresh_freq="1 day",
)
```
**How it works:** A Dynamic Table stores pre-computed partial aggregations (tiles). During dataset generation, tiles are merged for point-in-time correct results.
**Available functions:** `.sum()`, `.count()`, `.avg()`, `.min()`, `.max()`, `.std()`, `.var()`, `.approx_count_distinct()`, `.last_n()`, `.first_n()`
---
## Temporal Feature Patterns
Derive features from date, timestamp, or YYYYMM-encoded columns.
| Category | Features | SQL Pattern | When to Use |
|----------|----------|-------------|-------------|
| **Calendar extraction** | `ORIG_MONTH`, `ORIG_QUARTER`, `ORIG_YEAR` | `MOD(YYYYMM_COL, 100)`, `CEIL(MOD()/3.0)::INT`, `FLOOR(YYYYMM_COL/100)` | Any date/YYYYMM column |
| **Seasonality flags** | `IS_WINTER_ORIG`, `IS_Q4_ORIG`, `IS_WEEKEND` | `CASE WHEN month IN (...) THEN 1 ELSE 0 END` | Seasonal patterns expected |
| **Duration / span** | `LOAN_DURATION_YEARS`, `CONTRACT_MONTHS` | `DATEDIFF(...)` on static date pairs | Two date columns define a span |
| **Cyclical encoding** | `MONTH_SIN`, `MONTH_COS`, `DOW_SIN`, `DOW_COS` | `SIN(2 * PI() * val / period)`, `COS(...)` | Month/day-of-week should wrap |
| **Epoch / vintage** | `DAYS_SINCE_EPOCH`, `ORIG_YEAR_BUCKET` | `DATEDIFF('day', '1970-01-01', date_col)`, `FLOOR(YEAR/5)*5` | Absolute time position matters |
| **Relative position** | `MONTH_IN_QUARTER`, `WEEK_IN_YEAR` | `MOD(month - 1, 3) + 1`, `WEEKOFYEAR(...)` | Position within cycle matters |
---
## Combining Multiple Pattern Types
A feature view can combine multiple pattern types in a single Snowpark DataFrame pipeline:
```python
def build_customer_features(session, source_table):
df = session.table(source_table)
# Per-row: derive day of week
df = df.with_column("DAY_OF_WEEK", F.dayofweek(F.col("TX_DATETIME")))
# Per-group with window: rolling aggregates
window_7d = Window.partition_by("CUSTOMER_ID").order_by("TX_DATETIME").range_between(-7*86400, 0)
df = df.with_column("TX_SUM_7D", F.sum("TX_AMOUNT").over(window_7d))
# Lag: previous transaction amount
window_lag = Window.partition_by("CUSTOMER_ID").order_by("TX_DATETIME")
df = df.with_column("PREV_TX_AMOUNT", F.lag("TX_AMOUNT", 1).over(window_lag))
return df
```
---
## Incremental Refresh Compatibility
These patterns **block** incremental refresh on Dynamic Tables:
| Blocker | Fix |
|---------|-----|
| `MODE()` | Replace with `ROW_NUMBER() OVER (PARTITION BY ... ORDER BY COUNT(*) DESC)` in a separate FV |
| `RANDOM()`, `UUID()` | Remove or compute outside the FV |
| `CURRENT_DATE()`, `CURRENT_TIMESTAMP()` | Remove — compute at query/inference time instead |
| Float-typed aggregation + JOINs | Remove JOINs or split into separate FVs |
| Float-typed aggregation + CASE comparison | Cast INPUT columns to `DECIMAL` before aggregation: `AVG(COL::DECIMAL(10,2))` |
| `STDDEV()` / `AVG()` on float columns | Cast input to DECIMAL: `STDDEV(COL::DECIMAL(10,2))::DECIMAL(10,2)` |
feature-store/references/troubleshooting.md
# Feature Store Troubleshooting Reference
## Diagnostic Queries
### Check Failed Refreshes
```sql
SELECT name, refresh_version, state, state_message, refresh_start_time
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY())
WHERE schema_name = '<FEATURE_STORE_SCHEMA>'
AND state = 'FAILED'
ORDER BY refresh_start_time DESC;
```
### Check Dynamic Table Lag
```sql
SELECT
name,
target_lag,
actual_lag,
CASE WHEN actual_lag > target_lag THEN 'LAGGING' ELSE 'ON_TARGET' END AS lag_status
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLES())
WHERE schema_name = '<FEATURE_STORE_SCHEMA>';
```
### Health Dashboard
```sql
SELECT
name,
scheduling_state,
CASE
WHEN scheduling_state = 'ACTIVE'
AND DATEDIFF('minute', last_refresh_time, CURRENT_TIMESTAMP()) < 60
THEN 'HEALTHY'
WHEN DATEDIFF('minute', last_refresh_time, CURRENT_TIMESTAMP()) >= 60
THEN 'STALE'
ELSE 'WARNING'
END AS health_status,
target_lag,
actual_lag
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLES())
WHERE schema_name = '<FEATURE_STORE_SCHEMA>';
```
### Verify Change Tracking
```sql
SHOW TABLES LIKE '<TABLE_NAME>' IN <DATABASE>.<SCHEMA>;
-- Check the change_tracking column in results
```
### Check Refresh Performance
```sql
SELECT
name,
state,
DATEDIFF('second', refresh_start_time, refresh_end_time) AS duration_seconds,
statistics:numInsertedRows::INT AS rows_inserted,
statistics:numUpdatedRows::INT AS rows_updated
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY())
WHERE schema_name = '<FEATURE_STORE_SCHEMA>'
ORDER BY refresh_start_time DESC
LIMIT 50;
```
### Cost Monitoring
```sql
SELECT
name,
SUM(DATEDIFF('second', refresh_start_time, refresh_end_time)) / 3600.0 AS approx_compute_hours,
COUNT(*) AS refresh_count,
AVG(DATEDIFF('second', refresh_start_time, refresh_end_time)) AS avg_refresh_seconds
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY())
WHERE schema_name = '<FEATURE_STORE_SCHEMA>'
GROUP BY name
ORDER BY approx_compute_hours DESC;
```
### Warehouse Credit Usage
```sql
SELECT warehouse_name, SUM(credits_used) AS total_credits
FROM SNOWFLAKE.ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY
WHERE start_time > DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY warehouse_name
ORDER BY total_credits DESC;
```
### Object Dependencies
```sql
SELECT *
FROM SNOWFLAKE.ACCOUNT_USAGE.OBJECT_DEPENDENCIES
WHERE REFERENCED_OBJECT_NAME LIKE '%FEATURE%'
ORDER BY REFERENCING_OBJECT_NAME;
```
### Feature View Lineage (SQL)
```sql
SELECT *
FROM TABLE(INFORMATION_SCHEMA.OBJECT_LINEAGE(
'<DATABASE>.<SCHEMA>.<FEATURE_VIEW_NAME>',
'dynamic_table'
));
```
---
## Incremental Refresh Blockers
| Blocker | Fix |
|---------|-----|
| `MODE()` | Replace with `ROW_NUMBER() OVER (PARTITION BY ... ORDER BY COUNT(*) DESC)` in a separate FV |
| `RANDOM()`, `UUID()` | Remove or compute outside the FV |
| `CURRENT_DATE()`, `CURRENT_TIMESTAMP()` | Remove — compute at query/inference time instead (ODT) |
| Float-typed aggregation + JOINs | Split into aggregate-only FV (no JOINs) and a separate FV for joined lookups |
| Float-typed aggregation + CASE comparison | Cast INPUT columns to `DECIMAL` before aggregation: `AVG(COL::DECIMAL(10,2))` |
| `STDDEV()` / `AVG()` on float columns | Cast input to DECIMAL: `STDDEV(COL::DECIMAL(10,2))::DECIMAL(10,2)` |
---
## Common Issues
| Issue | Cause | Solution |
|-------|-------|----------|
| Full refresh instead of incremental | `MODE()` in query | Replace with ROW_NUMBER() ranked subquery in separate FV |
| Full refresh instead of incremental | Float aggregation + JOINs | Split into aggregate-only FV (no JOINs) |
| Full refresh instead of incremental | Float aggregation + CASE | Cast inputs to DECIMAL: `AVG(COL::DECIMAL(10,2))` |
| Full refresh instead of incremental | `CURRENT_DATE()` | Remove — compute at query time (ODT) |
| Full refresh instead of incremental | `RANDOM()`, `UUID()` | Remove non-deterministic functions |
| High refresh latency | Complex joins or large data | Increase warehouse size, optimize query |
| Permission denied | Missing privileges | Grant required privileges (see design-guide.md RBAC) |
| Feature view not found | Wrong database/schema context | Use fully qualified names |
| Database not authorized | Role can't access source | Use external FV (`refresh_freq=None`) or grant access |
| No "latest version" | No built-in alias | Sort version strings lexicographically (use V01, V02 zero-padded) |
| PIT retrieval wrong | Missing `timestamp_col` | Always set `timestamp_col` on FeatureView |
| Data leakage | Features from future | Validate: feature timestamps <= spine timestamp |
| MDT in FeatureView | Scaling/encoding in FV | Move to Model Registry Pipeline |
| Timestamp column error | timestamp_col in attach_feature_desc | Don't include timestamp_col in feature descriptions |
| Entity already exists | Duplicate registration | Warning only, registration continues |
| `state.upper()` fails | FeatureViewStatus is enum, not string | Use `"ACTIVE" in str(fv.status).upper()` |
| DT refresh fails after inference FV | Owner role mismatch | Inference FV must be owned by same role as source DTs, or GRANT SELECT |
| `resume_feature_view()` no effect | API may not trigger resume | Use `ALTER DYNAMIC TABLE <name> RESUME` directly |
---
## Warehouse Sizing
| Complexity | Warehouse | Use Case |
|------------|-----------|----------|
| Simple aggregations | X-Small to Small | Count, sum, avg |
| Complex joins | Medium to Large | Multi-table joins |
| ML transformations | Large to X-Large | UDF-based features |
Separate warehouses:
- `DEV_WH` (XSMALL) — development
- `TEST_WH` (SMALL) — testing
- `PROD_WH` (MEDIUM) — production FV refresh
- `PROD_OFT_WH` (SMALL) — dedicated OFT refresh
---
## Quick Commands
```sql
SHOW DYNAMIC TABLES IN SCHEMA <feature_store_schema>;
ALTER DYNAMIC TABLE <feature_view_name> REFRESH;
ALTER DYNAMIC TABLE <feature_view_name> SUSPEND;
ALTER DYNAMIC TABLE <feature_view_name> RESUME;
SHOW ONLINE FEATURE TABLES IN SCHEMA <feature_store_schema>;
SHOW DATASETS IN SCHEMA <feature_store_schema>;
DESCRIBE DATASET <dataset_name>;
SHOW MODELS IN SCHEMA <model_schema>;
SHOW FUNCTIONS IN MODEL <db>.<schema>.<model> VERSION <ver>;
```
feature-store/SKILL.md
---
name: feature-store
description: "**[REQUIRED]** Use for **ALL** Snowflake Feature Store operations: creating feature stores, registering entities/features, building feature views, feature pipelines, training dataset generation, online feature serving, preprocessing, inference feature views, feature lineage, auditing, monitoring, migration from other platforms, and CI/CD promotion. DO NOT attempt feature store work manually - invoke this skill first. Triggers: feature store, feature view, entity, feature pipeline, spine, point-in-time, online features, feature engineering, generate_dataset, retrieve_feature_values, FeatureStore, FeatureView, Entity, CreationMode, snowflake.ml.feature_store, feature freshness, feature serving, feature reuse, training/serving skew, audit feature view, validate features, feature lineage, which models use, inference feature view, model inference, preprocessing, aggregation API, promote features, migrate feature store, temporal features."
path: machine-learning/feature-store
---
# Snowflake Feature Store
Expert guidance for the full Snowflake Feature Store lifecycle: creating entities and feature views, building feature pipelines, generating training datasets with point-in-time correctness, enabling online serving, monitoring operations, tracing feature lineage, creating inference pipelines, and migrating from other platforms.
## When to Use
Use this skill when users ask about:
- Creating a feature store, entities, or feature views
- Building feature pipelines (managed or external)
- Generating training datasets with point-in-time correct feature retrieval
- Enabling and using online (low-latency) feature serving
- Feature transformation patterns (windowed aggregations, per-group, lag features)
- Using the Aggregation API (Feature class) for tiled time-windowed features
- Deriving temporal features from date/timestamp columns
- Preprocessing and encoding (MDT transforms)
- Monitoring feature freshness, pipeline health, or costs
- Auditing and validating feature views against best practices
- Tracing feature lineage — which models consume a feature view
- Creating inference feature views from model signatures
- Promoting features across environments (DEV → TEST → PROD)
- Migrating from Feast, Tecton, or other feature store platforms
- Integrating feature store with Snowflake Model Registry
## Mandatory Initialization
Before any workflow, you MUST:
### Step 1: Load API Reference
**Load**: `references/api-reference.md` — Python API quick reference for FeatureStore, Entity, FeatureView, Feature, OnlineConfig.
**DO NOT PROCEED until you have loaded this reference.**
### Step 2: Confirm Environment
**Ask user:**
```
To set up your feature store, I need:
1. Database name (must already exist)
2. Feature store schema name (will be created if needed)
3. Warehouse name
4. Are you working in a notebook, IDE, or CLI?
```
**⚠️ STOP**: Wait for user response before proceeding.
### Step 3: Establish Session
Confirm the user has a Snowpark session. If not, guide them:
```python
from snowflake.snowpark import Session
# Option A: Connection name (recommended for CLI/IDE)
session = Session.builder.config("connection_name", "<connection>").create()
# Option B: Explicit parameters
session = Session.builder.configs({
"account": "<account>",
"user": "<user>",
"password": "<password>",
"warehouse": "<warehouse>",
"database": "<database>",
"schema": "<schema>",
}).create()
session.sql_simplifier_enabled = True
```
---
## Intent Detection
When a user makes a request, detect their intent and route to the appropriate sub-skill:
### CREATE Intent
**Trigger phrases**: "create feature store", "register entity", "create feature view", "set up feature store", "new entity", "new feature view", "add features from a table", "temporal features", "aggregation API"
**→ Load**: `create/SKILL.md`
### PIPELINES Intent
**Trigger phrases**: "feature pipeline", "refresh_freq", "managed feature view", "external feature view", "dynamic table pipeline", "incremental refresh", "dbt features", "schedule features", "inference pipeline"
**→ Load**: `pipelines/SKILL.md`
### TRAINING Intent
**Trigger phrases**: "training dataset", "generate_dataset", "spine", "point-in-time", "training set", "retrieve features", "retrieve_feature_values", "AsOf join", "backfill", "preprocessing", "encoding", "scaling"
**→ Load**: `training/SKILL.md`
### ONLINE Intent
**Trigger phrases**: "online features", "online serving", "low latency", "real-time features", "OnlineConfig", "StoreType.ONLINE", "feature serving", "production serving", "online feature table"
**→ Load**: `online/SKILL.md`
### MONITOR Intent
**Trigger phrases**: "feature freshness", "refresh history", "pipeline health", "list feature views", "suspend feature view", "resume feature view", "feature store cost", "audit", "validate", "check feature store", "promote features", "DEV to PROD"
**→ Load**: `monitor/SKILL.md`
### LINEAGE Intent
**Trigger phrases**: "feature lineage", "which models use", "model consumers", "impact analysis", "inference feature view", "model inference", "serve features", "inference FV", "create inference view", "batch inference features", "model input features"
**→ Load**: `lineage/SKILL.md`
### MIGRATE Intent
**Trigger phrases**: "migrate from Feast", "migrate from Tecton", "migration", "convert feature store", "move to Snowflake feature store"
**→ Load**: `migrate/SKILL.md`
### PATTERNS Intent (no sub-skill — load reference directly)
**Trigger phrases**: "feature patterns", "windowed aggregation", "rolling average", "lag features", "cumulative features", "how to write features"
**→ Load**: `references/feature-patterns.md` and assist directly.
### DESIGN Intent (no sub-skill — load reference directly)
**Trigger phrases**: "naming conventions", "schema organization", "feature store design", "versioning strategy", "access control", "environment promotion", "RBAC", "MIT MDT ODT", "transformation taxonomy"
**→ Load**: `references/design-guide.md` and assist directly.
---
## Workflow Decision Tree
```
Start Session
↓
MANDATORY: Load references/api-reference.md
↓
MANDATORY: Confirm environment (database, schema, warehouse)
↓
MANDATORY: Establish Snowpark session
↓
Detect User Intent
↓
├─→ CREATE → Load create/SKILL.md
├─→ PIPELINES → Load pipelines/SKILL.md
├─→ TRAINING → Load training/SKILL.md
├─→ ONLINE → Load online/SKILL.md
├─→ MONITOR → Load monitor/SKILL.md
├─→ LINEAGE → Load lineage/SKILL.md
├─→ MIGRATE → Load migrate/SKILL.md
├─→ PATTERNS → Load references/feature-patterns.md (assist directly)
└─→ DESIGN → Load references/design-guide.md (assist directly)
```
---
## Sub-Skills
| Sub-Skill | Purpose | When to Load |
|-----------|---------|--------------|
| [create/SKILL.md](create/SKILL.md) | Create feature store, entities, feature views, temporal features, aggregation API | CREATE intent |
| [pipelines/SKILL.md](pipelines/SKILL.md) | Build and manage feature pipelines (managed, external, inference) | PIPELINES intent |
| [training/SKILL.md](training/SKILL.md) | Generate training datasets, preprocessing, Model Registry integration | TRAINING intent |
| [online/SKILL.md](online/SKILL.md) | Enable and use online feature serving, production patterns | ONLINE intent |
| [monitor/SKILL.md](monitor/SKILL.md) | Monitor health, audit/validate, promote, suspend/resume, cost | MONITOR intent |
| [lineage/SKILL.md](lineage/SKILL.md) | Feature lineage, inference feature views from model signatures | LINEAGE intent |
| [migrate/SKILL.md](migrate/SKILL.md) | Migrate from Feast, Tecton, or other platforms | MIGRATE intent |
## References
| Reference | Content | When to Load |
|-----------|---------|--------------|
| [references/api-reference.md](references/api-reference.md) | Python API quick reference | Always (mandatory init) |
| [references/feature-patterns.md](references/feature-patterns.md) | Feature transformation patterns + Aggregation API | PATTERNS intent or when writing feature logic |
| [references/design-guide.md](references/design-guide.md) | Naming, taxonomy (MIT/MDT/ODT), schema org, versioning, RBAC | DESIGN intent or when planning structure |
| [references/troubleshooting.md](references/troubleshooting.md) | Diagnostic queries, common issues, incremental refresh blockers | When diagnosing issues |
---
## Important Constraints
### 1. Package Versions
| Package | Min Version | For |
|---------|-------------|-----|
| `snowflake-ml-python` | `>= 1.5.0` | Core Feature Store |
| `snowflake-ml-python` | `>= 1.18.0` | Online features (OnlineConfig) |
| `snowflake-ml-python` | `>= 1.21.0` | Aggregation API (Feature class) |
| `snowflake-snowpark-python` | `>= 1.25.0` | Snowpark session |
### 2. Feature Store = Schema
A feature store is simply a Snowflake schema. The database must already exist; the schema is created by the API.
### 3. Entities Are Tags
Entities are implemented as Snowflake tags. Subject to the limit of 10,000 tags per account and 50 unique tags per object.
### 4. Managed Feature Views = Dynamic Tables
Snowflake-managed feature views use Dynamic Tables under the hood. All Dynamic Table constraints apply (change tracking, incremental refresh rules).
### 5. Point-in-Time Correctness
Always use `spine_timestamp_col` when generating training datasets with temporal features to prevent data leakage.
### 6. Transformation Taxonomy
Classify every transformation before deciding where it belongs:
| Type | Full Name | Where | Reusable? | Examples |
|------|-----------|-------|-----------|----------|
| **MIT** | Model-Independent | FeatureView | Yes, across models | Aggregations, joins, derived columns |
| **MDT** | Model-Dependent | Model Registry (Pipeline) | No, tied to model | Scaling, encoding, imputation. **Fit on training data only** |
| **ODT** | On-Demand | Inference time | N/A | Time-since-last, distance, current weather |
---
## Stopping Points Summary
All sub-skills follow this philosophy: **NO Snowflake object creation without explicit user approval.**
- **READ-ONLY queries**: Can run freely (listing, monitoring)
- **ANY mutation** (register entity, register feature view, create dataset): Requires stopping point and user approval
---
## Context Preservation Between Skills
When transitioning between sub-skills (e.g., CREATE → PIPELINES → TRAINING):
**Information to preserve:**
- Feature store instance details (database, schema, warehouse)
- Registered entity names and join keys
- Registered feature view names and versions
- Session object reference
**How:** Carry forward the `fs` (FeatureStore) object and entity/feature view references across workflow steps.
feature-store/training/SKILL.md
---
name: feature-store-training
description: "Generate training datasets from Snowflake Feature Store with point-in-time correct feature retrieval, preprocessing, and Model Registry integration."
parent_skill: feature-store
path: machine-learning/feature-store/training
---
# Training Datasets, Preprocessing & Model Registry
## When to Load
Parent skill routes here for TRAINING intent: "training dataset", "generate_dataset", "spine", "point-in-time", "retrieve features", "AsOf join", "backfill", "preprocessing", "encoding", "scaling".
## Prerequisites
- `../references/api-reference.md` loaded
- Feature store (`fs`) initialized with registered entities and feature views
---
## Core Concepts
### Spine
A DataFrame containing entity keys and (optionally) timestamps that define **which entities at which points in time** you need features for. The Feature Store joins features to the spine using AsOf joins for point-in-time correctness.
### Point-in-Time Correctness
When `spine_timestamp_col` is specified, the Feature Store retrieves feature values **as they existed at the spine timestamp** — preventing future data from leaking into training data.
### Transformation Taxonomy (MIT / MDT / ODT)
Before generating training data, classify transforms:
- **MIT** (Model-Independent): In FeatureView — reusable aggregations, joins, derived columns
- **MDT** (Model-Dependent): In Model Registry Pipeline — scaling, encoding, imputation (fit on training only)
- **ODT** (On-Demand): At inference time — time-since-last, distance, current context
---
## Workflow
### Step 1: Identify Feature Views
**List available feature views:**
```python
fs.list_feature_views().select("NAME", "VERSION", "DESC").show()
```
**Ask user:**
```
Which feature view(s) do you want to use for training?
(You can combine features from multiple feature views)
```
**⚠️ STOP**: Wait for user response.
**Retrieve the feature views:**
```python
fv1 = fs.get_feature_view("<FV_NAME_1>", "<VERSION>")
fv2 = fs.get_feature_view("<FV_NAME_2>", "<VERSION>")
# Optional: slice to use only specific features
fv1_slice = fv1.slice(["FEATURE_A", "FEATURE_B"])
```
---
### Step 2: Build the Spine
**Spine structure:**
| Use Case | Spine Columns |
|----------|---------------|
| Training | entity keys + timestamp + label |
| Batch inference | entity keys + timestamp |
| Online inference | entity keys only |
**Ask user:**
```
How would you like to define your training spine?
1. From an existing table (e.g., labeled events table)
2. From a SQL query
3. Build from feature view data (group by entity keys)
4. Manual construction
```
**⚠️ STOP**: Wait for user response.
**Option 1: From existing table**
```python
spine_df = session.table("<SPINE_TABLE>").select(
"<ENTITY_KEY>",
"<TIMESTAMP_COL>",
"<LABEL_COL>",
)
```
**Option 2: From SQL**
```python
spine_df = session.sql("""
SELECT customer_id, event_timestamp, label
FROM training_events
WHERE event_timestamp BETWEEN '2024-01-01' AND '2024-12-31'
""")
```
**Option 3: From feature view data**
```python
import snowflake.snowpark.functions as F
spine_df = fv1.feature_df.group_by("<ENTITY_KEY>").agg(
F.max("<TIMESTAMP_COL>").alias("ASOF_DATE")
)
```
**Option 4: Manual construction**
```python
spine_df = session.create_dataframe(
[("1", "3937", "2024-07-01 00:00"), ("2", "2", "2024-07-01 00:00")],
schema=["INSTANCE_ID", "CUSTOMER_ID", "EVENT_TIMESTAMP"]
)
```
---
### Step 3: Generate Training Dataset
**⚠️ MANDATORY CHECKPOINT**: Present configuration before generating.
```
I will generate a training dataset with:
- Name: <DATASET_NAME>
- Spine: <description of spine>
- Feature views: <list>
- Timestamp column: <col> (for point-in-time correctness)
- Label columns: <cols> (if any)
Approve? (Yes/No/Modify)
```
**Option A: Versioned Dataset (immutable, for reproducible training)**
```python
dataset = fs.generate_dataset(
name="<DATASET_NAME>",
version="V01_20250115",
spine_df=spine_df,
features=[fv1, fv2],
spine_timestamp_col="<TIMESTAMP_COL>",
spine_label_cols=["<LABEL_COL>"],
desc="Training dataset for <purpose>",
)
training_df = dataset.read.to_pandas()
```
**Option B: Training Set (returns DataFrame directly)**
```python
training_set = fs.generate_training_set(
spine_df=spine_df,
features=[
fv1.slice(["FEATURE_A", "FEATURE_B"]),
fv2,
],
timestamp_col="<TIMESTAMP_COL>",
spine_label_cols=["<LABEL>"],
)
training_df = training_set.to_pandas()
```
**Column prefixing** when joining multiple FVs:
- `auto_prefix=True` → columns become `USER_ORDERS_FV__TOTAL_SPEND_7D`
- `.with_name("ord")` → columns become `ord$TOTAL_SPEND_7D` (takes precedence)
**Important parameters:**
- `spine_timestamp_col`: Set whenever features are temporal. Omitting retrieves latest values only.
- `spine_label_cols`: Columns in spine that are labels/targets (excluded from features).
- `exclude_columns`: Columns to exclude from output.
- `include_feature_view_timestamp_col`: Set `True` to include the FV's timestamp in output.
- `output_type`: `"dataset"` (default, immutable) or `"table"` (returns DataFrame).
**⚠️ CRITICAL**: Always define `spine_timestamp_col` for temporal features — without it, PIT retrieval won't work and you risk data leakage.
---
### Step 4: Feature Retrieval for Inference
For batch inference (not training), use `retrieve_feature_values`:
```python
inference_spine = session.create_dataframe(
[("1",), ("2",), ("3",)],
schema=["CUSTOMER_ID"]
)
enriched_df = fs.retrieve_feature_values(
spine_df=inference_spine,
features=[fv1, fv2],
spine_timestamp_col="EVENT_TIMESTAMP",
)
enriched_df.show()
```
**Difference from generate_dataset:**
- `generate_dataset` → Creates a persistent, immutable Dataset object (for reproducible training)
- `retrieve_feature_values` → Returns a transient DataFrame (for inference/exploration)
---
### Step 5: Preprocessing & Model Training
**Important:** Preprocessing (scaling, encoding, imputation) is **Model-Dependent (MDT)** — it belongs with the model, NOT in the FeatureView. After generating your training dataset:
- For preprocessing and model training → **Load** `../../ml-development/SKILL.md`
- For logging the trained model to the registry → **Load** `../../model-registry/SKILL.md`
Include feature store provenance in the model comment (e.g., "Trained using CUSTOMER_ORDER_FV V01, TRANSACTION_FV V03").
---
### Step 6: Use the Dataset
**Convert to Snowpark DataFrame:**
```python
training_df = dataset.read.to_snowpark_dataframe()
```
**Convert to Pandas:**
```python
training_pdf = dataset.read.to_pandas()
```
**Check dataset versions:**
```python
dataset.list_versions()
```
**Retrieve feature views used in dataset:**
```python
fvs = fs.load_feature_views_from_dataset(dataset)
```
**Temporal validation** (always validate PIT correctness before training):
```python
assert (training_df["FEATURE_TS"] <= training_df["EVENT_TIMESTAMP"]).all()
```
---
## Common Pitfalls
| Pitfall | Symptom | Fix |
|---------|---------|-----|
| Missing `spine_timestamp_col` | Future data leaks into training | Always set for temporal features |
| Spine keys don't match entity join keys | Empty or misaligned results | Ensure spine columns match entity `join_keys` |
| Wrong timestamp column | Features retrieved at wrong point in time | Verify timestamp column exists in both spine and feature view |
| Using `generate_dataset` for inference | Unnecessary persistent storage | Use `retrieve_feature_values` for inference |
| MDT in FeatureView | Training/serving skew | Move scaling/encoding to Model Registry Pipeline |
---
## Stopping Points
- ✋ Step 1: Feature view selection
- ✋ Step 2: Spine definition approach
- ✋ Step 3: Before generating dataset (mandatory approval)
## Output
- Training Dataset object (immutable, versioned) or DataFrame
- Data ready for preprocessing and model training (via ml-development skill)
## Next Skill
- If user wants to train a model → **Load** `../../SKILL.md` (machine-learning parent skill)
- If user wants online serving → **Load** `online/SKILL.md`
- If user wants lineage/inference FV → **Load** `lineage/SKILL.md`
guides/cli-environment.md
# CLI Environment Guide
**IMPORTANT:** Do NOT write `.ipynb` file or use `notebook_actions` tools unless the user has explicitly asked to work in a notebook. Default to writing and running Python scripts via the CLI.
This guide applies when you are operating on the **CLI** (origin_application=snova or snowflake_coco_desktop). Follow these instructions for session setup, environment management, package installation, and code execution across all ML sub-skills.
---
## Execution
All code execution on the CLI runs **LOCALLY** on the user's machine using `bash` and `write` tools. Code is written as Python scripts.
**DO NOT present "Snowflake Notebook" or "Snowflake compute" as options.** For Snowflake compute, route to `../ml-jobs/SKILL.md`.
---
## Python Environment Setup
**Before running any Python script, you MUST set up the correct environment.**
### Step 1: Find or Create Environment
**ALWAYS** use the built-in cortex tools first:
```bash
cortex env detect
```
**If no environment is found**, create a virtual environment. Always ask the user for permission first.
Check if UV is available:
```bash
which uv
```
Then create the environment:
- **If UV is available** (preferred):
```bash
uv venv .venv --python 3.10
```
- **If UV is NOT available:**
```bash
python3 -m venv .venv
```
> **Note:** Python 3.10 is the recommended default for best compatibility with `snowflake-ml-python`. Python 3.11 is also supported.
After creating the venv, use it for all subsequent commands:
- UV-created venv: `uv run python <script>`
- Standard venv: `.venv/bin/python <script>`
### Step 2: Check and Install Required ML Packages
The following packages are needed for ML workflows:
| Package | Purpose |
|---------|---------|
| `snowflake-ml-python` | Model Registry, ML Jobs, Snowflake ML |
| `numba` | Required alongside snowflake-ml-python (prevents runtime errors) |
| `tomli` | TOML parsing for session creation (required for Python 3.10) |
| `snowflake-snowpark-python` | Snowflake data access |
| `scikit-learn` | ML algorithms |
| `pandas`, `numpy` | Data manipulation |
| `plotly`, `seaborn`, `matplotlib` | Data Visualization |
**Before running any script, check whether the core packages are already installed:**
```bash
<python_cmd> -c "from importlib.metadata import version; print(version('snowflake-ml-python'))"
```
If the package is **not found**, install it (along with the other required packages) using the correct command for the project type:
| Condition | Install Command |
|-----------|----------------|
| `pyproject.toml` exists AND uses uv (has `uv.lock` or `[tool.uv]`) | `uv add snowflake-ml-python numba tomli` |
| `pyproject.toml` exists AND uses poetry (has `poetry.lock`) | `poetry add snowflake-ml-python numba tomli` |
| `.venv` exists but NO `pyproject.toml` (bare venv) | `uv pip install snowflake-ml-python numba tomli` (if uv available) or `.venv/bin/pip install snowflake-ml-python numba tomli` |
> **Warning:** Do NOT use `uv add` without a `pyproject.toml` — it will fail. Use `uv pip install` instead for bare venvs.
---
## Session Setup
**Use the `snowpark_session.py` helper script** shipped with this skill. It handles all auth methods (password, externalbrowser, private key, token), reads `connections.toml` / `config.toml`, respects `$SNOWFLAKE_HOME`, and filters out unsupported config keys.
> **Note:** This script requires `tomli` on Python 3.10. Always install `tomli` alongside other ML packages.
### How to use in generated scripts
**Step 1: Copy** the helper script to the user's working directory, explain to the user on the reason for this cp before you do this:
```bash
cp <SKILL_DIR>/scripts/snowpark_session.py <WORKING_DIR>/snowpark_session.py
```
**Step 2: Import** in your generated Python code:
```python
from snowpark_session import create_snowpark_session
session = create_snowpark_session()
```
The connection name is resolved automatically in this order:
1. `$SNOWFLAKE_CONNECTION_NAME` environment variable
2. `$SNOWFLAKE_DEFAULT_CONNECTION_NAME` environment variable
3. `default_connection_name` from `connections.toml` or `config.toml`
4. Cortex Code agent settings (`~/.snowflake/cortex/settings.json`)
5. First available connection
You can also pass an explicit connection name: `create_snowpark_session("my_conn")`
### Test connectivity (optional)
```bash
python <WORKING_DIR>/snowpark_session.py --test
```
---
## Running Code
### Scripts
```bash
# With uv project
SNOWFLAKE_CONNECTION_NAME=<connection> uv run python /abs/path/script.py
# With poetry project
SNOWFLAKE_CONNECTION_NAME=<connection> poetry run python /abs/path/script.py
# With system python (after verifying packages)
SNOWFLAKE_CONNECTION_NAME=<connection> python3 /abs/path/script.py
```
**Always use absolute paths. Never `cd` then run.**
---
## Mandatory Checkpoints
**Before executing any script:**
1. Present summary of what will be executed
2. Wait for user confirmation (Yes/No)
3. **NEVER** execute without explicit approval
## Output Reporting
**After writing code, always tell the user:**
```markdown
Code written to: /absolute/path/to/file.py
```
After execution completes, report:
1. File location where code was saved
2. Execution results/metrics
3. Any artifacts created (models, outputs, etc.)
## Error Recovery
If execution fails:
1. Read the COMPLETE error output
2. Identify root cause
3. Fix the specific issue
4. **Ask user again** before re-executing
---
## Common Pitfalls
### sklearn Version Compatibility (1.6+)
The `squared` parameter was removed from `mean_squared_error` in sklearn 1.6. Use `root_mean_squared_error` instead:
```python
# WRONG - raises TypeError in sklearn >= 1.6
from sklearn.metrics import mean_squared_error
rmse = mean_squared_error(y_test, y_pred, squared=False)
# CORRECT
from sklearn.metrics import root_mean_squared_error
rmse = root_mean_squared_error(y_test, y_pred)
```
guides/snowsight-environment.md
# Snowsight Environment Guide
This guide applies when you are operating inside **Snowsight**. Follow these instructions for session setup, code execution, and package management across all ML sub-skills.
---
## Prerequisites
**Before starting any data science task, verify the following:**
1. **Check that the user is working in a Snowflake Notebook environment.**
- The user must have an active notebook (`.ipynb`) open and connected to a Snowflake workspace.
2. **If no notebook is detected:**
- **Do NOT proceed** with the data science task.
- Politely ask the user to:
1. Navigate to their Snowflake workspace.
2. Create or open a Notebook.
3. Ensure the notebook is connected to a running compute resource.
- Once the user confirms the notebook is open and connected, proceed with the task.
3. **If a notebook is detected:**
- Proceed directly with the task — do not ask for confirmation.
---
## Session Setup
Always obtain a Snowflake session using:
```python
from snowflake.snowpark.context import get_active_session
session = get_active_session()
```
**Tool usage guidance:**
- **SQL queries** (verification, status checks, SHOW commands): Use your `execute_sql` tool directly
- **Python code** (model operations, complex logic): Use notebook actions
---
## Code Execution
### Execution Strategy Decision Tree
**Before executing any cells, determine the notebook state:**
1. **Check kernel connection**: Use `get_notebook_state` to check `connectionState` and `sessionState`
2. **Check for existing variables**: Use `get_variables` to see if variables exist from prior execution
3. **Check cell execution history**: Look at `executionCounter` and `hasResultsFromPreviousRun` in cell results
**Decision Matrix:**
| Notebook State | Variables Exist? | Action |
|----------------|------------------|--------|
| Fresh notebook (no prior runs) | No | Run all cells (`run_type: 'all'`) |
| Connected kernel, variables exist | Yes | Run only NEW/MODIFIED cells (`run_type: 'single'`) |
| Disconnected/new kernel session | Stale (`hasResultsFromPreviousRun: true`) | Run all cells to rebuild state |
| Editing existing cells | Yes (want to preserve) | Run only the edited cell (`run_type: 'single'`) |
### Why This Matters
- **Avoid re-training models**: Training cells can take minutes/hours. Never re-run them unnecessarily.
- **Preserve expensive computations**: Data loading, transformations, and model objects persist in memory.
- **Incremental development**: Add new cells and run only those to build on existing state.
### Workflow Pattern
```python
# Step 1: Check notebook state FIRST
get_notebook_state(notebook_path) # Check connectionState, sessionState
get_variables(notebook_path) # Check if df, model, etc. already exist
# Step 2: Decide execution strategy
if variables_exist and kernel_connected:
# Only run newly added/modified cells
run_notebook(run_type='single', cell_id='new_cell_id')
else:
# Fresh start - run all cells
run_notebook(run_type='all')
```
### Rules
- **Always import all necessary packages at the start of your code**
- Check the execution environment (CPU/GPU) before running compute-intensive code
- Write direct executable code (no function wrappers)
- Reference variables from previous executions (they persist)
- Use `print()` statements to observe outputs
- Build incrementally on previous results
- Check the workspace for all existing .py and .ipynb files to better understand the workspace
- Show a bias for working with existing notebooks rather than creating new notebooks
### Cell Management
#### Append-Only by Default
**NEVER delete or replace existing notebook cells.** Always append new cells to the end of the notebook. Existing cells represent the user's work history and may contain valuable context, notes, or prior results.
**The only exception:** You may remove a cell if it produced an execution exception **and** you are replacing it with a corrected version. In that case, delete the errored cell and append the fix as a new cell.
**Why this matters:**
- Preserves the user's iterative exploration and thought process
- Avoids accidentally destroying working code the user may reference later
- Keeps the notebook as a readable log of the full analysis workflow
#### Naming Python Cells
When adding new Python cells, **always provide a descriptive cell name** (if the notebook action tool supports a `name` or `displayName` parameter). The name should briefly describe the cell's purpose, e.g.:
- `"Load and preview dataset"`
- `"Train XGBoost model"`
- `"Feature importance plot"`
- `"Evaluate model on test set"`
Good cell names make it easy for the user to navigate large notebooks and understand the workflow at a glance.
### Detecting Modified Cells
Use `get_notebook_state` to check each cell's `isModified` field:
- `isModified: true` → Cell has changes that need re-running
- `isModified: false` → Cell output is current
Only re-run cells where `isModified: true` when preserving existing state.
### Do NOT
- **Delete or replace existing notebook cells** (only remove a cell if it has an execution exception)
- Use `if __name__ == "__main__":` blocks
- Use argparse or command-line arguments
- Try to save files or modify the filesystem
- Attempt to install packages (use pre-installed ones)
---
## Pre-installed Libraries
You have access to these packages **without installation**:
**Machine Learning:** scikit-learn, xgboost, lightgbm, shap
**Time Series & Statistics:** statsmodels, prophet, scipy
**Data Processing:** pandas, numpy, snowflake-snowpark-python
**Deep Learning:** torch, transformers
**Data Visualization:** plotly, seaborn, matplotlib
---
## Working with Snowflake Data
### Prefer Snowpark Pushdown Operations
**Avoid loading entire tables into pandas unless necessary.** Use Snowpark pushdown operations to filter, aggregate, and transform data in Snowflake before pulling to Python.
#### Quick Data Inspection (ALWAYS start here)
```python
# Get row count without loading data
row_count = session.table("MY_TABLE").count()
print(f"Total rows: {row_count}")
# Get column names
columns = session.table("MY_TABLE").columns
print(f"Columns: {columns}")
# Preview first 5 rows
sample = session.table("MY_TABLE").limit(5).to_pandas()
print(sample)
```
#### Efficient Data Access
```python
from snowflake.snowpark.functions import col
# PREFERRED: Push down filters and aggregations to Snowflake
table = session.table("MY_TABLE")
df = table.filter(col("STATUS") == "ACTIVE").select(["COL1", "COL2"]).limit(10000).to_pandas()
# Aggregate in Snowflake, not pandas
summary = table.group_by("CATEGORY").agg({"AMOUNT": "sum", "COUNT": "count"}).to_pandas()
# AVOID: Loading entire large tables
# df = session.table("MY_TABLE").to_pandas() # Only for small tables
```
#### Loading Full Datasets
Use the DataConnector API for large data:
```python
df = session.table("MY_DATASET")
pandas_df = DataConnector.from_dataframe(df).to_pandas()
```
Only load full data when:
- The table is small (< 100k rows)
- You need the entire dataset for model training
- Pushdown operations cannot achieve the goal
---
## GPU Check
Before running compute-intensive tasks, check available hardware:
```python
import torch
if torch.cuda.is_available():
device = torch.device("cuda")
print(f"GPU available: {torch.cuda.get_device_name(0)}")
print(f"GPU memory: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB")
else:
device = torch.device("cpu")
print("Running on CPU")
```
---
## Container Runtime
- Container Runtime provides preconfigured, customizable environments for machine learning on Snowpark Container Services, supporting both interactive experimentation and batch ML workloads.
- It includes popular ML and deep learning frameworks, and lets you install additional packages from PyPI or internal repositories.
- Workloads run on CPU or GPU compute pools, with distributed processing that automatically uses all available resources.
- The DataConnector API makes it easy to load Snowflake data into frameworks like TensorFlow, PyTorch, or Pandas.
- You can use distributed training APIs for LightGBM, PyTorch, and XGBoost.
For more details, search the product documentation for "Container Runtime".
inference-logs/SKILL.md
---
name: inference-logs
description: "View and analyze captured inference data from model services with Auto-Capture enabled. Use when: querying INFERENCE_TABLE, viewing inference logs, analyzing request/response data, debugging inference history, checking captured predictions. Triggers: inference logs, inference table, captured inference, autocapture data, view inference history, inference requests, inference responses."
parent_skill: machine-learning
---
# Inference Logs (Auto-Capture Data)
Query and analyze captured inference data from model services that have Auto-Capture enabled.
## Prerequisites
- A model service with `autocapture=True` enabled during creation
- OWNERSHIP privilege on the model (to read inference table data)
- USAGE privilege on the service and gateway (if filtering by those)
**Note:** Auto-Capture is not supported for vLLM or HuggingFace inference engines.
---
## Step 1: Identify the Model
**Ask user:**
```
Which model would you like to view inference logs for?
Please provide:
- Model name
- Database and schema where the model is registered
```
**⚠️ STOP**: Wait for user response.
---
## Step 2: Check Auto-Capture Status
Verify the model has services with autocapture enabled:
```python
from snowflake.ml.registry import Registry
session = <SESSION_SETUP> # Per environment guide
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
services_df = mv.list_services()
print(services_df[['name', 'status', 'autocapture_enabled']])
```
If no services have `autocapture_enabled=True`:
```
None of the services for this model have Auto-Capture enabled.
Auto-Capture must be enabled when creating the service (it cannot be added later).
To enable it, you would need to recreate the service with autocapture=True.
Would you like help creating a new service with Auto-Capture enabled?
```
If yes, load `../spcs-inference/SKILL.md`.
---
## Step 3: Collect Filter Options
**Ask user:**
```
You can filter the inference logs. All filters are optional — press Enter to skip any:
- Model version (e.g., V1):
- Service name (e.g., MY_SERVICE):
- Gateway name (if using a gateway):
- Time range: How far back? (e.g., "1 hour", "24 hours", "7 days")
```
**⚠️ STOP**: Wait for user response.
---
## Step 4: Query Inference Data
### Basic Query (All Logs)
```sql
SELECT *
FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>'));
```
### Filtered Query
Include only the filters the user provided:
```sql
SELECT *
FROM TABLE(
INFERENCE_TABLE(
'<MODEL_NAME>',
VERSION => '<VERSION>',
SERVICE => '<SERVICE_NAME>',
GATEWAY => '<GATEWAY_NAME>'
)
)
WHERE TIMESTAMP > DATEADD('<unit>', -<N>, CURRENT_TIMESTAMP());
```
### Filter by Function Name
```sql
SELECT *
FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>'))
WHERE RECORD_ATTRIBUTES:"snow.model_serving.function.name" = '<function_name>';
```
### Sample Recent Logs
```sql
SELECT *
FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>'))
WHERE TIMESTAMP > DATEADD('hour', -1, CURRENT_TIMESTAMP())
ORDER BY TIMESTAMP DESC
LIMIT 100;
```
---
## Step 5: Present Results and Offer Analysis
After running the query, present the results and ask:
```
I found <N> inference records. What would you like to do?
1. View the raw data
2. Summarize request/response patterns
3. Analyze input feature distributions
4. Done
```
**⚠️ STOP**: Wait for user response.
### Request Volume Over Time
```sql
SELECT
DATE_TRUNC('hour', TIMESTAMP) as hour,
COUNT(*) as request_count
FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>'))
WHERE TIMESTAMP > DATEADD('day', -1, CURRENT_TIMESTAMP())
GROUP BY 1
ORDER BY 1;
```
---
## Inference Table Data Schema
| Field | Description |
|-------|-------------|
| `RECORD_ATTRIBUTES:"snow.model_serving.request.data.<column>"` | Input features sent to the model |
| `RECORD_ATTRIBUTES:"snow.model_serving.response.data.<column>"` | Inference output returned by the model |
| `RECORD_ATTRIBUTES:"snow.model_serving.request.timestamp"` | When the request hit the inference service |
| `RECORD_ATTRIBUTES:"snow.model_serving.response.code"` | HTTP status code |
| `RECORD_ATTRIBUTES:"snow.model_serving.truncation_policy"` | `NONE` or `TRUNCATED_DEFAULT` if data exceeded 1MB limit |
| `RECORD_ATTRIBUTES:"snow.model_serving.last_hop_id"` | Last gateway ID the request passed through |
| `RECORD_ATTRIBUTES:"snow.model_serving.hop_ids"` | List of gateway IDs showing request path |
| `TIMESTAMP` | Event timestamp (use for time-range filtering) |
---
## Important Notes
- **Only successful requests are captured.** Failed requests are not logged.
- **Filter arguments must reference existing entities.** If you recreated a service with the same name, queries only return data from the current service.
- **Data is retained after deletion.** Inference data persists even after deleting a service or version, as long as the model still exists.
- **Deleting the model permanently deletes all inference data.**
- **1MB limit per event.** Data exceeding this is progressively truncated (strings shortened, then payload dropped).
- **Performance tip:** Always filter by `TIMESTAMP` for large inference tables.
---
## Querying Historical Data for Deleted Entities
Even after deleting a service, version, or gateway, you can still query the historical data:
```sql
-- All logs for model (includes deleted services)
SELECT * FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>'));
-- Filter by deleted version
SELECT * FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>', MODEL_VERSION => '<VERSION>'));
-- Filter by deleted service
SELECT * FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>', MODEL_VERSION => '<VERSION>', SERVICE => '<SERVICE>'));
```
---
## Common Use Cases
### Debugging Unexpected Predictions
```sql
SELECT
RECORD_ATTRIBUTES:"snow.model_serving.request.data" as input,
RECORD_ATTRIBUTES:"snow.model_serving.response.data" as output,
TIMESTAMP
FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>'))
WHERE TIMESTAMP > DATEADD('hour', -1, CURRENT_TIMESTAMP())
ORDER BY TIMESTAMP DESC
LIMIT 10;
```
### Building a Retraining Dataset
```sql
CREATE TABLE training_data_from_prod AS
SELECT
RECORD_ATTRIBUTES:"snow.model_serving.request.data.feature1"::FLOAT as feature1,
RECORD_ATTRIBUTES:"snow.model_serving.request.data.feature2"::FLOAT as feature2,
RECORD_ATTRIBUTES:"snow.model_serving.response.data.prediction"::FLOAT as prediction,
TIMESTAMP
FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>'))
WHERE TIMESTAMP > DATEADD('day', -30, CURRENT_TIMESTAMP());
```
### Comparing Model Versions (A/B Testing)
```sql
SELECT
RECORD_ATTRIBUTES:"snow.model_serving.version" as version,
AVG(RECORD_ATTRIBUTES:"snow.model_serving.response.data.score"::FLOAT) as avg_score,
COUNT(*) as request_count
FROM TABLE(INFERENCE_TABLE('<MODEL_NAME>'))
WHERE TIMESTAMP > DATEADD('day', -7, CURRENT_TIMESTAMP())
GROUP BY 1;
```
ml-development/SKILL.md
---
name: ml-development
description: "**[REQUIRED]** for ALL data science, machine learning, data analysis, and statistical tasks. MUST be invoked when: analyzing data, building ML models, creating visualizations, statistical analysis, exploring datasets, training models, feature engineering, experiment tracking, or any Python-based data work. DO NOT attempt data science tasks without this skill."
---
# Data Science Expert Skill
You are now operating as a **Data Science Expert**. You specialize in solving problems using Python.
**IMPORTANT:** DO NOT SKIP ANY STEPS ON THIS WORKFLOW. EACH STEP MUST BE REASONED AND COMPLETED.
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
---
## Core Workflow
### 1. UNDERSTAND the Request
- Read the user's request carefully
- Identify what data is available (tables, files, variables)
- Determine the goal: exploration, analysis, modeling, or answering a question
### 2. PLAN Your Approach
Before writing code, think through your approach step by step:
- What data do I need to load?
- What preprocessing might be required?
- What analysis or model is appropriate?
- How will I evaluate success?
### 3. EXECUTE Incrementally
**[MANDATORY] Do ONE small step at a time.**
Break tasks down into small targeted steps and only work on one at a time. Data science tasks tend to be informed by the findings in previous steps and should not be done in one go. After each step:
- Observe the output
- Decide if you need to iterate or continue
- Don't try to do everything in one code block
### 4. ITERATE When Needed
- If results are unexpected, investigate why
- If errors occur, analyze and fix them
- Track what you've tried to avoid redundant attempts
- Remember findings from previous steps
### 5. COMPLETE with Quality
When providing a final solution:
- Include a summary of what was accomplished
- Report all evaluation metrics
- Provide end-to-end executable code
---
## Data Access Patterns
**CRITICAL: Prefer Snowpark Pushdown Operations**
Always start with quick data inspection WITHOUT loading full tables:
```python
# Get row count
row_count = session.table("MY_TABLE").count()
# Preview first 5 rows
sample = session.table("MY_TABLE").limit(5).to_pandas()
```
### Efficient Data Access
```python
from snowflake.snowpark.functions import col
# PREFERRED: Filter and aggregate in Snowflake
df = session.table("MY_TABLE").filter(col("STATUS") == "ACTIVE").select(["COL1", "COL2"]).limit(10000).to_pandas()
# AVOID: Loading entire large tables
# df = session.table("MY_TABLE").to_pandas() # Only for small tables (<100k rows)
```
**Always use Snowpark Session, NOT snowflake.connector.**
---
## CLI Workflow Steps
Use when operating on the CLI. Code is written as a local script. See your environment guide for execution details.
### Step 1: Ask About Experiment Tracking (for model training)
Check if the user has specified if they want to use experiment tracking.
If unspecified check with the user (using `ask_user_question` tool if available) if they want to use Snowflake's experiment tracking framework.
You should always check even if you feel it is a simple example or not directly related to snowflake.
**MANDATORY ASK:**
```markdown
Would you like to track this experiment using Snowflake's experiment tracking framework?
1. Yes - Track this model training experiment
2. No - Just train and evaluate
```
If the user mentions that they want to use experiment tracking you will need to do a few different things.
**IF THE USER SAYS YES**
You will need to ask a for the following information. Once again please use the `ask_user_question` tool if it is available.
Ask user for:
1) Database and schema for storing runs
2) Experiment name
3) Model framework if autologging or What parameters/metrics to track if manual
You can check what experiments are available by using either of the following commands
```SQL
SHOW EXPERIMENTS IN SCHEMA DATABASE.SCHEMA;
```
Below is provided an example question to prompt the user in order to ask them which of their experiments they want to use based on ones they have access to.
**Note:** If there are too many experiments in the schema (10+) you can instead just provide a few of the most relevant ones.
```markdown
What experiment name should be used for this experiment?
1. EXAMPLE_EXP_1
2. EXAMPLE_EXP_2
3. EXAMPLE_EXP_3
...
N. Other - You will be prompted to provide a name
```
Once you have collected this information load in the information from the skill `../experiment-tracking/SKILL.md`.
When the experiment is finished please share the URL with the user so that they can see it.
**Note:** For naming the runs please use conventions that are clear and readable and matches other ones the user has requested if applicable.
### Step 2: Ask About Model Serialization (for model training)
**⚠️ IMPORTANT:** This is about SAVING locally, NOT deployment.
**Do NOT ask:**
- "How would you like to deploy the model?"
- "Local only vs Register in Snowflake?"
**Do ask:**
```markdown
Would you like to save the trained model to a file (using `ask_user_question` tool if available)?
1. Yes - Save as pickle file (.pkl) for later use
2. No - Just train and evaluate
If yes, where should I save it? (default: ./model.pkl)
```
### Step 3: Analyze Data First
**⚠️ MANDATORY:** Understand data before writing code:
```sql
DESCRIBE TABLE <table_name>;
SELECT COUNT(*) FROM <table_name>;
SELECT * FROM <table_name> LIMIT 10;
```
### Step 4: Plan and Present
Plan the COMPLETE approach:
- Data loading strategy
- Data Visualization (Snowsight notebooks only; on CLI, save plots to files instead)
- Preprocessing steps
- Model selection
- Evaluation metrics
**Present your plan to the user before writing code.**
### Step 5: Write Complete Code
Set up the session following your loaded environment guide, then write the code:
```python
# Session setup per environment guide
# ...
# Load data using Snowpark
df = session.table("MY_TABLE").to_pandas()
# OR with filtering
df = session.table("MY_TABLE").select(["COL1", "COL2"]).filter(...).to_pandas()
```
#### Data Visualization Notes
- Ensure Visualizations are coherent, well labeled, and aesthetically pleasing
- **Snowsight only:** Render visualizations inline in notebook cells
- **CLI only:** Save visualizations to files (e.g., `plt.savefig("plot.png")`) — do NOT use notebooks on CLI
- Well done Visualizations help the user follow along the code and better understand the data and should be used frequently
### Step 6: Ask Before Executing
**⚠️ MANDATORY:** Before executing, ask user:
```markdown
I've written the complete script with:
- [Summary of what it does]
- [Data: X rows, Y columns]
- [Model: algorithm choice]
- [Expected output: metrics to report]
- [Model serialization: Yes/No, path if yes]
Ready to execute? (Yes/No)
```
### Step 7: Execute
Follow the execution instructions in your loaded environment guide.
### Step 8: Report Model Artifacts and Offer Next Steps
**⚠️ IMPORTANT:** After successful execution, if a model was saved:
1. **Report details:**
```markdown
Model saved successfully:
- File path: /absolute/path/to/model.pkl
- Framework: sklearn/xgboost/lightgbm/pytorch/tensorflow
- Sample input schema: [columns and types]
```
2. **Offer next step:**
```markdown
The model has been saved locally. Would you like to register it to Snowflake Model Registry?
```
3. **If user says yes:**
- Load `model-registry/SKILL.md`
- Pass along context: model file path, framework, sample input schema
- Tell model-registry: "User just trained this model, use this context"
---
### For Model Tasks
- [ ] Train/test split is proper (no data leakage)
- [ ] Appropriate metrics are used
- [ ] Model is evaluated on holdout data
- [ ] Feature importance is analyzed
- [ ] Performance is clearly reported
---
## Memory and Context
### Track Your Progress
- Remember what code you've executed
- Keep track of variables in memory
- Note what approaches you've tried
- Don't repeat failed attempts
### Reference Previous Work
When the user asks about previous experiments:
- Reference specific findings with metrics
- Mention which approach worked best
- Provide context from earlier analysis
ml-jobs/SKILL.md
---
name: ml-jobs
description: "Transform local Python scripts into Snowflake ML Jobs. Use when: running ML workloads on Snowflake compute pools, GPU training, submitting Python scripts to run remotely, converting local scripts to ML jobs, distributed training. Triggers: ml job, submit job, run on compute pool, remote execution, GPU training."
parent_skill: data-science-machine-learning
---
# Transform Python Script to Snowflake ML Job
Guide users step-by-step to convert a local Python script into a Snowflake ML Job that runs on Snowflake compute pools.
## When to Use
- User wants to run a Python script on Snowflake compute (GPU/high-memory)
- User wants to offload resource-intensive ML training to Snowflake
- User has a local script and wants it to run as an ML Job
- User mentions: "ml job", "compute pool", "remote execution", "GPU training"
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
---
## Prerequisites
- `snowflake-ml-python>=1.9.2` (refers to the env guide in the parent skill to setup `snowflake-ml-python`)
- Python 3.10 client environment
- Snowflake account with compute pool access
## Workflow
### Step 1: Understand the Script
**Ask user:**
```
To convert your Python script to a Snowflake ML Job, I need to understand it:
1. **Script location**: What is the path to your Python script?
2. **Script purpose**: What does the script do? (training, inference, data processing)
3. **Multiple files?**: Does your project have multiple Python files or just one?
```
**⚠️ STOP**: Wait for user response.
After user responds:
- **Read the script** to understand its structure
- Identify: imports, dependencies, data sources, outputs, return values
- Note any hardcoded paths or configurations that need modification
- **Infer GPU vs CPU requirements** (see below)
#### GPU/CPU Inference from Script
| Suggest GPU if ANY of these present | Suggest CPU if no GPU patterns |
|-------------------------------------|-------------------------------|
| `.cuda()`, `.to('cuda')`, `.to(device)` | `sklearn` only |
| `torch.cuda.is_available()` | `xgboost`, `lightgbm` (no GPU config) |
| `import cupy`, `cudf`, `cuml` (RAPIDS) | `pandas` processing |
| `device_map="auto"` or `"cuda"` | Pure numpy operations |
| `transformers` with large models | Traditional ML algorithms |
| `.half()`, `.bfloat16()` precision | |
| `DataParallel`, `DistributedDataParallel` | |
#### Training Package Selection
**Default to OSS (Open Source)**: Use standard open-source libraries (XGBoost, LightGBM, PyTorch, scikit-learn, etc.) directly.
**Only if user explicitly asks for distributed training**, use Snowflake's Container Runtime Distributed API with trainers from `snowflake.ml.modeling.distributors`:
- `XGBEstimator` / `XGBScalingConfig` for XGBoost
- `LightGBMEstimator` / `LightGBMScalingConfig` for LightGBM
- `PyTorchDistributor` / `PyTorchScalingConfig` for PyTorch
- Reference: https://docs.snowflake.com/en/developer-guide/snowflake-ml/distributed-training
### Step 2: Identify Compute and Stage Requirements
#### 2a: Query Available Compute Pools
**Run this query to discover available compute pools:**
```sql
SHOW COMPUTE POOLS;
```
This returns columns including:
- `name`: Compute pool name
- `state`: ACTIVE, IDLE, STARTING, etc.
- `instance_family`: CPU_X64_S, GPU_NV_S, GPU_NV_M, HIGHMEM_X64_S, etc.
- `min_nodes`, `max_nodes`: Pool size limits
- `active_nodes`: Currently running nodes
- `idle_nodes`: Warm nodes ready for immediate use
- `auto_suspend_secs`: Auto-suspend configuration
#### 2b: Select Best Compute Pool
**Use this logic to suggest a compute pool:**
1. **Determine required type** from Step 1 inference:
- If GPU needed → filter pools where `instance_family` contains "GPU"
- If CPU needed → filter pools where `instance_family` contains "CPU" or "HIGHMEM"
2. **For generic workloads (no special requirements):**
- CPU workloads → default to `SYSTEM_COMPUTE_POOL_CPU`
- GPU workloads → default to `SYSTEM_COMPUTE_POOL_GPU`
3. **For optimized selection, rank pools by:**
- **Idle nodes** (highest priority): Pools with `idle_nodes > 0` provide warm starts
- **Free capacity**: Calculate `max_nodes - active_nodes` for headroom
- **State**: Prefer `ACTIVE` over `IDLE` (already running), both over `SUSPENDED`
- **Instance family match**: Match workload intensity to instance size
**Ranking formula:**
```
score = (idle_nodes * 10) + (max_nodes - active_nodes) + (3 if state == 'ACTIVE' else 1 if state == 'IDLE' else 0)
```
#### 2c: Ask User for Confirmation
**Use `ask_user_question` to confirm:**
1. **Instance type** (GPU/CPU) - include brief reasoning from script analysis
2. **Compute pool** - recommend best match, include system pools as fallback
3. **Stage** - ask if they have an existing stage or need to create one
**⚠️ STOP**: Wait for user response.
**Instance Family Reference:**
| Workload | Instance Family |
|----------|-----------------|
| General ML | CPU_X64_S |
| GPU Training | GPU_NV_S, GPU_NV_M |
| Large Data | HIGHMEM_X64_S |
**If user needs new resources:**
```sql
CREATE COMPUTE POOL IF NOT EXISTS <POOL_NAME>
MIN_NODES = 1 MAX_NODES = 5 INSTANCE_FAMILY = <FAMILY>;
CREATE STAGE IF NOT EXISTS <DATABASE>.<SCHEMA>.<STAGE_NAME>;
```
### Step 3: Analyze Dependencies
**Automatically extract dependencies from the script you read in Step 1.**
1. **Check for existing requirements.txt** in the project directory
- If found and using `submit_directory`: dependencies will be installed automatically—no need to specify `pip_requirements`
- If found: review contents to understand what external packages are needed (for EAI detection in Step 4)
2. **Parse all import statements** from the script
3. **Map imports to pip packages** (e.g., `import sklearn` → `scikit-learn`, `import cv2` → `opencv-python`)
4. **Classify each package:**
| Category | Examples | Action |
|----------|----------|--------|
| Pre-installed in ML Runtime | pandas, numpy, scikit-learn, xgboost, torch, tensorflow, transformers | No pip_requirements needed |
| Standard library | os, sys, json, pathlib, typing | Ignore |
| Custom packages | Any not in above | Add to pip_requirements (unless requirements.txt exists) |
**Pre-installed packages in ML Runtime (do NOT add to pip_requirements):**
- pandas, numpy, scipy
- scikit-learn, xgboost, lightgbm
- torch, tensorflow, transformers
- snowflake-snowpark-python, snowflake-ml-python
**Present to user:**
**If requirements.txt exists:**
```
I found a requirements.txt in your project directory. When using submit_directory,
these dependencies will be installed automatically—no need to specify pip_requirements.
**requirements.txt contents:**
<list packages from file>
**Note:** Custom packages still require PyPI access via an External Access Integration.
```
**If NO requirements.txt:**
```
I analyzed your script and found these dependencies:
**Pre-installed (no action needed):**
- <list packages>
**Custom packages to install:**
- <list packages>
Any additional packages or specific versions needed?
```
**⚠️ STOP**: Wait for user confirmation or additions.
### Step 4: Detect External Network Access Requirements
ML Jobs run in an isolated environment. External network access requires an External Access Integration (EAI).
**⚠️ IMPORTANT**: Scripts often require MULTIPLE types of external access. Scan for ALL patterns below—don't stop at the first match.
#### 4a: Identify ALL External Access Patterns
Build a complete list by checking each category:
| Category | Indicators | Access Needed |
|----------|-----------|---------------|
| **Package Installation** | `requirements.txt`, `pip_requirements`, custom packages not in ML Runtime | PyPI |
| **Hugging Face** | `from_pretrained()`, `AutoModel`, `AutoTokenizer`, `SentenceTransformer()`, `pipeline()` | huggingface.co |
| **NLTK** | `nltk.download()` | nltk.org, github.com |
| **PyTorch Hub** | `torch.hub.load()`, `pretrained=True` in torchvision | pytorch.org, github.com |
| **TensorFlow Hub** | `hub.load()`, keras pretrained models | tfhub.dev |
| **Experiment Tracking** | `import wandb`, `import mlflow`, `import comet_ml` | wandb.ai, mlflow server, comet.ml |
| **LLM APIs** | `import openai`, `import anthropic` | api.openai.com, api.anthropic.com |
| **Cloud Storage** | `boto3`, `s3fs`, `google.cloud`, `azure` | AWS/GCP/Azure endpoints |
| **Generic HTTP** | `requests`, `httpx`, `urllib` with external URLs | Various (check URLs in code) |
**Example: A training script might need ALL of these:**
1. PyPI access - for custom packages
2. Hugging Face access - for `from_pretrained()` model downloads
3. W&B access - for `wandb.log()` experiment tracking
#### 4b: Query Available EAIs
```sql
SHOW EXTERNAL ACCESS INTEGRATIONS;
```
#### 4c: ALWAYS Confirm EAI Selection with User
**Use `ask_user_question` with `multiSelect: true`** to let user select all applicable EAIs.
Present:
1. ALL detected external access requirements (not just the first one)
2. Available EAIs from the query as options
3. Any suggested matches based on EAI names (but don't be overconfident—naming varies by account)
Example question:
```
**External Access Requirements Detected:**
1. Package installation (PyPI) - requirements.txt with custom packages
2. Model downloads (Hugging Face) - from_pretrained() calls detected
3. Experiment tracking (W&B) - import wandb detected
**Which EAI(s) cover these requirements?** (Select all that apply)
```
Options should include each available EAI from `SHOW EXTERNAL ACCESS INTEGRATIONS`, plus "None needed" if applicable.
**⚠️ STOP**: Wait for user to confirm EAI selection. Do not proceed without explicit confirmation.
#### 4d: Handle Missing EAIs
Creating EAIs requires **ACCOUNTADMIN** privileges. If needed EAIs don't exist:
**For PyPI access** (uses built-in network rule):
```sql
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION PYPI_EAI
ALLOWED_NETWORK_RULES = (snowflake.external_access.pypi_rule)
ENABLED = true;
GRANT USAGE ON INTEGRATION PYPI_EAI TO ROLE <USER_ROLE>;
```
**For other services** (requires custom network rule):
```sql
CREATE OR REPLACE NETWORK RULE <SERVICE>_RULE
TYPE = HOST_PORT
VALUE_LIST = ('<hostname1>', '<hostname2>')
MODE = EGRESS;
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION <SERVICE>_EAI
ALLOWED_NETWORK_RULES = (<SERVICE>_RULE)
ENABLED = true;
GRANT USAGE ON INTEGRATION <SERVICE>_EAI TO ROLE <USER_ROLE>;
```
**Common hostnames:**
| Service | Hostnames |
|---------|-----------|
| Hugging Face | `huggingface.co`, `cdn-lfs.huggingface.co` |
| Weights & Biases | `api.wandb.ai`, `wandb.ai` |
| OpenAI | `api.openai.com` |
| Anthropic | `api.anthropic.com` |
If user lacks privileges, provide the SQL for them to share with their admin.
### Step 5: Determine Submission Method
Based on gathered information, select the appropriate method:
| Scenario | Method | When to Use |
|----------|--------|-------------|
| Single file | `submit_file()` | One Python script, no local imports |
| Multi-file project | `submit_directory()` | Multiple modules, local imports |
| Already on stage | `submit_from_stage()` | Code already uploaded to Snowflake |
| Single function | `@remote` decorator | Quick test, simple function |
### Step 6: Generate the Job Submission Code
**Present to user before generating:**
```
I'll submit the ML Job with these settings:
- **Script**: <script_path>
- **Compute Pool**: <pool_name>
- **Instance Type**: <instance_family>
- **Dependencies**: <pip_requirements>
- **External Access**: <list of EAIs, or "None required">
- **Nodes**: <target_instances>
- **Method**: <submit_file|submit_directory|remote>
Ready to proceed? (Yes/No)
```
**⚠️ MANDATORY CHECKPOINT**: Wait for user approval.
**⚠️ CRITICAL**: The submission code must set database and schema context:
```python
session.use_database("<DATABASE>")
session.use_schema("<SCHEMA>")
```
#### Template: submit_file (Single Script)
Use `snowpark_session.py` from parent skill (`machine-learning/SKILL.md` → Session Setup Patterns). Copy the helper script to the working directory and import it.
```python
from snowflake.ml.jobs import submit_file
from snowpark_session import create_snowpark_session
DATABASE = "<DATABASE>"
SCHEMA = "<SCHEMA>"
session = create_snowpark_session()
session.use_database(DATABASE)
session.use_schema(SCHEMA)
job = submit_file(
"<SCRIPT_PATH>",
"<COMPUTE_POOL>",
stage_name="<PAYLOAD_STAGE>",
args=["--arg1", "value1"], # Command line args if needed
pip_requirements=[<DEPS>], # ["custom-package==1.0"]
external_access_integrations=[<EAIS>],
session=session,
)
print(f"Job submitted successfully!")
print(f"Job ID: {job.id}")
print(f"Status: {job.status}")
```
#### Template: submit_directory (Multi-File Project)
```python
from snowflake.ml.jobs import submit_directory
from snowpark_session import create_snowpark_session
DATABASE = "<DATABASE>"
SCHEMA = "<SCHEMA>"
session = create_snowpark_session()
session.use_database(DATABASE)
session.use_schema(SCHEMA)
job = submit_directory(
"<PROJECT_DIR>",
"<COMPUTE_POOL>",
entrypoint="<MAIN_SCRIPT.py>",
stage_name="<PAYLOAD_STAGE>",
pip_requirements=[<DEPS>], # Omit if requirements.txt exists in directory
external_access_integrations=[<EAIS>],
session=session,
)
print(f"Job submitted successfully!")
print(f"Job ID: {job.id}")
print(f"Status: {job.status}")
```
**Note:** If `requirements.txt` exists in the project directory, omit `pip_requirements`—dependencies install automatically.
#### Template: @remote Decorator (Function)
```python
from snowflake.ml.jobs import remote
from snowpark_session import create_snowpark_session
DATABASE = "<DATABASE>"
SCHEMA = "<SCHEMA>"
session = create_snowpark_session()
session.use_database(DATABASE)
session.use_schema(SCHEMA)
@remote(
"<COMPUTE_POOL>",
stage_name="<PAYLOAD_STAGE>",
pip_requirements=[<DEPS>],
external_access_integrations=[<EAIS>],
session=session,
)
def train_model(data_table: str):
# Your ML code here
from snowflake.snowpark import Session
session = Session.builder.getOrCreate()
df = session.table(data_table).to_pandas()
# ... training logic ...
return model
job = train_model("<TABLE_NAME>")
print(f"Job submitted successfully!")
print(f"Job ID: {job.id}")
```
### Step 7: Script Modifications (If Needed)
**Common modifications for ML Jobs:**
1. **Snowpark Session Access** (inside job):
```python
from snowflake.snowpark import Session
session = Session.builder.getOrCreate() # Auto-available in jobs
```
2. **Remove hardcoded paths** - use arguments or Snowflake data sources
#### 7a: Results and Artifacts
ML Jobs support two mechanisms for outputs:
- **Return values**: Retrieved via `job.result()` after job completion (assign to `__return__` variable)
- **Artifact files**: Saved to `MLRS_STAGE_RESULT_PATH` environment variable (e.g., model weights)
**Prefer minimal modifications.** Check if the user's script already supports the needed behavior before making changes.
**Step 1: Analyze existing script for output handling**
Look for:
- Existing CLI argument parsing (`argparse`, `click`, `typer`, etc.)
- Output path arguments (e.g., `--output-dir`, `--model-path`, `--save-path`)
- Existing artifact saving logic
**Step 2: Use `args` in job submission if possible**
If the script already accepts an output path argument, simply pass the stage result path via `args`:
```python
job = submit_file(
"train.py",
"MY_COMPUTE_POOL",
args=["--output-dir", "$MLRS_STAGE_RESULT_PATH", "--epochs", "10"],
# ... other params ...
)
```
Common output arguments to look for:
- `--output`, `--output-dir`, `--output-path`
- `--model-path`, `--model-dir`, `--save-path`
- `--checkpoint-dir`, `--results-dir`
**Step 3: Only modify script if necessary**
Modify the user's script only if:
1. It has no existing output path argument, AND
2. The user needs artifact persistence or return values
**Minimal modification pattern** (add to end of script):
```python
# For ML Jobs: capture return value
if __name__ == "__main__":
result = main() # or existing entry point
import os
if os.environ.get("MLRS_STAGE_RESULT_PATH"):
__return__ = result
```
**For artifact saving** (if script has hardcoded paths):
```python
import os
# Use stage path if in ML Job, otherwise use local path
output_dir = os.environ.get("MLRS_STAGE_RESULT_PATH", "./output")
model_path = os.path.join(output_dir, "model.pkl")
```
#### 7b: Retrieving Results and Artifacts
**After job completes (status = DONE):**
```python
# Get return value (from __return__ assignment)
result = job.result()
print(f"Result: {result}")
# Artifacts saved to MLRS_STAGE_RESULT_PATH are stored in the job's result stage
```
### Step 8: Execute the Job Submission
Execute the submission code using inline Python. After successful submission, report:
1. **Job ID** - from `job.id`
2. **Job Status** - initial status (usually PENDING or RUNNING)
**Example success message:**
```
Job submitted successfully!
- **Job ID**: TRAIN_ABC123
- **Status**: PENDING
You can check job status later using the job ID.
### Step 9: Ask About Waiting for Job Completion
**Ask user:**
```
Would you like me to wait for the job to complete? (Yes/No)
- **Yes**: I'll wait for the job to finish and show you the results and logs.
- **No** [Default]: You can check job status later using the job ID.
```
**⚠️ STOP**: Wait for user response. Default to No if user is unsure.
#### If user says Yes: Wait for Job Completion
```python
from snowflake.ml.jobs import get_job
from snowpark_session import create_snowpark_session
session = create_snowpark_session()
session.use_database("<DATABASE>")
session.use_schema("<SCHEMA>")
job = get_job("<JOB_ID>")
job.wait()
print(f"Job ID: {job.id}")
print(f"Status: {job.status}") # FAILED, DONE
if job.status == "DONE":
# Retrieve return value (from __return__ in script)
result = job.result()
print(f"\\n--- Result ---")
print(result)
print("\\n--- Logs ---")
print(job.get_logs())
```
**Present job status and results to user.**
#### If user says No or checks later
**When user asks for job status or results later:**
```python
from snowflake.ml.jobs import get_job
from snowpark_session import create_snowpark_session
session = create_snowpark_session()
session.use_database("<DATABASE>")
session.use_schema("<SCHEMA>")
job = get_job("<JOB_ID>")
print(f"Job ID: {job.id}")
print(f"Status: {job.status}")
if job.status == "DONE":
print(f"\n--- Result ---")
print(job.result())
print("\n--- Logs ---")
print(job.get_logs())
```
### Step 10: Troubleshooting
**If job fails:**
1. **Get full logs**:
```python
print(job.get_logs())
```
2. **Common issues:**
| Error | Cause | Fix |
|-------|-------|-----|
| ModuleNotFoundError | Missing dependency | Add to `pip_requirements` |
| Connection timeout | No EAI for pip | Create PYPI_EAI integration |
| PermissionError | Missing privileges | Grant USAGE on compute pool |
| FileNotFoundError | Wrong path | Use absolute paths in args |
3. **For multi-node jobs**, check specific instance logs:
```python
print(job.get_logs(instance_id=0))
print(job.get_logs(instance_id=1))
```
## Multi-Node Jobs (Distributed Training)
If user needs distributed training across multiple nodes:
```python
job = submit_file(
"<SCRIPT_PATH>",
"<COMPUTE_POOL>",
stage_name="<PAYLOAD_STAGE>",
target_instances=3, # Number of nodes
min_instances=2, # Minimum to start (optional)
session=session,
)
```
**Script must use distributed APIs:**
- Snowflake Distributed Modeling Classes (XGBEstimator, etc.)
- Ray for custom distribution
## Quick Reference
### Required Privileges
| Privilege | Object | Purpose |
|-----------|--------|---------|
| USAGE | Database | Access database |
| USAGE | Schema | Access schema |
| CREATE SERVICE | Schema | Create ML Jobs |
| USAGE | Compute Pool | Run on compute |
| USAGE | Stage | Upload payloads |
### Job Management
```python
from snowflake.ml.jobs import list_jobs, get_job, delete_job
jobs_df = list_jobs(limit=10) # List recent jobs
job = get_job("<JOB_ID>") # Get specific job
delete_job(job) # Delete job
```
ml-pipeline-orchestration/references/operations.md
# Operations Reference
Scheduling, monitoring, error handling, and permissions for Snowflake Task Graph DAGs.
---
## Scheduling
### Schedule Types
| Type | Example | Use Case |
|------|---------|----------|
| `timedelta` | `schedule=timedelta(hours=6)` | Simple intervals |
| `Cron` | `schedule=Cron("0 9 * * *", "America/Los_Angeles")` | Production, timezone-aware |
| `None` | `schedule=None` | Manual-only, event-driven |
### Pause, Resume, Manual Execution
```sql
-- Suspend/resume (alter ROOT task only)
ALTER TASK <DATABASE>.<SCHEMA>.<ROOT_TASK_NAME> SUSPEND;
ALTER TASK <DATABASE>.<SCHEMA>.<ROOT_TASK_NAME> RESUME;
-- Manual trigger
EXECUTE TASK <DATABASE>.<SCHEMA>.<ROOT_TASK_NAME>;
```
```python
dag_op.run(dag) # Trigger immediate run
dag_op.run(dag, retry_last=True) # Retry last failed
```
---
## Monitoring
### Snowsight UI
**Monitoring > Task History** → Filter by DAG name.
### TASK_HISTORY() Queries
```sql
-- Last 24 hours of failures
SELECT *
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
SCHEDULED_TIME_RANGE_START => DATEADD('day', -1, CURRENT_TIMESTAMP()),
ROOT_TASK_ID => '<ROOT_TASK_ID>',
ERROR_ONLY => TRUE
))
ORDER BY SCHEDULED_TIME DESC;
-- All runs for a DAG (last 7 days)
SELECT NAME, STATE, SCHEDULED_TIME, COMPLETED_TIME, ERROR_MESSAGE
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
SCHEDULED_TIME_RANGE_START => DATEADD('day', -7, CURRENT_TIMESTAMP()),
ROOT_TASK_ID => '<ROOT_TASK_ID>'
))
ORDER BY SCHEDULED_TIME DESC;
```
### Task States
| State | Description |
|-------|-------------|
| `SCHEDULED` | Queued |
| `EXECUTING` | Running |
| `SUCCEEDED` | Completed |
| `FAILED` | Error occurred |
| `SKIPPED` | Branch not taken or predecessor failed |
### Python Monitoring
```python
failed_runs = dag_op.get_complete_dag_runs(dag, error_only=True)
all_runs = dag_op.get_complete_dag_runs(dag, error_only=False)
current_runs = dag_op.get_current_dag_runs(dag)
```
> For ML Job monitoring (logs, GPU utilization), see **`../ml-jobs/SKILL.md`** → `get_job()`, `get_logs()`.
---
## Error Handling & Retry
### Task Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `task_auto_retry_attempts` | `0` | Auto retries on failure |
| `suspend_task_after_num_failures` | `10` | Auto-suspend after N failures |
| `user_task_timeout_ms` | `3600000` | Max execution time (1 hour) |
```python
# Set on DAG (applies to all tasks)
dag = DAG(
name="MY_PIPELINE",
task_auto_retry_attempts=2,
suspend_task_after_num_failures=5,
user_task_timeout_ms=7200000, # 2 hours
...
)
# Or set on individual task
train_task = DAGTask("TRAIN", train_model, warehouse=WH, user_task_timeout_ms=7200000)
```
### Finalizer Tasks
Always run regardless of success/failure:
```python
cleanup_task = DAGTask("CLEANUP", cleanup, warehouse=WAREHOUSE, is_finalizer=True)
```
---
## Permissions
### Task-Specific Privileges (Commonly Missing)
```sql
GRANT CREATE TASK ON SCHEMA <database>.<schema> TO ROLE <role>;
GRANT EXECUTE TASK ON ACCOUNT TO ROLE <role>;
GRANT EXECUTE MANAGED TASK ON ACCOUNT TO ROLE <role>; -- for serverless schedules
```
### Task Ownership Rule
All tasks in a DAG must have the **same owner role**.
```sql
SHOW TASKS LIKE '%MY_DAG%' IN SCHEMA <DATABASE>.<SCHEMA>;
-- Check "owner" column — all must match
```
ml-pipeline-orchestration/SKILL.md
---
name: ml-pipeline-orchestration
description: "Create and deploy ML pipelines using Snowflake Task Graphs (DAGs). Use when: orchestrating ML workflows, scheduling training/inference, converting notebooks to production DAGs, automating model retraining, building task graphs. Triggers: pipeline, DAG, task graph, schedule training, orchestrate, productionize, automate retraining."
path: machine-learning/ml-pipeline-orchestration
parent_skill: machine-learning
---
# ML Pipeline Orchestration
Orchestrate ML workflows using Snowflake Task Graphs (DAGs). **Use `@remote` for ML tasks** (compute pool) — cheaper, ML dependencies included.
## When to Use
- User wants to **schedule** ML training or inference
- User wants to **orchestrate** multiple ML steps (prep → train → evaluate → deploy)
- User wants to **convert a notebook** into a production pipeline
- User mentions: "DAG", "task graph", "pipeline", "schedule training", "productionize"
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding**, verify you have the environment guide from `machine-learning/SKILL.md` → Step 0.
---
## Key Concepts
```python
from snowflake.core.task.dagv1 import DAG, DAGTask, DAGTaskBranch, DAGOperation
from snowflake.core.task.context import TaskContext
from snowflake.core.task import Cron
from snowflake.ml.jobs import remote
```
- **Compute pool tasks:** Run on compute pool via `@remote` or `MLJobDefinition` — cheaper, ML packages included
- **Warehouse tasks:** Run as stored procedures — use for lightweight orchestration, branching
- **`TaskContext`:** Pass data between tasks via `get_predecessor_return_value()` / `set_return_value()`
- **Dependencies:** `task_a >> task_b` (sequential), `task >> [a, b]` (fan-out)
---
## API Gotchas
| Wrong | Correct |
|-------|---------|
| `DAGTask("T", my_remote_fn, warehouse=WH)` | `DAGTask("T", definition=my_remote_fn)` — no warehouse for `@remote` |
| `DAG(database=X, schema=Y)` | `DAGOperation(schema_ref)` — database/schema set via DAGOperation |
| `schedule="0 * * * *"` | `schedule=Cron("0 * * * *", "UTC")` or `timedelta(hours=1)` |
| Branch returns `"deploy"` | Branch returns `"PROMOTE_MODEL"` — must match task name exactly |
| `return result` in `@remote` | `ctx.set_return_value(json.dumps(result))` — must use TaskContext |
---
## Pipeline Template
```python
import json
from datetime import timedelta
from snowflake.core import Root
from snowflake.core.task.dagv1 import DAG, DAGTask, DAGTaskBranch, DAGOperation
from snowflake.core.task.context import TaskContext
from snowflake.core.task import Cron
from snowflake.ml.jobs import remote
from snowflake.snowpark import Session
# Configuration
DATABASE = "<DATABASE>"
SCHEMA = "<SCHEMA>"
WAREHOUSE = "<WAREHOUSE>"
COMPUTE_POOL = "<COMPUTE_POOL>"
DAG_STAGE = f"@{DATABASE}.{SCHEMA}.DAG_STAGE"
JOB_STAGE = f"@{DATABASE}.{SCHEMA}.JOB_STAGE"
# --- Warehouse task: lightweight data prep ---
def prepare_data(session: Session) -> str:
"""Prepare datasets, return info as JSON for downstream tasks."""
# ... data prep logic ...
return json.dumps({"train_table": "TRAIN_DATA", "test_table": "TEST_DATA"})
# --- @remote task: heavy ML training on compute pool ---
@remote(COMPUTE_POOL, stage_name=JOB_STAGE, database=DATABASE, schema=SCHEMA)
def train_model() -> None:
"""Train model on compute pool. Access predecessor data via TaskContext."""
session = Session.builder.getOrCreate()
ctx = TaskContext(session)
# Get data from predecessor task
data_info = json.loads(ctx.get_predecessor_return_value("PREPARE_DATA"))
# ... training logic using data_info ...
model_metrics = {"accuracy": 0.95, "model_path": "models/v1.pkl"}
# Pass results to successor tasks
ctx.set_return_value(json.dumps(model_metrics))
# --- Warehouse task: branching logic ---
def check_quality(session: Session) -> str:
"""Check model quality, return next task name."""
ctx = TaskContext(session)
metrics = json.loads(ctx.get_predecessor_return_value("TRAIN_MODEL"))
if metrics["accuracy"] >= 0.90:
return "PROMOTE_MODEL" # Must match task name exactly (case-sensitive)
return "SEND_ALERT"
# --- Warehouse tasks: conditional paths ---
def promote_model(session: Session) -> str:
ctx = TaskContext(session)
metrics = json.loads(ctx.get_predecessor_return_value("TRAIN_MODEL"))
# ... register model to registry ...
return "Model promoted"
def send_alert(session: Session) -> None:
# ... send notification ...
pass
def cleanup(session: Session) -> None:
"""Finalizer: always runs regardless of success/failure."""
# ... cleanup temp artifacts ...
pass
# --- Build the DAG ---
with DAG(
name="ML_TRAINING_PIPELINE",
schedule=timedelta(days=1),
stage_location=DAG_STAGE,
use_func_return_value=True,
) as dag:
prep = DAGTask("PREPARE_DATA", prepare_data, warehouse=WAREHOUSE)
train = DAGTask("TRAIN_MODEL", definition=train_model) # @remote: no warehouse!
check = DAGTaskBranch("CHECK_QUALITY", check_quality, warehouse=WAREHOUSE)
promote = DAGTask("PROMOTE_MODEL", promote_model, warehouse=WAREHOUSE)
alert = DAGTask("SEND_ALERT", send_alert, warehouse=WAREHOUSE)
final = DAGTask("CLEANUP", cleanup, warehouse=WAREHOUSE, is_finalizer=True) # runs automatically
prep >> train >> check >> [promote, alert]
# --- Deploy (session setup per environment guide) ---
root = Root(session)
dag_op = DAGOperation(root.databases[DATABASE].schemas[SCHEMA])
dag_op.deploy(dag, mode="orReplace")
```
**Key patterns:**
- `@remote` runs on compute pool — use `definition=` parameter, no warehouse
- Warehouse tasks for orchestration, branching, lightweight ops
- `TaskContext` passes data between tasks via JSON
- Branch return values must match task names exactly
**Alternative:** For file-based payloads, use `MLJobDefinition.register()`:
```python
from snowflake.ml.jobs import MLJobDefinition
job_def = MLJobDefinition.register("/path/to/script.py", compute_pool=POOL, stage_name=STAGE, session=session)
train_task = DAGTask("TRAIN", definition=job_def)
```
---
## Workflows
### Workflow 1: Create Pipeline
1. **Gather requirements** — What does it do? How often? Which database.schema?
2. **Generate code** — Use Pipeline Template above
3. **Deploy** — `dag_op.deploy(dag, mode="orReplace")`
4. **Ask about execution** — Offer to run the DAG to test it — `dag_op.run(dag)`
5. **Verify** — Snowsight → Monitoring → Task History
**⚠️ MANDATORY STOPPING POINT**: Wait for user requirements before generating code.
### Workflow 2: Add Task to Existing DAG
1. **Get structure** — `SHOW TASKS LIKE '%DAG_NAME%' IN SCHEMA ...`
2. **Add to DAG definition:**
```python
with DAG(...) as dag:
# ... existing tasks ...
new_ml_task = DAGTask("NEW_ML", definition=new_remote_fn) # @remote
new_orch_task = DAGTask("NEW_ORCH", new_warehouse_fn, warehouse=WH) # warehouse
existing >> new_ml_task >> new_orch_task
```
3. **Redeploy** — `dag_op.deploy(dag, mode="orReplace")`
**⚠️ MANDATORY STOPPING POINT**: Present changes and wait for approval before deploying.
### Workflow 3: Convert Notebook
1. **Identify notebook** — Confirm which notebook to convert
2. **Extract functions** — Each cell group becomes a function (`@remote` for ML, warehouse for orchestration)
3. **Deploy** — Follow Pipeline Template above
**⚠️ MANDATORY STOPPING POINT**: Wait for user confirmation.
---
## Quick Reference
| Pattern | Use Case |
|---------|----------|
| `@remote` + `definition=` | ML tasks on compute pool |
| `warehouse=` | Orchestration, branching |
| `ctx.set_return_value()` | Pass data from `@remote` task |
| `ctx.get_predecessor_return_value("TASK")` | Read data from predecessor |
| `DAGTaskBranch` | Conditional paths |
| `is_finalizer=True` | Cleanup (always runs) |
### Scheduling (in `DAG()`)
```python
DAG(name="MY_DAG", schedule=timedelta(days=1), ...) # Daily
DAG(name="MY_DAG", schedule=Cron("0 9 * * *", "America/Los_Angeles"), ...) # 9am PT
DAG(name="MY_DAG", schedule=None, ...) # Manual only
```
### Required Privileges
```sql
GRANT EXECUTE TASK ON ACCOUNT TO ROLE <role>;
GRANT EXECUTE MANAGED TASK ON ACCOUNT TO ROLE <role>; -- if using managed schedules
```
---
## Related Skills & References
- `ml-jobs/SKILL.md` — Compute pools, `@remote` details, EAI setup
- `model-registry/SKILL.md` — Model registration
- `references/operations.md` — Scheduling, monitoring, error handling
**External:** [Official e2e_task_graph sample](https://github.com/Snowflake-Labs/sf-samples/tree/main/samples/ml/ml_jobs/e2e_task_graph) | [Task Graph Docs](https://docs.snowflake.com/en/developer-guide/snowflake-python-api/snowflake-python-managing-tasks)
model-monitor/SKILL.md
---
name: model-monitor
description: "Set up and manage ML Observability for Snowflake Model Registry models. Use when: creating model monitors, tracking drift, viewing performance metrics, managing monitor lifecycle. Triggers: model monitor, ML observability, monitor model, drift detection, model performance, track predictions, add monitoring, enable monitoring, start monitoring, observability."
parent_skill: machine-learning
---
# Model Monitor Operations
## Intent Detection
Route based on user intent:
| User Says | Route To |
|-----------|----------|
| "create monitor", "set up monitoring", "track model", "monitor my model", "add monitoring", "enable monitoring", "start monitoring" | [Workflow A: Create Model Monitor](#workflow-a-create-model-monitor) |
| "check drift", "view metrics", "model performance", "query metrics" | [Workflow B: Query Monitor Metrics](#workflow-b-query-monitor-metrics) |
| "suspend monitor", "resume monitor", "add segment", "set baseline" | [Workflow C: Manage Monitor](#workflow-c-manage-monitor) |
| "monitor not working", "suspended", "refresh failing" | [Workflow D: Troubleshoot Monitor](#workflow-d-troubleshoot-monitor) |
---
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
---
## Prerequisites
Before using model monitors:
- Model must be registered in Snowflake Model Registry
- Model task must be `tabular_binary_classification`, `tabular_regression`, or `tabular_multi_classification`
- Source table with predictions and timestamps must exist
- `snowflake-ml-python >= 1.7.1` required
---
## Workflow A: Create Model Monitor
### Step 0: Check for Recent Model Context
**⚠️ IMPORTANT:** Before asking questions, check if you have context from a recent session:
**Context sources:**
- **From model-registry:** Model name, version, database, schema, framework
- **From spcs-inference:** Model name, version, database, schema, service name
**Context to look for:**
- Model name and version
- Database and schema
- Source table (if predictions are being logged)
- Inference service name (if coming from SPCS)
**If context exists:**
- Use known model name/version - don't ask again
- Skip to Step 1, pre-filling the checklist with known values
- Only ask for missing required parameters (source table, warehouse, timestamp column, etc.)
**If no context:** Proceed to Step 1 and Step 2 normally.
### Step 1: Parse Intent and Build Parameter Checklist
Analyze the user's request to determine which parameters to collect:
**Feature Detection Table:**
| If user mentions... | Add to collection |
|---------------------|-------------------|
| "baseline", "drift", "PSI" | BASELINE |
| "segment", "subgroup", "slice" | SEGMENT_COLUMNS |
| "accuracy", "performance", "ground truth", "actual" | ACTUAL columns |
| "id column", "unique identifier", "row id" | ID_COLUMNS |
**Build checklist:**
- [ ] Mandatory: monitor name, model, version, function, source, warehouse, refresh interval, aggregation window, timestamp, prediction columns
- [ ] BASELINE (if drift/baseline mentioned)
- [ ] SEGMENT_COLUMNS (if segment/subgroup mentioned)
- [ ] ACTUAL columns (if performance/accuracy mentioned)
- [ ] ID_COLUMNS (if unique identifier mentioned)
### Step 2: Collect Mandatory Parameters
**If recent context exists**, only ask for parameters not already known:
```
I see you just registered model <MODEL_NAME> version <VERSION>. To set up monitoring, I need:
1. Monitor name: [identifier for the monitor]
2. Function name: [e.g., 'predict']
3. Source table: [table with predictions, fully qualified]
4. Warehouse: [for monitor compute]
5. Refresh interval: [e.g., '1 day', '6 hours', min: '60 seconds']
6. Aggregation window: [days only, e.g., '1 day', '7 days']
7. Timestamp column: [column name, must be TIMESTAMP_NTZ]
8. Prediction column(s): [score or class columns]
```
**If no context**, ask for all parameters:
```
To create your model monitor, I need the following information:
1. Monitor name: [identifier for the monitor]
2. Model name: [must be in Model Registry]
3. Model version: [version to monitor]
4. Function name: [e.g., 'predict']
5. Source table: [fully qualified: DATABASE.SCHEMA.TABLE]
6. Warehouse: [for monitor compute]
7. Refresh interval: [e.g., '1 day', '6 hours', min: '60 seconds']
8. Aggregation window: [days only, e.g., '1 day', '7 days']
9. Timestamp column: [column name, must be TIMESTAMP_NTZ]
10. Prediction column(s): [score or class columns]
```
**⚠️ STOP**: Wait for user response.
### Step 3: Collect Feature-Specific Parameters
Based on Step 1 analysis, ask for additional parameters:
**If BASELINE needed:**
```
For drift detection, I also need:
- Baseline table: [fully qualified: DATABASE.SCHEMA.TABLE]
```
**If SEGMENT_COLUMNS needed:**
```
For segmentation, I also need:
- Segment column(s): [column names, must be STRING type, max 5, <25 unique values recommended, no special characters in values]
```
**If ACTUAL columns needed:**
```
For performance metrics, I also need:
- Actual/ground truth column(s): [score or class columns]
```
**⚠️ STOP**: Wait for user response (only if additional parameters needed).
### Step 4: Validate Prerequisites
```sql
-- Verify model and version exist
SHOW MODELS LIKE '<MODEL_NAME>' IN SCHEMA <DATABASE>.<SCHEMA>;
```
**If model doesn't exist (empty result):** Direct user to register model first using `../model-registry/SKILL.md`.
**If model exists:** Confirm the expected version appears in the results before proceeding.
```sql
-- Check source table columns
DESCRIBE TABLE <SOURCE_TABLE>;
```
Verify:
- Timestamp column exists (must be `TIMESTAMP_NTZ`)
- Prediction columns exist:
- Binary classification/regression: `NUMBER` type
- Multi-class classification: `STRING` type
- Actual columns (if provided): same type rules as prediction columns
- Segment columns (if provided): must be `STRING` type
### Step 5: Generate and Execute SQL
Build CREATE MODEL MONITOR statement with collected parameters:
```sql
CREATE MODEL MONITOR <MONITOR_NAME> WITH
MODEL = <MODEL_NAME>
VERSION = '<VERSION_NAME>'
FUNCTION = '<FUNCTION_NAME>'
SOURCE = <SOURCE_TABLE>
WAREHOUSE = <WAREHOUSE_NAME>
REFRESH_INTERVAL = '<REFRESH_INTERVAL>'
AGGREGATION_WINDOW = '<AGGREGATION_WINDOW>'
TIMESTAMP_COLUMN = <TIMESTAMP_COL>
PREDICTION_SCORE_COLUMNS = ('<PRED_COL>')
-- Include if ACTUAL columns collected:
ACTUAL_SCORE_COLUMNS = ('<ACTUAL_COL>')
-- Include if BASELINE collected:
BASELINE = <BASELINE_TABLE>
-- Include if SEGMENT_COLUMNS collected:
SEGMENT_COLUMNS = ('<SEGMENT_COL_1>', '<SEGMENT_COL_2>');
```
**⚠️ MANDATORY:** Present the generated SQL (with only the relevant clauses) to user and wait for approval before executing.
### Step 6: Verify Monitor Created
```sql
-- List all monitors in schema
SHOW MODEL MONITORS IN SCHEMA <DATABASE>.<SCHEMA>;
-- Filter by name pattern
SHOW MODEL MONITORS LIKE '%<PATTERN>%' IN SCHEMA <DATABASE>.<SCHEMA>;
-- Check specific monitor details
DESC MODEL MONITOR <MONITOR_NAME>;
```
Verify `monitor_state` is `ACTIVE` (not SUSPENDED, PARTIALLY_SUSPENDED, or UNKNOWN).
**⚠️ STOP**: Confirm monitor is running before proceeding.
### Suggested Next Actions
Present context-aware options based on what was configured:
**If baseline was NOT configured:**
```
Your model monitor is active! To enable drift detection:
- "Set a baseline for drift detection"
```
**If actual columns were NOT configured:**
```
To track model accuracy, you can add ground truth data later:
- "Add actual columns to my monitor" (requires recreating monitor)
```
**Always show:**
```
View your monitor: Snowsight → AI&ML → Models → Select Model → Monitors
Query metrics: "Show me metrics for my monitor"
```
---
## Workflow B: Query Monitor Metrics
### Step 1: Identify Monitor and Metric Type
**Ask user:**
```
What metrics would you like to view?
1. Drift metrics (PSI, distribution shifts)
2. Performance metrics (accuracy, precision, recall, F1, MAE, RMSE)
3. Statistical metrics (counts, nulls, distributions)
```
**⚠️ STOP**: Wait for user response.
### Step 2: Query Metrics
**Drift Metrics:**
```sql
SELECT *
FROM TABLE(MODEL_MONITOR_DRIFT_METRIC(
'<MONITOR_NAME>',
'<METRIC_NAME>', -- e.g., 'PSI', 'KL_DIVERGENCE'
'<COLUMN_NAME>', -- feature column to check drift
'DAY', -- granularity
'<START_TIME>'::TIMESTAMP_NTZ,
'<END_TIME>'::TIMESTAMP_NTZ
));
```
**Performance Metrics:**
```sql
SELECT *
FROM TABLE(MODEL_MONITOR_PERFORMANCE_METRIC(
'<MONITOR_NAME>',
'<METRIC_NAME>', -- 'ACCURACY', 'PRECISION', 'RECALL', 'F1', 'MAE', 'RMSE'
'DAY',
'<START_TIME>'::TIMESTAMP_NTZ,
'<END_TIME>'::TIMESTAMP_NTZ
));
```
**Statistical Metrics:**
```sql
SELECT *
FROM TABLE(MODEL_MONITOR_STAT_METRIC(
'<MONITOR_NAME>',
'<METRIC_NAME>', -- 'COUNT', 'NULL_COUNT', etc.
'DAY',
'<START_TIME>'::TIMESTAMP_NTZ,
'<END_TIME>'::TIMESTAMP_NTZ
));
```
**Query Metrics for Specific Segment:**
```sql
SELECT *
FROM TABLE(MODEL_MONITOR_DRIFT_METRIC(
'<MONITOR_NAME>',
'PSI',
'<COLUMN_NAME>',
'DAY',
'<START_TIME>'::TIMESTAMP_NTZ,
'<END_TIME>'::TIMESTAMP_NTZ,
'{"SEGMENTS": [{"column": "<SEGMENT_COL>", "value": "<SEGMENT_VALUE>"}]}'
));
```
### Step 3: Present Results
Format results clearly showing:
- Time period analyzed
- Metric values and trends
- Any concerning drift or performance degradation
**Suggest next actions** based on results:
- High drift → Investigate data pipeline, consider retraining
- Low performance → Review model, check for concept drift
- Missing data → Check source table updates
---
## Workflow C: Manage Monitor
### Suspend/Resume Monitor
```sql
-- Suspend monitoring
ALTER MODEL MONITOR <MONITOR_NAME> SUSPEND;
-- Resume monitoring
ALTER MODEL MONITOR <MONITOR_NAME> RESUME;
```
### Set Monitor Properties
Set one or more properties in a single statement (each is optional):
```sql
ALTER MODEL MONITOR <MONITOR_NAME> SET
BASELINE = '<BASELINE_TABLE_NAME>'
REFRESH_INTERVAL = '<REFRESH_INTERVAL>'
WAREHOUSE = <WAREHOUSE_NAME>;
```
**Examples:**
```sql
-- Set baseline only
ALTER MODEL MONITOR <MONITOR_NAME> SET BASELINE = '<BASELINE_TABLE>';
-- Set refresh interval only
ALTER MODEL MONITOR <MONITOR_NAME> SET REFRESH_INTERVAL = '6 hours';
-- Set warehouse only
ALTER MODEL MONITOR <MONITOR_NAME> SET WAREHOUSE = <NEW_WAREHOUSE>;
-- Set multiple properties at once
ALTER MODEL MONITOR <MONITOR_NAME> SET
REFRESH_INTERVAL = '6 hours'
WAREHOUSE = <NEW_WAREHOUSE>;
```
### Add/Remove Segment Columns
```sql
-- Add segment (max 5 segments, must be STRING type)
ALTER MODEL MONITOR <MONITOR_NAME> ADD SEGMENT_COLUMN = '<COLUMN_NAME>';
-- Remove segment
ALTER MODEL MONITOR <MONITOR_NAME> DROP SEGMENT_COLUMN = '<COLUMN_NAME>';
```
---
## Drop Model Monitor
**⚠️ MANDATORY:** Before dropping, confirm with user:
```
Are you sure you want to delete monitor <MONITOR_NAME>?
This cannot be undone. All monitor history and metrics will be lost.
Type "yes" to confirm:
```
**⚠️ STOP**: Wait for explicit confirmation before proceeding.
**If confirmed:**
```sql
DROP MODEL MONITOR <MONITOR_NAME>;
```
---
## Workflow D: Troubleshoot Monitor
### Step 1: Check Monitor Status
```sql
DESC MODEL MONITOR <MONITOR_NAME>;
```
Check `monitor_state`:
- `ACTIVE` - Operating correctly
- `SUSPENDED` - Monitoring paused (manual or after 5 failures)
- `PARTIALLY_SUSPENDED` - One underlying table stopped refreshing
- `UNKNOWN` - State cannot be identified
**Additional fields for debugging:**
- `aggregation_status` - Current aggregation state
- `aggregation_last_error` - Error details if suspended
### Step 2: Common Issues and Solutions
| Issue | Cause | Solution |
|-------|-------|----------|
| SUSPENDED | Manual suspend or 5 consecutive refresh failures | Fix root cause, then `ALTER MODEL MONITOR ... RESUME` |
| PARTIALLY_SUSPENDED | One underlying table stopped refreshing | Check DESC for `aggregation_last_error`, fix issue, then RESUME |
| UNKNOWN | State cannot be determined | Check DESC for errors, may need to recreate monitor |
| No drift metrics | No baseline set | Set baseline with `ALTER MODEL MONITOR ... SET BASELINE` |
| Missing performance metrics | No actual columns | Re-create monitor with `ACTUAL_SCORE_COLUMNS` |
| Segment query fails | Invalid segment value | Check segment column has expected values, case sensitive |
| Invalid data errors | NULLs, NaNs, or out-of-range values | Clean source data, remove invalid rows |
### Step 3: Resume After Fixing
```sql
-- After fixing root cause
ALTER MODEL MONITOR <MONITOR_NAME> RESUME;
-- Verify it's running
DESC MODEL MONITOR <MONITOR_NAME>;
```
---
## CREATE MODEL MONITOR Parameters
### Mandatory Parameters (Always Required)
| Parameter | Type | Description |
|-----------|------|-------------|
| `MONITOR_NAME` | identifier | Name for the monitor |
| `MODEL` | identifier | Model name (same schema as monitor) |
| `VERSION` | string | Model version name |
| `FUNCTION` | string | Function name (e.g., 'predict') |
| `SOURCE` | identifier | Table/view with predictions (fully qualified) |
| `WAREHOUSE` | identifier | Warehouse for compute |
| `REFRESH_INTERVAL` | string | How often to refresh (min: '60 seconds', e.g., '1 day') |
| `AGGREGATION_WINDOW` | string | Aggregation period (days only, e.g., '1 day') |
| `TIMESTAMP_COLUMN` | identifier | Timestamp column (must be TIMESTAMP_NTZ) |
| `PREDICTION_*_COLUMNS` | string list | At least one of: PREDICTION_CLASS_COLUMNS or PREDICTION_SCORE_COLUMNS |
### Feature-Triggered Parameters (Ask When User Mentions Feature)
| Parameter | Trigger Keywords | Type | Description |
|-----------|------------------|------|-------------|
| `BASELINE` | "baseline", "drift", "PSI" | identifier | Baseline table for drift detection |
| `SEGMENT_COLUMNS` | "segment", "subgroup", "slice" | string list | Columns for segmentation (max 5, STRING type, <25 unique values recommended) |
| `ACTUAL_CLASS_COLUMNS` | "accuracy", "performance", "ground truth", "actual" | string list | Ground truth class columns |
| `ACTUAL_SCORE_COLUMNS` | "accuracy", "performance", "ground truth", "actual" | string list | Ground truth score columns |
### Optional Parameters (Only If User Explicitly Requests)
| Parameter | Type | Description |
|-----------|------|-------------|
| `ID_COLUMNS` | string list | Columns that uniquely identify each row in source data |
| `CUSTOM_METRIC_COLUMNS` | string list | Additional numeric columns to track |
---
## Usage Notes
- **Column uniqueness:** Each column can only be used in one parameter (e.g., an ID column cannot also be a prediction column), except for segment columns which can overlap with other columns as long as they are STRING type
- **Single output only:** Multiple-output models not supported; prediction/actual column arrays must have exactly one element
- **Segment values:** Case sensitive, special characters not supported in segment queries
---
## Monitor Status Commands
### List Monitors
```sql
-- All monitors in current schema
SHOW MODEL MONITORS;
-- Filter by pattern
SHOW MODEL MONITORS LIKE '%mymodel%';
-- Specific scope
SHOW MODEL MONITORS IN SCHEMA <DATABASE>.<SCHEMA>;
SHOW MODEL MONITORS IN DATABASE <DATABASE>;
SHOW MODEL MONITORS IN ACCOUNT;
```
**Key output column:** `monitor_state` (ACTIVE, SUSPENDED, PARTIALLY_SUSPENDED, UNKNOWN)
### Describe Monitor
```sql
DESC MODEL MONITOR <MONITOR_NAME>;
```
**Key output columns for debugging:**
- `monitor_state` - Current state
- `aggregation_status` - Aggregation state details
- `aggregation_last_error` - Error message if suspended
- `columns` - JSON with all configured columns
---
## Supported Metrics
**Drift Metrics:**
- `PSI` (Population Stability Index)
- `KL_DIVERGENCE` (Kullback-Leibler Divergence)
**Performance Metrics (Classification):**
- `ACCURACY`, `PRECISION`, `RECALL`, `F1`
- `AUC`, `LOG_LOSS`
**Performance Metrics (Regression):**
- `MAE` (Mean Absolute Error)
- `RMSE` (Root Mean Square Error)
- `MAPE` (Mean Absolute Percentage Error)
**Statistical Metrics:**
- `COUNT`, `NULL_COUNT`
- `MEAN`, `STDDEV`, `MIN`, `MAX`
---
## Limitations
- Max 250 monitors per account
- Max 500 features monitored per monitor
- Max 5 segment columns per monitor
- Segment columns must be STRING type, <25 unique values recommended
- Segment values are case sensitive, no special characters supported
- Multiple-output models not supported (arrays must have one element)
- Supports regression, binary classification, and multi-class classification
- Monitor must be in same schema as model
- Cannot change model/source after creation (must drop and recreate)
---
## Output
- Model monitor created and tracking predictions
- Metrics available via SQL functions or Snowsight UI
- Alerts can be configured using Snowflake alerting features
model-registry/hugging-face-models/SKILL.md
---
name: model-registry-hugging-face-models
description: "Deploy Hugging Face models to Snowflake Model Registry. Use when: Logging Hugging Face models. Triggers: log model, model to snowflake, hugging face, huggingface, transformers pipeline."
parent_skill: model-registry
---
# Logging Hugging Face Models
The Model Registry has built-in support for any Hugging Face model loadable via `transformers.Pipeline`. There are two approaches to log a HF model, and the behavior is controlled by the `compute_pool_for_log` parameter on `TransformersPipeline`.
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
## When to Use
- Deploying Hugging Face Transformers models to Snowflake
- User mentions: "log HuggingFace model", "deploy transformers", "ProsusAI/finbert", model IDs with forward slashes
- Working with `transformers.Pipeline` compatible models
- Need to choose between remote vs local model logging strategies for hugging face models
### Logging Modes
| Mode | `compute_pool_for_log` value | What happens | When to use |
|------|------------------------------|--------------|-------------|
| **Remote** (default) | A compute pool name string (defaults to system CPU pool) | Model metadata (ID, task, revision) is captured; actual weights are downloaded server-side via `SYSTEM$IMPORT_MODEL` during `log_model()` | Standard path — no local download needed, works without `huggingface_hub` installed |
| **Local (snapshot download)** | `None` | Model artifacts are downloaded locally via `huggingface_hub.snapshot_download()` then uploaded to registry | When you need to pre-filter files (`allow_patterns`/`ignore_patterns`), work offline, or avoid server-side download |
### Approach 1: Using `TransformersPipeline` (recommended)
Create a `TransformersPipeline` wrapper — the model is NOT loaded into memory. Only metadata is captured.
**Remote logging (default):**
```python
from snowflake.ml.model.models import huggingface
from snowflake.ml.registry import Registry
session = <SESSION_SETUP>
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
model = huggingface.TransformersPipeline(
task="text-classification",
model="ProsusAI/finbert",
)
mv = reg.log_model(
model,
model_name="<MODEL_NAME>",
version_name="<VERSION>",
)
```
The default CPU compute pool is used to download the model server-side. To specify a different compute pool:
```python
model = huggingface.TransformersPipeline(
task="text-classification",
model="ProsusAI/finbert",
compute_pool_for_log="MY_COMPUTE_POOL",
)
```
**Local logging (snapshot download):**
Pass `compute_pool_for_log=None` to download model files locally before uploading. Requires `huggingface_hub` to be installed.
**⚠️** For `text-generation` models logged locally, you must provide `signatures` — see [OpenAI Signatures for Text Generation](#openai-signatures-for-text-generation).
```python
from snowflake.ml.model.models import huggingface
from snowflake.ml.registry import Registry
session = <SESSION_SETUP>
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
model = huggingface.TransformersPipeline(
task="text-classification",
model="ProsusAI/finbert",
compute_pool_for_log=None,
allow_patterns=["*.safetensors", "*.json", "*.txt"], # optional: filter downloaded files
ignore_patterns=["*.bin", "*.msgpack"], # optional: exclude files
)
mv = reg.log_model(
model,
model_name="<MODEL_NAME>",
version_name="<VERSION>",
)
```
### Approach 2: Using `transformers.pipeline()` directly
Load the model fully into memory, then log it. The model artifacts are uploaded from local memory.
**⚠️** For `text-generation` models, you must provide `signatures` — see [OpenAI Signatures for Text Generation](#openai-signatures-for-text-generation).
```python
import transformers
from snowflake.ml.model.openai_signatures import OPENAI_CHAT_WITH_PARAMS_SIGNATURE
from snowflake.ml.registry import Registry
session = <SESSION_SETUP>
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
model = transformers.pipeline(
task="text-generation",
model="bigscience/bloom-560m",
token="<HF_TOKEN>",
return_full_text=False,
max_new_tokens=100,
)
mv = reg.log_model(
model,
model_name="<MODEL_NAME>",
version_name="<VERSION>",
signatures=OPENAI_CHAT_WITH_PARAMS_SIGNATURE,
)
```
### `TransformersPipeline` Parameters
| Parameter | Description | Default |
|-----------|-------------|---------|
| `task` | Pipeline task (e.g., `"text-classification"`, `"text-generation"`, `"summarization"`). Inferred from model if `None`. | `None` |
| `model` | HuggingFace model identifier (e.g., `"ProsusAI/finbert"`). | `None` |
| `revision` | Branch, tag, or commit ID for the model version. | `None` |
| `token_or_secret` | HF token string or a fully qualified Snowflake secret name (e.g., `"DB.SCHEMA.MY_HF_SECRET"`). | `None` |
| `trust_remote_code` | Allow custom code from the HF Hub. Only set `True` for trusted repos. | `None` |
| `model_kwargs` | Extra keyword arguments passed to the model's `from_pretrained()`. | `None` |
| `compute_pool_for_log` | Compute pool name for remote logging. Set to `None` for local snapshot download. | Default CPU pool |
| `allow_patterns` | File patterns to include when downloading (local mode only). | `None` |
| `ignore_patterns` | File patterns to exclude when downloading (local mode only). | `None` |
### HF-specific `log_model()` Options
When calling `reg.log_model()` with a HF model, the `options` dict supports:
| Option key | Description | Default |
|------------|-------------|---------|
| `target_methods` | List of methods on the model. HF models use `__call__` by default. | `["__call__"]` |
| `cuda_version` | CUDA runtime version for GPU deployment. Set to `None` to disable GPU. | `"12.4"` |
The registry auto-infers `signatures` for most tasks (`fill-mask`, `question-answering`, `summarization`, `table-question-answering`, `text2text-generation`, `text-classification`, `sentiment-analysis`, `text-generation`, `token-classification`, `ner`, `translation`, `translation_xx_to_yy`, `zero-shot-classification`). For other tasks, provide `signatures` or `sample_input_data` explicitly.
**⚠️ `text-generation` and local logging:** When logging a HF model with task `text-generation` **locally** (via `compute_pool_for_log=None` or `transformers.pipeline()`), you **must** provide an OpenAI-compatible signature explicitly. Remote logging applies the signature automatically. See [OpenAI Signatures for Text Generation](#openai-signatures-for-text-generation).
`sample_input_data` is ignored for HF models — use `signatures` if the task is not in the auto-inferred list.
### OpenAI Signatures for Text Generation
The `snowflake.ml.model.openai_signatures` module provides pre-built signatures that emulate OpenAI chat completion behavior. These are **required** when logging `text-generation` HF models locally and **automatically applied** when logging remotely.
**Available signatures:**
| Signature | Description |
|-----------|-------------|
| `OPENAI_CHAT_SIGNATURE` | Default — content as structured parts (text, image, video, audio) |
| `OPENAI_CHAT_SIGNATURE_WITH_CONTENT_FORMAT_STRING` | Content as a plain string (most models support this) |
| `OPENAI_CHAT_WITH_PARAMS_SIGNATURE` | Default + inference params (`temperature`, `top_p`, etc.) as `ParamSpec` |
| `OPENAI_CHAT_WITH_PARAMS_SIGNATURE_WITH_CONTENT_FORMAT_STRING` | String content + inference params as `ParamSpec` |
**When to provide signatures:**
| Logging mode | Task `text-generation` | Other tasks |
|--------------|----------------------|-------------|
| **Remote** (`compute_pool_for_log` = compute pool name) | Auto-applied (uses `OPENAI_CHAT_WITH_PARAMS_SIGNATURE`) | Auto-inferred |
| **Local** (`compute_pool_for_log=None` or `transformers.pipeline()`) | **Must provide explicitly** | Auto-inferred |
**Example — local logging with OpenAI signature:**
```python
from snowflake.ml.model.models import huggingface
from snowflake.ml.model.openai_signatures import OPENAI_CHAT_WITH_PARAMS_SIGNATURE
from snowflake.ml.registry import Registry
session = <SESSION_SETUP>
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
model = huggingface.TransformersPipeline(
task="text-generation",
model="meta-llama/Llama-2-7b-chat-hf",
compute_pool_for_log=None,
token_or_secret="MY_DB.MY_SCHEMA.HF_TOKEN_SECRET", # required for gated models
)
mv = reg.log_model(
model,
model_name="<MODEL_NAME>",
version_name="<VERSION>",
signatures=OPENAI_CHAT_WITH_PARAMS_SIGNATURE,
)
```
### Authentication for Private/Gated Models
**Using a HF token directly for local download:**
```python
model = huggingface.TransformersPipeline(
task="text-generation",
model="meta-llama/Llama-2-7b-chat-hf",
token_or_secret="hf_xxxYOUR_TOKEN_HERExxxx",
)
```
**Using a Snowflake secret (recommended for production):**
```python
model = huggingface.TransformersPipeline(
task="text-generation",
model="meta-llama/Llama-2-7b-chat-hf",
token_or_secret="MY_DB.MY_SCHEMA.HF_TOKEN_SECRET",
)
```
### External Access for Snowflake Notebooks
When downloading HF models from a Snowflake Notebook, attach an external access integration allowing egress to HuggingFace hosts:
```sql
CREATE NETWORK RULE huggingface_network_rule
TYPE = HOST_PORT
VALUE_LIST = (
'huggingface.co',
'hub-ci.huggingface.co',
'cdn-lfs-us-1.hf.co',
'cdn-lfs-eu-1.hf.co',
'cdn-lfs.hf.co',
'transfer.xethub.hf.co',
'cas-server.xethub.hf.co',
'cas-bridge.xethub.hf.co'
)
MODE = EGRESS;
CREATE EXTERNAL ACCESS INTEGRATION huggingface_access_integration
ALLOWED_NETWORK_RULES = (huggingface_network_rule)
ENABLED = true;
```
Attach this integration to your Notebook before running model download code.
## Troubleshooting
- Many HF models are large — use a Snowpark-optimized warehouse.
- Snowflake warehouses do not have GPUs. Use CPU-optimized models for warehouse inference, or deploy via SPCS for GPU support.
- Task names are **case-sensitive** (e.g., `"text-classification"`, not `"Text-Classification"`).
## Stopping Points
- Before `log_model()` - confirm model name, version, and logging mode with user
- If gated model detected (e.g., Llama) - verify authentication is configured
- After logging - wait before proceeding to inference unless user explicitly requests it
## Output
- Model logged to Snowflake Model Registry
- Model version object (`ModelVersion`) ready for deployment
- Model accessible via `reg.get_model("<MODEL_NAME>").version("<VERSION>")`
model-registry/partitioned-inference/SKILL.md
---
name: partitioned-inference
description: Partitioned inference with CustomModel and @partitioned_api decorator
path: machine-learning/model-registry/partitioned-inference
parent_skill: model-registry
---
# Partitioned Inference
Parallelize inference across data partitions using the Model Registry. Train and predict on partition-specific submodels.
> **Execution Environment**: Partitioned inference runs on **virtual warehouse** (not compute pool). For compute pool-based distributed processing, see `../../distributed-training/SKILL.md`.
## When to Use
- Dataset has natural partitions (store, region, customer segment)
- Partitions are independent (uncorrelated data)
- Each partition has sufficient data for training
- Need to parallelize training/inference
## Stateless Partitioned Model
Training and inference happen together; no stored fit state.
```python
import pandas as pd
from snowflake.ml.model.custom_model import CustomModel, partitioned_api
class StatelessPartitionedModel(CustomModel):
@partitioned_api
def predict(self, input_df: pd.DataFrame) -> pd.DataFrame:
import xgboost
# Train on partition data
X = input_df[['feature1', 'feature2']]
y = input_df['target']
model = xgboost.XGBRegressor()
model.fit(X, y)
# Generate predictions
predictions = model.predict(X)
return pd.DataFrame({'prediction': predictions})
my_model = StatelessPartitionedModel()
```
## Stateful Partitioned Model
Pre-trained submodels loaded from context. **Two-phase workflow**: train models per partition first, then package for inference.
> **Training models per partition**: If you need to train models first, load `../../distributed-training/mmt/SKILL.md`. This skill covers only the inference side — packaging and running predictions with pre-trained models.
### Step 1: Package Pre-Trained Models into ModelContext
After training models per partition (via MMT or any other method), create a `ModelContext`. There are two options depending on model size:
**Option A: In-memory models** (default — models loaded into memory):
```python
from snowflake.ml.model.custom_model import CustomModel, ModelContext, partitioned_api
# models dict maps partition_id -> fitted model object
models = {
"store_1": fitted_model_1,
"store_2": fitted_model_2,
}
model_context = ModelContext(models=models)
stateful_model = StatefulPartitionedModel(context=model_context) # Note: parameter is 'context', not 'model_context'
```
**Option B: File artifacts** (for large models that are expensive to hold in memory):
```python
model_context = ModelContext(
artifacts={
"store_1": "/path/to/model1.pkl",
"store_2": "/path/to/model2.pkl",
}
)
```
### Step 2: Define the CustomModel with Partition Lookup
The `predict` method extracts the partition ID from input data and retrieves the corresponding pre-trained submodel.
**For in-memory models (Option A)** — use `self.context.model_ref()`:
```python
class StatefulPartitionedModel(CustomModel):
@partitioned_api
def predict(self, input: pd.DataFrame) -> pd.DataFrame:
partition_id = input["STORE_NUMBER"][0]
model = self.context.model_ref(partition_id)
predictions = model.predict(input[['feature1', 'feature2']])
return pd.DataFrame({'prediction': predictions})
```
**For file artifacts (Option B)** — use `self.context.path()` to load on demand:
```python
class ArtifactPartitionedModel(CustomModel):
@partitioned_api
def predict(self, input: pd.DataFrame) -> pd.DataFrame:
partition_id = input["STORE_NUMBER"][0]
model_path = self.context.path(partition_id)
import joblib
model = joblib.load(model_path)
predictions = model.predict(input[['feature1', 'feature2']])
return pd.DataFrame({'prediction': predictions})
```
## Logging Partitioned Models
### Automatic Dependency Inference
Snowflake automatically infers dependencies during `log_model()` when you provide `sample_input_data`. This means `conda_dependencies` is usually not needed — frameworks like xgboost, sklearn, lightgbm are detected automatically.
```python
from snowflake.ml.registry import Registry
reg = Registry(session=session, database_name="ML", schema_name="REGISTRY")
# No conda_dependencies needed
model_version = reg.log_model(
my_model,
model_name="my_partitioned_model",
version_name="v1",
options={"function_type": "TABLE_FUNCTION"},
sample_input_data=train_features, # Required for dependency inference
)
```
### Manual Dependencies (Fallback)
If automatic inference fails or you need specific versions, pin `conda_dependencies`:
```python
model_version = reg.log_model(
my_model,
model_name="my_partitioned_model",
version_name="v1",
options={"function_type": "TABLE_FUNCTION"},
sample_input_data=train_features,
conda_dependencies=["xgboost==1.7.6", "cloudpickle==2.2.1"],
)
```
> **Serialization errors?** Pin `conda_dependencies` to match the versions used during training. Common symptoms:
> - `AttributeError: Can't get attribute '_class_setstate'` → cloudpickle version mismatch (try `cloudpickle==2.2.1`)
> - `ModuleNotFoundError: No module named 'numpy._core.numeric'` → numpy 2.x incompatibility (try `numpy<2`)
## Running Partitioned Inference
### Python API
Call the model version with a partition column to distribute inference across partitions:
```python
model_version.run(
input_df,
function_name="PREDICT",
partition_column="STORE_NUMBER"
)
```
### SQL
Equivalent SQL using `PARTITION BY` to route rows to the correct submodel:
```sql
SELECT output1, output2, partition_column
FROM input_table,
TABLE(
my_model!predict(input_table.input1, input_table.input2)
OVER (PARTITION BY input_table.store_number)
)
ORDER BY input_table.store_number;
```
## Key Differences: Partitioned Models vs Many Model Training
| Aspect | Partitioned Inference | Many Model Training (MMT) |
|--------|----------------------|---------------------------|
| Primary use | Inference parallelization | Training parallelization |
| Storage | Model Registry | Snowflake Stage |
| Decorator | `@partitioned_api` | N/A |
| Output | Predictions per partition | Trained models per partition |
| Framework | CustomModel subclass | Training function |
**How they work together (Stateful Workflow):**
```
MMT (train per partition) → get_model() → ModelContext → Partitioned Inference
```
1. **MMT trains** models per partition → outputs to stage
2. **get_model()** retrieves fitted models
3. **ModelContext** packages models for CustomModel
4. **Partitioned Inference** runs predictions using pre-fitted models
## Key Classes
| Class/Decorator | Import |
|-----------------|--------|
| `CustomModel` | `snowflake.ml.model.custom_model` |
| `@partitioned_api` | `snowflake.ml.model.custom_model` |
| `ModelContext` | `snowflake.ml.model.custom_model` |
| `Registry` | `snowflake.ml.registry` |
## Quickstarts
- [Partitioned Model Quickstart](https://quickstarts.snowflake.com/guide/partitioned-ml-model/)
- [Many Model Inference Quickstart](https://quickstarts.snowflake.com/guide/many-model-inference-in-snowflake/)
model-registry/SKILL.md
---
name: model-registry
description: "Deploy models to Snowflake Model Registry and route to inference deployment. Use when: registering serialized models, deploying trained models, logging models, logging Hugging Face models. Triggers: model registry, deploy model, register model, log model, model to snowflake, hugging face, huggingface, transformers pipeline."
---
# Model Registry Operations
## Intent Detection
Route based on user intent:
| User Says | Route To |
|-----------|----------|
| "register model", "log model", "deploy pickle", "save model to registry" | [Workflow A: Register Model](#workflow-a-register-model) |
| "deploy model", "deploy model for inference", "deploy for inference" | [Workflow B: Deploy Model Decision Tree](#workflow-b-deploy-model-decision-tree) |
| "create inference service", "SPCS inference", "inference endpoint", "serve model", "snowpark container services" | `../spcs-inference/SKILL.md` |
| "partitioned inference", "@partitioned_api", "partition-aware model", "model per partition inference", "CustomModel partition" | `partitioned-inference/SKILL.md` |
---
## Workflow B: Deploy Model Decision Tree
Use this workflow when user says "deploy a model" or "deploy model for inference".
### Step 1: Choose Deployment Target
**Ask user:**
```
Where would you like to deploy your model for inference?
1. Warehouse - Run inference via SQL queries (simpler, no extra infrastructure)
2. Snowpark Container Services (SPCS) - REST endpoints, GPU support, scalable
```
**⚠️ STOP**: Wait for user response.
**If Warehouse:** Route to [Workflow A: Register Model](#workflow-a-register-model)
**If SPCS:** Load `../spcs-inference/SKILL.md` and follow its workflow.
---
## When to Use
**Register Model (Workflow A):**
- User has a model object in memory (just trained in the current session, snowsight only)
- User has a serialized model file (`.pkl`, `.ubj`, `.json`, `.pt`, `.h5`, etc.)
- User wants to register/log a model to Snowflake Model Registry
**SPCS Inference Service (`../spcs-inference/SKILL.md`):**
- User has a model already registered in the registry
- User wants to deploy the model for real-time inference via SPCS
- User wants to create an HTTP endpoint for model predictions
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
**⚠️ Conda Environment for WAREHOUSE Target:** When targeting WAREHOUSE, use a conda environment with `snowflake-ml-python` installed via conda (not pip). **Use the same Python version the model was trained with** to avoid pickle compatibility issues. Create with: `conda create -n snowml python=<VERSION> snowflake-ml-python -c https://repo.anaconda.com/pkgs/snowflake`
## API Reference
For understanding `ModelVersion` methods and their signatures (e.g., `show_functions()`, `get_metric()`, `run()`, `create_service()`), refer to the official documentation:
**ModelVersion API Reference:** https://docs.snowflake.com/en/developer-guide/snowpark-ml/reference/latest/api/model/snowflake.ml.model.ModelVersion
---
## Workflow A: Register Model
### Step 0: Check for Recent Model Context
**⚠️ IMPORTANT:** Check if you have context from a recent training session:
- Model variable name or file path
- Framework used (sklearn, xgboost, lightgbm, pytorch, tensorflow, huggingface/transformers)
- Training data that can be used as sample input
**If model context exists:** Skip to Step 2 — only ask for model name and database/schema.
**If no recent context:** Proceed to Step 1.
### Step 1: Gather Information
**If no recent context**, ask user for:
- Model file path (e.g., `.pkl`, `.ubj`, `.json`, `.pt`) OR confirm model variable in notebook kernel memory
- Model name for Snowflake
- Database and Schema to register this model (Do not use ask_user_question tool for this one, just stop and wait for user response)
- Framework (sklearn, xgboost, lightgbm, pytorch, tensorflow, huggingface/transformers, or other)
- Sample input data or schema description (if needed)
- Additional dependencies
**⚠️ STOP**: Wait for user response.
### Step 2: Check if Model Version Exists
**⚠️ CRITICAL:** The user's intent is to register a NEW model. Do NOT treat an existing model or version as the task being complete. Always proceed with registration — resolve any naming/version conflicts by proposing the next available version.
```sql
SHOW MODELS LIKE '<MODEL_NAME>' IN SCHEMA <DATABASE>.<SCHEMA>;
```
- **If version exists**: Ask user to choose new version (v2, v3...) or new model name
- **If "does not exist" error**: Proceed with "v1"
**⚠️ STOP**: Wait for user choice if model already exists.
### Step 3: Determine Model Type
Based on the framework:
| Framework | Model Type | Approach |
|-----------|------------|----------|
| sklearn, xgboost, lightgbm, pytorch, tensorflow | Built-in | Direct `log_model()` |
| Hugging Face (transformers) | Built-in | `hugging-face-models/SKILL.md` |
| Other (pycaret, custom, etc.) | Custom | Requires `CustomModel` wrapper |
### Step 4: Generate Registration Code
Set up the session following your loaded environment guide, then generate the registration code.
**For in-memory model (model just trained in current session): only in snowsight/snowflake platform**
```python
import pandas as pd
from snowflake.ml.registry import Registry
# Session setup per environment guide
# e.g., get_active_session() or create_snowpark_session()
session = <SESSION_SETUP>
# setup database and schema
session.use_database("<DATABASE>")
session.use_schema("<SCHEMA>")
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
# Use the model variable from the current session (e.g., model, clf, regressor)
# Use training data as sample input (e.g., X_train.head(5))
sample_input = <SAMPLE_DATA_FROM_TRAINING> # e.g., X_train.head(5)
mv = reg.log_model(
<MODEL_VARIABLE>, # e.g., model, clf, xgb_model
model_name="<MODEL_NAME>",
version_name="<VERSION_NAME>", # e.g., "v1", "v2" - determined in Step 2
sample_input_data=sample_input,
conda_dependencies=["<FRAMEWORK>", "<OTHER_DEPS>"], # Snowflake conda channel (warehouse) or conda-forge (SPCS)
target_platforms=["WAREHOUSE", "SNOWPARK_CONTAINER_SERVICES"],
comment="<DESCRIPTION>"
)
print(f"Model registered: {mv.model_name} version {mv.version_name}")
```
**For model loaded from file:**
```python
import pandas as pd
from snowflake.ml.registry import Registry
# Session setup per environment guide
session = <SESSION_SETUP>
# setup database and schema
session.use_database("<DATABASE>")
session.use_schema("<SCHEMA>")
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
# Load model using framework-appropriate method
# sklearn/lightgbm (pickle): pickle.load() or joblib.load()
# xgboost (.ubj/.json): xgb.Booster(); booster.load_model()
# pytorch (.pt): torch.load()
# tensorflow (.h5): tf.keras.models.load_model()
model = <LOAD_MODEL_CODE>
sample_input = pd.DataFrame(<SAMPLE_DATA>)
mv = reg.log_model(
model,
model_name="<MODEL_NAME>",
version_name="<VERSION_NAME>", # e.g., "v1", "v2" - determined in Step 2
sample_input_data=sample_input,
conda_dependencies=["<FRAMEWORK>", "<OTHER_DEPS>"], # Snowflake conda channel (warehouse) or conda-forge (SPCS)
target_platforms=["WAREHOUSE", "SNOWPARK_CONTAINER_SERVICES"],
comment="<DESCRIPTION>"
)
print(f"Model registered: {mv.model_name} version {mv.version_name}")
```
**For Custom/Unsupported Model Types:**
```python
import pandas as pd
from snowflake.ml.registry import Registry
from snowflake.ml.model import custom_model
# Session setup per environment guide
session = <SESSION_SETUP>
model_context = custom_model.ModelContext(
model_file="<MODEL_FILE_PATH>"
)
class MyCustomModel(custom_model.CustomModel):
def __init__(self, context: custom_model.ModelContext) -> None:
super().__init__(context)
# Load model using framework-appropriate method
self.model = <LOAD_MODEL_CODE>
@custom_model.inference_api
def predict(self, input_df: pd.DataFrame) -> pd.DataFrame:
predictions = self.model.predict(input_df)
return pd.DataFrame({"prediction": predictions})
my_model = MyCustomModel(model_context)
sample_input = pd.DataFrame(<SAMPLE_DATA>)
output = my_model.predict(sample_input)
reg = Registry(session=session, database_name="<DATABASE>", schema_name="<SCHEMA>")
mv = reg.log_model(
my_model,
model_name="<MODEL_NAME>",
version_name="<VERSION_NAME>", # e.g., "v1", "v2" - determined in Step 2
sample_input_data=sample_input,
conda_dependencies=["<DEPS>"], # Snowflake conda channel (warehouse) or conda-forge (SPCS)
target_platforms=["WAREHOUSE", "SNOWPARK_CONTAINER_SERVICES"],
comment="<DESCRIPTION>"
)
print(f"Model registered: {mv.model_name} version {mv.version_name}")
```
### Step 5: Execute and Verify
**Snowsight (Notebook)**: Test model loading → test prediction → run registration → verify with `reg.show_models()`
**CLI (Script)**: Write complete script, then ask user confirmation before executing.
**⚠️ MANDATORY:** Present summary and wait for user approval before executing.
Follow the execution instructions in your loaded environment guide. If execution fails, read complete error, fix, and ask user again before re-executing.
## log_model() Parameters
| Parameter | Description | Required |
|-----------|-------------|----------|
| `model` | Python model object | Yes |
| `model_name` | Name in registry | Yes |
| `version_name` | Version identifier | Recommended |
| `sample_input_data` | DataFrame for schema inference | Yes* |
| `conda_dependencies` | List of conda packages (for warehouse) | See below |
| `pip_requirements` | List of pip packages (requires artifact_repository_map for warehouse) | See below |
| `target_platforms` | Target deployment platforms | See below |
| `artifact_repository_map` | Map of package indexes for non-conda packages | See below |
*Or provide `signatures` instead.
## Dependencies for Warehouse vs SPCS
**For WAREHOUSE target:**
- Use `conda_dependencies` for packages in Snowflake conda channel
- OR use `pip_requirements` + `artifact_repository_map` for PyPI packages
**For SPCS only:**
- Can use `pip_requirements` directly without `artifact_repository_map`
- `conda_dependencies` are loaded from conda-forge (not Snowflake conda channel)
## target_platforms Strategy
**Default approach:** Try `["WAREHOUSE", "SNOWPARK_CONTAINER_SERVICES"]` first to enable both warehouse inference and SPCS deployment.
**Fallback:** If `log_model()` fails with warehouse target (e.g., due to unsupported dependencies or model size), retry with `["SNOWPARK_CONTAINER_SERVICES"]` only.
```python
# First attempt: try both platforms (use conda_dependencies for warehouse compatibility)
try:
mv = reg.log_model(
model,
model_name="<MODEL_NAME>",
version_name="<VERSION>",
sample_input_data=sample_input,
conda_dependencies=["<DEPS>"], # Snowflake conda channel (warehouse) or conda-forge (SPCS)
target_platforms=["WAREHOUSE", "SNOWPARK_CONTAINER_SERVICES"],
)
except Exception as e:
# Fallback: SPCS only (can use pip_requirements directly)
mv = reg.log_model(
model,
model_name="<MODEL_NAME>",
version_name="<VERSION>",
sample_input_data=sample_input,
pip_requirements=["<DEPS>"],
target_platforms=["SNOWPARK_CONTAINER_SERVICES"],
)
```
## Using artifact_repository_map for Non-Conda Packages
When your model depends on packages **not available in the Snowflake conda channel**, use `artifact_repository_map` to specify PyPI as the package source.
Use the shared `pypi_shared_repository` for public PyPI packages:
```python
mv = reg.log_model(
model,
model_name="<MODEL_NAME>",
version_name="<VERSION>",
sample_input_data=sample_input,
pip_requirements=["scikit-learn", "shap>=0.42.0"],
target_platforms=["WAREHOUSE", "SNOWPARK_CONTAINER_SERVICES"],
artifact_repository_map={
"shap": "pypi_shared_repository" # Map non-conda packages to PyPI
},
)
```
- **Keys**: Package names (must also be listed in `pip_requirements`)
- **Values**: Use `pypi_shared_repository` for public PyPI packages
## Common Issues (Workflow A)
- **Version exists**: Use `SHOW MODELS LIKE '<MODEL_NAME>' IN SCHEMA <DATABASE>.<SCHEMA>` to check if model and versions exist. Increment version or rename
- **Not serializable**: Ensure saved with `pickle.dump()` or `joblib.dump()`
- **Schema inference fails**: Provide explicit `signatures`
- **Package not found in Snowflake channel**: Use `artifact_repository_map` to specify PyPI or custom repository (see [Using artifact_repository_map](#using-artifact_repository_map-for-non-conda-packages))
- **Inference errors after registration**: For inference issues (dtype errors, TypeError, service failures, OOM), see `../debug-inference/SKILL.md`
### Step 6: Post-Registration Verification
**⚠️ MANDATORY:** After registration completes, verify the model was registered correctly before proceeding.
**Run verification checks:**
```sql
-- 1. Verify model and version exist
SHOW MODELS LIKE '<MODEL_NAME>' IN SCHEMA <DATABASE>.<SCHEMA>;
-- 2. Check available functions/methods (only run after confirming model exists above — errors if model not found)
-- Include VERSION = '<VERSION>' to check a specific version; omit to use the default version
SHOW FUNCTIONS IN MODEL <DATABASE>.<SCHEMA>.<MODEL_NAME> VERSION '<VERSION>';
```
Confirm the model name and version appear in Step 1 output before running Step 2.
**Verification checklist:**
| Check | Expected Result |
|-------|-----------------|
| `SHOW MODELS LIKE` includes model | Model name and version listed in results (empty = not registered) |
| `SHOW FUNCTIONS IN MODEL` returns methods | At least one method (e.g., `PREDICT`, `PREDICT_PROBA`) |
**If verification fails:**
- Model not found: Check database/schema context, re-run registration
- Version not found: Registration may have failed silently, check for errors
- No functions: Sample input may have been invalid, re-register with correct schema
**⚠️ STOP**: Only proceed to next steps after all verification checks pass.
---
### Step 7: Next Steps
**If target_platforms includes WAREHOUSE:**
Ask user what they'd like to do:
1. **Test warehouse inference** - Run a sample prediction query
2. **Deploy to SPCS** - Create an inference service (Workflow B)
3. **Set up model monitoring** - Track drift and performance (load `../model-monitor/SKILL.md`)
4. **Done** - Finish here
**If target_platforms is SPCS only:**
Warehouse inference is not available. Ask user:
1. **Deploy to SPCS** - Create an inference service (Workflow B)
2. **Set up model monitoring** - Track drift and performance (load `../model-monitor/SKILL.md`)
3. **Done** - Finish here
**⚠️ STOP**: Wait for user response.
**If user chooses to test warehouse inference:**
**⚠️ Always specify the version explicitly.** Use the version from Step 2 (e.g., `V1`, `V2`)—do not rely on the default version.
Run a sample prediction using SQL or Python. Use the method name from the model (e.g., `PREDICT`, `PREDICT_PROBA`, `TRANSFORM`).
**SQL Syntax:**
Use `MODEL(model_name, version)!METHOD(...)` syntax. Version names are **unquoted identifiers**.
```sql
SELECT MODEL(<DATABASE>.<SCHEMA>.<MODEL_NAME>, <VERSION>)!<METHOD_NAME>(col1, col2, col3) AS result
FROM <INPUT_TABLE>
LIMIT 10;
-- Extract specific output field
SELECT MODEL(<DATABASE>.<SCHEMA>.<MODEL_NAME>, <VERSION>)!<METHOD_NAME>(col1, col2, col3):output_feature_0 AS result
FROM <INPUT_TABLE>;
```
**⚠️ Important:** Do NOT quote version names. Use `V2` not `'V2'`.
**Python:**
```python
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
# Check available methods
print(mv.show_functions())
# Run inference
result = mv.run(test_data, function_name="<method_name>")
print(result)
```
**If user chooses SPCS deployment:** Proceed to Workflow B.
**When to use Warehouse vs SPCS Inference:**
| Use Case | Recommendation |
|----------|----------------|
| Ad-hoc queries, testing | Warehouse inference |
| Batch predictions | Warehouse inference |
| Real-time API endpoint | SPCS inference (Workflow B) |
| High-throughput, low-latency | SPCS inference (Workflow B) |
## Output
- Model registered in Snowflake Model Registry
- Model name and version for reference
- Ready for warehouse inference (SQL) or SPCS deployment (load `../spcs-inference/SKILL.md`)
## Sub-Skills
### partitioned-inference
Partitioned inference using `@partitioned_api` decorator. Deploy models that run different submodels per data partition (store, region, etc.). Supports both stateless (train+predict together) and stateful (pre-trained models via `ModelContext`) workflows.
**When to route here:**
- User wants to run inference with different models per partition
- User mentions `@partitioned_api`, `partitioned_inference_api`, or `CustomModel` with partitions
- User has trained models per partition (via MMT) and wants to deploy for inference
scripts/snowpark_session.py
"""
Snowpark session helper — create a Snowpark Session from local Snowflake CLI config.
Usage as a module (import in your scripts or notebooks):
from snowpark_session import create_snowpark_session
session = create_snowpark_session()
Usage as a CLI (test connectivity):
python snowpark_session.py
python snowpark_session.py --connection my_conn
python snowpark_session.py --test
Reads ~/.snowflake/connections.toml (or config.toml fallback) and handles all
authentication methods including private_key_path, externalbrowser, token, etc.
Respects:
$SNOWFLAKE_HOME — config directory (default: ~/.snowflake)
$SNOWFLAKE_CONNECTION_NAME — override connection name
$SNOWFLAKE_DEFAULT_CONNECTION_NAME — fallback connection name
"""
from __future__ import annotations
import json
import os
import sys
from pathlib import Path
from typing import Optional
from snowflake.snowpark import Session
# ---------------------------------------------------------------------------
# Keys that Snowpark Session.builder.configs() accepts.
# Unknown keys cause errors, so we filter the TOML config to this set.
# ---------------------------------------------------------------------------
_SNOWPARK_ALLOWED_KEYS = {
"account",
"user",
"password",
"authenticator",
"host",
"port",
"protocol",
"role",
"database",
"schema",
"warehouse",
"token",
"private_key",
}
# Path to Cortex Code agent settings (contains active connection name)
_AGENT_SETTINGS_PATH = Path("~/.snowflake/cortex/settings.json").expanduser()
# ---------------------------------------------------------------------------
# TOML loading — uses tomllib (Python 3.11+) or tomli (Python 3.10)
# ---------------------------------------------------------------------------
def _load_toml(path: Path) -> dict:
"""Load a TOML file and return its contents as a dict."""
try:
import tomllib
except ModuleNotFoundError:
try:
import tomli as tomllib # type: ignore[no-redef]
except ModuleNotFoundError:
raise ImportError(
"TOML parsing requires the 'tomli' package on Python < 3.11. "
"Install it with: pip install tomli"
)
with open(path, "rb") as f:
return tomllib.load(f)
# ---------------------------------------------------------------------------
# Config file discovery
# ---------------------------------------------------------------------------
def _load_all_connections(snowflake_home: Path) -> tuple[dict[str, dict], Optional[str]]:
"""Load all connection configs and default_connection_name from TOML files.
Priority:
1. connections.toml (flat structure, each top-level key is a connection)
2. config.toml (connections under [connections] section)
Returns:
(connections_dict, default_connection_name_or_None)
"""
connections_path = snowflake_home / "connections.toml"
config_path = snowflake_home / "config.toml"
if connections_path.exists():
data = _load_toml(connections_path)
default_name = data.get("default_connection_name")
connections = {
k: v for k, v in data.items()
if k != "default_connection_name" and isinstance(v, dict)
}
# Also check config.toml for default_connection_name if not in connections.toml
if default_name is None and config_path.exists():
cfg = _load_toml(config_path)
default_name = cfg.get("default_connection_name")
return connections, default_name
if config_path.exists():
data = _load_toml(config_path)
default_name = data.get("default_connection_name")
connections = data.get("connections", {})
return connections, default_name
raise FileNotFoundError(
f"No connections.toml or config.toml found in {snowflake_home}. "
f"Configure a connection with: snow connection add"
)
# ---------------------------------------------------------------------------
# Connection name resolution
# ---------------------------------------------------------------------------
def _read_agent_connection_name() -> Optional[str]:
"""Read the active connection name from Cortex Code agent settings."""
if not _AGENT_SETTINGS_PATH.exists():
return None
try:
data = json.loads(_AGENT_SETTINGS_PATH.read_text())
return data.get("cortexAgentConnectionName")
except (json.JSONDecodeError, OSError):
return None
def _resolve_connection_name(
explicit: Optional[str],
default_from_toml: Optional[str],
available: list[str],
) -> str:
"""Resolve which connection name to use.
Priority: explicit arg > $SNOWFLAKE_CONNECTION_NAME >
$SNOWFLAKE_DEFAULT_CONNECTION_NAME > TOML default >
agent settings > first available connection.
"""
name = (
explicit
or os.getenv("SNOWFLAKE_CONNECTION_NAME")
or os.getenv("SNOWFLAKE_DEFAULT_CONNECTION_NAME")
or default_from_toml
or _read_agent_connection_name()
)
if name and name in available:
return name
if name and name not in available:
raise KeyError(
f"Connection '{name}' not found. Available: {available}"
)
# Fall back to first available
if available:
return available[0]
raise KeyError("No connections found in Snowflake config files.")
# ---------------------------------------------------------------------------
# Auth handling
# ---------------------------------------------------------------------------
def _resolve_private_key(config: dict) -> dict:
"""Load private key from file path if specified, handling PEM and DER formats."""
pk_path = (
config.pop("private_key_path", None)
or config.pop("private_key_file", None)
or config.pop("privatekeypath", None)
)
if not pk_path:
return config
from cryptography.hazmat.primitives import serialization
key_path = Path(pk_path).expanduser()
if not key_path.exists():
raise FileNotFoundError(
f"Private key file not found: {key_path}. "
f"Check private_key_path in your Snowflake connection config."
)
passphrase = config.pop("private_key_passphrase", None)
password = passphrase.encode() if passphrase else None
key_data = key_path.read_bytes()
# Detect PEM vs DER format
if b"-----BEGIN" in key_data:
private_key = serialization.load_pem_private_key(key_data, password=password)
else:
private_key = serialization.load_der_private_key(key_data, password=password)
config["private_key"] = private_key
return config
def _resolve_token_file(config: dict) -> dict:
"""Read token from token_file_path if specified (used in SPCS / container environments)."""
token_file = config.pop("token_file_path", None)
if token_file and not config.get("token"):
token_path = Path(token_file)
if token_path.exists():
config["token"] = token_path.read_text().strip()
return config
# ---------------------------------------------------------------------------
# Main API
# ---------------------------------------------------------------------------
def create_snowpark_session(connection_name: Optional[str] = None) -> Session:
"""Create a Snowpark session from local Snowflake CLI config files.
Handles all authentication methods (password, externalbrowser, private_key,
token, etc.) and filters config to only keys that Snowpark accepts.
Args:
connection_name: Explicit connection name. If None, resolved from env
vars, TOML defaults, or agent settings.
Returns:
A connected Snowpark Session.
"""
snowflake_home = Path(
os.environ.get("SNOWFLAKE_HOME", "~/.snowflake")
).expanduser()
all_connections, default_name = _load_all_connections(snowflake_home)
conn_name = _resolve_connection_name(
explicit=connection_name,
default_from_toml=default_name,
available=list(all_connections.keys()),
)
raw_config = dict(all_connections[conn_name])
# Handle auth-specific keys before filtering
raw_config = _resolve_private_key(raw_config)
raw_config = _resolve_token_file(raw_config)
# Filter to only keys Snowpark accepts — prevents errors from unknown keys
config = {k: v for k, v in raw_config.items() if k in _SNOWPARK_ALLOWED_KEYS}
return Session.builder.configs(config).create()
# ---------------------------------------------------------------------------
# CLI entry point — for testing connectivity
# ---------------------------------------------------------------------------
def main() -> None:
import argparse
parser = argparse.ArgumentParser(
description="Test Snowflake Snowpark connectivity using local config."
)
parser.add_argument(
"--connection", "-c",
help="Connection name from connections.toml / config.toml",
)
parser.add_argument(
"--test", "-t",
action="store_true",
help="Run a test query (SELECT CURRENT_USER(), CURRENT_ROLE())",
)
args = parser.parse_args()
try:
print(f"Creating Snowpark session...")
session = create_snowpark_session(connection_name=args.connection)
print(f"✅ Connected successfully!")
print(f" Account: {session.get_current_account()}")
print(f" User: {session.get_current_user()}")
print(f" Role: {session.get_current_role()}")
print(f" Database: {session.get_current_database()}")
print(f" Schema: {session.get_current_schema()}")
print(f" Warehouse: {session.get_current_warehouse()}")
if args.test:
print("\nRunning test query: SELECT 1 AS test_col")
result = session.sql("SELECT 1 AS test_col").collect()
print(f" Result: {result}")
session.close()
except Exception as e:
print(f"❌ Error: {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
SKILL.md
---
name: machine-learning
description: "**[REQUIRED]** For **ALL** data science and machine learning tasks. This skill should ALWAYS be loaded in even if only a portion of the workflow is related to machine learning. Use when: analyzing data, training models, deploying models to Snowflake, registering models, working with ML workflows, running ML jobs on Snowflake compute, model registry, model service, model inference, log model, deploy pickle file, experiment tracking, model monitoring, ML observability, tracking drift, model performance analysis, distributed training, XGBoost, LightGBM, PyTorch, DPF, distributed partition function, many model training, hyperparameter tuning, HPO, compute pools, train at scale, feature store, feature views, entities, training datasets, online features, pipeline orchestration, DAG, task graph, schedule training. Routes to specialized sub-skills."
---
# Data Science & Machine Learning Skills
This skill routes to specialized sub-skills for data science and machine learning tasks.
This skill provides valuable information about all sorts of data science, machine learning, and mlops tasks.
It MUST be loaded in if any part of the user query relates to these topics❗❗❗
## Step 0: Load Environment Guide
**⚠️ CRITICAL: Before routing to any sub-skill, you MUST load the environment guide for your surface.**
Your system prompt indicates which surface you are operating on. Load the matching guide:
| Surface | Condition | Guide to Load |
|---------|-----------|---------------|
| **Snowsight** | You are operating inside the Snowflake Snowsight web interface | `guides/snowsight-environment.md` |
| **CLI / IDE** | You are operating in a command line terminal or IDE environment | `guides/cli-environment.md` |
The environment guide provides surface-specific instructions for **session setup, package management, and code execution** that apply to ALL sub-skills below. Sub-skills will reference these patterns rather than repeating them.
## Routing Behavior
**⚠️ CRITICAL: Route AUTOMATICALLY based on the user's request. Do NOT ask the user which sub-skill to use or how they want to deploy.**
When a user asks to "train a model", "build a model" or inquires about a similar task:
- **IMMEDIATELY** load `ml-development/SKILL.md` and start working
- Do NOT ask about deployment options upfront
- Do NOT ask "Local only vs Register in Snowflake vs End-to-end"
- Training and deployment are SEPARATE tasks - handle them sequentially if needed
## Intent Detection
### Dynamic Service Detection (Model Inference Services)
**⚠️ CRITICAL:** When a user mentions a **service name**, check if it's a model inference service:
1. Run `DESCRIBE SERVICE <DB>.<SCHEMA>.<SERVICE_NAME>`
2. If `managing_object_domain = 'Model'` → Route to `spcs-inference/SKILL.md`
This applies to ANY task involving the service (testing, REST API calls, latency profiling, benchmarking, debugging, management).
---
### Disambiguation: batch-inference vs spcs-inference (Online)
**⚠️ CRITICAL:** When user mentions "inference" without clear signals, you MUST ask for clarification.
There is a decision matrix located in the public docs `https://docs.snowflake.com/en/developer-guide/snowflake-ml/inference/inference-overview`.
**Inference Disambiguation Workflow:**
When user says something like "run inference on my model" or "inference" without batch/online signals:
```
I can help you run inference on your model. There are three approaches:
1. **Native Batch Inference (SQL)** - Embed inference in SQL pipelines
- <add decision points from docs matrix here>
2. **Job-Based Batch (SPCS)** - Run large-scale inference jobs
- <add decision points from docs matrix here>
3. **Real-Time Inference (SPCS)** - Deploy a REST endpoint
- <add decision points from docs matrix here>
Which approach fits your use case?
```
**⚠️ STOP**: Wait for user response before routing.
### Disambiguation: batch-inference vs ml-jobs
**⚠️ CRITICAL:** These two skills are commonly confused. Use this logic:
| User Intent | Key Signals | Route To |
|-------------|-------------|----------|
| Run inference on a **registered model** | "model registry" + ("inference", "predictions", "scoring", "run()", "run_batch()") | `batch-inference-jobs/SKILL.md` |
| Run a **Python script** on Snowflake compute | "script", "submit", "file", "directory", training code | `ml-jobs/SKILL.md` |
**Decision tree:**
1. Does the user want to run inference on an **existing model in the registry**?
- **YES** → `batch-inference-jobs/SKILL.md` (covers both `mv.run()` and `mv.run_batch()`)
- **NO** → Continue to step 2
2. Does the user want to run **custom Python code** (training, processing, or scripts) on Snowflake compute?
- **YES** → `ml-jobs/SKILL.md` (uses `submit_file()` or `submit_directory()`)
- **NO** → Continue to step 3
3. Does the user want to **orchestrate multiple steps** on a schedule (pipeline, DAG)?
- **YES** → `ml-pipeline-orchestration/SKILL.md`
- **NO** → Ask clarifying question
### Disambiguation: ml-jobs vs ml-pipeline-orchestration
| User Intent | Key Signals | Route To |
|-------------|-------------|----------|
| Run a **single ML job** | "submit", "run script", "compute pool", one-off execution | `ml-jobs/SKILL.md` |
| **Orchestrate multiple steps** on a schedule | "pipeline", "DAG", "schedule", "automate", "task graph", multi-step workflow | `ml-pipeline-orchestration/SKILL.md` |
### Disambiguation: Partitioned Modeling (Full Workflow)
**⚠️ CRITICAL:** When user says "partitioned modeling", "partitioned model", "model per partition", or similar ambiguous phrases, clarify their stage in the workflow:
**Partitioned Modeling Workflow:**
```
[1] Train models per partition → [2] Register → [3] Run partitioned inference
(MMT) (Registry) (@partitioned_inference_api)
```
**Decision tree:**
1. Does the user need to **train** models per partition?
- **YES** → Load `distributed-training/SKILL.md` (Many Model Training section)
- After training completes, ask if they want to proceed to inference
2. Does the user already have trained models and needs **inference**?
- **YES** → Load `model-registry/partitioned-inference/SKILL.md`
3. Does the user want the **full end-to-end workflow**?
- **YES** → Load `distributed-training/SKILL.md` first, then `model-registry/partitioned-inference/SKILL.md`
**Clarification prompt (when ambiguous):**
```
I can help with partitioned modeling. Where are you in the workflow?
1. **Train models per partition** - Use Many Model Training (MMT) to train one model per partition (store, region, etc.)
2. **Run partitioned inference** - Already have trained models, need to run predictions per partition
3. **Full workflow** - Train per partition → Register → Run partitioned inference
Which step do you need help with?
```
**⚠️ STOP**: Wait for user response before routing.
### Routing Table
| User Says | Route To | Action |
|-----------|----------|--------|
| "analyze data", "train model", "build model", "feature engineering", "predict", "classify", "regression" | `ml-development/SKILL.md` | Load immediately, start training |
| "register model", "model registry", "log model", "pickle to snowflake", "save model to snowflake", "upload model", ".pkl file", ".ubj file" | `model-registry/SKILL.md` | Load immediately, start registration (Workflow A) |
| "deploy model", "deploy model for inference", "deploy for inference" | `model-registry/SKILL.md` | Load immediately, ask deployment target (Workflow B) |
| "create inference service", "inference endpoint", "serve model", "snowpark container services", "model endpoint", "deploy in container", "deploy model service", "real-time inference", "online inference" | `spcs-inference/SKILL.md` | Load immediately, create SPCS service |
| "batch inference", "bulk predictions", "run_batch", "run()", "offline scoring", "score dataset", "batch predictions", "inference on registered model", "run predictions on registry model", "score with registered model", "offline inference", "SQL inference", "dbt inference", "dynamic table inference" | `batch-inference-jobs/SKILL.md` | Load immediately, set up batch inference |
| **"inference", "run inference"** (ambiguous, no batch/online signals) | **ASK USER** | Use disambiguation workflow above to clarify batch vs online |
| "ml job", "ml jobs", "run on snowflake compute", "submit job", "submit script", "submit file", "remote execution", "GPU training", "run python script on snowflake" | `ml-jobs/SKILL.md` | Load immediately, set up job |
| "pipeline", "DAG", "task graph", "schedule training", "schedule inference", "orchestrate", "productionize", "automate retraining", "convert notebook to pipeline" | `ml-pipeline-orchestration/SKILL.md` | Load immediately, set up DAG |
| "experiment tracking", "track experiment", "log metrics", "log parameters", "autolog", "training callback", "XGBoost callback", "LightGBM callback" | `experiment-tracking/SKILL.md` | Load immediately, set up experiment tracking |
| "model monitor", "monitor model", "add monitoring", "enable monitoring", "ML observability", "track drift", "model performance", "monitor predictions", "observability" | `model-monitor/SKILL.md` | Load immediately, set up monitoring |
| "inference logs", "inference table", "captured inference", "autocapture data", "view inference history", "INFERENCE_TABLE", "inference requests", "inference responses", "view captured predictions" | `inference-logs/SKILL.md` | Load immediately, query inference data |
| "inference error", "mv.run fails", "service failing", "OOM", "debug inference", "inference not working" | `debug-inference/SKILL.md` | Load immediately, diagnose issue |
| "distributed training", "distributed XGBoost", "distributed LightGBM", "XGBEstimator", "LightGBMEstimator", "PyTorchDistributor", "multi-node training", "multi-GPU training", "train at scale", "DPF", "distributed partition function", "many model training", "MMT", "train per partition", "ManyModelTraining", "partition by", "hyperparameter tuning", "hyperparameter optimization", "HPO", "Tuner", "TunerConfig", "search space", "grid search", "random search", "bayesian optimization", "tune model", "tune hyperparameters", "num_trials", "search_alg" | `distributed-training/SKILL.md` | Load immediately, distributed training/processing/tuning |
| "partitioned inference", "@partitioned_api", "inference per partition", "CustomModel partition" | `model-registry/partitioned-inference/SKILL.md` | Load immediately, partitioned inference |
| **"partitioned modeling", "partitioned model", "model per partition", "per-partition models"** (ambiguous) | **ASK USER** | Use partitioned modeling disambiguation above |
| "feature store", "feature view", "entity", "training data", "generate_training_set", "generate_dataset", "online features", "feature engineering for ML", "point-in-time features", "ASOF join" | `feature-store/SKILL.md` | Load immediately, route to feature store sub-skill |
**Sub-skill path aliases** (for routing resolution):
- `ml-job` → `ml-jobs/SKILL.md` (singular form routes to plural directory)
- `ml-jobs` → `ml-jobs/SKILL.md`
- `mljob` → `ml-jobs/SKILL.md`
- `mljobs` → `ml-jobs/SKILL.md`
## Workflow
```markdown
User Request → Load Environment Guide → Detect Intent → Load appropriate sub-skill → Execute
Examples:
- "Train a classifier" → Load ml-development → Train locally → Done
- "Deploy my model.pkl" → Load model-registry → Register to Snowflake → Done
- "Train AND deploy" → Load ml-development → Train → Save model → Report artifacts → Ask about deployment → If yes, load model-registry WITH CONTEXT (file path, framework, schema)
```
**Key principle**: Complete ONE task at a time. Only ask about the next step after the current step is done.
## Context Preservation Between Skills
**⚠️ CRITICAL:** When transitioning from ml-development to model-registry:
**Information to preserve and pass along:**
- Model file path (absolute path to serialized model file)
- Framework used (sklearn, xgboost, lightgbm, pytorch, tensorflow, etc.)
- Sample input schema (columns and types from training data)
- Any other relevant training context
**Why this matters:**
- Avoids asking the user to repeat information they just provided
- Prevents accidental retraining of the model
- Prevents modification of the training script
- Improves user experience with seamless workflow
**How to do it:**
1. When ml-development saves a model, it reports all details
2. When loading model-registry, explicitly mention this context
3. Model-registry checks for this context before asking questions
4. Use the preserved context instead of asking user again
**Example handoff:**
```markdown
ml-development: "Model saved to /path/to/model.pkl (sklearn). Would you like to register it?"
User: "Yes"
[Load model-registry with context: path=/path/to/model.pkl, framework=sklearn, schema=[...]]
model-registry: "I see you just trained a sklearn model. What should I call it in Snowflake?"
```
## Sub-Skills
### ml-development
Data exploration, statistical analysis, model training, and evaluation. Covers the full ML development workflow from data loading to model evaluation.
### model-registry
Deploy serialized models to Snowflake Model Registry. Supports various model formats (`.pkl`, `.ubj`, `.json`, `.pt`, etc.) depending on framework. Routes to `spcs-inference` sub-skill for inference service creation. Includes `partitioned-inference` sub-skill for partition-aware model deployment.
### experiment-tracking
Skills for tracking model training experiments using Snowflake's experiment tracking framework.
### spcs-inference
Deploy registered models to Snowpark Container Services for real-time inference. Handles compute pool selection, GPU/CPU configuration, num_workers, and service creation.
### batch-inference-jobs
Run batch inference on models **already registered** in the Snowflake Model Registry. Covers **two approaches**:
- **Native SQL Batch** (`mv.run()`): Warehouse-based, integrates with SQL pipelines
- **Job-based Batch** (`mv.run_batch()`): SPCS compute pools, for large-scale and unstructured data
### ml-jobs
Transform local **Python scripts** into Snowflake ML Jobs that run on Snowflake compute pools. Uses `submit_file()` or `submit_directory()`. Also includes compute pool reference (instance families, sizing).
### ml-pipeline-orchestration
Orchestrate multi-step ML workflows using Snowflake Task Graphs (DAGs) with the Python DAG API. Covers DAG creation, scheduling (Cron/timedelta), inter-task data passing, and notebook-to-pipeline conversion. Uses `@remote` for ML tasks on compute pools and warehouse tasks for orchestration.
### model-monitor
Set up ML Observability for models in the Snowflake Model Registry. Track drift, performance metrics, and prediction statistics over time.
### distributed-training
**Consolidated skill** covering all distributed ML training, processing, and tuning:
- **Distributed Estimators**: `XGBEstimator`, `LightGBMEstimator`, `PyTorchDistributor` for training one large model across nodes/GPUs
- **Many Model Training (MMT)**: Train one model per partition with auto-serialization and `get_model()`
- **DPF (Distributed Partition Function)**: General-purpose distributed processing for custom workflows
- **Tuner API**: Distributed hyperparameter tuning (Ray Tune on SPCS) with RandomSearch, GridSearch, BayesOpt
> **Note**: These APIs run server-side — either inside ML Jobs (submitted via CLI) or in Snowflake Notebooks with Container Runtime (Snowsight). For CLI submission, see ml-jobs.
### partitioned-inference (under model-registry)
Partitioned inference in the Model Registry using `@partitioned_api` decorator. Run inference with different submodels per data partition. Located at `model-registry/partitioned-inference/SKILL.md`.
### feature-store
Centralized feature management for ML workflows. Create feature stores, define entities, build managed (Dynamic Table) and external (View) feature views, generate training datasets with point-in-time correctness, and enable online feature serving for low-latency inference. Includes sub-skills for create, pipelines, training, online, monitor, lineage, and migrate.
### inference-logs
Query and analyze captured inference data from model services with Auto-Capture enabled. View historical request/response data logged via `INFERENCE_TABLE()`. Useful for debugging unexpected predictions, building retraining datasets, and A/B testing model versions.
## Reminders & Common Mistakes
### ❌ Don't assume a database/schema — always ask
When the workflow involves creating or writing to any Snowflake object (table, stage, model registry entry, experiment, etc.), **never silently pick a database/schema**. Always confirm with the user first.
- If a `DATABASE.SCHEMA` has already been used in this session, offer it as the default:
```
I'll need to create [object] in Snowflake. I see we've been working with `<DATABASE>.<SCHEMA>`.
Should I use that, or would you prefer a different database/schema?
```
- If no database/schema has been used yet, ask explicitly:
```
Which database and schema should I use for [object]? (format: DATABASE.SCHEMA)
```
- **Carry the confirmed choice forward** — reuse it for subsequent objects in the session, but still confirm each time.
- **⚠️ Personal databases (e.g. `USER$VINAY`) are not supported** for ML workflows. If the user picks a personal database, warn them:
```
Personal databases like `USER$<USERNAME>` don't support creating tables, model registry operations, or inference services. Please provide a standard database/schema instead.
```
- **⚠️ STOP**: Wait for the user's response before proceeding with any object creation.
spcs-inference/SKILL.md
---
name: spcs-inference
description: "Deploy models from Snowflake Model Registry to Snowpark Container Services for real-time inference. Use when: creating inference services, SPCS deployment, REST endpoints for models, GPU inference. Triggers: create inference service, SPCS inference, inference endpoint, serve model, deploy to SPCS, model endpoint."
parent_skill: model-registry
---
# SPCS Inference Service Deployment
Deploy a registered model to Snowpark Container Services for real-time inference.
## ⚠️ CRITICAL: Environment Guide Check
**Before proceeding, check if you already have the environment guide (from `machine-learning/SKILL.md` → Step 0) in memory.** If you do NOT have it or are unsure, go back and load it now. The guide contains essential surface-specific instructions for session setup, code execution, and package management that you must follow.
---
## Prerequisites
- Model already registered in Snowflake Model Registry (see `../model-registry/SKILL.md`)
- Access to a compute pool (GPU or CPU)
- `BIND SERVICE ENDPOINT` privilege for HTTP endpoints
---
## Workflow: Create Inference Service
### Step 1: Identify the Model
If coming from model registration, use that model reference. Otherwise ask for:
- Model name and version
- Database/Schema where the model is registered
**⚠️ STOP**: Wait for response if not already known.
### Step 2: Choose Service Database and Schema
**Ask user:**
```
Which database and schema would you like to deploy the inference service in?
Note: This can be different from where the model is registered.
```
**⚠️ STOP**: Wait for user response.
### Step 3: Select Compute Pool
```sql
SHOW COMPUTE POOLS;
```
Present available compute pools to the user, indicating GPU vs CPU, nodes, and services running:
**Ask user:**
```
Available compute pools:
| Pool Name | Instance Family | GPUs/Node | Min/Max Nodes | Active Nodes | State | Services |
|-----------|-----------------|-----------|---------------|--------------|-------|----------|
| POOL_A | GPU_NV_M | 4 x A10G | 1 / 4 | 2 | ACTIVE | 2 |
| POOL_B | CPU_X64_M | None | 1 / 2 | 0 | SUSPENDED | 0 |
| ... | ... | ... | ... | ... | ... | ... |
Which compute pool would you like to use?
Recommendation: Use a GPU compute pool for models that require GPU inference
(e.g., deep learning, transformers, large embeddings).
```
**⚠️ STOP**: Wait for user response.
**GPU Reference:**
| Instance Family | GPUs per Node | GPU Type |
|-----------------|---------------|----------|
| GPU_NV_S | 1 | A10G |
| GPU_NV_M | 4 | A10G |
| GPU_NV_L | 8 | A100 |
If no suitable pool exists, offer to create one:
```sql
CREATE COMPUTE POOL IF NOT EXISTS <POOL_NAME>
MIN_NODES = 1 MAX_NODES = <N>
INSTANCE_FAMILY = '<INSTANCE_FAMILY>'
AUTO_RESUME = TRUE;
```
### Step 4: Configure Max Instances
**Ask user:**
```
How many max instances would you like for the service?
Max instances controls horizontal scaling - each instance is a separate container
replica that can handle inference requests in parallel. More instances = higher
throughput for concurrent requests.
- 1 instance: Suitable for development/testing or low traffic
- 2+ instances: Recommended for production workloads expecting higher concurrent load
The service will automatically scale between min_instances and max_instances based on demand.
Enter max_instances (default: 1):
```
**⚠️ STOP**: Wait for user response.
### Step 5: Check Existing Service
```sql
SHOW SERVICES LIKE '<SERVICE_NAME>' IN SCHEMA <DATABASE>.<SCHEMA>;
```
If exists, ask user: rename, delete & recreate, or keep existing.
### Step 5b: Configure Auto-Capture (Inference Logging)
**Ask user:**
```
Would you like to enable Auto-Capture for this inference service?
Auto-Capture automatically logs every request and response processed by the service
into an inference table. This gives you:
- Historical inference data for debugging and analysis
- Real-world production data for retraining and improving models
- Data for A/B testing and shadow testing
Important notes:
- Auto-Capture is IMMUTABLE — you cannot enable or disable it on an existing service.
You must recreate the service to change this setting.
- Not supported for vLLM or HuggingFace inference engines.
Enable Auto-Capture? (Yes/No, default: No):
```
**⚠️ STOP**: Wait for user response.
**If user wants autocapture on a legacy model** (service creation fails with inference table error), guide them to clone the model first:
```sql
CREATE MODEL <NEW_MODEL_NAME> WITH VERSION <VERSION_NAME> FROM MODEL <OLD_MODEL_NAME> VERSION <OLD_VERSION>;
```
Then use the new cloned model for service creation with `autocapture=True`.
### Step 6: Create Service
**⚠️ MANDATORY:** Present summary and get user confirmation before executing:
```
Summary:
- Model: <MODEL_DATABASE>.<MODEL_SCHEMA>.<MODEL_NAME> (version <VERSION>)
- Service: <SERVICE_DATABASE>.<SERVICE_SCHEMA>.<SERVICE_NAME>
- Compute Pool: <COMPUTE_POOL> (GPU/CPU)
- Max Instances: <MAX_INSTANCES>
- GPU Requests: <VALUE or N/A>
- Auto-Capture: <Enabled/Disabled>
Proceed? (Yes/No)
```
**⚠️ STOP**: Wait for user confirmation before proceeding.
---
#### Service Creation Code
Set up the session following your loaded environment guide, then generate the service creation code.Service creation takes 5-15 minutes. You MUST:
Use `snowpark_session.py` from parent skill (`machine-learning/SKILL.md` → Session Setup Patterns). Copy the helper script to the working directory and import it.
**DO NOT** combine service creation with model registration in the same script.
**DO NOT** run service creation inline - it will timeout.
**For GPU compute pool:**
```python
from snowflake.ml.registry import Registry
# Session setup per environment guide
session = <SESSION_SETUP>
session.use_database("<SERVICE_DATABASE>")
session.use_schema("<SERVICE_SCHEMA>")
reg = Registry(session=session, database_name="<MODEL_DATABASE>", schema_name="<MODEL_SCHEMA>")
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
print("Creating service...")
mv.create_service(
service_name="<SERVICE_NAME>",
service_compute_pool="<COMPUTE_POOL>",
ingress_enabled=True,
gpu_requests="<MAX_GPUS_FOR_NODE>",
max_instances=<MAX_INSTANCES>,
autocapture=<True if user enabled Auto-Capture in Step 5b, otherwise omit this parameter>,
)
print("Service created successfully.")
```
**For CPU compute pool:**
```python
from snowflake.ml.registry import Registry
# Session setup per environment guide
session = <SESSION_SETUP>
session.use_database("<SERVICE_DATABASE>")
session.use_schema("<SERVICE_SCHEMA>")
reg = Registry(session=session, database_name="<MODEL_DATABASE>", schema_name="<MODEL_SCHEMA>")
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
print("Creating service...")
mv.create_service(
service_name="<SERVICE_NAME>",
service_compute_pool="<COMPUTE_POOL>",
ingress_enabled=True,
max_instances=<MAX_INSTANCES>,
autocapture=<True if user enabled Auto-Capture in Step 5b, otherwise omit this parameter>,
)
print("Service created successfully.")
```
---
#### Execution: CLI
**⚠️ CRITICAL - MUST FOLLOW THIS PATTERN for CLI:**
Service creation takes 5-15 minutes. You MUST:
1. Write the code to a **separate Python script file** (e.g., `/path/to/create_service.py`)
2. Execute it in **background mode**
3. Monitor via **SQL** while it runs (Step 7)
**DO NOT** combine service creation with model registration in the same script.
**DO NOT** run service creation inline - it will timeout.
Execute the script using the Bash tool with `run_in_background=true`:
```
Tool: Bash
Command: SNOWFLAKE_CONNECTION_NAME=<connection> python /absolute/path/to/create_service.py
run_in_background: true
```
This returns a `shell_id` immediately. Use `bash_output` tool with that `shell_id` to check progress.
#### Execution: Snowsight (Notebook)
Run the service creation code in a notebook cell. Service creation takes 5-15 minutes. Wait for the cell execution to finish
---
### Step 7: Monitor Service Status (CLI Only)
**⚠️ CRITICAL:** While the service creation runs:
1. **Check script progress** (CLI: use `bash_output` tool with the `shell_id`)
2. **Monitor service status** using SQL
Service creation typically takes 5-15 minutes. Poll every 60 seconds.
---
**Check service status (SQL):**
```sql
DESCRIBE SERVICE <SERVICE_DATABASE>.<SERVICE_SCHEMA>.<SERVICE_NAME>;
```
**Or use pattern matching for multiple services:**
```sql
SHOW SERVICES LIKE '<SERVICE_NAME>' IN SCHEMA <SERVICE_DATABASE>.<SERVICE_SCHEMA>;
```
**Service Status Reference:**
| Status | Meaning | Action |
|--------|---------|--------|
| `PENDING` | Service registered, waiting for resources | Wait 60s, poll again |
| `STARTING` | Containers being pulled and started | Wait 60s, poll again |
| `RUNNING` | Service is ready for inference | Success - proceed to Step 8 |
| `FAILED` | Deployment failed | Fetch logs (see below) |
| `SUSPENDED` | Service suspended | Run `ALTER SERVICE <SERVICE_NAME> RESUME;` |
---
**When status is RUNNING, get endpoint URL:**
```sql
SHOW ENDPOINTS IN SERVICE <SERVICE_DATABASE>.<SERVICE_SCHEMA>.<SERVICE_NAME>;
```
Report the `ingress_url` to user.
---
**Check instance-level status (optional, more detail):**
```sql
SELECT SYSTEM$GET_SERVICE_STATUS('<SERVICE_DATABASE>.<SERVICE_SCHEMA>.<SERVICE_NAME>');
```
Returns JSON with each container's status. Look for `"status":"READY"` and `"message":"Running"` on all instances before testing.
---
**If status is FAILED, fetch logs for debugging:**
```sql
CALL SYSTEM$GET_SERVICE_LOGS('<SERVICE_DATABASE>.<SERVICE_SCHEMA>.<SERVICE_NAME>', 0, 'model-inference');
```
**⚠️ CRITICAL: Route to debug-inference on any error.** After fetching logs, if they contain ANY error (e.g., `AttributeError`, pickle/deserialization failures, `TypeError`, `ImportError`, `OOMKilled`, ufunc errors, or any other runtime exception), **immediately load `../debug-inference/SKILL.md`** and follow its diagnostic workflow. Do NOT attempt to fix the issue directly without loading the debug skill first.
---
**Timeout:** If service hasn't reached RUNNING after ~20 minutes, inform user and provide manual check command.
### Step 8: Validate and Test Service
**⚠️ CRITICAL: Only test inference when service is fully ready**
Before testing or validating the inference endpoint:
1. Service status must be **RUNNING** (not PENDING or STARTING)
2. All instances must be ready - check with `SYSTEM$GET_SERVICE_STATUS()`:
```sql
SELECT SYSTEM$GET_SERVICE_STATUS('<DATABASE>.<SCHEMA>.<SERVICE_NAME>');
```
Verify all containers show `"status":"READY"` and `"message":"Running"`
3. If some instances show `"status":"PENDING"`, wait for them to become READY
**Why this matters:** Testing with partial instances can cause timeouts or inconsistent results. The service load balancer may route requests to instances that are still starting up.
---
#### Step 8a: Confirm Model Version and Functions
**If coming from a continuous workflow** (Steps 1-8 in this session):
- Model name, version, and functions are already known from earlier steps
- Skip to Step 8b using the known values
**If validating an existing service without prior context:**
First, identify which model version the service is using:
```sql
DESCRIBE SERVICE <DATABASE>.<SCHEMA>.<SERVICE_NAME>;
```
Look at `managing_object_name` column to find the model (e.g., `SSARDANA_DB.NEW_EXP.MY_MODEL`).
Then verify the model exists and discover available functions:
**SQL:**
```sql
-- 1. Verify model exists (returns empty list if not found — safe, never errors)
SHOW MODELS LIKE '<MODEL_NAME>' IN SCHEMA <DATABASE>.<SCHEMA>;
-- 2. Discover available functions (only run after confirming model exists above — errors if model not found)
-- Include VERSION = '<VERSION>' if version is known; omit to use the default version
SHOW FUNCTIONS IN MODEL <DATABASE>.<SCHEMA>.<MODEL_NAME> VERSION '<VERSION>';
```
Look at the output from Step 2 for available function names (e.g., `PREDICT`, `PREDICT_PROBA`). Use one of these function names in all subsequent queries.
**Python:**
```python
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
print(mv.show_functions())
```
This returns a list of available functions with their input/output signatures.
**Reference:** [ModelVersion API Documentation](https://docs.snowflake.com/en/developer-guide/snowpark-ml/reference/latest/api/model/snowflake.ml.model.ModelVersion) for additional methods like `mv.get_model_signature()`.
**⚠️ IMPORTANT:** Do NOT assume `PREDICT` exists. Different models expose different inference methods.
---
#### Step 8b: Test Inference
**Use the function name discovered in Step 8a for all paths:**
**SQL:**
```sql
-- Replace <FUNCTION_NAME> with actual function from Step 8a
SELECT <SERVICE_NAME>!<FUNCTION_NAME>(col1, col2, ...) FROM input_table;
```
**Python (mv.run):**
```python
# function_name must match one from show_functions() (lowercase)
result = mv.run(test_data, function_name="<function_name>", service_name="<SERVICE_NAME>")
```
**REST API:**
There are two ways to call the REST API depending on where you're calling from:
| Calling From | Endpoint Type | Authentication Required |
|--------------|---------------|------------------------|
| **Snowsight Notebook** | Internal endpoint | No (session context) |
| **External (CLI, apps)** | Public ingress URL | Yes (PAT token) |
---
**REST API from Snowsight Notebook (No Auth Required):**
When calling from a Snowsight Notebook, use the **internal endpoint** which requires no authentication.
First, get the internal endpoint using `mv.list_services()`:
```python
# Get all services for this model version
services_df = mv.list_services()
print(services_df)
```
**`list_services()` returns a DataFrame with these columns:**
- `name`: The name of the service
- `status`: The status of the service
- `inference_endpoint`: The public endpoint (gives privatelink endpoint if session uses privatelink connection)
- `internal_endpoint`: The internal endpoint of the service (use this from notebooks!)
- `autocapture_enabled`: Whether service has autocapture enabled
**Reference:** [ModelVersion.list_services() API Documentation](https://docs.snowflake.com/en/developer-guide/snowpark-ml/reference/latest/api/model/snowflake.ml.model.ModelVersion)
Then make REST calls using the `internal_endpoint`:
```python
import requests
# Get the internal endpoint from list_services() output
services_df = mv.list_services()
internal_endpoint = services_df[services_df['name'] == '<SERVICE_NAME>']['internal_endpoint'].iloc[0]
# Internal endpoint format: http://<service-name>.<namespace>.svc.spcs.internal:5000
# Call the endpoint - NO authorization header needed!
url = f"{internal_endpoint}/<function_name>" # e.g., /predict or /predict-proba
payload = {"data": [[0, val1, val2, val3]]}
response = requests.post(url, json=payload)
print(response.json())
```
**Key points for Snowsight Notebook:**
- Use `mv.list_services()` to get the `internal_endpoint` column value
- Internal endpoints use HTTP (not HTTPS) on port 5000
- **No `Authorization` header needed** - the notebook session context handles authentication
- This only works from within Snowflake (notebooks, stored procedures, UDFs)
---
**REST API from External Clients (Auth Required):**
Requires: network policy allowing client IP, PAT token, service role grant.
See [REST API Access Setup](#rest-api-access-setup) for full setup instructions.
**⚠️ IMPORTANT: URL Path Transformation**
In REST URLs, underscores (`_`) in method names are replaced by dashes (`-`).
| Python Method | REST Endpoint |
|---------------|---------------|
| `predict()` | `/predict` |
| `predict_proba()` | `/predict-proba` |
| `predict_log_proba()` | `/predict-log-proba` |
| `my_custom_method()` | `/my-custom-method` |
```python
import requests
url = "https://<endpoint-url>/<function_name>" # Use function from Step 8a (with dashes)
headers = {"Authorization": "Snowflake Token=\"<PAT>\""}
response = requests.post(url, json={"data": [[0, val1, val2]]}, headers=headers)
```
**Note:** When calling from within Snowflake (e.g., a Snowflake notebook), no authentication headers are needed — the request is already authenticated by the session context.
See [REST API Access Setup](#rest-api-access-setup) for details.
#### Step 8b-ii: View Captured Inference Data
**⚠️ This step only applies when the service has Auto-Capture enabled.**
After running test inference, check if the service has autocapture enabled:
```python
services_df = mv.list_services()
print(services_df[['name', 'autocapture_enabled']])
```
If `autocapture_enabled` is `True` for this service, **ask the user:**
```
Your service has Auto-Capture enabled. Inference requests and responses are being
automatically logged to the model's inference table.
Would you like to view the captured inference data? (Yes/No)
```
**⚠️ STOP**: Wait for user response.
**If yes:** Load `../inference-logs/SKILL.md` and pass along the model name, version, and service name from this workflow as context.
**If no:** Continue to Step 8c.
---
#### Step 8c: Handle Inference Errors
**⚠️ CRITICAL:** If any test inference call in Step 8b fails (SQL error, HTTP 500, TypeError, ufunc error, dtype error, or any other runtime exception), **immediately load `../debug-inference/SKILL.md`** and follow its diagnostic workflow. Do NOT attempt to fix the issue directly — the debug skill has specific diagnosis and resolution paths for common inference failures (nullable dtype issues, pickle errors, OOM, etc.).
### Step 9: Setup REST API Access
**Ask user:**
```
Would you like to set up REST API access to call this service from outside Snowflake?
This is needed if you want to call the inference endpoint from external apps,
scripts, or services (not via SQL or Python SDK).
```
**⚠️ STOP**: Wait for user response.
**If yes:** Continue with the REST API Access Setup flow below.
**If no:** Deployment complete.
### Next Steps
Ask user:
```
Your inference service is running! What would you like to do next?
1. Set up model monitoring - Track drift and performance
2. Done - Finish here
```
**If monitoring:** Load `../model-monitor/SKILL.md`
**If done:** Skip to Service Management Reference.
---
## REST API Access Setup
To access the inference endpoint from outside Snowflake (e.g., external apps, services, or local scripts), you need proper authentication and network access configured.
### Network Policy (Required)
**Ask user:**
```
Do you have a network policy that allows your client IP to access Snowflake?
```
**⚠️ STOP**: Wait for user response.
**If yes:** Skip to [Service Role Grant](#service-role-grant).
**If no or unsure:** Users calling the endpoint need a network policy allowing their client IP. If user has ACCOUNTADMIN/SECURITYADMIN, help them create one:
```sql
-- Create network rule for client IP
CREATE NETWORK RULE <RULE_NAME> MODE = INGRESS TYPE = IPV4 VALUE_LIST = ('<CLIENT_IP>/32');
-- Create and apply policy
CREATE NETWORK POLICY <POLICY_NAME> ALLOWED_NETWORK_RULE_LIST = ('<RULE_NAME>');
ALTER USER <USERNAME> SET NETWORK_POLICY = <POLICY_NAME>;
```
### Service Role Grant
```sql
GRANT SERVICE ROLE <SERVICE_NAME>!ALL_ENDPOINTS_USAGE TO ROLE <ROLE_NAME>;
```
### Authentication (CLI Only)
**⚠️ CRITICAL: Always use PAT (Programmatic Access Token) for REST API authentication from CLI. Snowsight Notebooks do not require PAT - use the internal endpoint instead (see Step 8b).**
PAT tokens are the standard authentication method for SPCS REST endpoints. Create one in Snowsight under User Menu > Settings > Authentication.
**Do NOT attempt other authentication methods** (JWT, session tokens, etc.) - always ask the user for their PAT token first.
### Test with PAT
**⚠️ MANDATORY: Always ask for PAT token FIRST before any REST endpoint testing.**
**Step 1: Get the endpoint URL:**
```sql
SHOW ENDPOINTS IN SERVICE <SERVICE_NAME>;
```
**Step 2: Understand the model signature:**
Before testing, check the model's input/output signature to understand expected columns and data types:
```python
mv = reg.get_model("<MODEL_NAME>").version("<VERSION>")
print(mv.show_functions())
```
This returns the function signatures showing input columns, output columns, and their data types. See [ModelVersion API Documentation](https://docs.snowflake.com/en/developer-guide/snowpark-ml/reference/latest/api/model/snowflake.ml.model.ModelVersion) for details.
**Step 3: Ask user for PAT token:**
```
To test the REST endpoint, I need a PAT (Programmatic Access Token).
Please provide your PAT token. If you don't have one:
1. Go to Snowsight
2. Click your username (bottom left)
3. Go to Settings > Authentication
4. Create a new Programmatic Access Token
Please paste your PAT token:
```
**⚠️ STOP**: Wait for user to provide PAT token. Do NOT proceed without it.
**Step 4: Generate test script:**
```python
import requests
import json
url = "<ENDPOINT_URL>/<FUNCTION_NAME>"
pat = "<PAT_TOKEN>"
headers = {
"Authorization": f'Snowflake Token="{pat}"',
"Content-Type": "application/json"
}
payload = {"data": [[0, <SAMPLE_INPUT>]]}
response = requests.post(url, json=payload, headers=headers)
print(f"Status: {response.status_code}")
print(f"Response: {json.dumps(response.json(), indent=2)}")
```
### REST API Request Format
**⚠️ CRITICAL: The request payload format is specific and must be followed exactly.**
The SPCS inference REST API follows Snowflake's [External Functions Data Format](https://docs.snowflake.com/en/sql-reference/external-functions-data-format).
Each row is a JSON array where the **first element** is the row number (0-based index within the batch). The remaining elements contain the input data, which can be in one of two formats:
**Flat format** - Arguments as separate array elements:
```python
# Single row with 3 input columns (e.g., integer, string, timestamp)
payload = {"data": [[0, 10, "Alex", "2024-01-01 16:00:00"]]}
# Multiple rows
payload = {"data": [
[0, 10, "Alex", "2024-01-01 16:00:00"],
[1, 20, "Steve", "2024-02-01 16:00:00"],
[2, 30, "Alice", "2024-03-01 16:00:00"]
]}
```
**Wide format** - Arguments as a single dict object:
```python
# Single row with named columns
payload = {"data": [[0, {"col1": value1, "col2": value2, "col3": value3}]]}
# Multiple rows
payload = {"data": [
[0, {"feature1": 5.1, "feature2": 3.5, "feature3": 1.4}],
[1, {"feature1": 4.9, "feature2": 3.0, "feature3": 1.4}],
[2, {"feature1": 4.7, "feature2": 3.2, "feature3": 1.3}]
]}
```
**Common mistakes to avoid:**
```python
# WRONG - missing row index
payload = {"data": [[value1, value2, value3]]}
# Error: row index missing
# WRONG - data as dict instead of array of arrays
payload = {"data": {"col1": value1, "col2": value2}}
# Error: various parsing errors
# CORRECT - flat format
payload = {"data": [[0, value1, value2, value3]]}
# CORRECT - wide format
payload = {"data": [[0, {"col1": value1, "col2": value2}]]}
```
The row index (first element) is returned in the response, allowing you to match requests with responses when sending batches.
### REST API Response Format
**⚠️ CRITICAL: Responses are always in wide format (dict), regardless of request format.**
The SPCS inference REST API always returns responses in wide format. Each row contains the row index followed by a dict with the output values. The output column names and types are determined by the model signature (use `mv.show_functions()` to inspect).
**Standard response structure (always wide format):**
```json
{
"data": [
[0, {"output_feature_0": <value>, "output_feature_1": <value>}],
[1, {"output_feature_0": <value>, "output_feature_1": <value>}]
]
}
```
- `data[0][0]` = row index (matches the first element in your request)
- `data[0][1]` = dict containing output features
**Correct parsing pattern:**
```python
result = response.json()
data = result.get('data', [[]])[0]
# Output is always a dict at data[1]
if len(data) >= 2 and isinstance(data[1], dict):
output = data[1]
# Access specific output features based on model signature
value_0 = output.get('output_feature_0', 0.0)
value_1 = output.get('output_feature_1', 0.0)
```
**Common mistake to avoid:**
```python
# WRONG - data[1] is a dict, not a float!
result = float(data[1]) # TypeError: float() argument must be a string or real number, not 'dict'
# CORRECT - access the dict key
result = data[1].get('output_feature_0', 0.0)
```
**Debugging tip:** If you encounter parsing errors, first inspect the raw response:
```python
print(json.dumps(response.json(), indent=2))
```
**⚠️ STOP**: Wait for user response before proceeding.
---
## Debugging Issues
**⚠️ CRITICAL: Auto-route to debug-inference on ANY error encountered during this workflow.**
At any point in this workflow — service FAILED status, container crashes, inference test errors (500s, TypeErrors, ufunc errors, dtype issues, AttributeErrors, OOM, etc.) — you MUST:
1. **Immediately load `../debug-inference/SKILL.md`**
2. Follow its diagnostic workflow to identify the root cause
3. Apply the recommended fix from that skill
Do NOT attempt to diagnose or fix inference errors directly. The debug-inference skill has specific triage paths, known error patterns, and tested fixes (e.g., nullable signature re-registration for ufunc errors, pickle class resolution for AttributeErrors).
---
## Anti-Patterns to Avoid
**Do NOT use `CREATE SERVICE` SQL syntax for model inference** - always use the Python SDK:
```sql
-- WRONG - bypasses model registry integration
CREATE SERVICE my_service IN COMPUTE POOL my_pool ...
```
**Instead:** Use `mv.create_service()` from the model version object - this properly links the service to the registered model.
**Do NOT use `block=False` in `create_service()`** - run the script in background instead:
```python
# WRONG - async mode can cause issues with status tracking
mv.create_service(..., block=False)
```
**Instead:** Run the entire script as a background process and monitor via SQL.
**Do NOT use `RESULT_SCAN(LAST_QUERY_ID())` to filter SHOW results** - column name casing issues cause failures:
```sql
-- WRONG - fragile, causes "invalid identifier" errors
SELECT * FROM TABLE(RESULT_SCAN(LAST_QUERY_ID())) WHERE name = 'X'
```
**Do NOT use SHOW as a subquery** - invalid SQL syntax:
```sql
-- WRONG - syntax error
SELECT * FROM (SHOW SERVICES) WHERE name = 'X'
```
**Instead:** Run `DESCRIBE SERVICE` or `SHOW` commands directly and read the output.
**Do NOT attempt JWT or session token authentication for REST endpoints** - these approaches waste time and don't work reliably:
```python
# WRONG - JWT requires keypair setup that users rarely have
token = snow_connection.generate_jwt()
# WRONG - session tokens are for internal Snowflake use
token = conn.rest.token
```
**Instead:** Always ask the user for their PAT (Programmatic Access Token) first. PAT is the standard, supported method for SPCS REST authentication.
---
## Service Management Reference
**Suspend/Resume:**
```sql
ALTER SERVICE <SERVICE_NAME> SUSPEND;
ALTER SERVICE <SERVICE_NAME> RESUME;
```
**Auto-suspend (default 30 min):**
```sql
ALTER SERVICE <SERVICE_NAME> SET AUTO_SUSPEND_SECS = <seconds>;
```
**Delete service:**
```sql
DROP SERVICE <SERVICE_NAME>;
```
**Scale service:**
```sql
ALTER SERVICE <SERVICE_NAME> SET MIN_INSTANCES = <N>, MAX_INSTANCES = <M>;
```