references/aggregation-optimization.md
# Principles
Aggregation pipelines process documents through sequential stages. Focus on:
- Reducing documents early in the pipeline
- Minimizing data moved between stages
- Leveraging indexes where possible
- Managing memory usage
## Memory limits and disk spilling
Blocking stages (such as in-memory `$sort` and `$group`) have a 100MB memory limit per stage. Default behavior when this limit is exceeded is to spill to disk automatically (`allowDiskUse` defaults to `true`).
**Better solutions:**
- Filter more aggressively early in pipeline
- Add indexes to enable `$sort` to use index order
- Use `$limit` with `$sort` to reduce the amount of data the sort must process in memory for unindexed sorts
- Consider materialized views for repeated aggregations
# Optimization Examples
These examples are not exhaustive but representative of some common optimization patterns.
## Unindexed $lookup vs. Indexed $lookup
**Bad** — No index on the foreign collection's join field:
```javascript
db.orders.aggregate([
{ $lookup: {
from: "products",
localField: "productId",
foreignField: "sku", // no index on products.sku!
as: "product"
}}
])
```
**Good** — Index on `foreignField` in the foreign collection:
```javascript
db.products.createIndex({ sku: 1 })
db.orders.aggregate([
{ $lookup: {
from: "products",
localField: "productId",
foreignField: "sku",
as: "product"
}}
])
```
**Why:** Each `$lookup` executes a find on the `from` collection. Without an index on `foreignField`, every join does a full collection scan. This is the single most critical $lookup optimization.
## Early $project Defeating Optimization vs. Late $project
**Bad** — Early `$project` prevents the optimizer from pruning unused fields, forgets to exclude `_id` which is unneeded, and includes `name` which is not used:
```javascript
db.collection.aggregate([
{ $project: { name: 1, status: 1, amount: 1 } },
{ $match: { status: "active" } },
{ $group: { _id: "$status", total: { $sum: "$amount" } } }
])
```
**Good** — Let the optimizer handle field pruning; use `$project` only at the end for reshaping:
```javascript
db.collection.aggregate([
{ $match: { status: "active" } },
{ $group: { _id: "$status", total: { $sum: "$amount" } } },
{ $project: { _id: 0, status: "$_id", total: 1 } } // reshape at the end
])
```
**Why:** MongoDB's pipeline optimizer automatically analyzes which fields are used and avoids fetching unused ones. An early `$project` defeats this optimization, and can inadvertently request the wrong fields.
## $facet for Divergent Processing vs. $unionWith
**Bad** — `$facet` sends all documents to every branch, even if branches need very different subsets:
```javascript
db.collection.aggregate([
{ $facet: {
"top10": [{ $sort: { score: -1 } }, { $limit: 10 }],
"totalCount": [{ $count: "n" }] // gets ALL docs even though it's just counting
}}
])
```
**Good** — Separate pipelines via `$unionWith` let each branch optimize independently:
```javascript
db.collection.aggregate([
{ $sort: { score: -1 } }, { $limit: 10 },
{ $unionWith: {
coll: "collection",
pipeline: [{ $count: "n" }]
}}
])
```
**Why:** `$facet` funnels every document into every branch. `$unionWith` runs independent pipelines that each benefit from their own index usage and optimization.
## $sort \+ $limit as Separate Concerns vs. Top-N Sort
**Bad** — Large sort, then limit (MongoDB may sort entire dataset):
```javascript
db.collection.aggregate([
{ $group: { _id: "$category", total: { $sum: "$amount" } } },
{ $sort: { total: -1 } },
// ... many stages later ...
{ $limit: 10 }
])
```
**Good** — Place `$limit` immediately after `$sort`:
```javascript
db.collection.aggregate([
{ $group: { _id: "$category", total: { $sum: "$amount" } } },
{ $sort: { total: -1 } },
{ $limit: 10 }
])
```
**Why:** When `$sort` is immediately followed by `$limit`, MongoDB performs a *top-N sort* — it only tracks the top N values instead of sorting the full dataset. Far less memory.
## $unwind Best Practices
**When $unwind is needed**, filter before unwinding so that the $match stage allows index usage:
```javascript
[
{ $match: { "items.category": "electronics" } }, // Reduce documents first
{ $unwind: "$items" }, // Then unwind
{ $match: { "items.category": "electronics" } } // Filter unwound elements
]
```
**Never $unwind to re-group by `_id`:** If you are using `$unwind` followed by `$group` with `_id:` you can replace it with an array operator like `$filter`, `$map` or `$reduce` to match or transform array elements without unwinding.
## Optimize $lookup operations
`$lookup` performs collection joins and can be expensive. Strategies to improve performance:
1. **Filter before lookup** to reduce left-side documents
2. **Use indexed fields** in the lookup `localField`/`foreignField`
3. **Add $match in the lookup pipeline** to reduce right-side documents early
4. **Add $project last in the lookup pipeline** to keep only the fields you need
5. **$unwind immediately after lookup** when you need `as` result flattened
```javascript
[
{ $match: { active: true } }, // Reduce left side
{ $lookup: {
from: "inventory",
localField: "product_id",
foreignField: "_id", // _id is always indexed
pipeline: [
{ $match: { inStock: true } }, // Reduce right side
{ $project: { _id: 0, name: 1, price: 1 } }
],
as: "product"
}},
{ $unwind: "$product" }
]
```
**Schema consideration:** Excessive `$lookup` usage may indicate over-normalization. Consider embedding frequently-joined data.
## $group efficiency
Group operations require accumulating result documents in memory. Keys to efficiency:
1. **Include only needed fields within the $group stage** \- reference only the fields you need in accumulators
2. **Be mindful of unbounded accumulators** \- `$push` and `$addToSet` grow as group size increases and can cause memory issues
**Bad** \- do not add $project before $group to "reduce fields":
```javascript
[
{ $match: { date: { $gte: ISODate("2024-01-01") } } },
{ $project: { category: 1, amount: 1 } },
{ $group: {
_id: "$category",
total: { $sum: "$amount" },
count: { $sum: 1 }
}}
]
```
**Good** \- reference only needed fields directly in $group:
```javascript
[
{ $match: { date: { $gte: ISODate("2024-01-01") } } },
{ $group: {
_id: "$category",
total: { $sum: "$amount" },
count: { $sum: 1 }
}}
]
```
**Why:** The $group stage only processes the fields referenced in its expressions. Adding a $project before it does not save memory.references/antipattern-examples.md
## $exists on Regular Index vs. Sparse Index
**Bad** — `$exists: true` on a regular index still requires a document fetch:
```javascript
db.collection.createIndex({ a: 1 })
db.collection.find({ a: { $exists: true } })
// Cannot efficiently answer — null semantics require checking each document
```
**Good** — Use a sparse index, which only contains entries where the field exists:
```javascript
db.collection.createIndex({ a: 1 }, { sparse: true })
db.collection.find({ a: { $exists: true } })
// Answered directly from the index — no document fetch needed
```
**Why:** Regular indexes store `null` for both missing and existing fields that are set to `null`, so `$exists` can't be answered from the index alone. Sparse indexes only store entries for documents where the field exists.
## Unanchored $regex vs. Anchored $regex
**Bad** — Unanchored case insensitive regex cannot use the index efficiently:
```javascript
db.collection.find({ name: { $regex: /smith/i } })
// Full index or collection scan — case-insensitive, not anchored
```
**Good** — Anchored, case-sensitive regex uses the index as a range query:
```javascript
db.collection.find({ name: { $regex: /^Smith/ } })
// Efficient index range scan on the "Smith" prefix
```
**Why:** Indexes store values in sorted order. Only a left-anchored, case-sensitive `$regex` can be converted into an efficient index range scan. For case-insensitive matching, use a case-insensitive collation index instead.
## $where / JavaScript vs. Native MQL Operators
**Bad** — Server-side JavaScript execution:
```javascript
db.collection.find({
$where: "this.price * this.quantity > 1000"
})
```
**Good** — Native aggregation expression:
```javascript
db.collection.find({
$expr: { $gt: [{ $multiply: ["$price", "$quantity"] }, 1000] }
})
```
**Why:** JavaScript executed on the server is always slower than native MQL, cannot use indexes. It's also a security risk and is deprecated. Use `$expr` with aggregation operators instead.
## In-Memory Sort vs. Index-Supported Sort
**Bad** — Sort on an unindexed field triggers in-memory sort:
```javascript
db.orders.find({ status: "processing" }).sort({ createdAt: -1 })
// Index: { status: 1 } — sort is done in memory
```
**Good** — Compound index supports both filter and sort:
```javascript
db.orders.createIndex({ status: 1, createdAt: -1 })
db.orders.find({ status: "processing" }).sort({ createdAt: -1 })
// No SORT stage in the plan — results come pre-sorted from the index
```
references/core-indexing-principles.md
# Core Index Principles
### Compound Index Guidelines
The first field of the index should be in the query's filter or sort condition.
**Equality → Sort → Range** order is most often preferred:
- **Equality** fields first (e.g. `{field: value}`, `{$in: [...]}` with \<= 200 elements, `{field: {$eq: value}}`)
- **Sort** fields next
- **Range** fields last (e.g. `$gt`, `$lt`, `$gte`, `$lte`, `{$in: [...]}` with \> 200 elements in the array, `$ne`, anchored case-sensitive `$regex`)
If equality is not very selective and range is, then ERS may perform better than ESR.
### Sort direction
Index `{a:1, b:1}` supports `sort({a:1, b:1})` and reverse `sort({a:-1, b:-1})`, but NOT mixed directions like `sort({a:1, b:-1})`. For mixed sorts, create index matching exact pattern.
### Collation Match
**Before** — Query collation differs from index collation, so the index cannot be used:
```javascript
db.users.createIndex({ name: 1 })
db.users.find({ name: "José" }).collation({ locale: "es", strength: 2 })
// Index cannot be used for query
```
**After** — Create the index with the same collation the query uses:
```javascript
db.users.createIndex({ name: 1 }, { collation: { locale: "es", strength: 2 } })
db.users.find({ name: "José" }).collation({ locale: "es", strength: 2 })
// Index can be used for query
```
**Why:** Collation must match between index and query.
# Covered Queries
A covered query retrieves data directly from the index, never accessing the actual documents. This is extremely fast and preferable when possible.
## Requirements
1. **All query fields** are in the index
2. **All returned fields** are in the index (includes sort fields)
3. **Inclusion projection required** \- you must use an inclusion projection (e.g., `{ field: 1 }`) that requests only indexed fields, plus `_id: 0` if `_id` is not in the index. Exclusion projections cannot produce covered queries.
4. **No `$exists` or null equality checks** \- queries using `$exists` or querying for null/missing values cannot usually be covered by an index
5. **Multikey index constraints** \- multikey indexes can cover queries under certain conditions, such as when the array field itself is not included in the projection and operators like `$elemMatch` are not used. If the array field must be projected, covering is not possible.
## Building a covered query
**Step 1:** Identify your query pattern
```javascript
db.products.find(
{ category: "electronics", inStock: true },
{ category: 1, inStock: 1, price: 1, _id: 0 }
).sort({ price: 1 })
```
**Step 2:** Create index with all accessed fields
Following ESR (Equality-Sort-Range):
```javascript
db.products.createIndex({
category: 1, // Equality
inStock: 1, // Equality
price: 1 // Sort
})
```
**Step 3:** Project only indexed fields
- Include indexed fields in projection
- **Exclude \_id** unless \_id is in the index (use `_id: 0`)
- Don't request fields not in the index
## Common mistakes
### Forgetting to explicitly exclude \_id
```javascript
// NOT COVERED - _id not in index but included in result
db.products.find(
{ category: "electronics" },
{ category: 1, price: 1 } // _id included by default!
)
```
**Fix:** Explicitly exclude \_id
```javascript
db.products.find(
{ category: "electronics" },
{ category: 1, price: 1, _id: 0 } // Now covered
)
```
### Requesting non-indexed fields
```javascript
// NOT COVERED - description not in index
db.products.find(
{ category: "electronics" },
{ category: 1, price: 1, description: 1, _id: 0 }
)
```
**Fix:** Only project indexed fields, or add description to index
### Array fields (multikey indexes)
```javascript
// NOT COVERED - tags is an array field and is included in projection
db.products.createIndex({ tags: 1, price: 1 })
db.products.find(
{ tags: "sale" },
{ tags: 1, price: 1, _id: 0 }
)
```
**Fix:** If the array field is not needed in the result, remove it from the projection:
```javascript
// COVERED - array field (tags) used in query but not projected
db.products.find(
{ tags: "sale" },
{ price: 1, _id: 0 }
)
```
Multikey indexes can cover queries when the array field itself is not projected and operators like `$elemMatch` are not used. If you must return the array field, the query cannot be covered.references/update-query-examples.md
# Update Query Examples
## replaceOne vs. updateOne with $replaceWith
**Bad** — Full document replacement generates a large oplog entry:
```javascript
db.coll.replaceOne({ _id: X }, entireNewDocument)
```
**Good** — Use aggregation-based update to generate smaller oplog deltas:
```javascript
db.coll.updateOne({ _id: X }, [{ $replaceWith: { $literal: entireNewDocument } }])
```
**Why:** `replaceOne` writes the full document to the oplog. The aggregation update syntax lets MongoDB compute deltas, resulting in smaller oplog entries when only a few fields are changed.
## findOneAndUpdate Misuse vs. updateOne
**Bad** — Using `findOneAndUpdate` when you don't need the document returned:
```javascript
db.coll.findOneAndUpdate(
{ _id: X },
{ $set: { status: "processed" } }
)
```
**Good** — Use `updateOne` when you don't need the result document:
```javascript
db.coll.updateOne(
{ _id: X },
{ $set: { status: "processed" } }
)
```
**Why:** `findOneAndUpdate` writes a copy of the pre-change document to a side collection for retryable writes. This overhead is unnecessary if you don't need the returned document.SKILL.md
---
name: mongodb-query-optimizer
description: >-
Help with MongoDB query optimization and indexing. Use only when the user asks for optimization or performance: "How do I optimize this query?", "How do I index this?", "Why is this query slow?", "Can you fix my slow queries?", "What are the slow queries on my cluster?", etc. Do not invoke for general MongoDB query writing unless user asks for performance or index help. Prefer indexing as optimization strategy. Use MongoDB MCP when available.
compatibility: >-
Best with MongoDB MCP server. Uses collection-indexes and explain when the connection string works; uses Atlas Performance Advisor when Atlas API is configured. Without either, suggest indexes from query shape only. User creates indexes in Atlas or migrations unless tooling allows otherwise.
license: Apache-2.0
metadata:
version: "1.0.0"
---
# MongoDB Query Optimizer
## When this skill is invoked
Invoke **only** when the user wants:
- Query/index **optimization** or **performance** help
- **Why** a query is slow or **how to speed it up**
- **Slow queries** on their cluster and/or **how to optimize them**
Do **not** invoke for routine query authoring unless the user has requested help with optimization, slow queries, or indexing.
## High Level Workflow
### General Performance Help
If the user wants to examine slow queries, or is looking for general performance suggestions (not regarding any particular query):
- Use MongoDB MCP server **atlas-get-performance-advisor** tool to fetch slow query logs and performance advisor output
- Make suggestions based on this information
If Atlas MCP Server for Atlas is not configured or you don’t have enough information to run **atlas-get-performance-advisor** against the correct cluster, tell the user that general performance analysis requires Atlas MCP Server configuration with API credentials, and suggest they configure it or ask about a specific query instead.
### Help with a Specific Query
If the user is asking about a particular query:
- Use **collection-indexes**, **explain**, and **find** MCP tools to get existing indexes on the collection, explain() output for the query, and a sample document from the collection
- Use **atlas-get-performance-advisor MCP** tool to fetch slow query logs and performance advisor output
Then make an optimization suggestion based on collected information and MongoDB best practices and examples from reference files. Prefer creating an index that fully covers the query if possible. If you cannot use MongoDB MCP Server then still try to make a suggestion.
## MCP: available tools
**How to invoke.** Call the **MongoDB MCP server** with the **exact tool name** as `toolName` and a single **arguments object** as `arguments`. Do not pass the tool name as an option, query param, or nested key; pass it as the MCP tool name and the parameters as the arguments object. Full MCP Server tool reference: [MongoDB MCP Server Tools](https://www.mongodb.com/docs/mcp-server/tools/).
**Database tools** (when the MCP cluster connection works):
| Tool name (exact) | Arguments object |
| :---- | :---- |
| `collection-indexes` | `{ "database": "<db>", "collection": "<coll>" }` — both required strings. |
| `explain` | `{ "database": "<db>", "collection": "<coll>", "method": [ { "name": "find", "arguments": { "filter": {...}, "sort": {...}, "limit": N } } ], "verbosity": "executionStats" }`. `method` is an array of one object: `name` is `"find"`, `"aggregate"`, or `"count"`; `arguments` holds that method's params (e.g. find: `filter`, `sort`, `limit`; aggregate: `pipeline`; count: `query`). Optional `verbosity`: `"queryPlanner"` (default), `"executionStats"`, `"queryPlannerExtended"`, `"allPlansExecution"`. |
| `find` | `{ "database": "<db>", "collection": "<coll>", "filter": {...}, "projection": {...}, "sort": {...}, "limit": N }` — `database`, `collection`, and `filter` are required. Optional: `projection`, `sort`, `limit`. |
**Atlas tools** (when Atlas API credentials are configured):
| Tool name (exact) | Arguments object |
| :---- | :---- |
| `atlas-list-projects` | `{}` or `{ "orgId": "<24-char hex>" }`. Returns projects with their IDs; use to get `projectId` for Performance Advisor. |
| `atlas-get-performance-advisor` | **Required:** `"projectId"` (24-character hex string), `"clusterName"` (string, 1–64 chars, alphanumeric/underscore/dash). **Optional:** `"operations"` — array of strings from `"suggestedIndexes"`, `"dropIndexSuggestions"`, `"slowQueryLogs"`, `"schemaSuggestions"` (request only what you need); for slowQueryLogs only: `"since"` (ISO 8601 date-time), `"namespaces"` (array of `"db.coll"` strings). |
For a user question, try to fetch information from both the connection string and Atlas API related to the query you are optimizing.
### 1\. DB connection string works for MongoDB MCP
Typical flow: call `collection-indexes` → `explain` → `find` (sample doc).
- **`collection-indexes`** — Use the result's `classicIndexes` (each has `name`, `key`) to see if the query can already use an existing index.
- **`explain`** — Run in `"queryPlanner"` mode first to check for COLLSCAN. If the query uses an index or the collection is very small, run again with `"executionStats"` (10-second timeout) to get docs scanned vs. returned.
### 2\. Atlas API access works for MongoDB MCP
If you need a project ID, call `atlas-list-projects` first. Then call `atlas-get-performance-advisor` with only the `operations` you need:
| Operation value | Use when |
| :---- | :---- |
| `slowQueryLogs` | Fetching slow queries—**prioritize by slowest and most frequent**. Optional: `namespaces` to scope to a collection; `since` for a time window. |
| `suggestedIndexes` | Fetching cluster index recommendations |
| `dropIndexSuggestions` | User asks what to remove or reduce index overhead |
| `schemaSuggestions` | User asks for schema/query-structure advice alongside indexes |
Do not pass the MCP tool name as an `operations` value—`operations` is a separate argument listing what data to fetch.
## Example workflow 1 (help with specific query)
**User:** "Why is this query slow? `db.orders.find({status: 'shipped', region: 'US'}).sort({date: -1})`"
**If MCP db connection is configured and the database + collection names are known**, run steps 1–3. Otherwise skip to step 4.
1. **Check existing collection indexes:**
- Call `collection-indexes` with database=`store`, collection=`orders`
- Result shows: `{_id: 1}`, `{status: 1}`, `{date: -1}`
2. **Run explain:**
- Call `explain` with method=`find`, filter=`{status: 'shipped', region: 'US'}`, sort=`{date: -1}`, verbosity=`queryPlanner` and `executionStats`
- Result: Uses `{status: 1}` index, then in-memory SORT, `totalKeysExamined: 50000`, `nReturned: 100`
3. **Run find:**
- Call `find` with limit=1 to fetch a sample document to impute the schema.
**If MCP Atlas connection is configured**, run step 4. Otherwise skip to step 5.
4. **Run atlas-get-performance-advisor:**
- Try to get the cluster name from the MCP connection string, or ask the user for projectId/clusterName
- Use slowQueryLogs to fetch slow query logs from database=`store`, collection=`orders` in the past 24 hours
- Use suggestedIndexes to check for index suggestions for the query
5. **Diagnose:** Based on explain output and slow query logs, this query targets 100 docs but scans 50K index entries (poor selectivity: 0.002). In-memory sort adds overhead. Index doesn't support both filter fields or sort.
6. **Recommend:** Create compound index `{status: 1, region: 1, date: -1}` following ESR (two equality fields, then sort). This eliminates in-memory sort and improves selectivity by filtering on both status and region.
If the MongoDB MCP server is not set up, follow best indexing practices.
## Example workflow 2 (general database performance help)
**User:** "Can you help with optimizing slow queries on my cluster?”
1. **Run atlas-get-performance-advisor:**
- Try to get the cluster name from the connection string and deduce the project name you need in atlas-list-projects; if you are not sure, then ask the user for cluster name and project id.
- Use slowQueryLogs to fetch slow query logs from the past 24 hours
- Use suggestedIndexes
- Use dropIndexSuggestions
- Use schemaSuggestions
2. **Diagnose and Recommend:** Based on slow query logs and performance advisor advice, you can create the compound index `{status: 1, region: 1, date: -1}` on the `db.orders` collection to optimize queries such as `find({status: 'shipped', region: 'US'}).sort({date: -1})`
Examine all performance advisor output as well as slow query logs. Provide information on what is being improved and why, and focus on suggestions that have the potential for greatest impact (e.g., indexes that affect the most queries, or queries that have the worst performance).
## Load references
Before beginning diagnosis and recommendation, load reference files.
Always load:
- `references/core-indexing-principles.md`
- `references/antipattern-examples.md`
Conditionally load these files:
- **If diagnosing aggregation pipelines** → `references/aggregation-optimization.md`
- **If diagnosing queries that change docs such as replaceOne, findOneAndUpdate, etc.** → `references/update-query-examples.md` for oplog-efficient updates and common update anti-patterns
## Output
- Keep answers short and clear: a few sentences on index and optimization suggestions, and reasoning behind them (e.g. general indexing principles, observing slow query logs in the cluster, or seeing advice in Performance Advisor)
- Focus on highest impact indexes or optimizations - if you've omitted some optimizations let the user know and present them if asked.
- Do not use strong language, such as saying “You should create these indexes and they will definitely improve application performance” \- Explain they are suggestions for certain queries, and give the reasoning behind them.
- Consider how many indexes already exist on the collection (if known) \- there shouldn’t generally be more than 20
- Suggest removing indexes only if the suggestion comes from Atlas Performance Advisor
- Do not create indexes directly via MCP unless the user gives approval