assets/examples/sample_commandstats.csv
cluster_name,command,calls,usec
demo-session-cache,get,500000000,650000000
demo-session-cache,set,200000000,400000000
demo-session-cache,hget,50000000,75000000
demo-session-cache,hset,30000000,90000000
demo-session-cache,hgetall,20000000,180000000
demo-session-cache,del,15000000,22000000
demo-session-cache,expire,80000000,72000000
demo-session-cache,ttl,25000000,30000000
demo-session-cache,exists,10000000,12000000
demo-session-cache,ping,5000000,2000000
demo-session-cache,info,1000,50000
demo-leaderboard,get,100000000,130000000
demo-leaderboard,set,50000000,100000000
demo-leaderboard,zadd,800000000,7200000000
demo-leaderboard,zrange,500000000,3500000000
demo-leaderboard,zrangebyscore,200000000,1800000000
demo-leaderboard,zrem,100000000,400000000
demo-leaderboard,zincrby,300000000,900000000
demo-leaderboard,zscore,50000000,60000000
demo-leaderboard,ping,10000000,4000000
assets/examples/sample_input_simple.csv
cluster_name,instance_type,region,engine,node_count,primary_nodes,avg_memory_gb,daily_commands
demo-session-cache,cache.r6g.xlarge,us-east-1,redis,6,3,12.5,850000000
demo-leaderboard,cache.r7g.2xlarge,us-east-1,valkey,4,2,45.2,2100000000
staging-cache,cache.t4g.medium,us-west-2,redis,2,1,0.3,5000000
demo-rate-limiter,cache.r7g.large,eu-west-1,valkey,2,1,0.05,15000000000
demo-user-profiles,cache.r5.4xlarge,ap-southeast-1,redis,6,3,85.0,500000000
references/data-modeling/command-availability.md
# Command and Feature Availability by Engine Version
Quick reference for which commands and features are available in each ElastiCache engine version. Use this when recommending data structures or patterns to verify they are supported on the customer's current engine.
## Command Availability Matrix
| Command / Feature | Redis OSS 6.x | Redis OSS 7.x | Valkey 7.2 | Valkey 8.0 | Valkey 8.1 | Valkey 8.2 |
|-------------------|:--------------:|:--------------:|:----------:|:----------:|:----------:|:----------:|
| **Per-Key TTL** | | | | | | |
| EXPIRE, TTL, PEXPIRE, PTTL | Yes | Yes | Yes | Yes | Yes | Yes |
| EXPIREAT, PEXPIREAT | Yes | Yes | Yes | Yes | Yes | Yes |
| EXPIRETIME, PEXPIRETIME | No | Yes | Yes | Yes | Yes | Yes |
| **Per-Field TTL (Hash)** | | | | | | |
| HEXPIRE, HPEXPIRE | No | No | No | No | No | No (available on ElastiCache Valkey 9.0) |
| HTTL, HPTTL | No | No | No | No | No | No (available on ElastiCache Valkey 9.0) |
| HEXPIREAT, HPEXPIREAT | No | No | No | No | No | No (available on ElastiCache Valkey 9.0) |
| HEXPIRETIME, HPEXPIRETIME | No | No | No | No | No | No (available on ElastiCache Valkey 9.0) |
| HPERSIST | No | No | No | No | No | No (available on ElastiCache Valkey 9.0) |
| HSETEX, HGETEX | No | No | No | No | No | No (available on ElastiCache Valkey 9.0) |
| **Bloom Filters** | | | | | | |
| BF.ADD, BF.EXISTS, BF.RESERVE | No | No | No | No | Yes | Yes |
| BF.INSERT, BF.MADD, BF.MEXISTS | No | No | No | No | Yes | Yes |
| BF.CARD, BF.INFO | No | No | No | No | Yes | Yes |
| **Vector Search** | | | | | | |
| FT.CREATE | No | No | No | No | No | Yes (node-based only) |
| FT.SEARCH | No | No | No | No | No | Yes (node-based only) |
| FT.INFO, FT.DROPINDEX | No | No | No | No | No | Yes (node-based only) |
| FT._LIST | No | No | No | No | No | Yes (node-based only) |
| **JSON** | | | | | | |
| JSON.SET, JSON.GET | Yes (6.2.6+) | Yes | Yes | Yes | Yes | Yes |
| JSON.MGET | Yes (6.2.6+) | Yes | Yes | Yes | Yes | Yes |
| JSON.ARRAPPEND, JSON.OBJKEYS | Yes (6.2.6+) | Yes | Yes | Yes | Yes | Yes |
| **Streams** | | | | | | |
| XADD, XREAD, XRANGE | Yes | Yes | Yes | Yes | Yes | Yes |
| XREADGROUP (consumer groups) | Yes | Yes | Yes | Yes | Yes | Yes |
| XAUTOCLAIM | Yes (6.2+) | Yes | Yes | Yes | Yes | Yes |
| **ACL (Access Control)** | | | | | | |
| ACL SETUSER, ACL DELUSER | Yes | Yes | Yes | Yes | Yes | Yes |
| ACL GETUSER, ACL LIST | Yes | Yes | Yes | Yes | Yes | Yes |
| ACL DRYRUN | No | Yes | Yes | Yes | Yes | Yes (available from Redis OSS 7.x; verify serverless availability — some ACL commands are restricted on serverless) |
| **Pub/Sub** | | | | | | |
| PUBLISH, SUBSCRIBE | Yes | Yes | Yes | Yes | Yes | Yes |
| Sharded Pub/Sub (SSUBSCRIBE) | No | Yes | Yes | Yes | Yes | Yes |
| **Functions and Scripting** | | | | | | |
| EVAL (Lua scripting) | Yes | Yes | Yes | Yes | Yes | Yes |
| FUNCTION LOAD (server-side functions) | No | Yes | Yes | Yes | Yes | Yes |
| FUNCTION LIST, FUNCTION DELETE | No | Yes | Yes | Yes | Yes | Yes |
| **Client and Connection** | | | | | | |
| CLIENT NO-EVICT | No | Yes | Yes | Yes | Yes | Yes |
| CLIENT NO-TOUCH | No | No | Yes | Yes | Yes | Yes |
| **Cluster** | | | | | | |
| CLUSTER SHARDS | No | Yes | Yes | Yes | Yes | Yes |
| CLUSTER MYSHARDID | No | Yes | Yes | Yes | Yes | Yes |
| **Other** | | | | | | |
| GETDEL, GETEX | Yes (6.2+) | Yes | Yes | Yes | Yes | Yes |
| LMPOP, ZMPOP | No | Yes | Yes | Yes | Yes | Yes |
| SINTERCARD | No | Yes | Yes | Yes | Yes | Yes |
| OBJECT FREQ, OBJECT IDLE | Yes | Yes | Yes | Yes | Yes | Yes |
## Deployment Model Restrictions
Some commands are further restricted by deployment model:
| Command / Feature | Serverless | Node-Based |
|-------------------|:----------:|:----------:|
| SELECT (multiple databases) | Not supported | Supported (cluster-mode disabled only) |
| Vector search (FT.*) | Not supported | Supported (Valkey 8.2 or above, excludes data tiering nodes; t2/t3/t4g require increased memory reserve) |
| WAIT | Restricted | Supported |
| CONFIG SET/GET | Not supported | Not supported (use parameter groups) |
| DEBUG | Not supported | Not supported |
| BGREWRITEAOF, BGSAVE, SAVE | Not supported | Not supported |
| MIGRATE, REPLICAOF, SLAVEOF | Not supported | Not supported |
| MONITOR | Not supported | Supported |
| SHUTDOWN, SYNC | Not supported | Not supported |
| PSUBSCRIBE, PUNSUBSCRIBE | Not supported | Supported |
| Keyspace notifications | Not supported | Supported |
| KEYS | Not supported | Available but dangerous |
| CLIENT NO-EVICT | Not supported | Supported (Redis OSS 7.x+ / Valkey 7.2+) |
| FUNCTION LOAD, FCALL, FCALL_RO | Not supported | Supported (Redis OSS 7.x+ / Valkey 7.2+) |
| Lua scripts (cross-slot) | Restricted (must have at least one KEY parameter; max 4 MiB script; max 3,999 arguments) | Supported (cluster-mode disabled) |
| SUBSCRIBE (global) | Supported | Supported (cluster-mode disabled) |
| SSUBSCRIBE (sharded) | Supported | Supported |
| OBJECT FREQ | Not supported | Supported |
| OBJECT IDLE | Not supported | Supported |
## Common Scenarios
**"I need per-field TTL on hashes"**: Available on ElastiCache Valkey 9.0. Per-field hash expiration (HEXPIRE, HTTL, HSETEX, HGETEX) was introduced in Valkey 9.0. On engine versions before 9.0, the workaround is to store each field as a separate key with its own EXPIRE, or use a sorted set with timestamps for expiration tracking.
**"I need vector search / semantic similarity"**: Requires Valkey 8.2 or above on node-based clusters (recommend 9.0). Not available on serverless. Not available in Redis OSS on ElastiCache. Use FT.CREATE to define a vector index, FT.SEARCH to query it.
**"I need server-side functions"**: Requires Redis OSS 7.x or Valkey 7.2+. FUNCTION LOAD is available from Redis OSS 7.x (node-based only; not available on serverless). FUNCTION LOAD replaces the older EVAL-only model (though EVAL still works). Functions persist across restarts, unlike EVALSHA-cached scripts.
**"I need sharded Pub/Sub"**: Requires Redis OSS 7.x or Valkey 7.2+. Use SSUBSCRIBE instead of SUBSCRIBE for cluster-mode-enabled deployments. Sharded Pub/Sub routes messages to the shard that owns the channel, reducing cross-node traffic.
references/data-modeling/common-patterns.md
# Common Patterns (Python & Node.js)
Canonical implementations for the three most requested patterns. Each includes connection setup, core operations, error handling, and TTL strategy. For advanced algorithms (XFetch, sliding-window rate limiting, tie-breaking leaderboards), see `recipe-gallery.md`.
All examples use lazy client initialization per the connection safety rule.
> **Cluster mode vs. standalone clients:** ElastiCache Serverless operates in cluster mode enabled only; clients must support cluster mode (e.g., `ValkeyCluster` in Python, `new Redis.Cluster()` in ioredis) to handle slot-based routing. The standalone `Valkey`/`new Redis()` clients shown below work for cluster mode disabled node-based clusters. For serverless or cluster mode enabled deployments, swap to the cluster-aware client variant.
---
## 1. Cache-Aside (DB Query Cache)
Read from cache first. On miss, read from DB, write to cache. On DB write, invalidate cache.
### Python (valkey-py)
```python
import valkey
import json
_client = None
def get_client():
global _client
if _client is None:
_client = valkey.Valkey(
host="mycluster.xxxxxx.use1.cache.amazonaws.com", # Replace with your endpoint; for clustercfg endpoints, use ValkeyCluster instead
port=6379, ssl=True, decode_responses=True,
username="appuser", password="<your-password-or-iam-token>",
socket_connect_timeout=5, socket_timeout=2,
)
return _client
CACHE_TTL = 300 # 5 minutes; tune per staleness tolerance
def cache_get(client, entity: str, entity_id: str, db_fetch_fn) -> dict | None:
key = f"cache:{entity}:{entity_id}"
cached = client.get(key)
if cached is not None:
return json.loads(cached)
result = db_fetch_fn(entity_id)
if result is not None:
client.setex(key, CACHE_TTL, json.dumps(result))
return result
def cache_invalidate(client, entity: str, entity_id: str) -> None:
client.delete(f"cache:{entity}:{entity_id}")
def cache_write_through(client, entity: str, entity_id: str, data: dict, db_write_fn) -> None:
db_write_fn(entity_id, data)
client.setex(f"cache:{entity}:{entity_id}", CACHE_TTL, json.dumps(data))
```
### Node.js (ioredis)
```javascript
const Redis = require("ioredis");
let client;
function getClient() {
if (!client) {
client = new Redis({
host: "mycluster.xxxxxx.use1.cache.amazonaws.com", port: 6379, tls: {}, // For clustercfg or serverless endpoints, use new Redis.Cluster([{host, port}], {redisOptions: {tls: {}}}) instead of standalone new Redis()
username: "appuser", password: "<your-password-or-iam-token>",
connectTimeout: 5000, commandTimeout: 2000,
});
}
return client;
}
const CACHE_TTL = 300;
async function cacheGet(entity, entityId, dbFetchFn) {
const key = `cache:${entity}:${entityId}`;
const cached = await getClient().get(key);
if (cached !== null) return JSON.parse(cached);
const result = await dbFetchFn(entityId);
if (result !== null && result !== undefined) {
await getClient().setex(key, CACHE_TTL, JSON.stringify(result));
}
return result;
}
async function cacheInvalidate(entity, entityId) {
await getClient().del(`cache:${entity}:${entityId}`);
}
```
---
## 2. Session Store
Store user session data as a hash. TTL matches session timeout.
### Python (valkey-py)
```python
import valkey
import json
import time
_client = None
def get_client():
global _client
if _client is None:
_client = valkey.ValkeyCluster(
host="mycluster.xxxxxx.clustercfg.use1.cache.amazonaws.com", # Replace with your endpoint
port=6379, ssl=True, decode_responses=True,
username="appuser", password="<your-password-or-iam-token>",
socket_connect_timeout=5, socket_timeout=2,
)
return _client
SESSION_TTL = 1800 # 30 minutes
def save_session(client, session_id: str, data: dict) -> None:
key = f"session:{session_id}"
data["updated_at"] = str(time.time())
client.hset(key, mapping=data)
client.expire(key, SESSION_TTL)
def get_session(client, session_id: str) -> dict | None:
key = f"session:{session_id}"
data = client.hgetall(key)
if not data:
return None
client.expire(key, SESSION_TTL) # slide expiry on access
return data
def delete_session(client, session_id: str) -> None:
client.delete(f"session:{session_id}")
```
### Node.js (ioredis)
```javascript
const SESSION_TTL = 1800;
async function saveSession(sessionId, data) {
const key = `session:${sessionId}`;
data.updated_at = String(Date.now() / 1000);
await getClient().hset(key, data);
await getClient().expire(key, SESSION_TTL);
}
async function getSession(sessionId) {
const key = `session:${sessionId}`;
const data = await getClient().hgetall(key);
if (!data || Object.keys(data).length === 0) return null;
await getClient().expire(key, SESSION_TTL);
return data;
}
async function deleteSession(sessionId) {
await getClient().del(`session:${sessionId}`);
}
```
---
## 3. Rate Limiter (Fixed Window)
Simple INCR + EXPIRE pattern. For sliding-window with better accuracy at window boundaries, see `recipe-gallery.md`.
### Python (valkey-py)
```python
import valkey
_client = None
def get_client():
global _client
if _client is None:
_client = valkey.ValkeyCluster(
host="mycluster.xxxxxx.clustercfg.use1.cache.amazonaws.com", # Replace with your endpoint
port=6379, ssl=True, decode_responses=True,
username="appuser", password="<your-password-or-iam-token>",
socket_connect_timeout=5, socket_timeout=2,
)
return _client
# Lua script: atomic INCR + conditional EXPIRE.
# Returns [count, ttl]. Sets TTL only when the key was just created (count == 1)
# or when TTL is missing (race condition recovery).
#
# NOTE: ElastiCache Serverless requires all Lua scripts to have at least one
# KEY parameter (scripts with 0 keys will fail). This script uses KEYS[1].
# In cluster mode (including Serverless), all keys used in a Lua script must
# hash to the same slot. If you extend this script to use multiple keys,
# use hash tags (e.g., {client_id}) to ensure they share a slot.
_RATE_LIMIT_SCRIPT = """
local count = redis.call('INCR', KEYS[1])
local ttl = redis.call('TTL', KEYS[1])
if count == 1 or ttl == -1 then
redis.call('EXPIRE', KEYS[1], ARGV[1])
end
return {count, ttl}
"""
def is_rate_limited(client, client_id: str, max_requests: int,
window_seconds: int) -> tuple[bool, int]:
"""Returns (is_limited, remaining_requests).
Uses a Lua script for atomicity: INCR and EXPIRE run in a single
server-side call, eliminating the race where a key could persist
without a TTL if EXPIRE failed or another request slipped in
between INCR and EXPIRE.
"""
key = f"ratelimit:{client_id}:{window_seconds}"
count, _ = client.eval(_RATE_LIMIT_SCRIPT, 1, key, window_seconds)
remaining = max(0, max_requests - count)
return count > max_requests, remaining
```
### Node.js (ioredis)
```javascript
// Lua script: atomic INCR + conditional EXPIRE.
// Sets TTL when key is new (count == 1) or TTL is missing (race recovery).
const RATE_LIMIT_SCRIPT = `
local count = redis.call('INCR', KEYS[1])
local ttl = redis.call('TTL', KEYS[1])
if count == 1 or ttl == -1 then
redis.call('EXPIRE', KEYS[1], ARGV[1])
end
return {count, ttl}
`;
async function isRateLimited(clientId, maxRequests, windowSeconds) {
const key = `ratelimit:${clientId}:${windowSeconds}`;
const [count] = await getClient().eval(RATE_LIMIT_SCRIPT, 1, key, windowSeconds);
const remaining = Math.max(0, maxRequests - count);
return { limited: count > maxRequests, remaining };
}
```
---
## Error Handling Pattern
Cache is an optimization, not a dependency. Wrap all cache calls so failures fall through to the data source.
```python
def safe_cache_get(client, entity, entity_id, db_fetch_fn):
try:
return cache_get(client, entity, entity_id, db_fetch_fn)
except Exception:
return db_fetch_fn(entity_id)
```
```javascript
async function safeCacheGet(entity, entityId, dbFetchFn) {
try {
return await cacheGet(entity, entityId, dbFetchFn);
} catch {
return dbFetchFn(entityId);
}
}
```
> **Reconnection strategy:** When a client disconnects due to a timeout or failover, retry with exponential backoff and jitter to avoid a thundering herd of reconnections that can overwhelm the server and cause prolonged outages. See the AWS best practices for [cluster client discovery and exponential backoff](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/BestPractices.Clients.Redis.Discovery.html).
references/data-modeling/instructions.md
# Data Modeling & Patterns
**When to use:** The user needs help designing key schemas, choosing data structures (strings, hashes, sorted sets, streams, etc.), implementing caching patterns (cache-aside, session store, leaderboard, rate limiting), or generating application code for a specific data access pattern.
**When not needed:** The user is setting up a new cache, choosing an engine or deployment model, configuring authentication, or working on monitoring and cost optimization.
Design key schemas and implement Valkey/Redis data structure patterns for application use cases.
## Loading
Read this file first. Other references in this folder load on demand when the current answer requires them. Scripts in `scripts/` run on demand.
## Routing
- For semantic cache, conversational memory, RAG, or vector-based recommendation, route to the `genai` sub-skill.
- If the user's engine is Memcached, load `memcached-recipe.md` for Memcached-specific patterns and constraints.
- If the user asks whether a specific command or feature is available on their engine version, load `command-availability.md` for the version compatibility matrix.
## Reference File Loading
Load `recipe-gallery.md` when the user asks about ANY of these patterns (do not rely on the Pattern Quick Reference table alone; the gallery has full working code):
| User asks about | Load |
|---|---|
| Cache-aside, read-through, write-through, write-behind | `common-patterns.md` (canonical starter), `recipe-gallery.md` (XFetch probabilistic refresh) |
| Session store, session management, user sessions | `common-patterns.md` (canonical starter) |
| Rate limiting, throttling, API rate control | `common-patterns.md` (fixed window), `recipe-gallery.md` (sliding window) |
| Leaderboard, ranking, scoreboard, sorted set patterns | `recipe-gallery.md` |
| Counters, analytics, HyperLogLog, unique counts | `recipe-gallery.md` |
| Pub/Sub, messaging, event notification | `recipe-gallery.md` |
| Streams, event sourcing, consumer groups, XADD/XREAD | `recipe-gallery.md` |
| Shopping cart, e-commerce cart | `recipe-gallery.md` |
| Job queue, task queue, priority queue | `recipe-gallery.md` |
| Distributed lock, mutex, Redlock | `recipe-gallery.md` |
| "Show me a working example", "give me code for" + any caching pattern | `recipe-gallery.md` |
Load `command-availability.md` when the user asks about command support on a specific engine or version, or when generating code that uses commands beyond basic GET/SET/HSET.
Load `memcached-recipe.md` when engine is Memcached or the user mentions Memcached-specific constraints.
## Check for existing context
Before starting, read `.elasticache/requirements.json` if it exists. Use the values (`engine`, `runtime`, `deployment_model`, `patterns`) as inputs rather than re-asking. If `engine` is `memcached`, load `memcached-recipe.md`. If `patterns` already lists the user's use case, skip pattern selection.
## Workflow
1. Identify the user's use case and match to a pattern below
2. Recommend the right data structures and key design
3. Provide working application code in the user's language
4. Include TTL strategy and invalidation approach
## After implementation
Update `.elasticache/requirements.json`: add or confirm the implemented pattern(s) in the `patterns` array. Read the existing file first, merge your updates, then write it back. Do not overwrite fields owned by other sub-skills.
## Handoff to monitoring
After implementing a pattern, prompt the user: "Want me to set up CloudWatch dashboards and alarms for this cache?" If yes, hand off to the `monitoring` sub-skill. For cache-aside patterns, the key metrics to monitor are hit rate, evictions, and latency. For rate limiters, monitor CPU and connection count.
## Pattern Quick Reference
| Pattern | Key Design | Structures / Commands | TTL |
|---------|-----------|----------------------|-----|
| DB Query Cache | `{table}:{pk}` or `{query_hash}` | Strings/JSON, cache-aside | 60s-3600s |
| Session Store | `session:{session_id}` | Hashes or Strings | Match session timeout |
| Leaderboard | `leaderboard:{game_id}:{period}` | Sorted Sets (ZADD, ZRANGE, ZRANK) | Per period |
| Rate Limiting | `ratelimit:{user_id}:{window}` | INCR+EXPIRE (fixed) or Sorted Sets (sliding) | Window duration |
| Counters & Analytics | `metric:{name}:{window}` | INCR, HINCRBY, HyperLogLog | Per window |
| Pub/Sub | channel names | PUBLISH/SUBSCRIBE (fire-and-forget); prefer SPUBLISH/SSUBSCRIBE (sharded pub/sub) for high-throughput on cluster-mode-enabled clusters | N/A |
| Event Streams | `stream:{topic}` | Streams + Consumer Groups | Per retention policy |
| Shopping Cart | `cart:{user_id}` | Hashes + TTL | Cart expiry |
| Job Queue | `queue:{name}` | Lists (FIFO) or Sorted Sets (priority) | Optional |
| Recommendation | `item:{id}:likes`, `item:{id}:ratings` | INCR/DECR, HSET | Optional |
## Key Design Principles
- Use colons as namespace separators: `{entity}:{id}:{attribute}`
- Include the entity type in the key: `user:u100:profile`, not just `u100`
- Keep keys short but readable
- Use TTL on everything that has a natural expiry. On serverless, Valkey uses a volatile-lru eviction policy and auto-scales storage up to the configured CacheUsageLimits maximum. OOM errors only occur when the maximum is reached AND no keys with a TTL are eligible for eviction. Always set a TTL on serverless caches.
- Prefer atomic operations (INCR, HINCRBY, ZADD) over read-modify-write
## Multi-Tenant Key Namespacing
In multi-tenant systems, prefix every key with the tenant identifier to prevent collisions:
```
{tenant_id}:{resource}:{id}
```
Example: `tenant42:session:abc123`, `tenant42:cart:user7`
Benefits of consistent tenant prefixing:
- Prevents data leaks between tenants; each tenant's keys are isolated by prefix.
- Enables per-tenant SCAN. Use `SCAN 0 MATCH tenant42:*` to enumerate only one tenant's keys.
- Supports per-tenant eviction or cleanup by scanning and deleting a single prefix.
**Hash tags for multi-key operations:** If you use MGET, pipelines, or transactions across multiple keys for the same tenant on cluster-mode-enabled caches (including serverless), wrap the tenant ID in curly braces to ensure all keys hash to the same slot: `{tenant42}:session:abc123`. Without hash tags, multi-key operations across different slots will fail with CROSSSLOT errors. **Hot-slot risk:** Hash tags concentrate all keys for a tenant onto a single shard. For large tenants with high key counts or throughput, this can create a hot slot that overloads one shard while others remain idle. Monitor per-shard CPU and memory metrics, and consider splitting very large tenants across multiple hash tags if hot-spotting occurs.
- Simplifies capacity analysis; count keys per tenant with `SCAN` + prefix match.
## Dangerous Commands -- Never Use KEYS in Production
**The `KEYS` command must never be used in production.** `KEYS *` (or any `KEYS` pattern) blocks the entire Redis/Valkey server while it scans every key in the database. On caches with millions of keys, this can block the server for seconds or longer, causing timeouts and cascading failures for all connected clients. This applies to both node-based and serverless ElastiCache deployments.
**Use `SCAN` with cursor-based iteration instead.** `SCAN` performs the same work incrementally without blocking the server. Each `SCAN` call returns a small batch of keys and a cursor to continue from.
Safe SCAN pattern in Python:
```python
# Using valkey-py (pip install valkey)
import valkey
r = valkey.Valkey(host="your-endpoint", port=6379, ssl=True, decode_responses=True)
# Or using redis-py (pip install redis) -- the AWS-validated Python client
# import redis
# r = redis.Redis(host="your-endpoint", port=6379, ssl=True, decode_responses=True)
def scan_keys(pattern="*", count=100):
"""Iterate over keys matching a pattern without blocking the server.
Uses SCAN with a cursor to retrieve keys in small batches.
The count parameter is a hint (not a hard limit) for batch size.
"""
cursor = 0
while True:
cursor, keys = r.scan(cursor=cursor, match=pattern, count=count)
for key in keys:
yield key
if cursor == 0:
break
# Example: find all session keys
for key in scan_keys("session:*"):
print(key)
```
If a team member or external reference suggests using `KEYS`, always redirect to `SCAN`. The same principle applies to other blocking commands like `SMEMBERS` on very large sets; prefer `SSCAN`, `HSCAN`, and `ZSCAN` for large collections.
## Code Generation
When generating application code:
- Always use TLS (ssl=True / tls:{}) for ElastiCache endpoints
- Prefer connection pooling; avoid creating a new connection per request. Use a module-level singleton pattern (initialize on first use via a `get_client()` function) to reuse connections across requests.
- Use pipelining for multi-command operations (reduces round trips)
- Wrap cache calls in try/except; cache is an optimization, not a dependency
- Reuse connections across requests (connection pooling). Create the connection pool once at application startup and share it; creating a new connection per request is ~13x slower (2.82 ms vs 0.21 ms per AWS benchmarks)
- Include the relevant SDK: redis-py or valkey-py (Python), ioredis (Node.js), Lettuce or Jedis (Java), go-redis (Go), or Valkey Glide (multi-language)
- Never use restricted ElastiCache commands in generated code. The following commands are unavailable on ElastiCache for clusters running Redis OSS or Valkey: `BGREWRITEAOF`, `BGSAVE`, `CONFIG`, `DEBUG`, `MIGRATE`, `REPLICAOF`, `SAVE`, `SLAVEOF`, `SHUTDOWN`, `SYNC`. Some restrictions may vary by engine version; check `command-availability.md` for details.
## Freshness disclaimer
When your response includes pricing, version constraints, or feature availability, include the freshness disclaimer per SKILL.md Global Rule #5: "For current pricing see https://aws.amazon.com/elasticache/pricing/. For current feature availability see https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/."
references/data-modeling/memcached-recipe.md
# Memcached Scale-Out Recipe
Guidance for using Amazon ElastiCache for Memcached: when it fits, how it scales, and what it lacks.
## When to Use Memcached
Memcached is the right choice when **all** of the following are true:
- You need simple get/set ephemeral caching (string key-value pairs)
- Your workload is read-heavy and benefits from multi-threaded read performance
- You do not need data persistence (data loss on restart is acceptable)
- You do not need replication or automatic failover
- You do not need advanced data structures (sorted sets, streams, hashes, lists)
- You do not need pub/sub, RBAC, IAM authentication, or vector search
**Typical use case:** Caching serialized database query results, HTML fragments, API responses, or computed objects where the source of truth is always the database and the cache is purely an acceleration layer.
## Serverless Memcached vs Node-Based Memcached
### Serverless Memcached
- Requires Memcached engine version 1.6.22 or higher
- Transparent Multi-AZ: data is automatically replicated across availability zones
- Supports automatic daily backups for recovery (must be explicitly enabled)
- No capacity planning or node selection required
- Automatic scaling within configured limits
- TLS always enabled: clients must support TLS connectivity to connect to serverless Memcached
- Simpler operational model: no node monitoring, no manual shard management
### Node-Based Memcached
- Manual scaling: you choose instance types and number of nodes
- No replication: each node holds a unique slice of data
- Node failure means partial data loss (client detects and routes to other nodes, but data is gone)
- Lower per-unit cost at steady-state if you can accurately predict load
**Recommendation:** Start with serverless Memcached unless you have a specific reason to manage nodes (e.g., cost optimization at very large steady-state scale).
## Auto Discovery
ElastiCache Memcached supports Auto Discovery, which lets clients automatically detect when nodes are added or removed. The client connects to the cluster's configuration endpoint (not individual node endpoints), polls for changes once per minute by default (this interval can be adjusted), and updates its node list without any application redeployment. See the [ElastiCache Auto Discovery documentation](https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/AutoDiscovery.html) for implementation details.
## Consistent Hashing
Memcached clients use consistent hashing to distribute keys across nodes:
- Each node is assigned positions on a hash ring
- A key is hashed and placed on the ring; it maps to the next node clockwise
- When a node is added or removed, only a fraction of keys need to remap (roughly `1/N` where N is the number of nodes), rather than all keys
This minimizes cache misses during scaling events.
## Scale-Out Pattern
Adding capacity to a Memcached cluster:
1. **Add nodes** via the console, AWS CLI, or SDK
2. **Auto Discovery** detects the new nodes within the polling interval
3. **Consistent hashing** redistributes a fraction of the key space to new nodes
4. **New nodes start cold**: cache misses for remapped keys will hit the database until the cache warms up
For node-based, scaling is done via `ModifyCacheCluster` (CLI: `aws elasticache modify-cache-cluster`). For serverless Memcached, scaling is automatic within the configured `CacheUsageLimits`; adjust limits via `modify-serverless-cache` with `--cache-usage-limits`. See AWS docs for full parameter reference.
## Node Failure Behavior
- **Data on the failed node is lost.** Node-based Memcached has no replication, so there is no replica to promote.
- The client removes the failed node from its hash ring; affected keys become cache misses that fall through to the database.
- Use serverless Memcached for transparent Multi-AZ redundancy, or design the application to tolerate cache misses gracefully.
## When NOT to Use Memcached
Switch to Valkey (or Redis OSS) if you need any of the following:
| Requirement | Why Memcached cannot help |
|-------------|--------------------------|
| Data persistence | Node-based Memcached is purely in-memory with no persistence. Serverless Memcached supports daily backup/restore but not RDB/AOF snapshots. |
| Replication / automatic failover | Node-based Memcached has no replicas. Serverless Memcached provides transparent Multi-AZ data redundancy. |
| Sorted sets, lists, streams, hashes | Memcached supports only string key-value pairs |
| Pub/sub messaging | Not supported |
| RBAC or IAM authentication | Not supported for Memcached |
| Vector search | Requires Valkey 8.2 or above node-based |
| Lua scripting or server-side logic | Not supported |
| Atomic data structure operations (INCR on hash fields, ZADD, etc.) | Limited to basic INCR/DECR on string counters |
**Rule of thumb:** If your use case goes beyond simple get/set/delete with TTL, use Valkey.
## Memcached Constraints
- Maximum item size: 1 MB
- Maximum 60 nodes per node-based Memcached cluster (default quota; can be increased via Service Quotas; serverless scales differently)
- No persistence (node-based); daily backups only (serverless)
- No pub/sub, no Lua scripting, no server-side logic
- No RBAC or IAM authentication
- Data model limited to string key-value pairs
references/data-modeling/recipe-gallery.md
# Recipe Gallery
Non-obvious algorithms that require specific implementation details. Standard patterns (session stores, counters, pub/sub, shopping carts, job queues, activity feeds) are omitted because the model generates those from constraints alone. These recipes contain fractional score encoding, probabilistic math, or cleanup semantics that must be implemented precisely.
```python
# Shared setup for all recipes
# Uses valkey-py (pip install valkey). Alternatively, redis-py 4.1.2+ is the
# AWS-validated client (pip install redis) and is API-compatible; replace
# `import valkey` with `import redis` and `valkey.Valkey` with `redis.Redis`.
import valkey, time, json, math, random, uuid
_client = None
def get_client():
"""Lazy client initialization. Never create connections at module level.
Uses ValkeyCluster because serverless is cluster-mode-enabled only.
For cluster-mode-disabled node-based clusters, use valkey.Valkey instead.
"""
global _client
if _client is None:
# Serverless requires cluster-mode client
_client = valkey.ValkeyCluster(
host="your-cache-endpoint.serverless.use1.cache.amazonaws.com",
port=6379, ssl=True, decode_responses=True,
socket_connect_timeout=5, socket_timeout=2,
)
return _client
```
---
## 1. Leaderboard Tie-Breaking
Sorted sets order equal scores lexicographically, which is rarely desired. To break ties by time (earliest score wins), encode an inverse timestamp into the fractional part of the score.
**Data structure:** Sorted set | **Key:** `leaderboard:{game_id}`
**Score format:** `points.inverse_timestamp` (e.g., `1500.300000000`) | **Member:** Player ID
```python
def add_score_with_tiebreak(game_id: str, player_id: str, points: int) -> None:
"""Score = points + fractional inverse-timestamp.
Example: 1500 pts at epoch 1700000000 -> 1500.300000000
Higher fractional = earlier timestamp, so ZREVRANGE ranks earlier achievers first.
"""
max_ts = 9999999999 # far-future ceiling
fractional = (max_ts - int(time.time())) / (max_ts + 1)
get_client().zadd(f"leaderboard:{game_id}", {player_id: points + fractional})
def get_top_n(game_id: str, n: int = 10) -> list[tuple[str, float]]:
return get_client().zrevrange(f"leaderboard:{game_id}", 0, n - 1, withscores=True)
def get_player_rank(game_id: str, player_id: str) -> int | None:
return get_client().zrevrank(f"leaderboard:{game_id}", player_id)
```
**Why this works:** The fractional part is always < 1, so it never changes integer-point ranking. Within the same points, an earlier player has a larger fractional part (`max_ts - earlier > max_ts - later`), so ZREVRANGE places them higher. No secondary data structure or Lua script needed.
---
## 2. Cache-Aside with XFetch (Probabilistic Early Refresh)
Probabilistic early refresh prevents thundering herd on popular keys. As TTL approaches zero, random callers refresh before actual expiry so the key never truly expires under load.
**Data structure:** String | **Key:** `cache:{entity}:{id}`
**TTL:** Varies by staleness tolerance (e.g., 300s for product data, 60s for inventory)
```python
CACHE_TTL = 300 # 5 minutes
def cache_aside_get(entity: str, entity_id: str, db_fetch_fn, ttl: int = CACHE_TTL) -> dict:
"""Cache-aside with XFetch. db_fetch_fn(entity_id) -> dict reads from the DB."""
key = f"cache:{entity}:{entity_id}"
c = get_client()
cached = c.get(key)
if cached is not None:
data = json.loads(cached)
remaining_ttl = c.ttl(key)
if remaining_ttl > 0 and _should_early_refresh(remaining_ttl, ttl):
result = db_fetch_fn(entity_id)
if result is not None:
c.setex(key, ttl, json.dumps(result))
return result
return data
# Cache miss
result = db_fetch_fn(entity_id)
if result is not None:
c.setex(key, ttl, json.dumps(result))
return result
def _should_early_refresh(remaining_ttl: int, total_ttl: int) -> bool:
"""XFetch: returns True with increasing probability as TTL approaches 0.
Only activates in the last 20% of TTL. beta controls aggressiveness."""
beta = 1.0
if total_ttl <= 0:
return False
threshold = total_ttl * 0.2
if remaining_ttl >= threshold:
return False
return random.random() < math.exp(-remaining_ttl * beta / threshold)
```
**Why probabilistic, not deterministic?** A fixed threshold (e.g., "refresh at 10% TTL remaining") causes all concurrent readers to hit the DB at once. The exponential probability curve means on average exactly one caller refreshes early while the rest serve the cached value.
**When NOT to use:** Do not use XFetch for consistency-critical data (inventory counts, account balances) where serving a stale value during the refresh window is a correctness bug, not just a performance tradeoff.
---
## 3. Sliding-Window Rate Limiter
Each request is a unique sorted set member scored by timestamp. The window slides continuously, and denied requests are cleaned up to avoid polluting the count.
**Data structure:** Sorted set | **Key:** `ratelimit:{client_id}:{window}`
**Score:** Epoch seconds | **Member:** `{timestamp}:{uuid}` (UUID prevents same-ms collisions)
```python
def sliding_window_rate_limit(
client_id: str, window_name: str, max_requests: int, window_seconds: int,
) -> tuple[bool, int]:
"""Returns (allowed, remaining)."""
key = f"ratelimit:{client_id}:{window_name}"
now = time.time()
member = f"{now}:{uuid.uuid4().hex[:8]}"
c = get_client()
pipe = c.pipeline()
pipe.zremrangebyscore(key, "-inf", now - window_seconds) # prune old
pipe.zadd(key, {member: now}) # optimistic add
pipe.zcard(key) # count window
pipe.expire(key, window_seconds + 1) # auto-cleanup
results = pipe.execute()
current_count = results[2]
allowed = current_count <= max_requests
if not allowed:
c.zrem(key, member) # remove denied request to keep count accurate
return allowed, max(0, max_requests - current_count)
```
> **Cluster mode note:** In cluster mode (including serverless), all keys in a pipeline must hash to the same slot. This recipe uses a single key per pipeline call, so it works as-is. If you extend pipelines to operate on multiple keys, use hash tags (e.g., `{prefix}:key1`, `{prefix}:key2`) to ensure slot co-location.
**Why UUID members?** Timestamp-only members silently drop concurrent requests (ZADD updates the score of an existing member identified by member name, so duplicate timestamps would collapse into one entry). Using a UUID suffix as part of the member name ensures each request is a distinct entry.
**Why remove denied requests?** Without cleanup, denied requests inflate the count, making the limiter progressively stricter under burst traffic.
---
## 4. Cache Invalidation (Write-Through + Event-Driven + TTL Hybrid)
No single invalidation method is reliable on its own: write-through misses changes from other services, event-driven delivery can lag or fail, and TTL alone allows stale reads. Combining all three bounds staleness even when individual layers fail.
* **Write-through:** Update cache on every DB write in the same code path.
* **Event-driven:** Subscribe to change events (SNS/SQS, DynamoDB Streams) to invalidate keys modified by other services.
* **TTL safety net:** Always set a TTL as backstop, so stale keys self-heal if both other layers miss.
```python
CACHE_TTL = 300 # 5-minute backstop
def write_through(entity: str, entity_id: str, data: dict, db_write_fn) -> None:
"""Write to DB then cache. TTL acts as safety net."""
db_write_fn(entity_id, data)
get_client().setex(f"cache:{entity}:{entity_id}", CACHE_TTL, json.dumps(data))
def handle_change_event(entity: str, entity_id: str) -> None:
"""Called by SNS/SQS consumer when another service modifies the entity."""
get_client().delete(f"cache:{entity}:{entity_id}")
```
**Why all three layers?** Write-through keeps the cache fresh for the owning service. Event-driven catches external writes. TTL guarantees bounded staleness even if events are lost or write-through fails. TTL is the only layer that requires zero operational trust.
references/genai/agent-memory.md
# Semantic Agent Memory Implementation Guide
User preferences, learned facts, and cross-session recall via vector similarity search. Requires node-based Valkey 8.2 or above (recommend 9.0).
For conversation persistence (session state, message history, resumable sessions), see `session-store.md` instead.
---
## 1. Semantic Memory with mem0 + Valkey
mem0 provides a memory layer that extracts facts from conversations, deduplicates them, and retrieves by semantic similarity. It has a first-class Valkey connector (`provider: "valkey"`) that uses native `FT.CREATE`/`FT.SEARCH`.
### Configuration
```python
from mem0 import Memory
from utils.embeddings import VECTOR_DIM # Your project's embedding dimension constant (e.g., 1024 for Titan V2)
config = {
"vector_store": {
"provider": "valkey",
"config": {
"index_name": "app_memory",
"embedding_model_dims": VECTOR_DIM,
"valkey_url": "valkeys://your-endpoint.cache.amazonaws.com:6379",
"index_type": "hnsw",
"hnsw_m": 16,
"hnsw_ef_construction": 200,
"hnsw_ef_runtime": 10,
}
},
"embedder": {
# Provider-specific. See embedding-providers.md for mem0 embedder
# configs for Bedrock Titan, Cohere, fastembed, sentence-transformers.
},
}
memory = Memory.from_config(config)
```
For HNSW parameter tuning guidance, see `elasticache-search.md`.
### Short-Term vs Long-Term Memory
| Dimension | Short-term | Long-term |
|-----------|-----------|-----------|
| Scope | Session/Recent (days) | User (90 days) |
| TTL | 720 hours (~30 days) | 90 days |
| Content | Current task context, active preferences | Persistent preferences, learned facts |
| Identity | session_id as user_id (anonymous), or user_id with session as run_id (authenticated) | user_id |
Use `memory_type` in metadata to distinguish:
```python
from datetime import datetime, timedelta
def add_short_term(messages, session_id, user_id=None):
metadata = {"memory_type": "short_term",
"expires_at": (datetime.now() + timedelta(hours=720)).timestamp()}
if user_id:
memory.add(messages, user_id=user_id, agent_id="myapp",
run_id=session_id, metadata=metadata)
else:
memory.add(messages, user_id=session_id, agent_id="myapp",
run_id="global", metadata=metadata)
def add_long_term(messages, user_id):
metadata = {"memory_type": "long_term",
"expires_at": (datetime.now() + timedelta(days=90)).timestamp()}
memory.add(messages, user_id=user_id, agent_id="myapp",
run_id="global", metadata=metadata)
```
> **Note:** The `expires_at` field is stored as metadata only. It does not automatically expire keys in Valkey. Your application must handle key expiration separately (e.g., by calling `EXPIRE` on the underlying keys, or by running a periodic cleanup job that filters on `expires_at` and deletes expired entries).
### Add and Search
```python
messages = [
{"role": "user", "content": "I prefer size 12 wide width shoes"},
{"role": "assistant", "content": "Noted, I'll filter for size 12 wide."},
]
memory.add(messages, user_id="user_001", agent_id="myapp", run_id="global")
results = memory.search(
query="What shoe size does this customer wear?",
user_id="user_001",
limit=5,
)
# mem0's search() returns a similarity score (higher = more similar),
# which is the inverse of raw FT.SEARCH COSINE distance (lower = more similar).
for entry in results["results"]:
if entry.get("score", 0) >= 0.7:
print(f"Memory: {entry['memory']} (score: {entry['score']:.2f})")
```
**Important:** Always pass a truthy `run_id` (default to `"global"`).
---
## 2. Semantic Memory Without mem0
For teams that don't want the mem0 dependency, use the Valkey search commands directly. The building blocks (FT.CREATE schema, binary encoding, FT.SEARCH with TAG filters, result parsing) are all in `elasticache-search.md`.
Key differences from the generic patterns there:
* **Prefix:** `memory:{uuid}` to scope the index
* **Schema fields:** `user_id` TAG (isolation), `memory_type` TAG (short/long), `created_at` NUMERIC, `memory` TAG.
> **Note:** ElastiCache supports three field types for FT.CREATE: TAG, NUMERIC, and VECTOR. TAG fields support exact-match filtering. The TEXT field type (which supports full-text search) is only available on MemoryDB, not ElastiCache.
* **Query pattern:** always pre-filter by `user_id` before KNN: `(@user_id:{user123})=>[KNN 5 @embedding $vec AS score]`
* **TTL:** 90 days for long-term, 30 days for short-term, applied via EXPIRE after HSET
Use `generate_embedding()` and `embedding_to_bytes()` from the shared embedding utility. See `embedding-providers.md`.
---
## 3. Key Design Principles
**Namespace by user_id.** Every memory query should filter by `user_id` to isolate data across users. In mem0, this is automatic (pass `user_id=` to every call). In raw Valkey, use a TAG filter: `@user_id:{user123}`.
**Scope by agent_id.** In multi-agent systems, use `agent_id` to separate memories per agent. One agent's learned facts should not bleed into another's context.
**Scope by run_id within a session.** Use `run_id` to associate memories with a specific session or run. This lets you query "what did the agent learn in this session?" separately from "what does the agent know about this user overall?"
**Search threshold: 0.7.** A cosine similarity of 0.7 is a good default for memory relevance. Below that, results tend to be tangentially related rather than genuinely useful. Tune based on your embedding model and use case.
---
## Cross-References
* Conversation persistence (session state, no vector search): see `session-store.md`
* Embedding model selection and configuration: see `embedding-providers.md`
* ElastiCache platform constraints, FT.SEARCH encoding, HNSW tuning: see `elasticache-search.md`
* LangChain/LlamaIndex framework integration: see `framework-guide.md`
references/genai/elasticache-search.md
# ElastiCache Vector Search Constraints
This file contains all ElastiCache-specific constraints for vector search. Loaded on-demand when any genai pattern needs vector search (Mode 2 server-side or Mode 3).
---
## Platform Gate
| Requirement | Detail |
|---|---|
| Engine | ElastiCache engine version 8.2 or above for Valkey (node-based only). Note: server reports `valkey_version` as 8.1.x. ElastiCache engine version 8.2 (shown in AWS Console and API) maps to the underlying Valkey OSS version 8.1.x in `INFO server` output. Use `FT._LIST` to confirm vector search availability. |
| Serverless | NOT supported. Vector search is unavailable on serverless. |
| Data-tiering nodes | NOT supported (r6gd family) |
This is the single most common mistake. Serverless ElastiCache cannot run vector search.
## Version Detection (MUST run before any FT.* command)
Before emitting any FT.CREATE, FT.SEARCH, or other FT.* command, detect the server version and confirm vector search support. Vanilla Valkey 8.0.x does NOT have FT.CREATE. Only Valkey 8.2 or above on node-based ElastiCache supports it.
```python
from valkey.exceptions import ResponseError
def get_valkey_version(client) -> tuple[int, int]:
"""Return (major, minor) version of the connected Valkey/Redis server."""
info = client.info("server")
ver = info.get("valkey_version", info.get("redis_version", "0.0"))
major, minor = int(ver.split(".")[0]), int(ver.split(".")[1])
return major, minor
def supports_ft_search(client) -> bool:
"""True if the server supports FT.* vector search commands.
Uses FT._LIST probing instead of version comparison because
ElastiCache v8.2 reports valkey_version as 8.1.x, making
version-based detection unreliable.
"""
try:
client.execute_command("FT._LIST")
return True
except ResponseError:
return False
```
**Usage pattern:** Call `supports_ft_search(client)` once at startup. If it returns `False`, use the Python-side fallback below instead of FT.* commands. Never assume FT.CREATE is available without checking.
**Note:** Because ElastiCache v8.2 reports `valkey_version` as `8.1.x`, the `get_valkey_version` helper above should NOT be used for vector search detection. The `supports_ft_search` function probes with `FT._LIST` which is reliable regardless of reported version.
---
## Python-Side Vector Search Fallback
When the server does not support FT.* commands (Valkey < 8.2, serverless, or any environment without search), use application-side cosine similarity. This approach stores vectors as binary HASH fields and performs brute-force KNN in Python. Suitable for datasets under ~50K vectors.
```python
import struct
import math
def python_cosine_similarity(a: list[float], b: list[float]) -> float:
"""Cosine similarity between two vectors using only stdlib."""
dot = sum(x * y for x, y in zip(a, b))
norm_a = math.sqrt(sum(x * x for x in a))
norm_b = math.sqrt(sum(x * x for x in b))
if norm_a == 0 or norm_b == 0:
return 0.0
return dot / (norm_a * norm_b)
def python_knn_search(client, prefix: str, query_vec: list[float],
vector_field: str = "embedding", k: int = 5,
filter_field: str = None, filter_value: str = None) -> list[dict]:
"""Brute-force KNN search using Python-side cosine similarity.
Scans all keys matching prefix, decodes stored FLOAT32 vectors,
computes cosine similarity, returns top-k results sorted by similarity.
"""
cursor = 0
results = []
dim = len(query_vec)
while True:
cursor, keys = client.scan(cursor=cursor, match=f"{prefix}*", count=200)
for key in keys:
data = client.hgetall(key)
if filter_field and filter_value:
field_key = filter_field.encode() if isinstance(filter_field, str) else filter_field
stored = data.get(field_key, b"").decode()
if stored != filter_value:
continue
vec_bytes = data.get(vector_field.encode() if isinstance(vector_field, str) else vector_field)
if not vec_bytes or len(vec_bytes) != dim * 4:
continue
stored_vec = list(struct.unpack(f"{dim}f", vec_bytes))
score = python_cosine_similarity(query_vec, stored_vec)
key_str = key.decode() if isinstance(key, bytes) else key
results.append({"key": key_str, "similarity": score, "data": data})
if cursor == 0:
break
results.sort(key=lambda x: -x["similarity"])
return results[:k]
```
**Integration pattern:** Use `supports_ft_search` to pick the right path at startup:
```python
client = get_client()
if supports_ft_search(client):
ensure_index(client)
hits = search_similar(client, query_embedding, top_k=5)
else:
hits = python_knn_search(client, PREFIX, query_embedding, k=5)
```
---
## Command Boundary
| Command | Status on ElastiCache |
|---|---|
| FT.CREATE | Documented, supported |
| FT.SEARCH | Documented, supported |
| FT.INFO | Documented, supported |
| FT._LIST | Documented, supported |
| FT.DROPINDEX | Documented, supported |
> **⚠️ FT.AGGREGATE is NOT available on ElastiCache.** FT.AGGREGATE is supported on Amazon MemoryDB but is not supported on ElastiCache. If you need server-side aggregation over vector search results, use MemoryDB or perform aggregation client-side after FT.SEARCH.
---
## Hard Limits
| Limit | Value |
|---|---|
| Max indexes per cluster | 10 |
| Max fields per index | 50 |
| Max vector dimensions | 32768 |
| HNSW M (max edges per node) | 2,000,000 (practical recommendation: 16–64 for most workloads; higher values increase memory usage and index build time) |
| HNSW EF_CONSTRUCTION | 4096 |
| HNSW EF_RUNTIME | 4096 |
| Max prefixes per index | 16 |
| Tag field max length | 10,000 |
| Numeric field max length | 256 |
**Transaction restriction:** FT.CREATE, FT.DROPINDEX, and alias commands CANNOT run inside MULTI/EXEC, Lua scripts, or functions.
---
## Index Lifecycle / Backfill
FT.CREATE triggers a background backfill for all existing keys matching the PREFIX. Query operations attempted while an index is undergoing backfill are not allowed and are terminated with an error.
> **Backfill types:** During initial index creation (FT.CREATE), queries against the index are blocked and return an error until backfill completes. However, during scaling events (e.g., adding shards), the index may undergo backfill with reduced recall for search queries — queries are allowed but may return incomplete results.
**Check readiness with FT.INFO.** Key fields to monitor:
| Field | Meaning |
|---|---|
| `backfill_in_progress` | Whether backfill is still running |
| `backfill_percent_complete` | Estimate of backfill completion, a fractional number in the range [0..1] |
| `mutation_queue_size` | Pending mutations waiting to be indexed |
| `recent_mutations_queue_delay` | Lag between writes and indexing |
| `state` | Must be `ready` before querying |
Wait for `state=ready` before issuing FT.SEARCH.
---
## Hash Slot Constraint (Cluster Mode)
All keys queried by a single FT.SEARCH must reside in the same hash slot. Use hash tags to guarantee slot co-location:
```
doc:{myprefix}:chunk_001
doc:{myprefix}:chunk_002
```
The `{myprefix}` portion determines the slot. Without hash tags on multi-shard clusters, FT.SEARCH only returns results from the shard it executes on.
---
## Client Connection for Vector Operations
When using valkey-py or redis-py for vector operations, set `decode_responses=False`. Vector data is binary (FLOAT32 bytes) and must not be decoded as UTF-8.
```python
import valkey
client = valkey.Valkey(
host="endpoint", port=6379,
ssl=True, ssl_cert_reqs="required", # validate the server cert (production default)
# ssl_ca_certs="/path/to/ca-bundle.pem", # only if your OS lacks a system CA store
# Dev/tunnel ONLY (e.g. SSH tunnel to localhost, where the cert name won't match):
# ssl_cert_reqs="none", # INSECURE: disables cert validation; never in production
decode_responses=False, # CRITICAL for vector operations
)
```
If you need `decode_responses=True` for non-vector operations (session store, counters), use a separate client instance.
**Do NOT** use the high-level `valkey.commands.search` or `redis.commands.search` Python wrappers for FT.* commands. Their parameter signatures vary across library versions and produce hard-to-debug failures. Use `client.execute_command()` for all FT.CREATE, FT.SEARCH, FT.INFO, and FT.DROPINDEX calls.
**Do NOT** create the Valkey/Redis client at module level (top of file, import time). Always initialize connections inside a function or on first use. Module-level connections crash applications that import the module before the cache is reachable.
**Do NOT** call FT.CREATE at module level or import time. Create indexes lazily in an explicit setup function, guarded by an "already exists" check.
---
## Vector Binary Encoding
HASH vectors must be stored as binary little-endian IEEE 754 FLOAT32.
**Python encoding:**
```python
import struct
binary = struct.pack(f"{len(vec)}f", *vec)
```
* JSON vectors are stored as arrays (no binary encoding needed).
* Query vectors passed via PARAMS are also FLOAT32 bytes.
**Do NOT** use numpy for vector byte packing. Use only `struct.pack` from the Python standard library. numpy may not be available in all deployment environments.
---
## Distance to Similarity Conversion
FT.SEARCH returns cosine DISTANCE, not similarity.
| Value | Meaning (distance) | Meaning (similarity) |
|---|---|---|
| 0 | Identical | 1.0 |
| 2 | Opposite | 0.0 |
**Conversion:** `similarity = 1.0 - (distance / 2.0)` # Normalized to [0,1]; standard cosine similarity = 1 - distance
Cosine distance ranges from 0 (identical) to 2 (opposite). This formula normalizes to [0, 1].
Many models and tutorials use similarity (0=opposite, 1=identical). Mixing these up inverts threshold logic.
---
## FT.CREATE Schema Reference
```
FT.CREATE idx ON HASH PREFIX 1 prefix:
SCHEMA
embedding VECTOR HNSW 6 TYPE FLOAT32 DIM <VECTOR_DIM> DISTANCE_METRIC COSINE
request_id TAG
state_tags TAG SEPARATOR ,
slot_budget_usd NUMERIC
```
Key details:
* `6` after HNSW means 3 key-value pairs follow (TYPE, DIM, DISTANCE_METRIC). The number is the total count of arguments, not the number of pairs.
* `PREFIX 1 prefix:` scopes which hashes get indexed. The `1` is the number of prefixes.
* `TAG SEPARATOR ,` is optional. The default separator is `,`. Only specify SEPARATOR if you need a different delimiter character.
---
## FT.SEARCH Query Reference
Hybrid query with pre-filters:
```
FT.SEARCH idx "(@tag:{value})=>[KNN 1 @embedding $vec AS score]"
PARAMS 2 vec <binary_vector>
RETURN 2 request_id score
DIALECT 2
```
* `DIALECT 2` is required for KNN queries.
* `PARAMS 2` means 1 key-value pair follows (param name, param value). The number is the total count of arguments.
* Pre-filter goes before `=>`. Only documents matching the filter enter the KNN stage.
**Parsing FT.SEARCH results (valkey-py / redis-py with decode_responses=False):**
```python
# result = [total_count, key_name_bytes, [field_bytes, value_bytes, ...], ...]
total = int(result[0])
if total == 0:
return None
key_name = result[1] # bytes
fields = result[2] # flat list: [b"field1", b"value1", b"field2", b"value2", ...]
doc = {}
for i in range(0, len(fields), 2):
k = fields[i].decode() if isinstance(fields[i], bytes) else fields[i]
v = fields[i+1].decode() if isinstance(fields[i+1], bytes) else fields[i+1]
doc[k] = v
distance = float(doc["score"])
similarity = 1.0 - (distance / 2.0) # Normalized to [0,1]; standard cosine similarity = 1 - distance
```
The result format is a flat list, not a dict. Field names and values alternate. With `decode_responses=False`, all values are bytes and must be decoded manually.
---
## FLAT vs HNSW
Use FLAT for small datasets (<10K vectors). Exact brute-force search, no graph overhead, zero tuning. Switch to HNSW when dataset grows and query latency matters.
---
## HNSW Tuning Defaults
| Parameter | Default | Effect |
|---|---|---|
| M | 16 (max 2,000,000 per AWS limits; practical recommendation: 16-64 for most workloads) | Graph connectivity; higher = better recall, more memory |
| EF_CONSTRUCTION | 200 (max 4096) | Build-time search depth; higher = better index quality, slower writes |
| EF_RUNTIME | 10 (max 4096) | Query-time search depth; higher = better recall, slower queries. Can be overridden per-query. |
---
## TAG Field Escaping
Hyphens in TAG values must be escaped in queries:
```python
escaped = value.replace("-", "\\-")
```
Spaces should be replaced with underscores before storing.
---
## Complete Vector Search Recipe
Self-contained working example. Copy and adapt. Uses only stdlib + valkey-py, lazy connection, execute_command for all FT.* calls.
```python
import struct
import valkey
VECTOR_DIM = 384 # match your embedding model's output dimensions
INDEX_NAME = "idx:items"
PREFIX = "item:"
def get_client():
"""Lazy connection. Never call at module level."""
return valkey.Valkey(
host="your-endpoint", port=6379,
ssl=True, ssl_cert_reqs="required", # validate the server cert (production default)
# ssl_ca_certs="/path/to/ca-bundle.pem", # only if your OS lacks a system CA store
# Dev/tunnel ONLY (e.g. SSH tunnel to localhost, where the cert name won't match):
# ssl_cert_reqs="none", # INSECURE: disables cert validation; never in production
decode_responses=False,
)
def ensure_index(client):
"""Create index if it doesn't exist. Call once at app startup, not at import time."""
try:
client.execute_command(
"FT.CREATE", INDEX_NAME,
"ON", "HASH",
"PREFIX", "1", PREFIX,
"SCHEMA",
"embedding", "VECTOR", "HNSW", "6",
"TYPE", "FLOAT32", "DIM", str(VECTOR_DIM), "DISTANCE_METRIC", "COSINE",
"category", "TAG",
)
except Exception as e:
if "already exists" not in str(e).lower():
raise
def store_vector(client, item_id, embedding, category=""):
"""Store a vector. embedding is a list of floats."""
key = f"{PREFIX}{item_id}"
client.hset(key, mapping={
"embedding": struct.pack(f"{VECTOR_DIM}f", *embedding),
"category": category,
})
def search_similar(client, query_embedding, top_k=5):
"""KNN search. Returns list of {key, similarity} dicts."""
query_bytes = struct.pack(f"{VECTOR_DIM}f", *query_embedding)
result = client.execute_command(
"FT.SEARCH", INDEX_NAME,
f"*=>[KNN {top_k} @embedding $vec AS score]",
"PARAMS", "2", "vec", query_bytes,
"RETURN", "1", "score",
"LIMIT", "0", str(top_k),
"DIALECT", "2",
)
if int(result[0]) == 0:
return []
hits = []
for i in range(1, len(result), 2):
key = result[i].decode() if isinstance(result[i], bytes) else result[i]
fields = result[i + 1]
field_dict = {}
for j in range(0, len(fields), 2):
k = fields[j].decode() if isinstance(fields[j], bytes) else fields[j]
v = fields[j+1].decode() if isinstance(fields[j+1], bytes) else fields[j+1]
field_dict[k] = v
distance = float(field_dict["score"])
similarity = 1.0 - (distance / 2.0) # Normalized to [0,1]; standard cosine similarity = 1 - distance
hits.append({"key": key, "similarity": similarity})
return hits
```
---
## Error Recovery
Common failures and how to fix them.
**App crashes on import / startup:**
Move all `valkey.Valkey()` calls and `FT.CREATE` calls out of module scope. Wrap in a `get_client()` function called inside route handlers or on first use.
**`ImportError: No module named 'numpy'`:**
Replace `numpy.array(...).tobytes()` with `struct.pack(f"{len(vec)}f", *vec)`. Never depend on numpy for vector byte packing.
**`FT.CREATE` returns an error:**
Verify the command matches the exact syntax: `FT.CREATE <name> ON HASH PREFIX 1 <prefix> SCHEMA <fields>`. The most common mistakes: missing `SCHEMA` keyword, wrong argument count after `HNSW` (must be `6` for 3 key-value pairs: TYPE, DIM, DISTANCE_METRIC), or using `INDEX` instead of `CREATE`.
**`FT.SEARCH` returns 0 results when data exists:**
1. Check index state: `FT.INFO <index_name>`, look for `state: ready`. If backfill is in progress, wait.
2. Check `DIALECT 2` is present in the query. KNN queries require it.
3. Check the query vector is FLOAT32 bytes (struct.pack), not a string or list.
4. Check the key prefix matches what FT.CREATE was given.
**`ResponseError` mentioning `valkey.commands.search`:**
Do not use the high-level Python search wrapper. Replace with `client.execute_command("FT.SEARCH", ...)` as shown in the recipe above.
**Score/similarity values seem inverted:**
COSINE distance is 0 (identical) to 2 (opposite). Convert with `similarity = 1.0 - (distance / 2.0)`. If your thresholds aren't working, verify you're comparing similarity (not distance) against the threshold.
references/genai/embedding-providers.md
# Embedding Provider Chain for GenAI Sub-Skill
Load this reference when the user needs to choose or configure an embedding provider for vector search with ElastiCache (Valkey).
> **Important:** Vector search (FT.CREATE / FT.SEARCH) requires node-based ElastiCache Valkey 8.2 or later (recommend 9.0). It is not available on ElastiCache Serverless. If using serverless, see the application-side comparison approach in `semantic-cache.md`.
## Provider Chain (Decision Order)
**First, check `.elasticache/requirements.json`.** If `infrastructure.embedding_provider`, `infrastructure.embedding_model`, and `infrastructure.embedding_dim` are already set, use those values. Do not re-ask.
**If not set, ask the user:** "Do you have a preferred embedding model or provider?"
* **User names a specific model** (OpenAI, Cohere, Titan, etc.) -> Use that model. Match the DIM in FT.CREATE to its output dimensions.
* **User says Bedrock / AWS** -> Use Bedrock Titan Embed v2 (Option 1 below)
* **User has no preference, or no API access** -> Default to open-source fastembed (Option 3 below). Zero setup, zero cost, works anywhere.
**After selection:**
### A. Persist the choice to `requirements.json`
```json
{
"infrastructure": {
"embedding_provider": "bedrock",
"embedding_model": "amazon.titan-embed-text-v2:0",
"embedding_dim": 1024,
"embedding_module": "utils/embeddings.py"
}
}
```
### B. Generate a reusable embedding utility in the user's project
Create a file (default: `utils/embeddings.py`, or wherever fits the user's project structure) that exports:
* `generate_embedding(text: str) -> list[float]`
* `embedding_to_bytes(embedding: list[float]) -> bytes`
* `VECTOR_DIM: int`
Use the provider-specific code from the Standard Functions section below. This file is generated ONCE. Every subsequent file the model generates imports from it:
```python
from utils.embeddings import generate_embedding, embedding_to_bytes, VECTOR_DIM
```
Save the file path in `requirements.json` as `infrastructure.embedding_module` so the model never regenerates it.
## On Return Visits
If `infrastructure.embedding_module` is set in `requirements.json`, read that file to confirm it exists. If it exists, import from it. Never regenerate. If the file was deleted, regenerate it from the stored provider/model/dim values.
---
## Provider Options
| Provider | Model | Dimensions | Requires API | Best for |
|----------|-------|-----------|-------------|----------|
| Bedrock Titan | amazon.titan-embed-text-v2:0 | 256/512/1024 | Yes (Bedrock) | Production |
| Bedrock Cohere | cohere.embed-english-v3 | 1024 | Yes (Bedrock) | English-only (use cohere.embed-multilingual-v3 for multilingual) |
| fastembed | BAAI/bge-small-en-v1.5 | 384 | No | Prototyping |
| sentence-transformers | all-MiniLM-L6-v2 | 384 | No | Prototyping with more model choice |
### Option 1: Amazon Bedrock Titan Embed Text v2 (Recommended for production)
* **Model ID:** `amazon.titan-embed-text-v2:0`
* **Dimensions:** 256, 512, or 1024 (recommend 1024 for best accuracy; 256 for cost-sensitive workloads)
* **IAM permissions needed:** `bedrock:InvokeModel` on the Titan Embed model ARN
### Option 2: Amazon Bedrock Cohere Embed
* **Model ID:** `cohere.embed-english-v3` or `cohere.embed-multilingual-v3`
* **Dimensions:** 1024
* **IAM permissions needed:** `bedrock:InvokeModel` on the Cohere Embed model ARN
* **Note:** Cohere uses `texts` (list) instead of `inputText` (string), and requires `input_type`. Use `"search_document"` when storing and `"search_query"` when querying.
### Option 3: fastembed (Open-Source, No API Key Needed)
* **Install:** `pip install fastembed`
* **Default model:** `BAAI/bge-small-en-v1.5` (384 dims) or `BAAI/bge-base-en-v1.5` (768 dims)
* **Tradeoff:** Lower accuracy than Titan, but zero cost and zero setup
* **Good for:** Prototyping, development, CI/CD tests, users without AWS accounts
### Option 4: Amazon Bedrock Cohere Embed v4 (Multimodal)
* **Model ID:** `cohere.embed-v4:0`
* **Dimensions:** 256, 512, 1024, or 1536 (default 1536)
* **Context:** Up to 128k tokens
* **Multimodal:** Supports interleaved text + image inputs via `inputs` field
* **IAM permissions needed:** `bedrock:InvokeModel` on the Cohere Embed v4 model ARN
* **Note:** Uses a different request format from v3. See AWS Bedrock docs for the `inputs` field schema.
### Option 5: sentence-transformers (Open-Source, More Model Choices)
* **Install:** `pip install sentence-transformers`
* **Popular model:** `all-MiniLM-L6-v2` (384 dims)
* **Tradeoff:** Wider model selection, but heavier dependency (pulls in PyTorch)
---
## mem0 Embedder Configs
Use these in the `embedder` block of mem0's config dict. See `agent-memory.md` for the full mem0 configuration.
**Bedrock Titan:**
```json
{
"embedder": {
"provider": "aws_bedrock",
"config": {
"model": "amazon.titan-embed-text-v2:0",
"aws_region": "us-east-1"
}
}
}
```
**Bedrock Cohere:**
```json
{
"embedder": {
"provider": "aws_bedrock",
"config": {
"model": "cohere.embed-english-v3",
"aws_region": "us-east-1"
}
}
}
```
**fastembed / sentence-transformers:**
```json
{
"embedder": {
"provider": "huggingface",
"config": {
"model": "BAAI/bge-small-en-v1.5"
}
}
}
```
---
## Standard Functions (Utility File Templates)
These are the canonical implementations for the reusable embedding utility file. Use the one matching the user's chosen provider.
**Bedrock Titan:**
```python
import boto3, json, struct
_bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
VECTOR_DIM = 1024
def generate_embedding(text: str) -> list[float]:
response = _bedrock.invoke_model(
modelId="amazon.titan-embed-text-v2:0",
# "embeddingTypes" is optional; float is the default. Include only if you need
# a specific type (e.g., "binary"). Omitting it returns float embeddings.
body=json.dumps({"inputText": text, "dimensions": VECTOR_DIM}),
)
return json.loads(response["body"].read())["embedding"]
def embedding_to_bytes(embedding: list[float]) -> bytes:
return struct.pack(f"{VECTOR_DIM}f", *embedding)
```
**Bedrock Cohere:**
```python
import boto3, json, struct
_bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
VECTOR_DIM = 1024
def generate_embedding(text: str, query: bool = False) -> list[float]:
response = _bedrock.invoke_model(
modelId="cohere.embed-english-v3",
body=json.dumps({
"texts": [text],
"input_type": "search_query" if query else "search_document",
"truncate": "END",
}),
)
return json.loads(response["body"].read())["embeddings"][0]
def embedding_to_bytes(embedding: list[float]) -> bytes:
return struct.pack(f"{VECTOR_DIM}f", *embedding)
```
**Cohere note:** Pass `query=True` when embedding a search query (retrieval), `query=False` (default) when embedding documents for storage. Other providers ignore this parameter.
**fastembed:**
```python
import struct
from fastembed import TextEmbedding
_model = TextEmbedding("BAAI/bge-small-en-v1.5")
VECTOR_DIM = 384
def generate_embedding(text: str) -> list[float]:
return list(_model.embed([text]))[0].tolist()
def embedding_to_bytes(embedding: list[float]) -> bytes:
return struct.pack(f"{VECTOR_DIM}f", *embedding)
```
**sentence-transformers:**
```python
import struct
from sentence_transformers import SentenceTransformer
_model = SentenceTransformer("all-MiniLM-L6-v2")
VECTOR_DIM = 384
def generate_embedding(text: str) -> list[float]:
return _model.encode(text).tolist()
def embedding_to_bytes(embedding: list[float]) -> bytes:
return struct.pack(f"{VECTOR_DIM}f", *embedding)
```
---
## Batch Embedding for Bulk Ingestion
For bulk ingestion (>1K documents), batch embedding calls to avoid per-request overhead. Titan accepts sequential calls (add exponential backoff for throttling). Cohere natively supports batching via `texts: [list]` with up to 96 texts per call.
---
## Dimension Compatibility Warning
The `FT.CREATE` index `DIM` must match your embedding model's output dimension **exactly**. If you change embedding providers, you must:
1. Drop the existing index: `FT.DROPINDEX <index_name>`
2. Delete all existing vector keys: `SCAN` + `DEL` by prefix
3. Recreate the index with the new `DIM`
4. Re-embed and re-ingest all data
This is destructive. Choose your embedding model before ingesting production data.
**Backfill warning:** After recreating an index with `FT.CREATE`, queries (`FT.SEARCH`) are **not allowed** while the index is backfilling and will return an error. Use `FT.INFO <index_name>` and check the `state` field -- wait until it reports `ready` before issuing queries.
references/genai/framework-guide.md
# Framework Integration Guide
How to connect popular AI/ML frameworks to ElastiCache Valkey. This file covers framework-specific wiring only. For full implementation patterns, see the dedicated guides linked in each section.
---
## 1. Strands Agents
**Package:** `strands-valkey-session-manager` (community package, v0.1.0+ — MIT license, maintained by jeromevdl)
Import: `from strands_valkey_session_manager import ValkeySessionManager`
Implements Strands' `SessionManager` interface. Persists conversation messages, agent state, and session metadata to Valkey automatically. Serverless OK.
For full setup code and key design patterns, see `session-store.md`.
Strands does not include a built-in semantic cache. Wrap the agent call with cache check/store logic using the approach in `semantic-cache.md`.
---
## 2. mem0
**Package:** `mem0`
Native Valkey vector store provider (`provider: "valkey"`). Handles index creation, embedding storage, and similarity search internally. Requires node-based Valkey 8.2 or later (recommend 9.0).
Key wiring points:
* Use `valkeys://` URL scheme (the `s` enables TLS). Port 6379 for node-based.
* The `llm` block is required for mem0's fact extraction. Use Bedrock:
```python
"llm": {
"provider": "aws_bedrock",
"config": {
"model": "us.anthropic.claude-sonnet-4-6-v1:0",
"max_tokens": 512,
}
}
```
* Always pass a `user_id` to `memory.add()` and `memory.search()` for user-scoped memory isolation.
* Key config fields: `embedding_model_dims` (e.g., `1024` for Titan V2) and `index_type` (`flat` or `hnsw`).
For full mem0 config, HNSW parameters, short/long-term memory patterns, and identity model, see `agent-memory.md`. For mem0 embedder configs per provider, see `embedding-providers.md`.
---
## 3. LangChain / LangGraph
**Package:** `langgraph-checkpoint-aws` (install with `pip install 'langgraph-checkpoint-aws[valkey]'`)
### Checkpointing (ValkeySaver)
Persist LangGraph agent state across invocations.
```python
from langgraph_checkpoint_aws import ValkeySaver
with ValkeySaver.from_conn_string(
"valkeys://your-cluster.serverless.use1.cache.amazonaws.com:6379",
ttl_seconds=3600,
) as checkpointer:
graph = builder.compile(checkpointer=checkpointer)
config = {"configurable": {"thread_id": "session-1"}}
result = graph.invoke({"messages": [HumanMessage(content="Hello")]}, config)
```
### LLM Caching (ValkeyCache)
Exact-match caching of LLM responses (no vector search needed, works on serverless).
```python
from langgraph_checkpoint_aws import ValkeyCache
cache = ValkeyCache.from_conn_string(
"valkeys://your-cluster.serverless.use1.cache.amazonaws.com:6379",
prefix="llm_cache:",
ttl=3600,
)
```
Use `valkeys://` URL scheme for TLS.
### Semantic Caching (ValkeyStore)
Vector-based semantic caching of LLM responses (requires node-based Valkey 8.2 or later; recommend 9.0).
```python
from langgraph_checkpoint_aws import ValkeyStore, ValkeyIndexConfig
index_config = ValkeyIndexConfig(
collection_name="semantic_cache",
embed=embeddings,
fields=["query"],
index_type="HNSW",
dims=1024,
)
store = ValkeyStore.from_conn_string(
"valkeys://your-cluster.cache.amazonaws.com:6379",
index=index_config,
)
store.setup()
```
Unlike ValkeyCache (exact-match), ValkeyStore uses vector search to match semantically similar queries. For full implementation, see `semantic-cache.md`.
---
## 4. ElastiCache TLS Connection Reference
All frameworks must use TLS when connecting to ElastiCache.
| Client / Framework | TLS mechanism | Example |
|---|---|---|
| valkey-py | `ssl=True, ssl_cert_reqs="required"` (use `"none"` only for tunnel/dev) | `valkey.Valkey(host=..., ssl=True, ssl_cert_reqs="required")` |
| valkey-glide | `use_tls=True` + `TlsAdvancedConfiguration(use_insecure_tls=True)` | See valkey-glide docs |
| URL-based (LangChain) | `valkeys://` scheme | `valkeys://endpoint:6379` |
| mem0 | `valkeys://` URL scheme in `valkey_url` config | `valkeys://your-cluster.cache.amazonaws.com:6379` |
| Strands session manager | Pass a TLS-configured `valkey.Valkey` client | See `session-store.md` |
### Port Reference
| Cluster type | Default port | Notes |
|---|---|---|
| Node-based (primary) | 6379 | Standard Valkey port |
| Node-based (reader) | 6379 | Same port as primary; use the reader endpoint address |
| Serverless (primary) | 6379 | Single endpoint |
| Serverless (reader) | 6380 | Eventually-consistent reads routed to closest node (could be primary). Obtain the address from the `ReaderEndpoint` attribute in `DescribeServerlessCaches`. |
**Security group note:** For serverless caches, your VPC security group must allow inbound TCP on both port 6379 (primary) and port 6380 (reader). If you only open 6379, reader-endpoint connections will fail silently.
For raw valkey-py and valkey-glide connection examples, see `elasticache-search.md`. For IAM authentication setup, see the setup sub-skill (`references/setup/auth-model-selector.md`).
references/genai/instructions.md
# GenAI, Search & Vector Workloads
**When to use:** The user wants to implement semantic caching, conversational memory, RAG, vector search, recommendation engines, or any GenAI/LLM pattern with ElastiCache. Also useful when the user asks about caching LLM responses (exact or semantic match).
**When not needed:** The user is working on traditional caching patterns (session store, leaderboard, rate limiting), general setup, monitoring, or migration without a GenAI component.
## Loading
Read this file first. Load files listed below on demand based on the routing decision.
| File | Load when |
|------|-----------|
| `elasticache-search.md` | Any pattern needing vector search (Mode 2 server-side or Mode 3). Contains platform gate, version detection, limits, encoding, backfill, AND a Python-side cosine fallback for when FT.* is unavailable. **ALWAYS load this file before generating vector code** so the version guard and fallback are included. |
| `semantic-cache.md` | User wants semantic caching for LLM/API responses. Step-by-step: FT.CREATE, embed, FT.SEARCH, threshold, store. |
| `session-store.md` | User wants session state, message history, resumable conversations, or Strands session management. Plain data structures, serverless OK. |
| `agent-memory.md` | User wants semantic agent memory, cross-session recall via similarity, or mem0 integration. Requires node-based Valkey 8.2 or above (recommend 9.0). |
| `rag-retrieval.md` | User wants RAG, knowledge base retrieval, or document search. Step-by-step: schema, index, ingest, hybrid query. |
| `framework-guide.md` | User mentions a specific framework (Strands, mem0, LangChain) or asks how to connect their AI app. |
| `embedding-providers.md` | User needs to choose or configure an embedding provider. Bedrock Titan v2 preferred, open-source fallback (fastembed, sentence-transformers). |
## Check for existing context
Before starting, read `.elasticache/requirements.json` if it exists. If the `genai` section is set (`mode`, `mode_2_path`, `framework`), use those values instead of re-asking. If `infrastructure.embedding_module` is set and the file exists, import from it.
## Three-Way GenAI Routing
Before recommending a pattern, classify the user's need into one of these three modes. Many teams asking for "RAG cache" actually need semantic response reuse (Mode 2), not full vector search (Mode 3). Ask a clarifying question before jumping to the heaviest option.
### Mode 1: Plain Cache
Standard caching of LLM or API responses by exact key match. No vector index needed.
When to use: the user wants to avoid repeated identical LLM calls. Responses are looked up by an exact key (prompt hash, request fingerprint, or deterministic cache key).
Deployment: **serverless Valkey**. This is just regular SET/GET. Route to `data-modeling` sub-skill for key schema and TTL guidance.
### Mode 2: Semantic Response Cache
Cache LLM responses with semantic similarity matching so "nearly identical" prompts return a cached response.
When to use: the user wants fuzzy cache hits, where semantically close prompts share a cached response.
Two implementation paths:
- **Application-side embedding comparison**: app computes embeddings, stores as plain keys, does similarity math client-side. Serverless OK. Works on ANY Valkey/Redis version.
- **Server-side vector similarity**: Valkey performs the similarity search via FT.SEARCH. Requires node-based Valkey 8.2 or above (recommend 9.0).
Ask the user which approach they prefer. Default to application-side unless they have high query volume AND confirmed Valkey 8.2 or above node-based. If the user's Valkey version is < 8.2, application-side is the ONLY option. Always load `elasticache-search.md` and use the `supports_ft_search()` version check before generating any FT.* code.
Deployment: **serverless Valkey** for application-side; **node-based Valkey 8.2 or above (recommend 9.0)** for server-side.
Load: `semantic-cache.md`, `elasticache-search.md` (if server-side), `embedding-providers.md`
### Mode 3: Full Vector Search
Vector indexing, KNN/ANN queries, RAG retrieval, recommendation via embeddings, agent memory with vector recall, or any workload that requires Valkey to maintain and query a vector index.
When to use: the user needs to store many embeddings and retrieve the top-K most similar. Typical for RAG knowledge bases, agent memory, recommendation engines, semantic search, catalog search.
Deployment: **node-based Valkey 8.2 or above** (mandatory; recommend 9.0). No serverless.
Load: `elasticache-search.md` plus the relevant pattern file (`rag-retrieval.md`, `agent-memory.md`, or `semantic-cache.md`), `embedding-providers.md`, `framework-guide.md` (if framework mentioned)
### How to classify
1. Does the user need similarity matching at all? If no, use **Mode 1**.
2. Is the similarity matching scoped to caching LLM responses? If yes, use **Mode 2** and ask application-side vs server-side.
3. Does the user need to index, store, and query a corpus of embeddings? If yes, use **Mode 3**.
## Hard Routing Rules (non-negotiable)
1. **Vector search MUST use node-based Valkey 8.2 or above (recommend 9.0).** Serverless does NOT support vector search. Never recommend serverless for vector search workloads, not even as a temporary or future option.
2. **Global Datastore MUST use node-based clusters.** If the GenAI workload requires multi-Region replication, it must be node-based.
3. **Vector search is NOT available on data tiering node types (r6gd family).** Do not recommend r6gd instances for any workload requiring vector search.
4. **Changing embedding providers requires full re-indexing.** If the user wants to switch embedding models, warn that ALL existing vectors must be deleted and re-embedded. This is destructive. Confirm before proceeding.
## Common Mistakes to Avoid
1. Do NOT generate FT.CREATE or FT.SEARCH code without first loading `elasticache-search.md` and including the `supports_ft_search()` guard.
2. Do NOT use numpy for vector byte packing. Use `struct.pack` only.
3. When using raw valkey-py/redis-py client directly, do NOT use the high-level `redis.commands.search` or `valkey.commands.search` Python wrappers for FT.* commands. Use `execute_command()` instead. When using supported frameworks like `langgraph-checkpoint-aws` (ValkeyStore) or Mem0, their built-in abstractions (e.g., `store.search()`, `store.put()`, `m.add()`, `m.search()`) are acceptable.
4. Do NOT recommend serverless for any Mode 3 workload or Mode 2 server-side.
5. Do NOT generate inline embedding code. Always import from the shared utility file (`infrastructure.embedding_module`).
6. Do NOT assume the user's Valkey version supports vector search. Always check with `supports_ft_search()`.
7. Do NOT skip the embedding provider selection step. The FT.CREATE DIM must match the embedding model's output dimensions exactly.
8. Be aware of an inconsistency in AWS docs regarding the HNSW M parameter: the FT.CREATE command doc states the maximum is 512, while the vector search limits page (vector-search-features-limits.md) states the maximum is 2,000,000. Treat the limits page as authoritative for parametric restrictions.
## Data-Plane Access
Mode 2 (server-side) and Mode 3 workloads need data-plane access to run FT.CREATE, FT.SEARCH, JSON operations, and other Valkey commands. The primary path is the valkey-py client (`pip install valkey`). Connect through the cache endpoint (via the jump host or SSM tunnel covered in setup). The agent writes Python that runs each command via `execute_command()`, consistent with the guidance in "Common Mistakes to Avoid" rule 3.
## Engine Requirement
All GenAI patterns require **Valkey**. Vector search patterns specifically require **Valkey 8.2 or above on node-based clusters (recommend 9.0)**.
> Vector search is available with Valkey 8.2 or above on node-based clusters in all AWS Regions at no additional cost. Not supported on data-tiering instances (r6gd) or serverless caches.
If the user doesn't have a cache yet, hand off to `setup` and ensure node-based Valkey 8.2 or above is selected (recommend 9.0) when vector search is needed.
## Workflow
1. Classify the user's need (Mode 1/2/3). If `genai.mode` is already set in `requirements.json`, skip classification.
2. Persist the mode to `requirements.json` under `genai.mode`. If Mode 2, also persist `genai.mode_2_path` (`"app-side"` or `"server-side"`).
3. If Mode 2 (server-side) or Mode 3: check `requirements.json` for `infrastructure.embedding_module`. If set and the file exists, import from it. If not set, load `embedding-providers.md`, ask the user for their preferred model, generate the reusable utility file in their project, and persist the choice.
4. Load the relevant reference files per the table above
5. If vector search is needed, verify node-based Valkey 8.2 or above (recommend 9.0; load `elasticache-search.md` for constraints)
6. Walk through the pattern-specific implementation guide. All generated code imports from the embedding utility created in step 3 rather than generating inline embedding code.
7. If the user mentions a framework, load `framework-guide.md` and persist the choice to `genai.framework` in `requirements.json`.
## After implementation
### Update requirements artifact
After the pattern is implemented, update `.elasticache/requirements.json`. GenAI owns the `genai` section. Read the existing file first, merge your updates, then write it back. Do not overwrite fields owned by setup or requirements.
## Freshness disclaimer
When your response includes pricing, version constraints, or feature availability, include the freshness disclaimer per SKILL.md Global Rule #5: "For current pricing see https://aws.amazon.com/elasticache/pricing/. For current feature availability see https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/."
references/genai/rag-retrieval.md
# RAG Retrieval with ElastiCache Valkey
## When to Use
RAG retrieval from ElastiCache Valkey. Best for:
- **Real-time knowledge bases**: Valkey indexes update automatically when matching keys are mutated. No batch re-indexing required.
- **Low-latency agentic AI retrieval**: Sub-millisecond vector search vs hundreds of ms from dedicated vector DBs.
- **Colocated vector + cache workloads**: The vector store lives alongside your application cache, eliminating a separate service hop.
## Deployment Requirement
Node-based Valkey 8.2 or above clusters only (recommend 9.0). Vector search is NOT available on serverless.
See `elasticache-search.md` for full platform constraints and limits.
**Cluster mode:** If running on a multi-shard cluster, use hash tags to ensure all document chunks for a single index land on the same shard:
```
doc:{corpus}:{doc_id}:{chunk_id}
```
The `{corpus}` hash tag forces slot co-location. Without this, FT.SEARCH returns partial results (it only queries the shard it executes on). See `elasticache-search.md` Hash Slot Constraint for details.
> **Note:** The code examples below use the simplified key pattern `doc:{doc_id}:{chunk_id}` which assumes a single-shard cluster. For multi-shard clusters, adapt the key pattern to include a hash tag, e.g., `doc:{corpus}:{doc_id}:{chunk_id}`, and update the index prefix accordingly.
---
## Step 1: Design the Document Schema
Use HASH keys with a consistent prefix. All keys matching the prefix are auto-indexed.
```
Key pattern: doc:{doc_id}:{chunk_id}
Fields:
embedding - FLOAT32 bytes (match your embedding model's dimensions)
text - the chunk text (plain string, not indexed for search)
source - source document name/path (TAG)
category - document category (TAG)
created_at - unix timestamp (NUMERIC)
chunk_index - position in source document (NUMERIC)
```
**Prefix-based scoping**: When the index uses `PREFIX 1 doc:`, every HASH key starting with `doc:` is automatically indexed. New keys, updated keys, and deleted keys are reflected in the index without manual intervention.
---
## Step 2: Create the Index
```python
from utils.embeddings import VECTOR_DIM
# Note: Maximum of 10 indexes can be created per cluster.
# For multi-tenant RAG, use TAG pre-filters on a shared index rather than per-tenant indexes.
client.execute_command(
"FT.CREATE", "idx:docs",
"ON", "HASH",
"PREFIX", "1", "doc:",
"SCHEMA",
"embedding", "VECTOR", "HNSW", "6",
"TYPE", "FLOAT32", "DIM", str(VECTOR_DIM), "DISTANCE_METRIC", "COSINE",
"source", "TAG",
"category", "TAG",
"created_at", "NUMERIC",
"chunk_index", "NUMERIC",
)
```
After creation, if matching keys already exist, Valkey backfills the index in the background. **Query operations attempted while an index is undergoing backfill are not allowed and are terminated with an error.** Wait for readiness before querying:
> **Backfill types:** During initial index creation (FT.CREATE), queries against the index are blocked and return an error until backfill completes. However, during scaling events (e.g., adding shards), the index may undergo backfill with reduced recall for search queries — queries are allowed but may return incomplete results.
```python
import time
def wait_for_index_ready(client, index_name, timeout=60):
"""Poll FT.INFO until index state is 'ready'.
Note: FT.INFO returns a flat array with nested sub-arrays for some fields.
This simple zip approach works for top-level fields like 'state' but is
fragile if the response structure changes. For production use, iterate
through the list looking for the 'state' key explicitly.
"""
start = time.time()
while time.time() - start < timeout:
info = client.execute_command("FT.INFO", index_name)
info_dict = dict(zip(info[::2], info[1::2]))
state = info_dict.get("state") or info_dict.get(b"state")
if state in ("ready", b"ready"):
return True
time.sleep(0.5)
raise TimeoutError(f"Index {index_name} not ready after {timeout}s")
```
**Replica backfill caveat:** Backfill completion is not synchronized between primary and replicas. If your application reads from replicas, verify backfill completion on all replicas before issuing search queries. See `elasticache-search.md` for details.
---
## Step 3: Ingest Documents
Chunking, embedding, and storing in a pipeline batch:
```python
from utils.embeddings import generate_embedding, embedding_to_bytes
def ingest_documents(client, documents, batch_size=50):
pipe = client.pipeline(transaction=False)
count = 0
for doc in documents:
chunks = chunk_document(doc["text"])
for i, chunk_text in enumerate(chunks):
embedding = generate_embedding(chunk_text)
key = f"doc:{doc['id']}:{i}"
pipe.hset(key, mapping={
"embedding": embedding_to_bytes(embedding),
"text": chunk_text,
"source": doc.get("source", ""),
"category": doc.get("category", ""),
"created_at": str(doc.get("created_at", -1)),
"chunk_index": str(i),
})
count += 1
if count % batch_size == 0:
pipe.execute()
pipe = client.pipeline(transaction=False)
pipe.execute()
return count
```
**Chunking guidance:**
| Content type | Chunk size | Overlap | Rationale |
|---|---|---|---|
| Factual Q&A, structured data, code | 256-512 tokens | 50 tokens | Precise retrieval; smaller chunks reduce noise in results |
| General documentation, how-to guides | 512-1024 tokens | 50-100 tokens | Balanced retrieval quality and context |
| Narrative content, long-form reasoning | 1024-2048 tokens | 100-200 tokens | Preserves reasoning chains and context |
Prefer chunking by semantic boundaries (paragraphs, sections, headers) over fixed-size splits when the source has structure. Fixed-size is acceptable for unstructured text.
Overlap prevents information loss at chunk boundaries. A sentence split across two chunks without overlap is lost to both. 10-20% overlap is the standard range.
Note: Bedrock Titan Embed v2 supports up to 8192 input tokens. For retrieval tasks, AWS recommends segmenting documents into logical segments such as paragraphs or sections rather than embedding at the maximum token length.
**Real-time updates**: To update a document chunk, just HSET the same key with new field values. Valkey re-indexes automatically. No rebuild needed.
---
## Step 4: Retrieve
Supports pure vector search or hybrid (vector + metadata pre-filters).
| Shape | Query String |
|-------|-------------|
| Pure vector | `*=>[KNN k @embedding $vec AS score]` |
| Vector + TAG filter | `(@category:{technical})=>[KNN k @embedding $vec AS score]` |
| Vector + numeric range | `(@created_at:[1700000000 +inf])=>[KNN k @embedding $vec AS score]` |
| Combined filters | `(@category:{technical} @created_at:[1700000000 +inf])=>[KNN k @embedding $vec AS score]` |
The filter expression is a **pre-filter**: it narrows the candidate set before KNN runs.
```python
from utils.embeddings import generate_embedding, embedding_to_bytes
def retrieve_chunks(client, query_text, category=None,
min_timestamp=None, top_k=5):
query_bytes = embedding_to_bytes(generate_embedding(query_text))
filter_parts = []
if category:
safe_cat = category.replace("-", "\\-")
filter_parts.append(f"@category:{{{safe_cat}}}")
if min_timestamp:
filter_parts.append(f"@created_at:[{min_timestamp} +inf]")
if filter_parts:
pre_filter = "(" + " ".join(filter_parts) + ")"
query_str = f"{pre_filter}=>[KNN {top_k} @embedding $vec AS score]"
else:
query_str = f"*=>[KNN {top_k} @embedding $vec AS score]"
results = client.execute_command(
"FT.SEARCH", "idx:docs", query_str,
"PARAMS", "2", "vec", query_bytes,
"RETURN", "3", "text", "source", "score",
"LIMIT", "0", str(top_k),
"DIALECT", "2",
)
chunks = []
if results[0] == 0:
return chunks
for i in range(1, len(results), 2):
fields = results[i + 1]
field_dict = dict(zip(fields[::2], fields[1::2]))
text = field_dict.get(b"text", b"").decode()
source = field_dict.get(b"source", b"").decode()
score = float(field_dict.get(b"score", b"0"))
# COSINE distance is [0, 2] (0=identical, 2=opposite). Convert to [0, 1] similarity.
similarity = 1.0 - (score / 2.0)
chunks.append({"text": text, "source": source, "similarity": similarity})
return chunks
```
---
## Step 5: Delete Documents
When source documents are deleted or fully replaced:
```python
def delete_document_chunks(client, doc_id):
"""Remove all chunks for a document. Index removes them automatically."""
cursor = "0"
prefix = f"doc:{doc_id}:"
while True:
cursor, keys = client.scan(cursor=cursor, match=f"{prefix}*", count=100)
if keys:
client.delete(*keys)
if cursor == 0:
break
```
For bulk refresh: delete existing chunks, then re-ingest the updated document.
---
## Cross-References
- Embedding providers and utility setup: see `embedding-providers.md`
- Platform constraints, FT.SEARCH encoding, HNSW tuning: see `elasticache-search.md`
- Framework integration (LangChain, Strands): see `framework-guide.md`
references/genai/semantic-cache.md
# Semantic Cache Implementation Guide
## When to use
Semantic cache for LLM/API responses. Avoids redundant inference calls when prompts are semantically similar (not just exact match). Two deployment options:
* **Application-side comparison** (serverless OK): Generate embeddings in app code, compare locally. No vector index needed in Valkey, just store/retrieve by key. Works with ElastiCache Serverless.
* **Server-side vector similarity** (node-based Valkey 8.2 or above required; recommend 9.0): FT.SEARCH with KNN finds the nearest cached prompt. Sub-millisecond lookup. Requires ElastiCache node-based with search enabled.
This guide covers the server-side approach. For the application-side approach (serverless OK, any Valkey version), use the `python_knn_search` fallback in `elasticache-search.md` with semantic cache key patterns below.
**Before using any FT.* code below**, call `supports_ft_search(client)` from `elasticache-search.md`. If it returns `False`, use the Python-side fallback instead.
## Key design
Use a dual-key pattern. Separate the vector index hash from the response payload:
```
semcache:vec:{request_id} # HASH: embedding, request_id, timestamp, filter fields
semcache:rr:{request_id} # HASH: request_text, response_text, created_at
```
Why separate keys: the vector index only scans the `vec:` prefix. Response payloads can be large (full LLM output) and do not need indexing. Keeping them out of the index reduces memory pressure on HNSW graph traversal.
**Cluster mode:** If running on a multi-shard cluster, use hash tags to ensure all keys for a single index land on the same shard:
```
semcache:vec:{myapp}:{request_id}
semcache:rr:{myapp}:{request_id}
```
The `{myapp}` hash tag forces slot co-location. Without this, FT.SEARCH returns partial results (it only queries the shard it executes on). See `elasticache-search.md` Hash Slot Constraint for details. For single-shard clusters, hash tags are optional since all keys land on the same shard.
```python
PREFIX_VECTOR = "semcache:vec:"
PREFIX_RR = "semcache:rr:"
INDEX_NAME = "idx:semcache"
```
## Step 1: Create the index
```python
import time
import uuid
from utils.embeddings import VECTOR_DIM
def create_index(client):
"""Create HNSW COSINE vector index. Idempotent."""
try:
client.execute_command(
"FT.CREATE", INDEX_NAME,
"ON", "HASH",
"PREFIX", "1", PREFIX_VECTOR,
"SCHEMA",
"embedding", "VECTOR", "HNSW", "6",
"TYPE", "FLOAT32",
"DIM", str(VECTOR_DIM),
"DISTANCE_METRIC", "COSINE",
"request_id", "TAG",
"scope", "TAG", "SEPARATOR", ",",
"timestamp", "NUMERIC",
)
except Exception as e:
if "already exists" not in str(e).lower():
raise
```
**Index limit:** A maximum of 10 indexes can be created per cluster. For multi-tenant designs, use TAG-based filtering within a single index rather than creating per-tenant indexes.
**HNSW tuning parameters:** The index above uses HNSW defaults (`M=16`, `EF_CONSTRUCTION=200`, `EF_RUNTIME=10`). For semantic cache workloads, the default `EF_RUNTIME` of 10 may yield suboptimal recall; consider increasing it (e.g., 50-200) via the `EF_RUNTIME` query modifier on `FT.SEARCH` to improve cache hit detection. Higher `M` values (e.g., 32+) improve recall at the cost of memory. `EF_CONSTRUCTION` values of 200-400 are generally sufficient. See the [vector search overview](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/vector-search-overview.html) for detailed guidance on tuning these parameters.
## Step 2: Cache lookup (FT.SEARCH)
All code below uses the shared embedding utility:
```python
from utils.embeddings import generate_embedding, embedding_to_bytes, VECTOR_DIM
```
```python
SIMILARITY_THRESHOLD = 0.90 # default cosine similarity; tune per use case
def cache_lookup(client, query_vec: bytes, threshold: float = SIMILARITY_THRESHOLD) -> dict | None:
"""Search for a semantically similar cached prompt using a precomputed query
embedding (bytes from embedding_to_bytes). Returns hit dict or None."""
result = client.execute_command(
"FT.SEARCH", INDEX_NAME,
"*=>[KNN 1 @embedding $vec AS score]",
"PARAMS", "2", "vec", query_vec,
"RETURN", "2", "request_id", "score",
"DIALECT", "2",
)
if not result or int(result[0]) == 0:
return None
# Parse result: [total_hits, key_name, [field, value, ...]]
# With decode_responses=False, all values are bytes
fields = result[2]
doc = {}
for i in range(0, len(fields), 2):
k = fields[i].decode() if isinstance(fields[i], bytes) else fields[i]
v = fields[i+1].decode() if isinstance(fields[i+1], bytes) else fields[i+1]
doc[k] = v
# Cosine distance to similarity: distance 0=identical, 2=opposite.
similarity = 1.0 - (float(doc["score"]) / 2.0)
if similarity < threshold:
return None
# Fetch the cached response from the rr key
request_id = doc["request_id"]
rr_key = f"{PREFIX_RR}{request_id}"
rr_data = client.hgetall(rr_key)
if not rr_data:
return None
# Decode bytes keys/values from hgetall
response_text = rr_data.get(b"response_text", b"").decode()
return {
"response": response_text,
"similarity": similarity,
"request_id": request_id,
}
```
## Step 3: Cache store (HSET)
```python
def cache_store(client, prompt: str, response: str, embedding_bytes: bytes,
scope: str = "", ttl: int = 3600):
"""Store prompt+response pair. Sets TTL on both keys."""
request_id = str(uuid.uuid4())
now = time.time()
# Vector key (indexed)
vec_key = f"{PREFIX_VECTOR}{request_id}"
client.hset(vec_key, mapping={
"embedding": embedding_bytes,
"request_id": request_id,
"timestamp": str(now),
"scope": scope if scope else "",
})
# Response key (not indexed)
rr_key = f"{PREFIX_RR}{request_id}"
client.hset(rr_key, mapping={
"request_text": prompt,
"response_text": response,
"created_at": str(now),
})
if ttl > 0:
# Add random jitter to spread out cache invalidations and prevent
# thundering herd when many entries expire simultaneously.
import random
jitter = random.randint(0, max(1, ttl // 10)) # up to 10% jitter
client.expire(vec_key, ttl + jitter)
client.expire(rr_key, ttl + jitter)
```
## Step 4: Full flow
```python
def semantic_cache_query(client, prompt: str, llm_fn, threshold: float = 0.90,
scope: str = "", ttl: int = 3600) -> dict:
"""
Complete semantic cache flow.
llm_fn: callable that takes a prompt string and returns response string.
"""
# Compute the query embedding once and reuse it for lookup and store.
embedding_bytes = embedding_to_bytes(generate_embedding(prompt))
# Lookup
hit = cache_lookup(client, embedding_bytes, threshold=threshold)
if hit:
return {"response": hit["response"], "source": "cache", "similarity": hit["similarity"]}
# Miss: call LLM
response = llm_fn(prompt)
# Store (reuses the embedding already computed above)
cache_store(client, prompt, response, embedding_bytes, scope=scope, ttl=ttl)
return {"response": response, "source": "llm", "similarity": 0.0}
```
## Similarity thresholds
> **Note:** ElastiCache vector search uses cosine distance (1 - cosine_similarity), where 0 = identical and 1 = orthogonal. A distance threshold of 0.10 corresponds to cosine similarity ≥ 0.90.
Starting recommendations:
| Use case | Threshold | Notes |
|----------|-----------|-------|
| Factual Q&A, API calls | 0.90 - 0.95 | Strict. Wrong answer is costly. |
| Customer support / FAQ | 0.85 - 0.90 | Moderate. Slightly paraphrased questions should hit. |
| General chat, creative | 0.70 - 0.85 | Lenient. Accept broader semantic matches. |
| Sub-agent (tool dispatch) | 0.65 - 0.70 | Very lenient. From production: THRESHOLD_SUBAGENT = 0.70 |
Lower threshold = more cache hits but higher risk of returning a semantically incorrect answer. Start strict (0.92) and lower based on observed false-hit rate.
## Advanced: hybrid filtering
Add TAG or NUMERIC pre-filters to scope cache hits before vector similarity runs. This narrows the candidate set so KNN only compares within a relevant subset.
```python
# Pre-filter by model version and user segment, then KNN
pre_filter = "(@scope:{bedrock_claude_v4} @timestamp:[1700000000 +inf])"
query = f"{pre_filter}=>[KNN 1 @embedding $vec AS score]"
result = client.execute_command(
"FT.SEARCH", INDEX_NAME, query,
"PARAMS", "2", "vec", query_vec,
"RETURN", "2", "request_id", "score",
"DIALECT", "2",
)
```
Filter patterns:
* `@scope:{model_v2}` filters TAG field to exact token
* `@timestamp:[{cutoff} +inf]` filters NUMERIC to recent entries only
* Multiple filters combine with implicit AND inside parentheses
* Escape hyphens in TAG values: `my\\-value`
references/genai/session-store.md
# Session Store Implementation Guide
Session state, message history, and resumable conversations with ElastiCache Valkey. Plain data structures, no vector search, no embedding model. Works on serverless.
> **Serverless eviction policy:** Serverless caches use `volatile-lru` (not configurable), which only evicts keys that have a TTL set. Always set a TTL on every key. Keys without a TTL will never be evicted and can cause OOM errors. The `allkeys-lru` policy recommended in some best-practice guides applies only to node-based clusters.
## Key Design
```
{conv:session_id}:messages # JSON-encoded list of message dicts
{conv:session_id}:metadata # JSON-encoded conversation metadata
conv:user:{user_id}:sessions # sorted set of session_ids, score = timestamp
```
**Hash tag note:** The `{conv:session_id}` hash tag ensures messages and metadata keys hash to the same slot on cluster-mode-enabled caches (including serverless). Without a hash tag, pipeline operations on these keys are non-atomic across slots and a mid-pipeline failure could leave partial state. The `conv:user:{user_id}:sessions` index key does not share a hash tag with the message/metadata keys, so it hashes to a different slot. This is acceptable because the index is advisory (listing sessions), not transactional.
## Store and Load Pattern (Custom)
Use this when you're not on the Strands framework and need direct control over session storage.
```python
# pip install valkey
# For serverless or cluster-mode-enabled node-based clusters:
# from valkey.cluster import ValkeyCluster
# For cluster-mode-disabled node-based clusters:
# import valkey (use valkey.Valkey)
import json
import time
import valkey
CONVERSATION_TTL = 30 * 24 * 60 * 60 # 30 days (adjust to your retention policy)
_client = None
def get_client():
"""Lazy client initialization. Never create connections at module level.
For serverless or cluster-mode-enabled node-based clusters, use
valkey.cluster.ValkeyCluster instead of valkey.Valkey.
Serverless is always cluster-mode-enabled. The single serverless
endpoint abstracts slots to one virtual node, so ValkeyCluster
works transparently. For cluster-mode-disabled node-based clusters,
valkey.Valkey is correct.
"""
global _client
if _client is None:
_client = valkey.ValkeyCluster(
host="your-endpoint.serverless.use1.cache.amazonaws.com", # serverless format; node-based differs
port=6379, ssl=True, ssl_cert_reqs="required", # verify server cert against trusted CAs (secure default)
# ssl_cert_reqs="none", # tunnel/dev ONLY (e.g. local SSH tunnel); never in production
decode_responses=True, socket_timeout=5,
)
return _client
def save_session(session_id: str, messages: list[dict], metadata: dict,
user_id: str = None):
"""Persist conversation state. Call after each turn."""
msg_key = f"{{conv:{session_id}}}:messages"
meta_key = f"{{conv:{session_id}}}:metadata"
client = get_client()
pipe = client.pipeline()
pipe.set(msg_key, json.dumps(messages, default=str))
pipe.expire(msg_key, CONVERSATION_TTL)
pipe.set(meta_key, json.dumps(metadata, default=str))
pipe.expire(meta_key, CONVERSATION_TTL)
if user_id:
# Note: idx_key hashes to a different slot than {conv:session_id} keys.
# On cluster-mode-enabled caches (including serverless), this pipeline
# becomes a multi-slot pipeline. ValkeyCluster handles this transparently
# by splitting commands across slots, but atomicity is only guaranteed
# within a single slot. If you need atomic user-index updates, issue
# the ZADD/EXPIRE separately outside the pipeline.
idx_key = f"conv:user:{user_id}:sessions"
pipe.zadd(idx_key, {session_id: time.time()})
pipe.expire(idx_key, CONVERSATION_TTL)
pipe.execute()
def load_session(session_id: str) -> dict | None:
"""Load persisted conversation. Returns {messages, metadata} or None."""
client = get_client()
raw_messages = client.get(f"{{conv:{session_id}}}:messages")
if not raw_messages:
return None
raw_metadata = client.get(f"{{conv:{session_id}}}:metadata")
return {
"messages": json.loads(raw_messages),
"metadata": json.loads(raw_metadata) if raw_metadata else {},
}
def list_user_sessions(user_id: str, limit: int = 20) -> list[str]:
"""Most recent sessions first."""
return get_client().zrevrange(f"conv:user:{user_id}:sessions", 0, limit - 1)
```
## Strands Integration
The `strands-valkey-session-manager` package (community package, v0.1.0+ — MIT license, maintained by jeromevdl) provides a drop-in `ValkeySessionManager`. Five lines of setup:
```python
import valkey
from strands import Agent
from strands_valkey_session_manager import ValkeySessionManager
_strands_client = None
def get_strands_client():
"""Lazy client initialization. Never create connections at module level.
Uses ValkeyCluster for serverless (cluster-mode-enabled).
For cluster-mode-disabled node-based clusters, use valkey.Valkey instead.
"""
global _strands_client
if _strands_client is None:
_strands_client = valkey.ValkeyCluster(
host="your-endpoint.serverless.use1.cache.amazonaws.com", port=6379, ssl=True,
ssl_cert_reqs="required", decode_responses=True, # in-VPC EKS with real cert verification
# tunnel/dev alternative: ssl_cert_reqs="none" (skips cert verification; not for production)
)
return _strands_client
def create_agent(session_id: str = "user-42") -> Agent:
client = get_strands_client()
session_mgr = ValkeySessionManager(session_id=session_id, client=client)
return Agent(system_prompt="You are a helpful assistant.", session_manager=session_mgr)
# Usage:
# agent = create_agent("user-42")
# response = agent("My name is Alex and I'm building a RAG pipeline.")
```
The session manager stores three key types in Valkey: `session:{id}` (session record), `session:{id}:agent:{agent_id}` (agent state), and `session:{id}:agent:{agent_id}:message:{msg_id}` (individual messages).
## TTL Recommendations
| Scope | Recommended TTL |
|-------|----------------|
| Active session | 24 hours |
| Resumable session | 720 hours (~30 days) |
| Session index per user | 90 days |
---
## Cross-References
* Semantic memory (vector-based recall across sessions): see `agent-memory.md`
* LangChain/LlamaIndex/Strands framework integration: see `framework-guide.md`
references/migration/auth-migration.md
# Authentication Migration Runbooks
Step-by-step procedures for migrating between ElastiCache authentication models and cluster topologies.
## AUTH Token to RBAC Migration
Legacy AUTH token authentication uses a single shared password for all clients. RBAC provides per-user access control with fine-grained command and key permissions. This migration moves from the legacy model to the recommended model.
### Prerequisites
- Node-based replication group with `TransitEncryptionEnabled: true`
- Engine version Redis OSS 6.0 or higher, or Valkey 7.2 or higher (RBAC is not supported on earlier versions)
- AUTH token currently configured on the replication group
- Application code ready to switch to username/password authentication
### Step 1: Create RBAC Users
Create one or more RBAC users that match your application's access needs.
```bash
# Create an application user with full access
aws elasticache create-user \
--user-id myapp-appuser \
--user-name appuser \
--engine <valkey|redis> \
--access-string "on ~* +@all" \
--authentication-mode Type=password,Passwords="<strong-password>" \
--region <region>
# Create a read-only user for monitoring or reporting
aws elasticache create-user \
--user-id myapp-readonly \
--user-name readonly \
--engine <valkey|redis> \
--access-string "on ~* +@read" \
--authentication-mode Type=password,Passwords="<strong-password>" \
--region <region>
```
For IAM auth instead of passwords:
```bash
aws elasticache create-user \
--user-id myapp-appuser \
--user-name myapp-appuser \
--engine <valkey|redis> \
--access-string "on ~* +@all" \
--authentication-mode Type=iam \
--region <region>
```
Store RBAC passwords in Secrets Manager:
```bash
aws secretsmanager create-secret \
--name elasticache/myapp/appuser \
--secret-string '{"username":"appuser","password":"<strong-password>"}' \
--tags Key=Purpose,Value=elasticache-auth \
--region <region>
```
### Step 2: Create User Group and Associate
```bash
# Create user group (must include the default user)
# Use --engine valkey for Valkey clusters or --engine redis for Redis OSS clusters.
aws elasticache create-user-group \
--user-group-id myapp-usergroup \
--engine valkey \
--user-ids default myapp-appuser myapp-readonly \
--region <region>
# Associate the user group with the replication group WITHOUT removing the AUTH token
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--user-group-ids-to-add myapp-usergroup \
--region <region>
```
> **Note:** At this point, both AUTH token and RBAC credentials are active. Existing clients using the AUTH token continue to work.
### Step 3: Update Application to Use RBAC Credentials
Update application code to pass `username` and `password` (or IAM token) instead of just the AUTH token.
Before (AUTH token):
```python
client = valkey.Valkey(host=endpoint, port=6379, ssl=True, password=auth_token)
```
After (RBAC):
```python
client = valkey.Valkey(host=endpoint, port=6379, ssl=True,
username="appuser", password=rbac_password)
```
Deploy the updated application. Verify that it connects successfully using RBAC credentials.
### Step 4: Verify RBAC Connectivity
Before removing the AUTH token, verify that all clients are successfully authenticating via RBAC:
```bash
aws elasticache describe-replication-groups \
--replication-group-id <cluster-id> \
--region <region>
# Confirm the output shows:
# "AuthTokenEnabled": true,
# "UserGroupIds": ["myapp-usergroup"]
```
Confirm that the application works correctly with RBAC credentials and that no clients still depend on the AUTH token.
### Step 5: Remove AUTH Token
Now that all clients are using RBAC, remove the AUTH token:
```bash
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--auth-token-update-strategy DELETE \
--apply-immediately \
--region <region>
```
### Step 6: Verify AUTH Token Removal
Verify that the AUTH token is disabled:
```bash
aws elasticache describe-replication-groups \
--replication-group-id <cluster-id> \
--region <region>
# Confirm the output shows:
# "AuthTokenEnabled": false,
# "UserGroupIds": ["myapp-usergroup"]
```
### Step 7: Verify and Secure the Default User
The `default` user is always present in RBAC user groups. Disable it or restrict its access to prevent unauthenticated connections:
**For Valkey clusters:** The default user can be removed from the user group entirely, or disabled:
```bash
# Option A: Remove default user from the user group (Valkey only)
aws elasticache modify-user-group \
--user-group-id myapp-usergroup \
--user-ids-to-remove default \
--region <region>
# Option B: Disable the default user (works for both Valkey and Redis OSS)
aws elasticache modify-user \
--user-id default \
--access-string "off ~* -@all" \
--region <region>
```
**For Redis OSS clusters:** The default user **must** remain in the user group (Redis OSS requires it), but should be disabled:
```bash
# Disable the default user (required approach for Redis OSS)
aws elasticache modify-user \
--user-id default \
--access-string "off ~* -@all" \
--region <region>
```
> **Note:** On Redis OSS, removing the default user from a user group will fail. On Valkey, the default user is optional in user groups, so removing it is the cleanest approach.
### Rollback Plan
If issues arise during migration:
1. **Before Step 5:** The AUTH token is still active. Simply revert application code to use AUTH token authentication and deploy. Both auth methods work simultaneously, so rollback is safe at any point before Step 5
2. **After Step 5:** The AUTH token has been removed. To roll back, re-set the AUTH token on the replication group, revert application code, and deploy
3. Remove the user group association if needed:
```bash
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--user-group-ids-to-remove myapp-usergroup \
--region <region>
```
## Cluster-Mode Disabled to Cluster-Mode Enabled Migration
Cluster-mode disabled uses a single shard (primary + replicas). Cluster-mode enabled uses multiple shards for horizontal scaling. This migration changes the endpoint model and requires client updates.
### Prerequisites
- Understand that cluster-mode enabled changes the client interaction model
- Client libraries must support `MOVED` and `ASK` redirect handling
- Multi-key commands must use hash tags to colocate keys on the same shard
- **For in-place migration:** Minimum engine version of Valkey 7.2 or Redis OSS 7.0 is required
- **For in-place migration:** Auto-failover must be enabled with at least 1 replica
- **For in-place migration:** The cluster may only have keys in database 0 (multiple databases are not supported)
- **For in-place migration:** CMD→CME is a one-way operation. Once cluster mode is set to `enabled`, it **cannot be reverted back to disabled**. You can only revert from `compatible` back to `disabled`
### Impact Assessment
| Aspect | Cluster-Mode Disabled | Cluster-Mode Enabled |
|--------|----------------------|---------------------|
| Shards | 1 | 1-500 (default limit: up to 90 nodes; 500 requires a limit increase and engine version 5.0.6+; versions below 5.0.6 limited to 250) |
| Max data | Limited by node memory | Distributed across shards |
| Endpoint | Single primary endpoint | Configuration endpoint (resolves to all shards) |
| Client | Standard client | Cluster-aware client required |
| Multi-key ops | Unrestricted | Keys must share a hash slot (use `{tag}`) |
| KEYS command | Works (not recommended) | Scoped to a single shard |
### Step 1: Assess Client Compatibility
Verify that all client applications can handle cluster mode:
| Runtime | Cluster-Aware Client |
|---------|---------------------|
| Python (valkey-py) | `valkey.cluster.ValkeyCluster` |
| Node.js (iovalkey) | `new Redis.Cluster([...])` |
| Java (Lettuce) | `RedisClusterClient.create(uri)` |
| Go (valkey-go) | Automatic (handles redirects natively) |
| CLI | `valkey-cli -c` |
Check for multi-key commands that span keys without hash tags:
- `MGET key1 key2` -- keys must be on the same shard
- `SUNION set1 set2` -- sets must be on the same shard
- Lua scripts accessing multiple keys -- all keys must be in the same slot
- `MULTI`/`EXEC` transactions -- all keys must be in the same slot
Fix: use hash tags like `{user:100}:profile` and `{user:100}:sessions` to ensure related keys land on the same shard.
### Step 2: Create a New Cluster-Mode Enabled Cluster
For Valkey 7.2+ or Redis OSS 7.0+, you can perform an in-place migration from cluster-mode disabled to cluster-mode enabled using the `modify-replication-group` CLI command (or `ModifyReplicationGroup` API). This is a two-step process: first set cluster mode to `compatible`, then to `enabled`. Requirements: auto-failover must be enabled with at least 1 replica, keys must exist only in database 0, and the migration is irreversible once set to `enabled` (you can revert from `compatible` back to `disabled`).
```bash
# Step 1: Set cluster mode to compatible
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--cache-parameter-group-name <cluster-enabled-parameter-group> \
--cluster-mode compatible \
--region <region>
# Wait for the cluster to become available, then update applications
# to use cluster protocol and the configuration endpoint.
# Step 2: Set cluster mode to enabled
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--cluster-mode enabled \
--region <region>
```
> **Note:** In compatible mode, other modification operations such as scaling and engine version upgrades are not allowed. The cluster endpoints will change once cluster mode is set to enabled. Make sure to update your applications with the new endpoints.
For older engine versions (below Redis OSS 7.0), you must create a new cluster and migrate data.
```bash
# Create a new cluster-mode enabled replication group
aws elasticache create-replication-group \
--replication-group-id <new-cluster-id> \
--replication-group-description "Cluster-mode enabled migration target" \
--engine valkey \
--engine-version <version> \
--cache-node-type <node-type> \
--num-node-groups 3 \
--replicas-per-node-group 1 \
--cache-subnet-group-name <subnet-group> \
--security-group-ids <sg-id> \
--transit-encryption-enabled \
--at-rest-encryption-enabled \
--automatic-failover-enabled \
--multi-az-enabled \
--region <region>
```
### Step 3: Migrate Data
Option A: Snapshot and restore (brief downtime acceptable)
```bash
# Snapshot the old cluster
aws elasticache create-snapshot \
--replication-group-id <old-cluster-id> \
--snapshot-name migration-snapshot \
--region <region>
# Restore to the new cluster (must delete the new cluster first, then recreate from snapshot)
# Note: restoring a snapshot from a cluster-mode disabled cluster into a cluster-mode enabled
# cluster requires the same number of node groups. Plan shard configuration carefully.
```
Option B: Dual-write migration (zero downtime)
1. Configure the application to write to both old and new clusters
2. Run a backfill to copy existing data from old to new (application-level copy)
3. Gradually shift reads to the new cluster
4. Stop writes to the old cluster
5. Decommission the old cluster
> **Warning:** Do NOT use DUMP/RESTORE for data migration between clusters. The serialization format is engine-version-specific and may produce corrupted data on version mismatches. Use application-level copy (read from source, write to target) or the ElastiCache online migration tools instead.
> **Note:** The ElastiCache online migration tool (`start-migration` / `complete-migration`) is designed exclusively for migrating data from self-hosted open-source Valkey or Redis OSS on Amazon EC2 to ElastiCache. It is **not** for moving data between ElastiCache clusters. Additionally, the target cluster must **not** have encryption in-transit enabled, must have Multi-AZ enabled, must not be part of a global datastore, must have data tiering disabled, and the number of shards in source and target must match. Online migration is not supported for ElastiCache serverless caches or clusters running on the r6gd node type. See the [Online Migration documentation](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/OnlineMigration.html) for details.
Option C: Online migration via ElastiCache (only if source is self-hosted Valkey/Redis on EC2, not an ElastiCache cluster)
```bash
# Ensure the target cluster does NOT have TLS enabled and meets all prerequisites above
aws elasticache start-migration \
--replication-group-id <new-cluster-id> \
--customer-node-endpoint-list Address=<self-hosted-endpoint>,Port=6379
# After data is in sync, finalize the migration:
aws elasticache complete-migration \
--replication-group-id <new-cluster-id>
```
### Step 4: Update Application Endpoints
The new cluster uses a configuration endpoint that resolves to all shards. Update connection strings:
Before:
```python
client = valkey.Valkey(host="old-primary-endpoint", port=6379, ssl=True, ...)
```
After:
```python
client = valkey.cluster.ValkeyCluster(
host="new-configuration-endpoint", port=6379, ssl=True, ...
)
```
### Step 5: Validate Application Behavior
- Verify all read and write operations succeed
- Check for `CROSSSLOT` errors (indicates multi-key operations on different shards)
- Monitor CloudWatch metrics for error rates, latency, and connection counts
- Run `valkey-cli -h <new-endpoint> -p 6379 --tls PING` for basic validation
### Rollback Plan
**For the in-place migration path (Valkey 7.2+/Redis OSS 7.0+):** Once cluster mode is set to `enabled`, the change is **irreversible** — you cannot convert back to cluster-mode disabled. You can only revert from `compatible` back to `disabled` before completing the final step. Take a snapshot before setting `enabled`.
**For the new-cluster migration path:** The old cluster remains running throughout the migration:
1. Revert application endpoints to the old cluster
2. Deploy the reverted application
3. Delete the new cluster-mode enabled cluster
4. No data loss (old cluster was never modified)
### Note on Endpoint Model Change
This migration changes the endpoint model:
- Cluster-mode disabled: single primary endpoint + single reader endpoint
- Cluster-mode enabled: configuration endpoint that the client uses to discover all shards
Applications must update their connection logic, not just the endpoint hostname. This is the most common source of issues during this migration.
references/migration/feature-comparison.md
# Migration Feature Comparison Matrix
Side-by-side comparison of features, capabilities, and constraints across migration paths. Use this reference to identify what is gained, lost, or changed when moving between engines and deployment models.
## Engine Comparison: Valkey vs Redis OSS vs Memcached
| Feature | Valkey (7.2+) | Redis OSS (7.x) | Memcached |
|---------|:------------:|:---------------:|:---------:|
| **Data Structures** | | | |
| Strings | Yes | Yes | Yes |
| Hashes | Yes | Yes | No |
| Lists | Yes | Yes | No |
| Sets | Yes | Yes | No |
| Sorted Sets | Yes | Yes | No |
| Streams | Yes | Yes | No |
| HyperLogLog | Yes | Yes | No |
| Bitmaps | Yes | Yes | No |
| Geospatial | Yes | Yes | No |
| JSON (native) | Yes | Yes | No |
| Vector Search | Yes (8.2+, node-based only) | No | No |
| **Persistence and Replication** | | | |
| Snapshots (RDB) | Yes | Yes | Serverless only (automatic backups) (verify against latest Memcached serverless documentation for current backup behavior) |
| AOF persistence | Deprecated (legacy only) | Deprecated (legacy only) | No |
| Replication | Yes | Yes | No |
| Multi-AZ failover | Yes | Yes | No |
| Global Datastore | Yes (node-based only) | Yes (node-based only) | No |
| **Cluster and Scaling** | | | |
| Cluster mode (sharding) | Yes | Yes | No (data partitioning via client-side auto-discovery) |
| Max shards (cluster mode) | 500 (default 90 nodes; increasable to 500 for v5.0.6+; 250 for earlier versions) | 500 (default 90 nodes; increasable to 500 for v5.0.6+; 250 for earlier versions) | N/A (up to 60 nodes per cluster) |
| Online resharding | Yes | Yes | No |
| Serverless deployment | Yes | Yes | Yes |
| **Authentication and Security** | | | |
| RBAC (per-user ACLs) | Yes | Yes | No |
| IAM authentication | Yes (7.2+) | Yes (7.0+) | No |
| AUTH token (legacy) | Yes (node-based only) | Yes (node-based only) | No |
| In-transit encryption (TLS) | Yes | Yes | Yes |
| At-rest encryption | Yes | Yes | Yes |
| **Pub/Sub and Messaging** | | | |
| Pub/Sub | Yes | Yes | No |
| Keyspace notifications | Yes | Yes | No |
| **Operational** | | | |
| Lua scripting | Yes | Yes | No |
| Transactions (MULTI/EXEC) | Yes | Yes | No (CAS via gets/cas) |
| Pipelining | Yes | Yes | Yes |
| Max item size | 512 MB | 512 MB | 1 MB (default) |
| Max connections | Varies by node type | Varies by node type | Varies by node type |
| **Pricing** | | | |
| Node-based discount vs Redis OSS | 20% lower | Baseline | Lowest node cost |
| Serverless discount vs Redis OSS | 33% lower | Baseline | Available |
### Migration Notes: Redis OSS to Valkey
**What you gain:**
* 20% cost savings on node-based, 33% on serverless
* Access to Valkey-specific features in future releases (vector search in 8.2)
* Active open-source community and roadmap
* Zero application code changes (API-compatible with Redis OSS 7.2)
**What you lose:**
* Nothing for standard Redis OSS workloads
* Redis Ltd. commercial module ecosystem (RediSearch, RedisBloom, RedisTimeSeries) is not available. Valkey 8.1 provides native Bloom filters (BF.* commands) and Valkey 8.2 provides native vector search
**What changes:**
* Engine identifier changes from `redis` to `valkey` in API calls and IaC
* Future version numbers follow Valkey versioning (7.2, 8.0, 8.1, 8.2, 9.0)
### Migration Notes: Memcached to Valkey
**What you gain:**
* Persistence (snapshots, AOF)
* Replication and automatic failover
* Rich data structures (hashes, lists, sets, sorted sets, streams, JSON)
* RBAC and IAM authentication
* Pub/Sub and keyspace notifications
* Lua scripting and transactions
* Serverless deployment option
* Online resharding
**What you lose:**
* Multi-threaded architecture (Memcached uses multiple threads per node; Valkey is primarily single-threaded per shard but compensates with sharding)
* Potentially lower per-node cost for simple key-value workloads (Memcached nodes can be cheaper)
* Slab-based memory allocator (Memcached's memory model differs; unlikely to matter in practice)
**What changes:**
* Client library must change (memcached client to Valkey/Redis client)
* Application code must be rewritten for Valkey commands (SET/GET vs set/get, different API)
* Connection model changes (Valkey uses persistent TCP connections)
* Max item size increases from 1 MB to 512 MB
---
## Deployment Model Comparison: Serverless vs Node-Based
| Feature | Serverless | Node-Based |
|---------|:----------:|:----------:|
| **Provisioning** | | |
| Setup time | Under 1 minute | 5-15 minutes |
| Capacity planning | Automatic | Manual (choose node type, count) |
| Scaling | Automatic (up and down) | Manual or policy-based |
| **Networking** | | |
| VPC required | Yes | Yes |
| Port | 6379 | 6379 (configurable) |
| TLS | Always enabled (cannot disable) | Optional (strongly recommended) |
| **Authentication** | | |
| RBAC | Yes | Yes |
| IAM auth | Yes | Yes |
| AUTH token | Not supported | Yes (legacy) |
| **Data and Performance** | | |
| Max data storage | Default 5 TB (adjustable via `CacheUsageLimits`); max 32 GiB per slot | Limited by node type and shard count |
| Max throughput | Default 15,000,000 ECPU/s (adjustable via `CacheUsageLimits`); 30K-90K ECPU/s per slot | Depends on node type and shard count |
| Latency | Single-digit milliseconds | Sub-millisecond |
| **Features** | | |
| Vector search | Not supported | Yes (Valkey 8.2 or above) |
| Global Datastore | Not supported | Yes |
| Cluster mode | Automatic (transparent) | Configurable (enabled/disabled) |
| Custom parameter groups | Not supported | Yes |
| Pub/Sub | Yes | Yes |
| Keyspace notifications | Not supported | Yes |
| Lua scripting | Yes (with restrictions) | Yes |
| **Operations** | | |
| Maintenance windows | None (transparent patching) | Weekly configurable window |
| Manual snapshots | Yes (`CreateServerlessCacheSnapshot`) | Yes (`create-snapshot`) |
| Snapshot restore | Yes (`create-serverless-cache --snapshot-arns-to-restore`) | Yes (`create-replication-group --snapshot-name`) |
| Node type selection | N/A | Full control |
| Reserved pricing | Not available | Yes (1-year or 3-year) |
| **Pricing Model** | | |
| Billing unit | ECPU + data storage (GB-hours) | Node-hours + optional data transfer |
| Minimum cost | ~$6/month (Valkey, approximate, us-east-1) | Varies by node type (starts ~$12/month for t4g.micro) |
| Cost optimization | Automatic right-sizing | Reserved nodes, right-sizing, Graviton |
### Migration Notes: Node-Based to Serverless
**What you gain:**
* Zero capacity planning and automatic scaling
* No maintenance windows or patching overhead
* Sub-minute provisioning
* Pay only for actual usage (beneficial for variable workloads)
* Always-on TLS and encryption
**What you lose:**
* Sub-millisecond latency (serverless is single-digit ms)
* Vector search support (Valkey 8.2 or above, node-based only)
* Global Datastore (node-based only)
* Custom parameter group tuning
* AUTH token authentication (must use RBAC or IAM)
* Reserved node pricing (no commitment discounts for serverless)
* Direct control over node count and placement
**What changes:**
* Endpoint format changes to `*.serverless.<region>.cache.amazonaws.com`
* Auth must be RBAC or IAM (AUTH tokens not supported)
* TLS is mandatory (update clients if not already using TLS)
* Billing model changes from node-hours to ECPU + storage
* Some Lua script restrictions may apply (scripts must have at least one KEY parameter; max script size 4 MiB; max 3,999 arguments per request)
* `SELECT` command (multiple databases) is not supported (serverless is always cluster-mode enabled, so only database 0 is available)
* Keyspace notifications are not supported on serverless caches
### Migration Notes: Serverless to Node-Based
**What you gain:**
* Sub-millisecond latency
* Vector search (Valkey 8.2 or above)
* Global Datastore for multi-region
* Custom parameter group tuning
* Reserved node pricing (approximately 30-55% savings depending on term length and payment option; run `python3 scripts/price_calculator.py --mode node --node-type <type> --nodes <N> --show-ri-options` for current estimates)
* AUTH token option (legacy, but available)
* Full control over topology and placement
**What you lose:**
* Automatic scaling (must manage capacity manually or via policies)
* Zero-maintenance patching (must configure maintenance windows)
* Sub-minute provisioning (node-based takes 5-15 minutes)
* Usage-based pricing (node-based charges for provisioned capacity, not actual use)
**What changes:**
* Endpoint format changes from serverless to standard replication group endpoint
* May switch from RBAC to AUTH token (or keep RBAC)
* TLS becomes optional (but strongly recommended to keep it enabled)
* Billing model changes from ECPU + storage to node-hours
* Must choose node type, shard count, and replica count
---
## Migration Path Quick Reference
| From | To | In-Place? | Data Migration Required? | Application Code Changes? | Auth Changes? |
|------|-----|:---------:|:------------------------:|:-------------------------:|:-------------:|
| Redis OSS (node-based) | Valkey (node-based) | Yes | No | No | No |
| Valkey 7.2 | Valkey 8.1 | Yes | No | No | No |
| Valkey 7.2 | Valkey 8.2 | Yes (direct upgrade supported) | No | No | No |
| Self-managed Redis | ElastiCache (node-based) | No | Yes (replication or snapshot) | Endpoint update | May need RBAC |
| Self-managed Redis | ElastiCache (serverless) | No | Yes (dual-write or app-level) | Endpoint + TLS + auth | Must use RBAC |
| Node-based | Serverless | No | Yes (dual-write or snapshot-restore) | Endpoint + TLS + auth | Must switch to RBAC if using AUTH token |
| Serverless | Node-based | No | Yes (dual-write) | Endpoint, possibly auth | Optional |
| Memcached | Valkey | No | Yes (full rewrite) | Full rewrite | Must add RBAC or IAM |
| Cluster-mode disabled | Cluster-mode enabled | Yes (Valkey 7.2+/Redis OSS 7.0+ via compatible mode); No (older versions) | No (7.0+); Yes (older versions) | Cluster-aware client required | No |
---
## Unsupported Migration Paths
The following transitions are not supported in-place and require special handling:
| Transition | Why Not Supported | Workaround |
|------------|-------------------|------------|
| Valkey to Redis OSS (downgrade) | Valkey 7.2 to Redis OSS 7.1 rollback is supported in-place (no downtime). Other Valkey versions cannot be downgraded in-place. | For Valkey 7.2 node-based: use modify-replication-group specifying Redis OSS 7.1. For Valkey 7.2 serverless: use modify-serverless-cache specifying Redis OSS 7.1. For other versions: restore from pre-upgrade snapshot to new Redis OSS cluster. |
| Higher to lower engine version | Version downgrade not supported | Restore from pre-upgrade snapshot to new cluster with older version |
| Serverless to node-based (direct) | No direct conversion path | Create new node-based cluster, migrate data, cut over |
| Node-based to serverless (direct) | No direct conversion path | Create new serverless cache, migrate data, cut over |
| Memcached to Valkey (direct) | Different engines, different data models | Create new Valkey cache, rewrite application, re-populate data |
references/migration/global-datastore-operations.md
# Global Datastore Operations
Operational guide for setting up, managing, and upgrading ElastiCache Global Datastore for cross-region replication.
## Prerequisites
Global Datastore has specific requirements that must be met before creation:
* **Node-based only.** Serverless caches do not support Global Datastore.
* **Supported instance families:** M5, M6g, M7g, R5, R6g, R6gd, R7g, C7gn in size **large and above** only. Prior-generation instances (M4, R4) and burstable instances (t2, t3, t4g) are not supported.
* **Multi-AZ and automatic failover required** on the primary replication group.
* **Same node type, engine version, and primary node count** across all regions. Each cluster in the global datastore can have a different number of read replicas to accommodate read traffic local to that cluster. When creating a secondary, the node type is inferred from the primary and should not be specified.
* **Same engine and version** across all regions at the time of creation. Engine upgrades follow a specific ordering (see "Upgrading Engine Version" below).
* **Both cluster mode disabled (CMD) and cluster mode enabled (CME) are supported.** If using cluster-mode enabled, the number of shards must match across all clusters. Pub/sub propagation behavior differs between the two modes.
* **Transit encryption (TLS)** settings must match across all regions (all enabled or all disabled).
* **At-rest encryption** settings must also match across all regions (all enabled or all disabled).
* **IPv6 not supported.** Global datastores do not support Internet Protocol version 6 (IPv6).
* **Maximum of three regions**: one primary and up to two secondary regions. The exception is the China (Beijing) and China (Ningxia) regions, where replication can only occur between those two regions.
* **China region restriction:** In China, Global Datastore replication can only occur between the Beijing (cn-north-1) and Ningxia (cn-northwest-1) regions. Cross-region replication to or from regions outside China is not supported.
## Creating a Global Datastore
### Step 1: Create the Global Datastore from an Existing Primary
The primary replication group must already exist and be in `available` status.
```bash
aws elasticache create-global-replication-group \
--global-replication-group-id-suffix my-global-ds \
--primary-replication-group-id <primary-cluster-id> \
--region <primary-region>
```
This registers the existing primary replication group as the primary member of the Global Datastore.
Wait for the Global Datastore status to become `available`:
```bash
aws elasticache describe-global-replication-groups \
--global-replication-group-id ldgnf-my-global-ds \
--region <primary-region>
```
Note: The full Global Datastore ID is prefixed with a short string derived from the primary region (e.g., `ldgnf-` for us-east-1). Use `describe-global-replication-groups` without the ID filter to discover the prefix if needed.
### Step 2: Add a Secondary Region
Create a replication group in the secondary region that joins the Global Datastore:
```bash
aws elasticache create-replication-group \
--replication-group-id <secondary-cluster-id> \
--replication-group-description "Secondary region for Global Datastore" \
--global-replication-group-id ldgnf-my-global-ds \
--cache-subnet-group-name <subnet-group-in-secondary-region> \
--security-group-ids <sg-id-in-secondary-region> \
--region <secondary-region>
```
Many parameters are inherited from the Global Datastore and must not be specified when creating a secondary, including: `PrimaryClusterId`, `AutomaticFailoverEnabled`, `NumNodeGroups`, `CacheParameterGroupName`, `CacheNodeType`, `Engine`, `EngineVersion`, `CacheSecurityGroupNames`, `EnableTransitEncryption`, `AtRestEncryptionEnabled`, `SnapshotArns`, and `SnapshotName`.
Monitor until the secondary reaches `available` and replication lag is minimal:
```bash
aws elasticache describe-global-replication-groups \
--global-replication-group-id ldgnf-my-global-ds \
--show-member-info \
--region <primary-region>
```
## Removing a Secondary Region
Removing a secondary region is a two-step process: disassociate, then delete.
### Step 1: Disassociate the Secondary
```bash
aws elasticache disassociate-global-replication-group \
--global-replication-group-id ldgnf-my-global-ds \
--replication-group-id <secondary-cluster-id> \
--replication-group-region <secondary-region> \
--region <primary-region>
```
Wait for the disassociation to complete. The secondary becomes a standalone replication group in its region.
### Step 2: Delete or Retain the Secondary
After disassociation, the secondary is independent. Delete it if no longer needed:
```bash
aws elasticache delete-replication-group \
--replication-group-id <secondary-cluster-id> \
--final-snapshot-identifier <secondary-cluster-id>-final-$(date +%Y%m%d) \
--region <secondary-region>
```
Or retain it as a standalone cluster in that region.
### Deleting the Global Datastore
To delete the Global Datastore itself, all secondaries must be disassociated first:
```bash
aws elasticache delete-global-replication-group \
--global-replication-group-id ldgnf-my-global-ds \
--retain-primary-replication-group \
--region <primary-region>
```
Use `--retain-primary-replication-group` to keep the primary as a standalone cluster after disassociation. Use `--no-retain-primary-replication-group` to delete the primary replication group as part of the Global Datastore deletion. If you want to keep the primary, you must specify `--retain-primary-replication-group` explicitly.
## Upgrading Engine Version on Global Datastore
Engine upgrades must be performed on the Global Datastore object, not on individual members. See `upgrade-patching.md` for the version compatibility matrix and general upgrade procedures.
```bash
aws elasticache modify-global-replication-group \
--global-replication-group-id ldgnf-my-global-ds \
--engine-version 8.0 \
--apply-immediately \
--region <primary-region>
```
The key difference from standalone upgrades: you cannot upgrade individual member replication groups independently. The upgrade applies to secondary regions first, then to the primary region.
## Failover Between Regions
Use regional failover to promote a secondary to primary. This is used for disaster recovery or planned region migration.
> **Important:** ElastiCache does not support automatic cross-region failover. When needed, you must promote a secondary cluster manually.
```bash
aws elasticache failover-global-replication-group \
--global-replication-group-id ldgnf-my-global-ds \
--primary-region <current-secondary-region> \
--primary-replication-group-id <current-secondary-cluster-id> \
--region <current-primary-region>
```
After failover:
* The former secondary becomes the new primary (read-write).
* The former primary becomes the new secondary (read-only).
* Application writes must target the new primary region's endpoint.
* Update application connection strings or DNS to point to the new primary region.
Monitor failover progress:
```bash
aws elasticache describe-global-replication-groups \
--global-replication-group-id ldgnf-my-global-ds \
--show-member-info \
--region <new-primary-region>
```
### Planned Failover vs Unplanned Scenarios
The `failover-global-replication-group` command performs a planned failover (both regions must be healthy). For unplanned scenarios where the primary region is unavailable:
1. Disassociate the secondary from the Global Datastore (this promotes it to a standalone primary).
2. Point applications to the promoted cluster.
3. When the original primary region recovers, assess data divergence before re-establishing replication.
## Node Type Changes on Global Datastore
Same principle as engine upgrades: modify the Global Datastore object, not individual members. See `upgrade-patching.md` for general node type change procedures.
```bash
aws elasticache modify-global-replication-group \
--global-replication-group-id ldgnf-my-global-ds \
--cache-node-type cache.r7g.xlarge \
--apply-immediately \
--region <primary-region>
```
references/migration/instructions.md
# Migration
**When to use:** The user wants to migrate from self-hosted Redis to ElastiCache, upgrade from Redis OSS to Valkey, move between node-based and serverless deployment models, or needs a preflight compatibility check for a planned migration.
**When not needed:** The user is creating a new cache from scratch (use create-secure-cache.md), asking about data modeling patterns, monitoring, or GenAI features.
Migrate to ElastiCache from self-hosted Redis, or between ElastiCache engines and deployment models.
## Loading
Read this file first. Other references in this folder load on demand when the current answer requires them. Scripts in `scripts/` run on demand (for example, `migration_preflight.py` before cutover).
## Check for existing context
Before starting, read `.elasticache/requirements.json` if it exists. Use `engine`, `deployment_model`, and `infrastructure.endpoint` to understand the current cache. If the file does not exist, gather this information from the user.
## References
- `references/migration/valkey-migration-guide.md` -- Redis-to-Valkey migration decision tree, client compatibility matrix, multi-step upgrade paths, extended support cost impact. Load when the user is planning or executing a Redis-to-Valkey engine switch.
- `references/migration/self-managed-migration.md` -- self-managed Redis to ElastiCache migration details.
- `references/migration/upgrade-patching.md` -- engine version upgrades, service update patching, node type changes, maintenance windows.
- `references/migration/feature-comparison.md` -- engine and deployment model comparison matrices.
- `references/migration/auth-migration.md` -- AUTH token to RBAC migration, RBAC user/group setup, IAM auth setup, cluster-mode disabled to cluster-mode enabled migration. Load when the user is changing authentication models or enabling cluster mode.
- `references/migration/global-datastore-operations.md` -- Global Datastore setup, adding/removing secondary regions, cross-region failover, engine upgrades on Global Datastore. Load when the user is working with cross-region replication or Global Datastore.
- `references/migration/rollback-procedures.md` -- rollback plans, undo migration, recovery from failed migrations.
- `references/migration/topology-validation.md` -- AZ, subnet, security group validation before migration.
- `references/migration/sizing-assessment.md` -- memory, CPU, connection, throughput assessment for target sizing.
## Intent-to-File Routing
**IMPORTANT:** When the user mentions migrating from self-managed, self-hosted, on-prem, EC2-hosted, or any non-ElastiCache Redis/Valkey/Memcached instance, ALWAYS load `self-managed-migration.md` before answering. This file contains critical constraints (e.g., DUMP/RESTORE is banned for cross-engine migration because serialization formats are engine-specific). Without it, models default to DUMP/RESTORE which will silently corrupt data.
| User asks about | Load |
|---|---|
| Redis to Valkey, engine switch, cost savings | `valkey-migration-guide.md` |
| Self-managed, self-hosted, on-prem, EC2, non-ElastiCache Redis to ElastiCache | `self-managed-migration.md` (FORCE LOAD) |
| Version upgrade, patching, node type change, maintenance window | `upgrade-patching.md` |
| Engine or deployment model comparison | `feature-comparison.md` |
| AUTH to RBAC, cluster mode disabled to enabled | `auth-migration.md` -- CMD to CME is a two-step process (disabled to compatible to enabled), requires Valkey 7.2+/Redis OSS 7.0+, auto-failover with 1+ replicas, keys in DB 0 only, and is **irreversible** once set to enabled |
| Global Datastore, cross-region replication | `global-datastore-operations.md` |
| Rollback, undo migration, something went wrong | `rollback-procedures.md` |
| AZ mismatch, subnet, security group port issues | `topology-validation.md` |
| Target sizing, memory/CPU assessment, cost estimation | `sizing-assessment.md` |
## Scripts
- `scripts/migration_preflight.py` -- preflight validation (version, modules, memory, cluster mode)
- `scripts/serverless_estimator.py` -- estimates what existing clusters would cost on serverless (uses actual metrics)
- `scripts/command_classifier.py` -- classifies commands for ECPU estimation (used by serverless_estimator)
- `scripts/pricing.py` -- pricing loader with built-in defaults (used by serverless_estimator)
- `scripts/collect_metrics.sh` -- collects memory, commandstats, replication info from a running cluster
- `assets/examples/sample_input_simple.csv` -- example cluster input for serverless_estimator
- `assets/examples/sample_commandstats.csv` -- example per-command stats for detailed mode
- `scripts/price_calculator.py` -- greenfield cost comparison (serverless vs node-based, engine vs engine)
## Workflow
### IMPORTANT: Validate-Before-Migrate Gate (Hard Requirement)
**The skill must NOT emit, suggest, or execute any migration command (test-migration, start-migration, complete-migration, modify-replication-group for engine switch, or any data-migration command via CLI or SDK) until ALL preflight checks pass.** This is a hard gate, not a recommendation. If any preflight check returns a FAIL status, the skill must refuse to proceed and instruct the user to resolve the failures first.
This gate applies regardless of user urgency, automation context, or IDE mode. The "Never-Auto-Execute List" includes: migration cutover execution, engine switches, replication group deletion, snapshot restoration to production clusters, and Global Datastore regional failover.
### Step 1: Identify the migration path
Determine which migration path applies (see "Migration Paths" below). This determines which preflight checks, topology validations, and rollback procedures are relevant.
### Step 2: Run preflight validation (MANDATORY, blocks all subsequent steps)
Run the preflight check against the source:
```bash
python3 scripts/migration_preflight.py --host <source-host> --port 6379
python3 scripts/migration_preflight.py --host <source-host> --port 6379 --tls # if TLS
```
This checks version compatibility, module usage, key count, memory, cluster mode, and sizing.
**Gate enforcement rules:**
- If the preflight script exits with a non-zero status (any FAIL finding), the skill MUST stop here.
- The skill MUST NOT proceed to Step 3 or any subsequent step.
- The skill MUST present each FAIL finding to the user with remediation guidance.
- The skill MUST ask the user to resolve all failures and re-run preflight before continuing.
- WARN findings should be presented to the user but do not block the migration.
### Topology and Sizing
After preflight passes, validate topology and sizing:
- Topology validation (AZ, subnet, security group compatibility): see `topology-validation.md`
- Sizing assessment (memory, CPU, connections, cost estimation): see `sizing-assessment.md`
### Step 3: Create pre-cutover snapshot
Before executing any migration command, create a snapshot of the source (if ElastiCache) or confirm a backup exists (if self-managed):
```bash
# For ElastiCache source
aws elasticache create-snapshot \
--replication-group-id <source-cluster> \
--snapshot-name pre-migration-$(date +%Y%m%d-%H%M%S) \
--region <region>
```
The skill MUST confirm the snapshot is in `available` status before proceeding.
### Step 4: Execute migration with AWS CLI or SDK
Only after Steps 2, 3, and topology/sizing validation pass. Use the migration commands described in the "Migration Paths" section below.
### Step 5: Validate and cut over
Before cutover, the skill must follow this approval gate:
- Show before/after configuration diff
- Confirm timing window with the user
- Require explicit "proceed" from the user
- Suggest snapshot before cutover (already done in Step 3)
**Connection draining:** Before switching endpoints, drain existing connections to the old cache. Stop sending new requests to the old endpoint, wait for in-flight operations to complete (typically 5-10 seconds), then switch. If using connection pools, close the old pool after all borrowed connections are returned. Do not kill active connections mid-operation.
### Step 6: Run security audit on the new cache
```bash
python3 scripts/security_audit.py --serverless <name>
python3 scripts/security_audit.py --replication-group <name>
```
### Step 7: Post-migration validation
Verify key count, spot-check data, confirm latency, and monitor error rates. See the "Pre-Migration Checklist" section and `references/migration/rollback-procedures.md` for the verification checklist.
### Step 8: Update requirements.json and hand off
After migration completes, update `.elasticache/requirements.json` with the new cache's details: `engine`, `deployment_model`, `infrastructure.cache_name`, `infrastructure.endpoint`, `infrastructure.auth_model`. This ensures downstream sub-skills (setup, data-modeling, monitoring) pick up the new cache without re-asking. Then prompt the user: "Want me to set up monitoring for the new cache?" If yes, hand off to `monitoring`.
### Always: Confirm with the user before each destructive step
The skill must never auto-execute destructive operations (engine switches, cluster deletion, snapshot restoration, migration cutover). Always require explicit user confirmation.
## Migration Paths
### Self-Hosted Redis to ElastiCache
Migration tools in sequence:
1. **Test first**: `test-migration` -- validates connectivity and compatibility without moving data
2. **Start migration**: `start-migration` -- begins data replication from source to ElastiCache
3. **Complete migration**: `complete-migration` -- finalizes and cuts over
```bash
aws elasticache test-migration \
--replication-group-id <target> \
--customer-node-endpoint-list Address=<source>,Port=6379 \
--region <region>
```
**Online migration prerequisites (test-migration / start-migration / complete-migration):**
*Target requirements:*
- Target must NOT have encryption in-transit enabled (you can enable TLS after migration completes)
- Target must have Multi-AZ enabled with automatic failover
- Target must be using Valkey, or Redis OSS 5.0.6 or higher
- Target must NOT be part of a global datastore
- Target must have data tiering disabled
- Number of shards in source and target must match
- Number of logical databases must be the same on source and target
*Source requirements:*
- Source must NOT have AUTH enabled
- Source `protected-mode` must be set to `no`
- Data-modification commands must not be renamed (e.g., `sync`, `psync`, `info`, `config`, `command`, `cluster`)
- If source has a `bind` configuration, it must allow requests from ElastiCache nodes
### Redis OSS to Valkey (In-Place Engine Upgrade)
Zero-downtime upgrade when upgrading from Redis OSS 5.0.6+. Upgrading from Redis OSS versions earlier than 5.0.6 may experience 30-60 seconds of failover during DNS propagation. No data migration needed.
```bash
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--engine valkey \
--engine-version 7.2 \
--apply-immediately \
--region <region>
```
Key facts:
- Valkey is designed as a drop-in replacement for Redis OSS 7. You can upgrade from Redis OSS 5.x, 6.x, or 7.x directly to Valkey
- No application code changes needed for default parameter groups. If you have a custom cache parameter group, you must pass a custom Valkey parameter group with the same static parameter values
- To upgrade a single-node Redis OSS (cluster mode disabled) cluster, you must first add it to a replication group before performing the cross-engine upgrade
- AWS CLI version requirements: AWS CLI v1 minimum 1.35.2, AWS CLI v2 minimum 2.18.2
- Approximate 20% cost savings on node-based pricing (see https://aws.amazon.com/elasticache/pricing/ for current pricing)
- Can then migrate to serverless for additional savings (see https://aws.amazon.com/elasticache/pricing/ for current pricing)
- **Rollback supported**: ElastiCache supports rolling back from Valkey 7.2 to Redis OSS 7.1. Any user group/user associated with the cache being rolled back must be configured with engine type `REDIS`
### Node-Based to Serverless
No in-place conversion. Requires creating a new serverless cache and migrating data.
Steps:
1. Create new Valkey serverless cache (use `setup` sub-skill)
2. Dual-write: application writes to both old and new cache
3. Warm up: let reads gradually shift to new cache
4. Cut over: point all reads to serverless
5. Stop writes to old cache
6. Delete old cluster: `delete-replication-group` (with optional final snapshot)
Alternative: snapshot-restore if brief downtime is acceptable. Note: ElastiCache Serverless snapshot restore has engine version requirements. Verify RDB version compatibility against the latest AWS docs before attempting a restore: https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/
**Auth changes**: Serverless requires RBAC (AUTH tokens not supported). Update application auth before cutover.
**TLS changes**: Serverless always has TLS enabled. Update client config to use `ssl=True` / `tls: {}`.
**Cluster mode changes**: Serverless operates in cluster mode enabled only. Clients must support cluster mode enabled to connect. If your application currently uses a cluster-mode-disabled client, you must update it before cutover.
### Serverless to Node-Based
Rare, but needed when workload outgrows serverless cost-efficiency or requires vector search (Valkey 8.2 or above, node-based only).
Steps:
1. Create new node-based replication group
2. Dual-write pattern (same as above)
3. Cut over and decommission serverless cache: `delete-serverless-cache`
## Pre-Migration Checklist
- [ ] Check source Redis version compatibility (Valkey supports Redis OSS 7.2 commands)
- [ ] Check for Redis module usage (RedisJSON, RediSearch, RedisTimeSeries) -- not all available in Valkey
- [ ] If migrating to serverless: auth must use RBAC (not AUTH tokens), TLS is mandatory
- [ ] Verify VPC/security group configuration for the target
- [ ] Plan for DNS/endpoint changes in application config
- [ ] Test connectivity from application to new cache endpoint (`valkey-cli -h <endpoint> -p 6379 --tls PING`)
- [ ] For online migration (test-migration/start-migration/complete-migration): Target must NOT have encryption in-transit enabled
- [ ] For online migration: Target must have Multi-AZ enabled with automatic failover
- [ ] Have rollback plan (keep old cache running until validated)
- [ ] Run `scripts/serverless_estimator.py` for node-based to serverless cost validation
- [ ] Run `scripts/price_calculator.py --mode serverless --data-gb <N> --ops-per-sec <N>` and `--mode node --node-type <type> --nodes <N> --show-ri-options` to validate cost expectations for greenfield sizing
## Freshness disclaimer
When your response includes pricing, version constraints, or feature availability, include the freshness disclaimer per SKILL.md Global Rule #5: "For current pricing see https://aws.amazon.com/elasticache/pricing/. For current feature availability see https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/."
references/migration/rollback-procedures.md
# Migration Rollback Procedures
Explicit rollback procedures for every supported migration path. Each procedure assumes the pre-cutover snapshot and the old environment are still available. The skill must reference these procedures during migration planning and present the relevant rollback plan to the user before cutover.
## General Rollback Principles
1. **Never decommission the source** until the validation period ends (minimum 48-72 hours post-cutover).
2. **Always create a snapshot** of the target immediately before cutover so you have a recovery point for the new environment as well.
3. **DNS or connection-string rollback** is the fastest path back. Use DNS TTLs of 60 seconds or less during migration windows.
4. **Data verification** must pass before and after any rollback to confirm consistency.
---
## Rollback: Redis OSS to Valkey (In-Place Engine Upgrade)
**Migration type:** In-place engine switch via `modify-replication-group`. The Valkey 7.2 to Redis OSS 7.1 rollback is zero-downtime and preserves the endpoint IP address. (Note: the 30-60 second failover caveat applies to forward upgrades from Redis OSS versions earlier than 5.0.6, not to this rollback path.)
**Preferred rollback method:** ElastiCache supports rolling back from Valkey 7.2 to Redis OSS 7.1 in-place using `modify-replication-group` (or `modify-serverless-cache` for serverless). You can perform a rollback using the same API/CLI steps as an engine upgrade, specifying Redis OSS 7.1 as the target engine version. The rollback is zero-downtime and preserves the endpoint IP address. Only Valkey 7.2 to Redis OSS 7.1 is supported, even if you originally upgraded from an earlier Redis OSS version. Any user group and user associated with the replication group must be configured with engine type `REDIS` for the rollback to work. If a custom parameter group is in use, a compatible Redis OSS parameter group must be provided.
**Alternative rollback method:** You can also restore a snapshot created from your Valkey 7.2 cache as a Redis OSS 7.1 cache by specifying Redis OSS 7.1 as the target engine version during restore. This creates a new cache from the snapshot and does not affect the Valkey cache.
**Note:** This section covers node-based (replication group) rollback. For serverless caches, use `modify-serverless-cache` with `--engine redis --major-engine-version 7` for in-place rollback, or restore a serverless cache snapshot specifying Redis OSS 7.1 as the target engine.
### Pre-Cutover Preparation
1. Create a manual snapshot before starting the engine switch:
```bash
aws elasticache create-snapshot \
--replication-group-id <cluster-id> \
--snapshot-name pre-valkey-upgrade-$(date +%Y%m%d-%H%M%S) \
--region <region>
```
2. Record the current engine, engine version, node type, parameter group, and security group configuration.
3. Confirm the snapshot status is `available` before proceeding with the upgrade.
### Rollback Steps
#### Option A: In-Place Rollback (Preferred, Zero Downtime)
1. **Roll back the engine in-place** using `modify-replication-group`:
```bash
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--engine redis \
--engine-version 7.1 \
--region <region>
```
For serverless caches, use:
```bash
aws elasticache modify-serverless-cache \
--serverless-cache-name <cache-name> \
--engine redis \
--major-engine-version 7
```
2. **Wait for the rollback to complete.** Monitor the replication group status until it returns to `available`. The endpoint IP address and all other aspects of the application will not change.
3. **Verify data integrity** and application behavior (see verification steps below).
**Requirements for in-place rollback:**
* Only Valkey 7.2 to Redis OSS 7.1 is supported (even if you upgraded from an earlier version).
* Any user group and user associated with the replication group must be configured with engine type `REDIS`.
* If a custom parameter group is in use, provide a compatible Redis OSS parameter group via `--cache-parameter-group-name`.
#### Option B: Snapshot Restore (Fallback)
Use this approach if in-place rollback is not possible (e.g., user group engine type issues).
1. **Stop application traffic** to the upgraded (Valkey) cluster if possible, or prepare to redirect traffic.
2. **Restore the pre-upgrade snapshot** to a new replication group running Redis OSS:
```bash
aws elasticache create-replication-group \
--replication-group-id <cluster-id>-rollback \
--replication-group-description "Rollback from Valkey to Redis OSS" \
--engine redis \
--engine-version 7.1 \
--cache-node-type <original-node-type> \
--snapshot-name pre-valkey-upgrade-<timestamp> \
--num-cache-clusters 2 \
--transit-encryption-enabled \
--automatic-failover-enabled \
--multi-az-enabled \
--cache-subnet-group-name <subnet-group> \
--security-group-ids <sg-id> \
--region <region>
```
3. **Wait for the new cluster** to reach `available` status:
```bash
aws elasticache describe-replication-groups \
--replication-group-id <cluster-id>-rollback \
--query "ReplicationGroups[0].Status" \
--region <region>
```
4. **Verify data integrity** on the restored cluster:
* Compare `DBSIZE` with the expected key count from before the upgrade.
* Spot-check a sample of keys to confirm values match.
5. **Update application connection strings** (or DNS) to point to the restored cluster's endpoint.
6. **Validate application behavior:**
* Confirm reads and writes succeed.
* Check latency is within expected range.
* Monitor error rates in application logs.
7. **Decommission the Valkey cluster** after the validation period:
```bash
aws elasticache delete-replication-group \
--replication-group-id <cluster-id> \
--final-snapshot-identifier <cluster-id>-final-$(date +%Y%m%d) \
--region <region>
```
### Data Gap Consideration
**If using in-place rollback (Option A):** No data loss occurs. The rollback preserves all data on the cluster.
**If using snapshot restore (Option B):** Any writes that occurred between the pre-upgrade snapshot and the rollback will be lost. To minimize data loss:
* Keep the migration window as short as possible.
* If the Valkey cluster is still running, consider exporting recent keys before restoring the snapshot.
### Rollback Limitations
* ElastiCache only supports rolling back from Valkey 7.2 to Redis OSS 7.1. This is true even if you upgraded to Valkey 7.2 from an earlier version than Redis OSS 7.1.
* Any user group and user associated with the replication group or serverless cache being rolled back must be configured with engine type `REDIS`.
* If a custom parameter group is applied, a compatible Redis OSS parameter group must be provided during rollback.
---
## Rollback: Self-Managed Redis to ElastiCache
**Migration type:** Replication-based (online) or snapshot-based (offline) migration to an ElastiCache replication group or serverless cache.
**Rollback advantage:** The self-managed Redis source is still running during migration. Rollback is straightforward because the source was never modified.
**Prerequisite note:** Online migration requires the target ElastiCache cluster to have transit encryption (TLS) disabled. If transit encryption was enabled on the target, online migration cannot be used; use backup/restore instead. See `topology-validation.md` for the full prerequisites checklist.
### Pre-Cutover Preparation
1. **Keep the self-managed Redis instance running** throughout the migration and validation period.
2. Create a snapshot of the ElastiCache target before cutover:
```bash
aws elasticache create-snapshot \
--replication-group-id <target-cluster> \
--snapshot-name pre-cutover-$(date +%Y%m%d-%H%M%S) \
--region <region>
```
3. Record the current application connection strings and DNS entries.
4. Set DNS TTL to 60 seconds or less if using DNS-based endpoint switching.
### Rollback Steps
1. **Revert application connection strings** to point back to the self-managed Redis endpoint.
* If using DNS: update the DNS record to the self-managed Redis IP/hostname.
* If using config files or environment variables: deploy with the old connection string.
2. **Deploy the reverted application** across all instances/containers.
3. **Verify connectivity** to the self-managed Redis:
```bash
valkey-cli -h <self-managed-host> -p 6379 --tls PING # omit --tls if TLS is not enabled on the source
```
4. **Verify data integrity:**
* Confirm `DBSIZE` matches expectations.
* Spot-check key values.
* Verify that writes during the cutover window are present (if the source was still receiving writes via dual-write).
5. **Handle data written only to ElastiCache during cutover:**
* If the application wrote exclusively to ElastiCache after cutover, those writes are not on the self-managed source.
* For critical data, export those keys from ElastiCache before decommissioning:
```bash
# Export individual keys via application-level read/write, or create a snapshot for bulk recovery
aws elasticache create-snapshot \
--replication-group-id <target-cluster> \
--snapshot-name post-cutover-backup-$(date +%Y%m%d) \
--region <region>
```
6. **Stop the ElastiCache migration** if it is still in progress: call `complete-migration` with the `--force` flag to stop replication from the source without waiting for sync to finish:
```bash
aws elasticache complete-migration \
--replication-group-id <target-cluster> \
--force \
--region <region>
```
Alternatively, delete the target replication group to cancel entirely:
```bash
aws elasticache delete-replication-group \
--replication-group-id <target-cluster> \
--no-retain-primary-replication-group \
--final-snapshot-identifier <target-cluster>-rollback-$(date +%Y%m%d) \
--region <region>
```
7. **Keep or delete the ElastiCache target** based on whether you plan to retry the migration later.
---
## Rollback: Node-Based to Serverless
**Migration type:** New serverless cache created, data migrated via dual-write or snapshot-restore, then cutover.
**Rollback advantage:** The original node-based cluster remains running during migration.
### Pre-Cutover Preparation
1. **Keep the node-based cluster running** until the validation period ends.
2. Create a snapshot of the node-based cluster before cutover:
```bash
aws elasticache create-snapshot \
--replication-group-id <node-based-cluster> \
--snapshot-name pre-serverless-cutover-$(date +%Y%m%d-%H%M%S) \
--region <region>
```
3. Document the differences between node-based and serverless that affect the application:
* Auth model: serverless requires RBAC (AUTH tokens not supported).
* TLS: serverless always has TLS enabled.
* Port: serverless uses port 6379 with mandatory TLS (different from node-based where TLS may be optional).
* Endpoint format: serverless endpoints use `<cache-name>-xxxxx.serverless.<region-code>.cache.amazonaws.com` (where `<region-code>` is an abbreviated form, e.g., `use1` for us-east-1).
* Cluster mode: serverless operates in cluster-mode-enabled only. Clients must support cluster protocol (MOVED/ASK redirects). When rolling back to node-based CMD, clients that were updated for cluster mode may need to revert to non-cluster connection handling.
### Rollback Steps
1. **Revert application auth configuration:**
* If the application was updated from AUTH token to RBAC for serverless, revert to AUTH token auth for the node-based cluster.
* If the application already used RBAC, revert the endpoint only.
2. **Revert application connection strings** to the node-based cluster endpoint:
```python
# Revert from serverless endpoint
# client = valkey.Valkey(host="<cache-name>-xxxxx.serverless.use1.cache.amazonaws.com", port=6379, ssl=True, username="appuser", password=pw)
# Back to node-based endpoint
client = valkey.Valkey(host="node-based-primary-endpoint", port=6379, ssl=True, password=auth_token)
```
3. **Revert TLS configuration** if the node-based cluster does not use TLS:
* Remove `ssl=True` from client configuration if the original cluster did not have transit encryption enabled.
4. **Deploy the reverted application.**
5. **Verify data on the node-based cluster:**
* Confirm `DBSIZE` matches expectations.
* Verify that data written only to serverless during the cutover window is handled (export or accept loss).
6. **Delete the serverless cache** after validation:
```bash
aws elasticache delete-serverless-cache \
--serverless-cache-name <serverless-cache> \
--final-snapshot-name <serverless-cache>-final-$(date +%Y%m%d) \
--region <region>
```
### Auth Rollback Detail
If you changed from AUTH token to RBAC as part of the serverless migration:
1. The node-based cluster should still accept AUTH token connections (if you did not remove the AUTH token).
2. Revert application code to use `password=auth_token` without a `username`.
3. See `references/migration/auth-migration.md` for the full AUTH-to-RBAC rollback procedure.
---
## Rollback: Serverless to Node-Based
**Migration type:** New node-based cluster created, data migrated via dual-write or snapshot-restore, then cutover.
**Rollback advantage:** The original serverless cache remains running during migration.
### Pre-Cutover Preparation
1. **Keep the serverless cache running** until the validation period ends.
2. Create a manual snapshot of the serverless cache before cutover using `CreateServerlessCacheSnapshot`:
```bash
aws elasticache create-serverless-cache-snapshot \
--serverless-cache-name <serverless-cache> \
--serverless-cache-snapshot-name pre-node-cutover-$(date +%Y%m%d-%H%M%S)
```
3. Document the configuration differences for rollback:
* Auth model: serverless uses RBAC; node-based may use AUTH token or RBAC.
* TLS: serverless always has TLS; node-based may or may not.
* Endpoint: serverless endpoint format differs from node-based.
### Rollback Steps
1. **Revert application connection strings** to the serverless cache endpoint:
```python
# Revert from node-based endpoint
# client = valkey.Valkey(host="node-based-primary-endpoint", port=6379, ssl=True, password=auth_token)
# Back to serverless endpoint
client = valkey.Valkey(host="<cache-name>-xxxxx.serverless.use1.cache.amazonaws.com", port=6379, ssl=True, username="appuser", password=rbac_password)
```
2. **Revert auth configuration** if you switched from RBAC to AUTH token:
* Restore `username` parameter in client configuration.
* Use RBAC credentials instead of AUTH token.
3. **Ensure TLS is enabled** in client configuration (`ssl=True`), since serverless requires it.
4. **Deploy the reverted application.**
5. **Verify connectivity and data:**
```bash
valkey-cli -h <serverless-endpoint> -p 6379 --tls PING
```
* Confirm reads and writes succeed.
* Check latency is within the expected serverless range (single-digit ms).
6. **Delete the node-based cluster** after the validation period:
```bash
aws elasticache delete-replication-group \
--replication-group-id <node-based-cluster> \
--final-snapshot-identifier <node-based-cluster>-final-$(date +%Y%m%d) \
--region <region>
```
---
## Rollback Verification Checklist
After completing any rollback, run through this checklist:
* [ ] Application is connected to the original (pre-migration) cache endpoint
* [ ] Authentication is working (correct auth model, credentials, username if RBAC)
* [ ] TLS configuration matches the original cache (enabled/disabled)
* [ ] `DBSIZE` matches expected key count
* [ ] Spot-check: sample keys return correct values
* [ ] Application read and write latency is within normal range
* [ ] No connection errors in application logs
* [ ] CloudWatch metrics on the original cache show healthy traffic
* [ ] Security audit passes: `python3 scripts/security_audit.py --replication-group <name>` or `--serverless <name>`
* [ ] Stakeholders notified of the rollback
* [ ] Post-mortem scheduled to analyze why the migration failed
## When Rollback Is Not Possible
In rare cases, rollback may not be straightforward:
* **Source was already decommissioned.** This is why the skill enforces keeping the source running during the validation period.
* **Data diverged significantly.** If the application wrote different data to source and target for an extended period, merging is complex. Use the ElastiCache snapshot as the recovery point and accept data written only to the source as lost.
* **Auth model was changed on the original cluster.** If you modified the original cluster's auth configuration (e.g., removed AUTH token) as part of migration prep, restoring auth requires reconfiguring the original cluster or restoring from snapshot.
In all cases, the pre-cutover snapshot provides a recovery point. The skill must confirm a snapshot exists before allowing cutover to proceed.
references/migration/self-managed-migration.md
# Self-Managed Redis Migration to ElastiCache
Migrate from self-managed Redis (running on EC2, on-premises, or another cloud provider) to Amazon ElastiCache. This guide covers assessment, migration approaches, and cutover.
## Assessment Phase
Before migrating, inventory your source environment and validate compatibility.
### 1. Source Inventory
Gather the following from your self-managed Redis instance:
| Item | How to check | Why it matters |
|------|-------------|----------------|
| Redis version | `INFO server` -> `redis_version` | Valkey 7.2 is compatible with Redis OSS up to version 7.2.4. Older source versions may use deprecated features. |
| Data size | `INFO memory` -> `used_memory_human` | Determines target cache sizing and migration duration |
| Key count | `DBSIZE` | Helps estimate migration time and validate completeness |
| Command usage | `INFO commandstats` | Identifies unsupported or uncommon commands |
| Persistence config | `CONFIG GET save`, `CONFIG GET appendonly` | Determines whether RDB snapshots or AOF are in use for snapshot-based migration |
| Cluster mode | `INFO cluster` -> `cluster_enabled` | Cluster mode affects migration tooling and target topology |
| Modules loaded | `MODULE LIST` | RedisJSON, RediSearch, RedisTimeSeries, etc. may not all be available in Valkey |
| Connected clients | `INFO clients` -> `connected_clients` | Helps size the target cache |
| Peak memory | `INFO memory` -> `used_memory_peak_human` | Size the target to handle peak load |
| Replication topology | `INFO replication` | Identifies primary/replica structure to replicate on the target |
### 2. Automated Preflight Check
Run the bundled preflight script against your source Redis:
```bash
# Without TLS
python3 scripts/migration_preflight.py --host <source-host> --port 6379
# With TLS
python3 scripts/migration_preflight.py --host <source-host> --port 6379 --tls
```
The script checks version compatibility, module usage, key count, memory, cluster mode, and sizing. Resolve any **FAIL** findings before proceeding.
### 3. Compatibility Check
Review potential compatibility issues:
- **Unsupported commands:** Some Redis commands may behave differently or be unavailable. Check `commandstats` output against [ElastiCache supported commands](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/RestrictedCommands.html).
- **Modules:** If you use RedisJSON, RediSearch, or RedisTimeSeries modules, verify whether Valkey provides equivalent functionality or plan application changes.
- **Lua scripts:** Test all Lua scripts against the target engine. Most standard Lua scripts work unchanged, but scripts that use module-specific commands need updates.
- **Restricted commands:** The following commands are unavailable on ElastiCache: `bgrewriteaof`, `bgsave`, `config`, `debug`, `migrate`, `replicaof`, `save`, `slaveof`, `shutdown`, and `sync`. Use parameter groups instead of `CONFIG` commands. Check your application code and scripts for usage of any of these commands.
### 4. Network Planning
ElastiCache runs inside a VPC. Plan connectivity from your source to the target:
- **Source on EC2 (same VPC):** Direct access. Ensure security groups allow traffic on port 6379.
- **Source on EC2 (different VPC):** Use VPC peering or Transit Gateway.
- **Source on-premises:** Use AWS Direct Connect or Site-to-Site VPN for replication-based migration. For snapshot-based, upload the RDB file to S3.
- **Source on another cloud:** Use VPN, Direct Connect, or snapshot-based migration via S3.
---
## Migration Approaches
### Approach 1: Snapshot-Based (Offline)
**Best for:** Small to medium datasets, acceptable downtime window, simplest execution.
**Steps:**
```bash
# 1. Create RDB snapshot on the source
valkey-cli -h <source-host> BGSAVE
# Wait for completion:
valkey-cli -h <source-host> LASTSAVE
# 2. Copy the RDB file to S3
aws s3 cp /var/lib/redis/dump.rdb s3://my-migration-bucket/dump.rdb
# 3. Grant ElastiCache access to the S3 bucket by adding a bucket policy:
# {
# "Version": "2012-10-17",
# "Statement": [{
# "Sid": "ElastiCacheSnapshotAccess",
# "Effect": "Allow",
# "Principal": { "Service": "<region>.elasticache-snapshot.amazonaws.com" },
# "Action": ["s3:GetObject", "s3:ListBucket", "s3:GetBucketAcl"],
# "Resource": [
# "arn:aws:s3:::my-migration-bucket",
# "arn:aws:s3:::my-migration-bucket/*"
# ]
# }]
# }
# 4. Create an ElastiCache replication group and seed from the snapshot
aws elasticache create-replication-group \
--replication-group-id my-new-cache \
--replication-group-description "Migrated from self-managed Redis" \
--engine valkey \
--engine-version 9.0 \
--cache-node-type cache.r7g.large \
--num-cache-clusters 2 \
--snapshot-arns arn:aws:s3:::my-migration-bucket/dump.rdb \
--transit-encryption-enabled \
--region <region>
```
**Downtime:** Equals the time from final snapshot to application cutover. For large datasets, this can be minutes to hours.
**Limitations:**
- Requires a downtime window
- Any writes during migration are lost
- RDB format must be compatible with the target engine version
- For node-based clusters, use `--snapshot-arns` with `create-replication-group`. For serverless caches, use `--snapshot-arns-to-restore` with `create-serverless-cache` instead.
### Approach 2: Replication-Based (Online)
> **Eligibility:** Online migration is not supported to ElastiCache serverless caches or clusters running on the r6gd node type. This approach is designed for migrating from self-hosted Redis/Valkey on EC2 to ElastiCache, not for moving data between ElastiCache clusters.
**Best for:** Large datasets, minimal downtime requirement, source is network-reachable from VPC.
**Steps:**
```bash
# 1. Create the target replication group (without snapshot seeding)
aws elasticache create-replication-group \
--replication-group-id my-new-cache \
--replication-group-description "Migration target" \
--engine valkey \
--engine-version 9.0 \
--cache-node-type cache.r7g.large \
--num-cache-clusters 2 \
--automatic-failover-enabled \
--multi-az-enabled \
--region <region>
# 2. Test the migration (validates connectivity and compatibility)
aws elasticache test-migration \
--replication-group-id my-new-cache \
--customer-node-endpoint-list Address=<source-host>,Port=6379 \
--region <region>
# 3. Start the migration (begins replication)
aws elasticache start-migration \
--replication-group-id my-new-cache \
--customer-node-endpoint-list Address=<source-host>,Port=6379 \
--region <region>
# 4. Monitor replication status
# Option A: Use INFO replication on the ElastiCache primary node
# to check master_link_status (should be 'up') and replication offset
valkey-cli -h <elasticache-primary-endpoint> -p 6379 INFO replication
# Option B: Monitor the ReplicationLag and PrimaryLinkHealthStatus
# CloudWatch metrics for the target replication group
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache \
--metric-name ReplicationLag \
--dimensions Name=CacheClusterId,Value=my-new-cache-001 \
--statistics Average \
--start-time $(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%S) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
--period 60 \
--region <region>
# 5. When lag is minimal, complete the migration (promotes the target)
aws elasticache complete-migration \
--replication-group-id my-new-cache \
--region <region>
# To abort migration without ensuring data is in sync, use --force:
# aws elasticache complete-migration \
# --replication-group-id my-new-cache \
# --force \
# --region <region>
# WARNING: --force stops migration without ensuring data is in sync.
# Use it only to abort migration, not for normal completion.
```
**Replication lag convergence:** "Minimal" means ReplicationLag < 1 second for 5 consecutive checks (1 minute apart). If lag does not converge below 1s within 30 minutes, check source write rate and network bandwidth between source and target. Do not proceed with `complete-migration` until lag is stable.
**Monitoring replication health:** In addition to the CLI command above, verify replication status using:
- Run `INFO replication` on the ElastiCache primary node and confirm `master_link_status` is `up`
- Monitor the **Primary Link Health Status** CloudWatch metric (value of 1 means data is in sync)
- Verify low client output buffer by running `CLIENT LIST` on your source Redis instances
**Downtime:** Near zero. Only the brief moment during the DNS/endpoint switch in your application.
**Important:** During migration, the ElastiCache cluster is read-only. You can use ElastiCache nodes for reads, but you cannot write to the ElastiCache cluster. All writes must continue going to the source Redis until `complete-migration` is executed.
**Requirements:**
*Target requirements:*
- Target must NOT have encryption in-transit enabled (TLS must be disabled during migration; you can enable it after migration completes) (verify against current online migration prerequisites documentation)
- Target must have Multi-AZ enabled (`--automatic-failover-enabled`)
- Target must be using Valkey, or Redis OSS 5.0.6 or higher
- Target must NOT be part of a global datastore
- Target must have data tiering disabled
- Number of shards in source and target must match (prerequisite).
- Number of logical databases must be the same on source and target (set via `databases` in Redis config)
- Target must have sufficient memory available to fit the data from the source cluster. See `sizing-assessment.md` for memory sizing guidance.
*Source requirements:*
- Source Redis must be accessible from the ElastiCache VPC (security groups, VPC peering, VPN, or Direct Connect)
- The security group attached to source Redis instances must allow inbound traffic from ElastiCache nodes
- Source must NOT have AUTH enabled
- Source `protected-mode` must be set to `no`
- If source has `bind` configuration, it must be updated to allow requests from ElastiCache nodes
- Data-modification commands must not be renamed (e.g., `sync`, `psync`, `info`, `config`, `command`, `cluster`)
- All source Redis instances must be running on the same port
- Source must have enough capacity to handle replication overhead
- For cluster-mode disabled, source can be Redis 2.8.21 onward (via CLI). For cluster-mode enabled, source can be any cluster-mode enabled version.
### Approach 3: Dual-Write (Application-Level)
**Best for:** Complex migrations, need full control, source not directly reachable from VPC, or need to transform data during migration.
> **Warning:** Do NOT use DUMP/RESTORE for cross-engine migration (e.g., Redis OSS to Valkey). The serialization format is engine-specific and may fail silently or produce corrupted data. Use application-level copy or the ElastiCache migration tools instead.
**Steps:**
```python
# Dual-write wrapper example
import valkey
old_client = valkey.Valkey(host="old-redis-host", port=6379, decode_responses=True)
new_client = valkey.Valkey(
host="new-cache.serverless.use1.cache.amazonaws.com",
port=6379,
ssl=True,
decode_responses=True,
)
# Feature flag to control read source
READ_FROM_NEW = False # Flip to True when new cache is warm
def cache_set(key: str, value: str, ttl: int = 300):
"""Write to both caches during migration."""
old_client.setex(key, ttl, value)
try:
new_client.setex(key, ttl, value)
except Exception:
# Log but don't fail -- old cache is still the primary
pass
def cache_get(key: str) -> str | None:
"""Read from the active cache."""
if READ_FROM_NEW:
result = new_client.get(key)
if result is None:
# Fall back to old cache during warm-up
result = old_client.get(key)
if result is not None:
# Backfill into new cache
ttl = old_client.ttl(key)
if ttl and ttl > 0:
new_client.setex(key, ttl, result)
return result
else:
return old_client.get(key)
```
**Downtime:** Zero, if the application handles the dual-write correctly.
**Trade-offs:**
- Requires application code changes
- Double write latency during the migration period
- Must handle failures on either cache gracefully
- Most flexible: works even when source is not directly reachable from the VPC
---
## Cutover Checklist
Complete each item before and during the cutover:
- [ ] Target cache provisioned and accessible from application VPC
- [ ] Data migrated or replication caught up (lag below acceptable threshold)
- [ ] Application updated with the new ElastiCache endpoint
- [ ] Authentication configured (RBAC or IAM on the target; serverless does not support AUTH tokens)
- [ ] TLS enabled on the target (mandatory for serverless; strongly recommended for node-based)
- [ ] Client connection code uses `ssl=True` / `tls: {}` for the new endpoint
- [ ] Monitoring configured on the target (CloudWatch metrics, alarms, dashboards -- see `monitoring` sub-skill)
- [ ] Security audit passed on the new cache (`python3 scripts/security_audit.py --serverless <name>` or `--replication-group <name>`)
- [ ] Rollback plan documented (keep old source running until validation period ends)
- [ ] DNS or application config switch executed
- [ ] Smoke tests passed (read/write operations, latency within expected range)
- [ ] Old source decommissioned after validation period (recommended: keep source running for 48-72 hours post-cutover)
---
## Choosing an Approach
| Factor | Snapshot-Based | Replication-Based | Dual-Write |
|--------|:-----------:|:-------------:|:--------:|
| Downtime | Minutes to hours | Near zero | Zero |
| Complexity | Low | Medium | High |
| Network requirement | S3 upload only | Source reachable from VPC | No direct connectivity needed |
| Data transformation | None (exact copy) | None (exact copy) | Possible during write |
| Data size limit | Practical limit ~100 GB | No practical limit | No practical limit |
| Application changes | Endpoint update only | Endpoint update only | Dual-write logic required |
| Best for | Small data, simple setup | Large data, minimal downtime | Complex scenarios, unreachable source |
---
## Post-Migration Validation
After cutover, verify the migration was successful:
1. **Key count:** Compare `DBSIZE` on source vs target
2. **Spot-check data:** Read a sample of keys and verify values match
3. **Latency:** Confirm read/write latency meets expectations (sub-ms for node-based, single-digit ms for serverless)
4. **Hit rate:** Monitor cache hit rate -- expect it to ramp up over the warm-up period
5. **Error rate:** Monitor application logs for connection errors or unexpected responses
6. **Run security audit:**
```bash
python3 scripts/security_audit.py --serverless <name>
# or
python3 scripts/security_audit.py --replication-group <name>
```
references/migration/sizing-assessment.md
# Pre-Migration Sizing Assessment
**When to use:** Before migrating, assess source workload to determine target sizing: memory, CPU, connections, throughput. Prevents undersizing (performance degradation) or oversizing (wasted cost).
Before migrating, assess the source workload to determine appropriate target sizing. Undersizing causes performance degradation; oversizing wastes cost.
## Memory Utilization and Peak Usage
Collect current and peak memory from the source:
```bash
# Self-managed Redis
valkey-cli -h <source-host> INFO memory | grep -E "used_memory_human|used_memory_peak_human|used_memory_dataset_perc|mem_fragmentation_ratio"
# ElastiCache (via CloudWatch)
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache \
--metric-name DatabaseMemoryUsagePercentage \
--dimensions Name=CacheClusterId,Value=<cache-cluster-id> Name=CacheNodeId,Value=0001 \
--start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
--period 3600 \
--statistics Maximum Average \
--region <region>
```
**Sizing rules:**
* **Node-based:** choose a node whose `maxmemory` is at least 2x current peak usage to account for growth, fragmentation, and the default 25% `reserved-memory-percent` (which leaves only 75% of `maxmemory` for data). Account for `reserved-memory-percent` (default 25% for accounts created on or after March 16, 2017; accounts created before that date default to `reserved-memory` with 0 bytes reserved). For burstable instance types, increase `reserved-memory-percent` up to 50% on micro instances and up to 30% on small instances to avoid swap usage during backup and replication.
* **Serverless:** set `DataStorage.Maximum` to at least 2x current usage. The higher multiplier accounts for auto-scaling buffer and the fact that you cannot manually tune memory allocation.
* If `mem_fragmentation_ratio` is above 1.5, the actual memory needed may be lower after migration (ElastiCache manages memory more efficiently).
### CPU Utilization and Command Mix
High CPU on the source may indicate the need for more shards or a larger node type on the target.
```bash
# Self-managed Redis
valkey-cli -h <source-host> INFO commandstats
# Identify expensive commands
valkey-cli -h <source-host> SLOWLOG GET 50
# ElastiCache (via CloudWatch)
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache \
--metric-name EngineCPUUtilization \
--dimensions Name=CacheClusterId,Value=<cache-cluster-id> Name=CacheNodeId,Value=0001 \
--start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
--period 3600 \
--statistics Maximum Average \
--region <region>
```
**Sizing rules:**
* If source CPU consistently exceeds 70%, consider adding shards (horizontal scaling) on the target rather than just matching the source topology.
* Heavy `KEYS`, `SORT`, or Lua script usage drives CPU. These patterns may need optimization regardless of migration.
* For serverless targets, CPU is managed automatically, but high-CPU command patterns will increase ECPU consumption and cost.
### Connection Count and Throughput
```bash
# Self-managed Redis
valkey-cli -h <source-host> INFO clients | grep -E "connected_clients|maxclients"
valkey-cli -h <source-host> INFO stats | grep -E "instantaneous_ops_per_sec|total_commands_processed"
# ElastiCache (via CloudWatch)
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache \
--metric-name CurrConnections \
--dimensions Name=CacheClusterId,Value=<cache-cluster-id> Name=CacheNodeId,Value=0001 \
--start-time $(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%S) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
--period 3600 \
--statistics Maximum \
--region <region>
```
**Sizing rules:**
* Both serverless caches and individual ElastiCache node-based nodes support up to 65,000 concurrent client connections (`maxclients`). Verify the peak connection count is within this limit.
* For serverless, connections scale automatically. ECPU consumption is per-request, not per-connection; idle connections do not consume ECPUs. However, use connection pooling to reduce overhead from connection setup.
* If connection count is high, recommend connection pooling in the application (most client libraries support this).
* Throughput (ops/sec) helps determine whether a single shard is sufficient or multiple shards are needed.
### Source-to-Target Sizing Recommendations
| Source Metric | Serverless Target | Node-Based Target |
|---------------|-------------------|-------------------|
| Memory < 1 GB | Default limits (~$6/month minimum) | cache.t4g.small (t4g.micro has only 0.5 GiB; with 25% reserved-memory, usable is ~0.375 GiB) |
| Memory 1-10 GB | Set DataStorage.Maximum to 2x peak | cache.r7g.large (1 shard) |
| Memory 10-50 GB | Set DataStorage.Maximum to 2x peak | cache.r7g.xlarge (1-2 shards) |
| Memory 50-200 GB | Evaluate cost vs node-based | cache.r7g.2xlarge (multiple shards) |
| Memory > 200 GB | Node-based recommended for cost | cache.r7g.4xlarge+ (multiple shards) |
| Ops/sec < 10,000 | Default ECPU limits | Single shard sufficient |
| Ops/sec 10,000-100,000 | Increase ECPUPerSecond.Maximum | 1-3 shards depending on node type |
| Ops/sec > 100,000 | Evaluate cost vs node-based | Multiple shards, consider cluster mode enabled |
| Connections < 1,000 | Default | Most node types support this |
| Connections 1,000-10,000 | Default (scales automatically) | cache.r7g.large+ |
| Connections > 10,000 | Default (but review ECPU cost) | cache.r7g.xlarge+ with connection pooling |
**Migration method constraints:**
* Online migration is **not supported** to ElastiCache serverless caches or clusters running on the r6gd (data tiering) node type. If you plan to use online migration, choose a non-r6gd node-based target. For serverless or r6gd targets, use backup/restore or another migration strategy.
* Online migration additionally requires: the target must have transit encryption (TLS) disabled, Multi-AZ enabled, and not be part of a Global Datastore. The source must not have AUTH enabled and must have `protected-mode` set to `no`. Shard counts should match between source and target. See `topology-validation.md` for the full prerequisites checklist.
Run cost estimation to compare current spend vs target:
```bash
# For node-based to serverless migrations (uses actual cluster metrics)
python3 scripts/serverless_estimator.py --input clusters.csv --commandstats stats.csv
# For greenfield cost comparison (serverless)
python3 scripts/price_calculator.py --mode serverless --data-gb <N> --ops-per-sec <N>
# For greenfield cost comparison (node-based with RI options)
python3 scripts/price_calculator.py --mode node --node-type <type> --nodes <N> --show-ri-options
```
To collect metrics from a running cluster for the estimator:
```bash
./scripts/collect_metrics.sh <endpoint> <port> [output_prefix]
```
references/migration/topology-validation.md
# Topology Validation
**When to use:** Before executing a migration, validate source/target topology compatibility: AZ configuration, cross-region constraints, subnet groups, security group ports.
Before executing a migration, validate that the source and target topologies are compatible. Mismatches in availability zones, subnets, or security groups are common causes of migration failure or degraded availability post-cutover.
## Single-AZ to Multi-AZ (and Vice Versa)
**Single-AZ source migrating to Multi-AZ target:**
* This is a common and recommended upgrade path (improves availability).
* Verify that the target VPC has subnets in at least two Availability Zones.
* The target subnet group must include subnets in multiple AZs.
* Check: `aws ec2 describe-subnets --filters "Name=vpc-id,Values=<vpc-id>" --query "Subnets[*].{SubnetId:SubnetId,AZ:AvailabilityZone}"` to confirm AZ coverage.
**Multi-AZ source migrating to Single-AZ target:**
* This is a downgrade in availability. The skill must warn the user that automatic failover will not be available and a node failure will cause downtime.
* Confirm the user explicitly accepts the reduced availability before proceeding.
**Detection commands:**
```bash
# Check source topology (if ElastiCache)
aws elasticache describe-replication-groups \
--replication-group-id <source-cluster> \
--query "ReplicationGroups[0].{MultiAZ:MultiAZ,AutoFailover:AutomaticFailover,NodeGroups:NodeGroups[*].NodeGroupMembers[*].PreferredAvailabilityZone}" \
--region <region>
# Check target subnet group AZ coverage
aws elasticache describe-cache-subnet-groups \
--cache-subnet-group-name <subnet-group> \
--query "CacheSubnetGroups[0].Subnets[*].SubnetAvailabilityZone.Name" \
--region <region>
```
### Cross-Region Replication Constraints
* ElastiCache does not support direct cross-region migration via `start-migration` or `test-migration`.
* For cross-region moves, use snapshot-based migration: export a snapshot to S3 in the source region, copy the RDB file to an S3 bucket in the target region, then restore from the S3 object in the target region.
* Global Datastore is available for ongoing cross-region replication but requires node-based clusters (not serverless).
**Online migration (`start-migration`/`test-migration`) prerequisites for the source:**
* The source Redis must NOT have AUTH enabled.
* The source must have `protected-mode` set to `no`.
* The source must update `bind` to allow inbound connections from ElastiCache nodes.
* The source must not have any renamed commands (e.g., via `rename-command` in redis.conf). ElastiCache uses standard commands during replication.
* The source must be listening on the same port as the target ElastiCache cluster (default 6379).
* The number of logical databases must match between source and target.
**Online migration (`start-migration`/`test-migration`) prerequisites for the target:**
* The target ElastiCache deployment must NOT have encryption in-transit enabled.
* The target must have Multi-AZ enabled.
* The target must be running Valkey 7.2+, or Redis OSS 5.0.6 or higher.
* The target must NOT be part of a Global Datastore.
* The target must have data tiering disabled.
**Cross-region snapshot migration** (three-step process):
Step 1: Export snapshot to S3 in the source region (the S3 bucket must be in the same region as the snapshot):
```bash
aws elasticache copy-snapshot \
--source-snapshot-name <snapshot-name> \
--target-snapshot-name <snapshot-name>-export \
--target-bucket <s3-bucket-in-source-region> \
--region <source-region>
```
Step 2: Copy the RDB file to an S3 bucket in the target region:
```bash
aws s3 cp s3://<s3-bucket-in-source-region>/<snapshot-name>-export-0001.rdb \
s3://<s3-bucket-in-target-region>/<snapshot-name>-export-0001.rdb
```
Step 3: Restore from the S3 object in the target region:
```bash
aws elasticache create-replication-group \
--replication-group-id <new-cluster-id> \
--replication-group-description "Restored from cross-region snapshot" \
--engine valkey \
--snapshot-arns arn:aws:s3:::<s3-bucket-in-target-region>/<snapshot-name>-export-0001.rdb \
--cache-node-type <node-type> \
--region <target-region>
```
Note: the `--target-bucket` parameter on `copy-snapshot` is reserved for exporting a snapshot to S3. Do not use it when making a copy of a backup within ElastiCache.
**S3 export** (exports snapshot to an S3 bucket for external use or archival):
```bash
aws elasticache copy-snapshot \
--source-snapshot-name <snapshot-name> \
--target-snapshot-name <snapshot-name>-export \
--target-bucket <s3-bucket-in-same-region> \
--region <source-region>
```
The ElastiCache backup and the S3 bucket must be in the same AWS Region.
### Subnet Group Compatibility
The target cache must use a subnet group within the same VPC, or a peered/connected VPC, as the application workloads.
**Checks to perform:**
* Confirm the target subnet group exists and has subnets in the required AZs.
* Verify that the target subnets have sufficient available IP addresses for the cache nodes (each node and replica consumes one IP).
* If migrating from self-managed Redis in a different VPC, confirm VPC peering or Transit Gateway connectivity.
```bash
# Check available IPs in target subnets
aws ec2 describe-subnets \
--subnet-ids <subnet-1> <subnet-2> \
--query "Subnets[*].{SubnetId:SubnetId,AZ:AvailabilityZone,AvailableIPs:AvailableIpAddressCount}" \
--region <region>
```
### Shard Count Compatibility
For online migration (`start-migration`), the number of shards in the source and target **must** match. This is a hard prerequisite — online migration will fail if shard counts differ. Verify shard counts match before starting migration.
```bash
# Check shard count on source (self-managed Redis cluster-mode enabled)
valkey-cli -h <source-host> -p <port> CLUSTER INFO | grep cluster_size
# Check shard count on target ElastiCache replication group
aws elasticache describe-replication-groups \
--replication-group-id <target-cluster> \
--query "ReplicationGroups[0].NodeGroups | length(@)" \
--region <region>
```
### Security Group Port Mismatches
Different ElastiCache deployment models use different ports. A security group misconfiguration will cause silent connectivity failures.
| Deployment Model | Default Port | TLS Required? |
|-----------------|:------------:|:-------------:|
| Node-based (no TLS) | 6379 | No |
| Node-based (with TLS) | 6379 | Yes |
| Serverless | 6379 | Yes (always) |
**Common mismatches to check:**
* Source security group allows port 6379 but target security group does not (or vice versa).
* Application security group has outbound rules that restrict the target port.
* When migrating from node-based to serverless, confirm TLS is configured in the client (serverless always requires TLS).
* When migrating from serverless to node-based, confirm the security group allows port 6379 (or the custom port if configured).
**Cross-VPC prerequisite:**
* If the source Redis instance and the target ElastiCache cluster are in different VPCs, VPC peering or Transit Gateway connectivity must be established before starting online migration. The ElastiCache nodes must be able to reach the source Redis IP address over the network.
**Migration-specific security group requirement:**
* During online migration (`start-migration`), ElastiCache nodes connect TO the source Redis instance to replicate data. The security group attached to your source Redis instances must allow **inbound** traffic from ElastiCache nodes on the Redis port.
```bash
# Check inbound rules on the target security group
aws ec2 describe-security-groups \
--group-ids <target-sg-id> \
--query "SecurityGroups[0].IpPermissions[?FromPort==\`6379\`]" \
--region <region>
# Check outbound rules on the application security group
aws ec2 describe-security-groups \
--group-ids <app-sg-id> \
--query "SecurityGroups[0].IpPermissionsEgress[?FromPort==\`6379\` || FromPort==\`0\`]" \
--region <region>
```
**Remediation:** If port mismatches are found, update the security group rules before starting migration. Require explicit user confirmation before modifying security groups.
references/migration/upgrade-patching.md
# Engine Upgrades, Patching, and Node Type Changes
Operational runbook for keeping ElastiCache clusters current with engine versions, service updates, and node type changes.
## Engine Version Upgrades
### Valkey and Redis OSS Engine Version Upgrades
Engine version upgrades are performed in-place with minimal downtime for replication groups running Redis OSS 5.0.6 or higher with Multi-AZ enabled. The cluster is available for reads during the entire upgrade and for writes during most of the upgrade, except during the failover operation which lasts a few seconds. For versions earlier than Redis OSS 5.0.6, you may experience a failover time of 30 to 60 seconds during DNS propagation. Single-node clusters experience primary unavailability during upgrades. Note: pending scale-up operations must complete before an engine upgrade can be applied.
```bash
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--engine-version 8.0 \
--apply-immediately \
--region <region>
```
Process (single shard / cluster mode disabled):
1. ElastiCache upgrades replicas first, one at a time
2. Performs a failover to promote an upgraded replica to primary
3. Upgrades the old primary last
4. Total time depends on cluster size and data volume (typically minutes to an hour)
Process (multiple shards / cluster mode enabled):
1. All shards are processed in parallel; only one upgrade operation is performed on a shard at any time
2. In each shard, all replicas are processed before the primary is processed
3. If there are fewer replicas in a shard, the primary in that shard might be processed before replicas in other shards finish
4. Across all shards, primary nodes are processed in series; only one primary node is upgraded at a time
### Valkey Major Version Upgrades (e.g., 7.2 -> 8.2)
Same in-place mechanism as minor upgrades. Major versions may introduce new features (e.g., vector search in 8.2) and deprecate old behaviors.
Pre-upgrade checklist:
- Review the Valkey release notes for breaking changes
- Test the upgrade in a non-production environment first
- Confirm application compatibility with the new version
- When upgrading major engine versions (e.g., from 5.0.6 to 6.0), select a new parameter group that is compatible with the new engine version
- Take a manual snapshot before upgrading (safety net)
- Schedule during a low-traffic window even though the upgrade is online
- Prepare your application for connection drops: during engine upgrades, ElastiCache will terminate existing client connections. Implement error retries with exponential backoff in your Redis/Valkey clients to minimize impact.
```bash
# Take a pre-upgrade snapshot
aws elasticache create-snapshot \
--replication-group-id <cluster-id> \
--snapshot-name pre-upgrade-$(date +%Y%m%d) \
--region <region>
# Upgrade
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--engine-version 8.2 \
--apply-immediately \
--region <region>
```
### Redis OSS to Valkey Migration (In-Place Engine Switch)
Valkey is designed as a drop-in replacement for Redis OSS 7. No application code changes required. When upgrading from Redis OSS 5.0.6 and higher, you will experience no downtime. When upgrading from earlier Redis OSS versions than 5.0.6, you may experience a failover time of 30 to 60 seconds during DNS propagation.
```bash
# If using the default parameter group:
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--engine valkey \
--engine-version 8.0 \
--apply-immediately \
--region <region>
# If using a custom parameter group, you must also pass a Valkey parameter group:
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--engine valkey \
--engine-version 8.0 \
--cache-parameter-group-name <valkey-param-group> \
--apply-immediately \
--region <region>
```
Prerequisites:
- Single-node Redis OSS (cluster mode disabled) clusters must first be added to a replication group before performing the cross-engine upgrade. See [Creating a replication group using an existing cluster](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Replication.CreatingReplGroup.ExistingCluster.html).
- If a custom cache parameter group is applied to the existing Redis OSS replication group, you must pass a custom Valkey cache parameter group with the same Redis OSS static parameter values (use `--cache-parameter-group-name`).
Benefits:
- Immediate 20% cost savings on node-based pricing
- Access to Valkey-specific features in future versions
- No-downtime, zero-data-loss process (from Redis OSS 5.0.6+)
- Can subsequently upgrade to Valkey 8.0 or 8.2 (8.2 required for vector search); Valkey 9.0 is the recommended target
### Cluster Mode Disabled to Cluster Mode Enabled (In-Place)
For Valkey 7.2+ and Redis OSS 7.0+ replication groups with automatic failover and at least one replica, you can convert from cluster mode disabled (CMD) to cluster mode enabled (CME) in-place without creating a new cluster. This is a **one-way operation** — once cluster mode is set to `enabled`, it cannot be disabled. CME→CMD conversion is not supported.
This is a two-step process:
1. Set cluster mode to `compatible` (allows both CMD and CME clients to connect)
2. Set cluster mode to `enabled` (CME only)
```bash
# Step 1: Enable compatible mode
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--cluster-mode compatible \
--apply-immediately \
--region <region>
# Wait for modification to complete, then:
# Step 2: Enable cluster mode
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--cluster-mode enabled \
--apply-immediately \
--region <region>
```
Prerequisites:
- Valkey 7.2+ or Redis OSS 7.0+
- Automatic failover enabled with at least one replica
- All keys must be in database 0 (DB 0)
**Important:** You can revert from `compatible` back to `disabled`, but once set to `enabled`, the change is **irreversible**. Test in a non-production environment first. For details on auth and ACL considerations during this migration, see `auth-migration.md`.
### Version Compatibility Matrix
| Source Engine | Source Version | Target Engine | Target Version | Method | Downtime |
|--------------|---------------|---------------|----------------|--------|----------|
| Redis OSS | 6.x | Redis OSS | 7.x | In-place upgrade | Zero (Multi-AZ) |
| Redis OSS | 7.x | Valkey | 7.2 | In-place engine switch | Zero (Multi-AZ) |
| Valkey | 7.2 | Valkey | 8.0 | In-place upgrade | Zero (Multi-AZ) |
| Valkey | 7.2 | Valkey | 8.2 | In-place upgrade | Zero (Multi-AZ) |
| Valkey | 8.0 | Valkey | 8.2 | In-place upgrade | Zero (Multi-AZ) |
| Redis OSS | 5.x | Redis OSS | 7.x | In-place (direct jump supported) | Zero (Multi-AZ, 5.0.6+); brief failover for older versions |
| Valkey | 7.2 | Redis OSS | 7.1 | In-place rollback | Zero (Multi-AZ) |
| Any | Any newer | Any | Any older | **Not supported** (except Valkey 7.2 to Redis OSS 7.1) | N/A |
Key rule: **Engine version downgrades are generally not supported.** You cannot roll back from 8.2 to 7.2 using an in-place operation. However, ElastiCache supports rolling back from Valkey 7.2 to Redis OSS 7.1 as a special case, using the same console, API, or CLI steps as an upgrade. This rollback is performed with zero downtime. Rollback from Valkey 8.0 or higher to Redis OSS is not supported. Always test upgrades in a non-production environment first.
### Extended Support for Older Redis OSS Versions
Redis OSS versions 4 and 5 will enter Extended Support on February 1, 2026 (end of standard support: January 31, 2026). Redis OSS v6 will enter Extended Support on February 1, 2027 (end of standard support: January 31, 2027). Running these versions past end of standard support incurs additional charges with escalating yearly premiums. Extended Support is available for up to 3 years. After Extended Support ends, AWS will attempt to upgrade caches still running those versions to a supported version of Valkey; if the upgrade fails, the cache may be deleted. We strongly recommend upgrading to Valkey or Redis OSS v6+ before the end of standard support.
### Rollback Considerations
General engine version downgrades are not supported, with one exception: ElastiCache supports rolling back from Valkey 7.2 to Redis OSS 7.1 using the same in-place process as an upgrade, with zero downtime. Requirements for this rollback:
- Only Valkey 7.2 to Redis OSS 7.1 is supported (even if you upgraded from an earlier version)
- Any user group and user associated with the replication group or serverless cache must be configured with engine type `REDIS`
- You can also restore a Valkey 7.2 snapshot as a Redis OSS 7.1 cache
For all other downgrade scenarios:
- **Before upgrading**: create a manual snapshot
- **If the upgrade causes issues**: restore the snapshot to a new cluster running the old version, update application endpoints, and decommission the upgraded cluster
- **For critical workloads**: run the new version in parallel (blue-green) and cut over only after validation
## Service Updates (Patching)
ElastiCache periodically releases service updates for security patches, bug fixes, and minor improvements.
### Self-Service Updates
Check for available updates:
```bash
aws elasticache describe-service-updates \
--service-update-status available \
--region <region>
# Check which clusters need an update
aws elasticache describe-update-actions \
--service-update-name <update-name> \
--update-action-status not-applied \
--region <region>
```
Apply an update:
```bash
aws elasticache batch-apply-update-action \
--replication-group-ids <cluster-id-1> <cluster-id-2> \
--service-update-name <update-name> \
--region <region>
```
### Auto-Apply Updates
Some service updates have an auto-apply date. If you do not apply them before that date, AWS applies them during your maintenance window.
- **Security updates**: typically auto-applied. Apply proactively during a convenient window.
- **Non-security updates**: may have a longer self-service window before auto-apply.
### Maintenance Windows
Node-based clusters have a configurable weekly maintenance window for auto-applied updates. Serverless caches do not have maintenance windows (updates are applied transparently with zero downtime).
Configure the maintenance window:
```bash
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--preferred-maintenance-window "sun:03:00-sun:04:00" \
--region <region>
```
Best practices:
- Set the maintenance window to a low-traffic period
- Multi-AZ clusters experience zero downtime during patching (replicas are patched first, then a failover occurs)
- Single-node clusters experience brief downtime during patching (avoid single-node in production)
- Monitor the `describe-events` output after patching to confirm completion
### Serverless Updates
Serverless caches automatically apply the latest minor and patch software versions transparently, with no downtime and no action required from the operator. There is no maintenance window to configure. However, when a new major version is available, ElastiCache Serverless sends a notification and the operator must choose to upgrade by modifying the cache. Major version upgrades are also performed without downtime.
## Node Type Changes (Vertical Scaling)
### Online Vertical Scaling
For supported node type transitions, ElastiCache performs online scaling with zero downtime (Multi-AZ required).
```bash
# Check which node types you can scale to
aws elasticache list-allowed-node-type-modifications \
--replication-group-id <cluster-id> \
--region <region>
# Scale up (or down)
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--cache-node-type cache.r7g.xlarge \
--apply-immediately \
--region <region>
```
Process:
1. ElastiCache provisions new nodes with the target type
2. Syncs data to the new nodes
3. Performs DNS failover to the new nodes
4. Removes old nodes
Time: depends on data volume. Large datasets (100+ GB) can take 30 minutes or more.
### Node Type Change Constraints
- Not all transitions are supported. Use `list-allowed-node-type-modifications` to check.
- Scaling down may fail if the target node type has insufficient memory for the current dataset.
- Graviton (r7g, m7g, t4g) node types are recommended for best price-performance.
- Cross-family changes (e.g., r6g to r7g) are supported for most combinations.
### Serverless Scaling
Serverless caches scale automatically. There is no node type to change. Adjust `CacheUsageLimits` to control maximum capacity and cost:
```bash
aws elasticache modify-serverless-cache \
--serverless-cache-name <name> \
--cache-usage-limits '{
"DataStorage": {"Maximum": 10, "Unit": "GB"},
"ECPUPerSecond": {"Maximum": 15000}
}' \
--region <region>
```
## Operational Checklist
Before any upgrade or patching operation:
- [ ] Verify Multi-AZ and automatic failover are enabled (required for zero-downtime operations)
- [ ] Take a manual snapshot (safety net for rollback)
- [ ] Review release notes for the target version
- [ ] Test in a non-production environment
- [ ] Schedule during a low-traffic window
- [ ] Notify stakeholders of the planned change
- [ ] Monitor CloudWatch metrics during and after the operation
- [ ] Run `python3 scripts/security_audit.py` after completion to confirm posture
references/migration/valkey-migration-guide.md
# Redis OSS to Valkey Migration Guide
Complete guide for planning and executing a Redis OSS to Valkey migration on ElastiCache, including client compatibility, upgrade paths, and extended support cost implications.
## Decision Tree: Should You Migrate to Valkey?
```
Is your cluster running Redis OSS on ElastiCache?
├── No (self-managed Redis) --> See "Self-Hosted Redis to ElastiCache" in instructions.md
└── Yes
├── Running Redis OSS 4.x or 5.x?
│ └── Yes --> URGENT: These versions are past EOL. You are paying Extended Support fees.
│ └── Direct upgrade to Valkey is supported (e.g., --engine valkey --engine-version 7.2).
│ Optionally upgrade to Valkey 8.0, 8.1, or 8.2 for new features (Valkey 9.0 is the recommended target).
├── Running Redis OSS 6.x?
│ └── Yes --> Extended Support fees begin after EOL (check lifecycle calendar).
│ └── Direct upgrade to Valkey is supported (e.g., --engine valkey --engine-version 7.2).
│ Optionally upgrade to Valkey 8.0, 8.1, or 8.2 for new features (Valkey 9.0 is the recommended target).
└── Running Redis OSS 7.x?
└── Yes --> Direct switch to Valkey 7.2 available (in-place, zero downtime).
└── After switch: optionally upgrade to Valkey 8.0, 8.1, or 8.2 for new features (Valkey 9.0 is the recommended target).
```
After reaching Valkey 7.2, further upgrades are available:
```
Valkey 7.2
├── Valkey 8.0 (in-place upgrade, adds performance improvements)
├── Valkey 8.1 (in-place upgrade, hash table memory improvements: up to 20% less overhead for common key/value patterns; suitable bridge between 8.0 and 8.2)
├── Valkey 8.2 (in-place upgrade, adds vector search)
└── Valkey 9.0 (in-place upgrade, recommended target version)
```
## Multi-Step Upgrade Paths
In-place engine version downgrades are not supported, except for Valkey 7.2 to Redis OSS 7.1 rollback (see `references/migration/rollback-procedures.md`). Plan upgrades forward only.
| Current Version | Target | Steps | Estimated Time |
|----------------|--------|-------|----------------|
| Redis OSS 4.x | Valkey 7.2 | Direct: `--engine valkey --engine-version 7.2` | 1 operation |
| Redis OSS 5.x | Valkey 7.2 | Direct: `--engine valkey --engine-version 7.2` | 1 operation |
| Redis OSS 6.x | Valkey 7.2 | Direct: `--engine valkey --engine-version 7.2` | 1 operation |
| Redis OSS 7.x | Valkey 7.2 | Direct: `--engine valkey --engine-version 7.2` | 1 operation (minutes) |
| Redis OSS (any) | Valkey 8.0 | Direct: `--engine valkey --engine-version 8.0` | 1 operation |
| Redis OSS (any) | Valkey 8.2 | Direct: `--engine valkey --engine-version 8.2` | 1 operation |
| Valkey 7.2 | Valkey 8.1 | Valkey 7.2 -> 8.1 (direct) | 1 operation (minutes) |
| Valkey 7.2 | Valkey 8.2 | Valkey 7.2 -> 8.2 (direct) | 1 operation (minutes) |
| Valkey 8.0 | Valkey 8.2 | Valkey 8.0 -> 8.2 (direct) | 1 operation (minutes) |
| Valkey 8.1 | Valkey 8.2 | Valkey 8.1 -> 8.2 (direct) | 1 operation (minutes) |
> **Note:** Cross-engine upgrades support jumping directly from any Redis OSS version to any available Valkey version (7.2, 8.0, 8.2, etc.) in a single operation. Verify the target version is available in your region. AWS documentation may not explicitly list every supported source→target combination.
Valkey 8.1 features (verify availability in your region): 20% less memory via new hash table (efficiency improvement), native Bloom filters, COMMANDLOG, SET IFEQ, ZRANK 45% lower latency, PFMERGE/PFCOUNT 12x faster. Note: Valkey 8.0 introduced 20% more data per node (capacity improvement); these are two distinct features.
Valkey 7.2.6 adds: WITHSCORE option for ZRANK/ZREVRANK, CLIENT NO-TOUCH, CLUSTER MYSHARDID.
Each step is an in-place upgrade. Direct cross-engine upgrades from any Redis OSS version to Valkey are supported (e.g., `modify-replication-group --engine valkey --engine-version 7.2`). Multi-step upgrades (e.g., upgrading to Redis OSS 7.x first) are optional for users who prefer incremental validation. When upgrading from Redis OSS 5.0.6 and higher, you will experience no downtime. When upgrading from earlier Redis OSS versions (e.g., 4.x), you may experience a failover time of 30 to 60 seconds during DNS propagation. Take a snapshot before each step. Allow the cluster to stabilize between steps (monitor CloudWatch metrics for 15-30 minutes).
## Extended Support Cost Impact
Redis versions past End-of-Life incur Extended Support charges on top of regular node pricing. These charges increase over time (Year 1, Year 2, Year 3+ tiers). Migrating to Valkey eliminates Extended Support charges entirely and provides 20% node-based pricing savings (33% on serverless).
Run the price calculator to see current Extended Support rates for your cluster:
```bash
# Extended Support surcharge for a 6-node Redis cluster
python3 scripts/price_calculator.py --engine redis --extended-support --nodes 6 --region <region>
# Serverless cost estimate
python3 scripts/price_calculator.py --engine redis --mode serverless --data-gb <your-data-size> --ops-per-sec <N> --region <region>
```
The calculator fetches live pricing from the AWS Bulk Pricing API. Rates are cached locally for 7 days.
## Client Library Compatibility
Valkey is wire-protocol compatible with Redis OSS 7.2. Most client libraries work without code changes. The table below lists minimum versions and any required configuration changes. Client library version requirements are community/vendor-specified and may change. Verify against the latest library documentation.
### Java Clients
| Client | Min Version for Valkey | Code Changes Required | Notes |
|--------|----------------------|----------------------|-------|
| Jedis | 4.0+ (recommended: 5.0+) | None | Works out of the box. 5.0+ has dedicated Valkey support. |
| Lettuce | 6.2+ (recommended: 6.3+) | None | Connection URI stays `redis://` or `rediss://` (TLS). |
| Redisson | Check [Redisson compatibility matrix](https://github.com/redisson/redisson#compatibility) (recommended: 3.30+) | None for basic use | Cluster-mode-enabled with Redisson requires testing: verify `ClusterServersConfig` compatibility. Redisson's distributed objects (locks, maps, queues) work unchanged. |
| Valkey GLIDE (Java) | 1.0+ | New client | Official Valkey client. Consider for new projects building on the Valkey ecosystem. Existing projects can stay on Jedis/Lettuce with no compatibility issues. |
**Redisson-specific notes:**
* Redisson 3.27+ is validated for Valkey 7.2 compatibility.
* If using `RedissonClient` with cluster mode enabled, test the `ClusterServersConfig` scan interval and DNS resolution against Valkey endpoints before production cutover.
* Redisson's `RLock`, `RMap`, `RQueue`, and other distributed objects are Valkey-compatible since they use standard Redis commands internally.
* If using Redisson PRO features (e.g., performance optimizations, additional codecs), verify with Redisson's compatibility matrix.
### Python Clients
| Client | Min Version for Valkey | Code Changes Required | Notes |
|--------|----------------------|----------------------|-------|
| redis-py | 4.5+ (recommended: 5.0+) | None | `redis.Redis()` and `redis.cluster.RedisCluster()` work unchanged. |
| valkey-py | 6.0+ | Import change | `from valkey import Valkey` instead of `from redis import Redis`. API is identical. |
| Valkey GLIDE (Python) | 1.0+ | New client | Official Valkey client. Consider for new projects building on the Valkey ecosystem. Existing projects can stay on redis-py/valkey-py. |
### Node.js Clients
| Client | Min Version for Valkey | Code Changes Required | Notes |
|--------|----------------------|----------------------|-------|
| ioredis | 5.0+ | None | Works out of the box. Connection URL stays `rediss://` for TLS. |
| node-redis | 4.5+ | None | `createClient()` works unchanged with Valkey endpoints. |
| Valkey GLIDE (Node.js) | 1.0+ | New client | Official Valkey client. Consider for new projects building on the Valkey ecosystem. Existing projects can stay on ioredis/node-redis. |
### Go Clients
| Client | Min Version for Valkey | Code Changes Required | Notes |
|--------|----------------------|----------------------|-------|
| go-redis/redis | v9.0+ | None | `redis.NewClusterClient()` and `redis.NewClient()` work unchanged. |
| valkey-go | v2.0+ | Import change | `github.com/valkey-io/valkey-go` instead of `github.com/redis/go-redis`. |
| Valkey GLIDE (Go) | 2.0+ | New client | Official Valkey client. Go support added in GLIDE 2.0. Consider for new projects building on the Valkey ecosystem. |
### .NET Clients
| Client | Min Version for Valkey | Code Changes Required | Notes |
|--------|----------------------|----------------------|-------|
| StackExchange.Redis | 2.7+ | None | `ConnectionMultiplexer.Connect()` works unchanged. |
## Pre-Migration Checklist
Before switching from Redis OSS to Valkey:
* [ ] **Verify source version.** Direct cross-engine upgrade from any Redis OSS version to Valkey is supported. No intermediate Redis OSS version upgrade is required.
* [ ] **Check for Redis module usage.** If using RediSearch, RedisBloom, or RedisTimeSeries modules: RediSearch functionality is replaced by native vector search in Valkey 8.2. RedisBloom is replaced by native Bloom filter support (BF.ADD, BF.EXISTS, BF.RESERVE) in Valkey 8.1+. RedisTimeSeries has no Valkey equivalent; evaluate whether the workload can be redesigned.
* [ ] **Verify client library version.** Check the compatibility table above. Upgrade the client library if below the minimum version.
* [ ] **Test in non-production.** Create a non-production cluster, switch it to Valkey, and run application integration tests.
* [ ] **Run `scripts/migration_preflight.py`.** Validates version compatibility, module usage, memory, and cluster configuration.
* [ ] **Take a manual snapshot.** In-place rollback is only supported from Valkey 7.2 to Redis OSS 7.1. For upgrades beyond Valkey 7.2 (e.g., to 8.0+), the snapshot is your rollback path.
* [ ] **Plan the maintenance window.** The engine switch is zero-downtime with Multi-AZ, but schedule during low traffic as a precaution.
* [ ] **Notify stakeholders.** The engine identifier changes from `redis` to `valkey` in API calls, CloudWatch metrics namespace, and IaC definitions.
## Execution
After all checklist items pass:
> **Important:** If your cluster is a Redis OSS (cluster mode disabled) single-node cluster (no replicas), you must first add it to a replication group before performing the cross-engine upgrade. See [Creating a replication group using an existing cluster](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/Replication.CreatingReplGroup.ExistingCluster.html). The `modify-replication-group` command below will fail on standalone single-node clusters.
```bash
# Take pre-migration snapshot
aws elasticache create-snapshot \
--replication-group-id <cluster-id> \
--snapshot-name pre-valkey-migration-$(date +%Y%m%d) \
--region <region>
# Switch engine (zero downtime from Redis OSS 5.0.6+)
# Note: Single-node Redis OSS (cluster mode disabled) clusters must first be
# added to a replication group before cross-engine upgrading.
# If using a custom parameter group, add --cache-parameter-group-name <valkey-param-group>
# The Valkey custom parameter group must have the same Redis OSS static parameter values.
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--engine valkey \
--engine-version 8.0 \
--apply-immediately \
--region <region>
```
Monitor the operation:
```bash
aws elasticache describe-replication-groups \
--replication-group-id <cluster-id> \
--query "ReplicationGroups[0].Status" \
--region <region>
```
Wait for status to return to `available`. Then optionally upgrade to a newer Valkey version. Valkey 9.0 is the recommended target:
```bash
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--engine-version 9.0 \
--apply-immediately \
--region <region>
```
## Post-Migration Validation
* [ ] Confirm engine shows `valkey` in `describe-replication-groups` output
* [ ] Spot-check application reads and writes
* [ ] Verify CloudWatch metrics are flowing (namespace remains `AWS/ElastiCache`; dimensions are `CacheClusterId` and `CacheNodeId`, not an engine dimension)
* [ ] Run `scripts/security_audit.py --replication-group <cluster-id>` to confirm security posture
* [ ] Update IaC definitions to reflect `engine: valkey` and the new engine version
* [ ] Keep the pre-migration snapshot for at least 7 days as a rollback safety net
## Rollback
**Valkey 7.2 rollback (in-place):** ElastiCache supports rolling back from Valkey 7.2 to Redis OSS 7.1 in-place with no downtime and no endpoint changes. Use the same `modify-replication-group` command specifying `--engine redis --engine-version 7.1`. Any user group and user associated with the replication group must be configured with engine type `REDIS`.
```bash
aws elasticache modify-replication-group \
--replication-group-id <cluster-id> \
--engine redis \
--engine-version 7.1 \
--apply-immediately \
--region <region>
```
Alternatively, you can restore a snapshot created from your Valkey 7.2 cache as a Redis OSS 7.1 cache.
**Valkey 8.0+ rollback (snapshot-based):** In-place rollback from Valkey 8.0 or higher to Redis OSS is not supported. If the migration causes issues:
1. Restore the pre-migration snapshot to a new Redis OSS cluster
2. Update application endpoints to point to the restored cluster
3. Decommission the Valkey cluster after confirming the rollback is stable
See `references/migration/rollback-procedures.md` for detailed rollback steps.
references/monitoring/alarm-packs.md
# Alarm Packs
CloudWatch alarm configurations for ElastiCache serverless and node-based deployments.
## Alarm Philosophy
- **Start conservative, tune from baseline.** Set initial thresholds based on the guidance below, then adjust after observing 1-2 weeks of production traffic patterns.
- **Use workload-dependent thresholds, not fixed generic values.** A cache with 90% hit rate serving web sessions has different alarm thresholds than a cache doing vector search with 60% hit rate. The numbers below are starting points.
- **Alarm on sustained conditions, not transient spikes.** Use multiple evaluation periods (e.g., 3 out of 5 datapoints) to avoid false alarms from brief traffic bursts.
- **Pair alarms with runbooks.** Every alarm should link to an action. If an alarm fires and the team does not know what to do, it is noise.
- **Escalate gradually.** Start with warning thresholds that notify. Add critical thresholds that page.
## Serverless Alarm Pack
Dimension all alarms on `ServerlessCacheName`. Use `ComparisonOperator: GreaterThanThreshold` except for CacheHitRate which uses `LessThanThreshold`. Set `TreatMissingData: notBreaching` for all.
| Alarm Name | Metric | Statistic | Period | Threshold | EvalPeriods / Datapoints | Starting Threshold Guidance |
|---|---|---|---|---|---|---|
| ThrottledCmds (P0) | ThrottledCmds | Sum | 60s | 0 | 3 / 2 | > 0 sustained across 2 of 3 minutes. Any throttling warrants investigation. |
| ECPUSpike | ElastiCacheProcessingUnits | Sum | 60s | (baseline-dependent) | 5 / 3 | Set to 2x your observed 1-minute average during normal traffic. Uses 60s period to match ThrottledCmds cadence. Tune after establishing a baseline. |
| StorageApproachingLimit | BytesUsedForCache | Maximum | 300s | (80% of MaxDataStorageGB) | 3 / 2 | Set to 80% of configured `MaxDataStorageGB`. For a 5 GB limit, alarm at ~4 GB (4,000,000,000 bytes). |
| ReadLatency | SuccessfulReadRequestLatency | Average | 60s | 5000 (microseconds) | 5 / 3 | > 5ms sustained. Adjust based on application SLA. Serverless latency is typically 1-3ms at p50. |
| LowHitRate | CacheHitRate | Average | 300s | 80 (LessThan) | 6 / 4 | < 80% sustained over 20 minutes. Adjust based on workload; caches warming up or doing write-heavy work will naturally have lower hit rates. |
## Node-Based Alarm Pack
Dimension alarms on `CacheClusterId` and `CacheNodeId` (the dimensions under which node-level metrics are published). To get replication-group-level visibility, create per-node alarms or use CloudWatch metric math to aggregate across nodes. Use `ComparisonOperator: GreaterThanThreshold` except for CacheHitRate which uses `LessThanThreshold`. Set `TreatMissingData: notBreaching` for all.
| Alarm Name | Metric | Statistic | Period | Threshold | EvalPeriods / Datapoints | Starting Threshold Guidance |
|---|---|---|---|---|---|---|
| EngineCPU (P0) | EngineCPUUtilization | Maximum | 60s | 90 | 5 / 3 | > 90% sustained across 3 of 5 minutes. For nodes with 2 or fewer vCPUs (t4g.micro, t4g.small), use `CPUUtilization` with threshold `90 / vCPU_count` instead, since EngineCPU can spike to 100% on a single thread while overall host capacity remains available. |
| MemoryHigh | DatabaseMemoryUsagePercentage | Maximum | 60s | 80 | 5 / 3 | > 80% sustained. Evictions may begin depending on `maxmemory-policy`. Set a critical alarm at 90%. |
| ReplicationLag | ReplicationLag | Maximum | 60s | 1 (seconds) | 5 / 3 | > 1 second sustained. For apps reading from replicas needing consistency, set tighter (e.g., 0.1s). Valkey 7.2+/Redis OSS 5.0.6+ measure sub-second precision. The CloudWatch metric unit is seconds, not milliseconds. |
| Evictions | Evictions | Sum | 300s | 100 | 3 / 2 | > 100 evictions per 5-minute period sustained. Some eviction is normal with volatile-* policies, but sustained eviction means the dataset outgrew the cache. For clusters using `volatile-lru` or `volatile-ttl` policies, eviction of TTL'd keys is expected behavior. Raise or disable this alarm for those policies. Not user-tunable on serverless (serverless manages eviction automatically). |
| NewConnections | NewConnections | Sum | 60s | 1000 | 3 / 2 | > 1000 new connections per minute sustained. Indicates connection storm or missing pooling. Tune based on your steady-state new connection rate. |
| ReadLatency | SuccessfulReadRequestLatency | Average | 60s | 5000 (microseconds) | 5 / 3 | > 5ms sustained. Adjust based on application SLA. Node-based latency is typically 0.5-2ms at p50. |
| WriteLatency | SuccessfulWriteRequestLatency | Average | 60s | 5000 (microseconds) | 5 / 3 | > 5ms sustained. Critical for write-heavy workloads (session stores, counters, rate limiters). Adjust based on application SLA. Node-based write latency is typically 0.5-2ms at p50. |
| LowHitRate | CacheHitRate | Average | 300s | 80 (LessThan) | 6 / 4 | < 80% sustained over 20 minutes. Investigate key design, TTL strategy, working set size. |
**Per-node alarms for hot-shard detection:** For cluster-mode clusters with 3+ shards, create a separate EngineCPU alarm per primary node (dimensioned on `CacheClusterId` and `CacheNodeId`) to detect hot shards. A single alarm aggregating across nodes can hide shard imbalance when multiple shards are hot but none is the single highest.
**Note on CPUUtilization vs. EngineCPUUtilization:** AWS recommends using `EngineCPUUtilization` rather than `CPUUtilization` for capacity planning. `CPUUtilization` may vary across engine versions and node types due to changes in how enhanced I/O features utilize available cores, making it an unreliable metric for capacity planning.
**Network saturation alarms (recommended for large instances):** Add alarms for `TrafficManagementActive`, `NetworkBandwidthInAllowanceExceeded`, `NetworkBandwidthOutAllowanceExceeded`, `NetworkConntrackAllowanceExceeded`, and `NetworkPacketsPerSecondAllowanceExceeded`. These fire before general performance degradation becomes visible and are the first signal of host-level saturation. `DatabaseCapacityUsagePercentage` is available on all node-based clusters (on data-tiering instances (r6gd), the formula includes SSD storage; on all other instances, it is calculated as `used_memory/maxmemory`); add it alongside `DatabaseMemoryUsagePercentage`.
## Alarm-to-Runbook Mapping
| Alarm | Runbook (troubleshooting.md) |
|-------|------------------------------|
| EngineCPU | High CPU |
| MemoryHigh | Memory Pressure |
| ReplicationLag | Replication Lag |
| Evictions | Memory Pressure (eviction policy) |
| LowHitRate | Low Hit Rate |
| ReadLatency | High Latency |
| WriteLatency | High Latency |
| ThrottledCmds | Throttling (Serverless) |
| ECPUSpike | Throttling (if ThrottledCmds also firing) or cost-reporting.md (if no throttling, indicates approaching limits) |
| StorageApproachingLimit | Memory Pressure |
## Deploying Alarm Packs
### Via the Generate Script
```bash
# Serverless (generates dashboard + alarms)
python3 scripts/generate_dashboards.py --serverless <cache-name> \
--sns-topic arn:aws:sns:us-east-1:123456789012:cache-alerts \
--output observability.json
# Node-based
python3 scripts/generate_dashboards.py --replication-group <cluster-id> \
--sns-topic arn:aws:sns:us-east-1:123456789012:cache-alerts \
--output observability.json
```
### Deploy
```bash
aws cloudformation deploy \
--template-file observability.json \
--stack-name <cache-name>-observability \
--region us-east-1
```
## Notification Routing
| Destination | Integration |
|---|---|
| Slack | SNS → AWS Chatbot → Slack channel |
| PagerDuty | SNS → PagerDuty Events API v2 (HTTPS subscription) |
| Email | SNS → Email subscription (confirm required) |
**Restrict who can subscribe.** Attach an SNS topic access policy that limits `sns:Subscribe` to authorized principals or accounts only. Operational alerts carry infrastructure details (cache names, ARNs, thresholds), so an open subscribe policy lets unauthorized parties receive them.
## Tuning Thresholds from Baseline
After deploying the initial alarm pack:
1. **Observe for 1-2 weeks** under normal production traffic.
2. **Review CloudWatch metric graphs** to identify the natural baseline (average, p95, p99) for each metric.
3. **Adjust thresholds** to sit above the p99 of normal operation. The goal is zero false alarms during normal traffic while catching genuine anomalies.
4. **Add anomaly detection alarms** for metrics with variable baselines (e.g., ECPU following business hours). Use CloudWatch anomaly detection alarms with `ANOMALY_DETECTION_BAND`. CloudWatch learns the pattern and alarms on deviation from the expected band.
references/monitoring/big-key-hunter.md
# Big-Key Hunter
**When to use:** Memory growing faster than key count, latency spikes on O(N) commands (HGETALL, LRANGE, SMEMBERS), network bandwidth saturation on one node, or client timeouts on reads of known keys.
**When not needed:** Memory pressure with many small keys (troubleshooting.md Memory Pressure). High request rate on normal-sized keys (hot-key-detection.md).
## Preconditions
| Deployment | Available Tiers |
|---|---|
| Serverless | Tier A + Tier B only. `MEMORY USAGE` blocked; estimate from cardinality x element size. |
| Node-based | All tiers. Tier C gives exact bytes via `MEMORY USAGE`. |
### Command availability
| Command | Node-based | Serverless | Purpose |
|---|---|---|---|
| STRLEN, HLEN, LLEN, ZCARD, SCARD, PFCOUNT, XLEN | Yes | Yes | Per-type cardinality |
| `valkey-cli --bigkeys` | Yes | Yes | Client-side SCAN + per-type length sampling |
| `valkey-cli --memkeys` | Yes | Fails | Depends on MEMORY USAGE |
| MEMORY USAGE \<key\> | Yes | Blocked | Exact bytes including overhead |
| DEBUG OBJECT | Blocked | Blocked | Restricted everywhere. Never recommend. |
| XINFO STREAM | Yes | Yes | Stream length + groups + entries |
### "Big" thresholds
| Type | Threshold |
|---|---|
| String | >= 100 KB |
| Hash | >= 1,000 fields |
| List | >= 10,000 items |
| Set | >= 1,000 members |
| Sorted set | >= 1,000 members |
| Stream | >= 100,000 entries |
Tune up for workloads that legitimately use large collections (event sourcing, leaderboards). Tune down if latency is sensitive.
---
## Tier A: Triage with CloudWatch
### Step 1: Memory-to-key-count ratio
Big-key: memory grows faster than key count. Volume: both grow together.
```bash
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache --metric-name BytesUsedForCache \
--dimensions Name=<dim>,Value=<id> \
--start-time <24h-ago> --end-time <now> \
--period 3600 --statistics Maximum --region <region>
```
Query both `BytesUsedForCache` and `CurrItems`. Compute bytes-per-key trend.
**Stop condition:** If bytes-per-key is flat and below 10KB, this is not a big-key problem. Route to troubleshooting.md Memory Pressure.
### Step 2: Network bandwidth correlation (node-based)
Big keys on read-heavy workloads show as `NetworkBytesOut` spikes on one node while peers are flat, correlated with elevated `EngineCPUUtilization`. Use `CacheClusterId` dimension (not `ReplicationGroupId` which hides imbalance).
### Step 3: Slow-log fingerprint (node-based only)
```bash
aws logs filter-log-events \
--log-group-name <your-slow-log-group-name> \
--filter-pattern "HGETALL SMEMBERS LRANGE SUNIONSTORE SORT" \
--start-time <1h-ago-ms> --region <region>
```
The log group name is user-specified when configuring log delivery (retrieve via `describe-replication-groups` → `LogDeliveryConfigurations`).
First argument of each entry is the key. Keys appearing repeatedly with O(N) commands are big-key candidates. Serverless: skip (slow-log not delivered).
---
## Tier B: Narrow with client-side sampling
`valkey-cli --bigkeys` is a client-side sampler (SCAN loop + TYPE + per-type length), not a server command. No native server-side big-key detection exists in Valkey (valkey-rfc #34 proposes TOPKEYS).
### Cost and impact
| Signal | Typical value |
|---|---|
| Runtime on 1M keys | 1-3 minutes |
| Runtime on 10M keys | 15-30 minutes |
| Throughput impact | 0.5% to 2% |
| EngineCPU impact | Minimal (reads only) |
Run against a replica on node-based. Run off-peak on serverless (every sampled key consumes ECPUs). Stop if EngineCPU rises >10 percentage points.
### Step 1: Run the sampler
```bash
valkey-cli -h <endpoint> -p 6379 --tls --bigkeys
```
### Step 2: Interpret
`--bigkeys` reports logical size (bytes for strings, element count for aggregates), not memory footprint. A hash with 50000 small fields may use less memory than a 2MB string.
### Step 3: Cross-reference with Tier A
If the big-key matches a slow-log fingerprint or is on the shard with elevated NetworkBytesOut, it is the root cause.
**Stop condition:** If top key per type is below the thresholds table, no remediation needed.
### Alternative: offline RDB analysis (node-based with backups)
For very large keyspaces where online sampling is too slow:
```bash
aws elasticache copy-snapshot \
--source-snapshot-name <snapshot> \
--target-snapshot-name <target> \
--target-bucket <s3-bucket> --region <region>
```
Download and analyze with RDB tools offline. Not available on serverless.
---
## Tier C: Verify with MEMORY USAGE (node-based only)
### Step 1: Measure candidates
```bash
valkey-cli -h <endpoint> -p 6379 --tls MEMORY USAGE <key>
```
Default sampling (5 nested elements) is fast and good enough for ranking. Use `SAMPLES 0` only for the final top-1 or top-2 candidates (O(N) in value size).
### Step 2: Rank and confirm
Note the hash slot via `CLUSTER KEYSLOT <key>` if cluster mode. A big-key on a hot slot (cross-reference hot-key-detection.md Tier B) is the worst case.
---
## Stream big-key detection
Streams need a different approach. Length alone undersells the problem; a stuck consumer group with growing PEL is pathological.
```bash
valkey-cli -h <endpoint> -p 6379 --tls XLEN <stream-key>
valkey-cli -h <endpoint> -p 6379 --tls XINFO STREAM <stream-key>
valkey-cli -h <endpoint> -p 6379 --tls XINFO GROUPS <stream-key>
```
If first-entry timestamp is hours old, consumers are not keeping up. Enforce XTRIM MAXLEN.
---
## Remediation options
**Critical constraint (cluster mode):** ElastiCache does not migrate slots containing items with serialized size larger than 256 MB during slot migration. You must decompose keys exceeding 256 MB before any resharding or scale-out operation.
In order of preference:
1. **Value decomposition** (default): break large value into smaller keys. Large hash becomes `key:part1`, `key:part2`. Long list gets LTRIM or converts to stream.
2. **Stream trimming** (streams only): `XTRIM <key> MAXLEN ~ <n>` with approximate flag. Enforce via scheduled task.
3. **Compression**: gzip/zstd at application layer before writing. Reduces network and memory at cost of CPU.
4. **Avoid O(N) reads**: replace HGETALL with HMGET for specific fields, LRANGE 0 -1 with paginated LRANGE, SMEMBERS with SSCAN. Does not reduce memory but eliminates latency spikes.
5. **TTL tuning**: shorter TTL prevents unbounded growth for aggregated data.
6. **Move to different store**: genuinely large blobs (video metadata, ML features) belong in S3 with a pointer in the cache.
---
## Cross-links
- Big-key also hot (high access frequency): `hot-key-detection.md`
- Memory pressure cluster-wide, not key-specific: `troubleshooting.md` Memory Pressure
- Big-keys causing replication lag: `troubleshooting.md` Replication Lag
- Per-shard memory imbalance from big-key bundling: `slot-memory-imbalance-detection.md`
- Alarm patterns: `alarm-packs.md`
references/monitoring/client-tuning-and-diagnostics.md
# Client Tuning & Diagnostics
Client-side timeout configuration, connection pooling, TLS setup per language, and CloudWatch metrics troubleshooting. This file covers the client configuration side of performance and connectivity issues. For server-side troubleshooting playbooks, see `troubleshooting.md`. For initial connectivity failures on new caches, see `../setup/connectivity-diagnostics.md`.
---
## Client Timeout Configuration
Poorly tuned timeouts cause two failure modes: too low triggers false failures during brief network blips or failovers; too high makes the application hang when the cache is unreachable.
### Recommended Starting Values
| Timeout | Starting Value | Rationale |
|---------|---------------|-----------|
| Connect timeout | 2-5 seconds | Time to establish TCP + TLS handshake. Increase if cross-AZ or through tunnel. |
| Command (socket) timeout | 1-2 seconds | Max wait for a single command response. Most commands return in under 5ms. |
| Retry attempts | 3 | With exponential backoff. Covers brief failover windows (~15-30s for node-based Multi-AZ). |
| Retry backoff base | 100-200ms | First retry after 100-200ms, then doubles. Avoids thundering herd. |
### Per-Library Configuration
#### Python (redis-py / valkey-py)
```python
import redis
pool = redis.ConnectionPool(
host="endpoint.cache.amazonaws.com",
port=6379,
ssl=True,
socket_connect_timeout=5, # seconds, TCP + TLS handshake
socket_timeout=2, # seconds, per-command timeout
retry_on_timeout=True,
health_check_interval=15, # seconds, sends PING on idle connections
)
r = redis.Redis(connection_pool=pool)
```
#### Node.js (ioredis)
```javascript
const Redis = require("ioredis");
const client = new Redis({
host: "endpoint.cache.amazonaws.com",
port: 6379,
tls: {},
connectTimeout: 5000, // ms, TCP + TLS handshake
commandTimeout: 2000, // ms, per-command timeout
maxRetriesPerRequest: 3,
retryStrategy(times) {
return Math.min(times * 200, 2000); // ms, exponential backoff capped at 2s
},
});
```
#### Java (Lettuce)
```java
// Socket connect timeout should be lower than command timeout for Lettuce.
// Set JVM DNS cache TTL to support failover DNS changes.
java.security.Security.setProperty("networkaddress.cache.ttl", "10");
RedisURI uri = RedisURI.builder()
.withHost("endpoint.cache.amazonaws.com")
.withPort(6379)
.withSsl(true)
.withTimeout(Duration.ofMillis(2000)) // per-command timeout
.build();
ClientResources clientResources = DefaultClientResources.builder()
.addressResolverGroup(new DirContextDnsResolver())
.reconnectDelay(
Delay.fullJitter(
Duration.ofMillis(100),
Duration.ofSeconds(10),
100, TimeUnit.MILLISECONDS))
.build();
ClientOptions options = ClientOptions.builder()
.socketOptions(SocketOptions.builder()
.connectTimeout(Duration.ofMillis(5000)) // TCP + TLS handshake; match table guidance of 2-5s
.keepAlive(true)
.build())
.timeoutOptions(TimeoutOptions.builder()
.fixedTimeout(Duration.ofMillis(2000))
.build())
.build();
RedisClient client = RedisClient.create(clientResources, uri);
client.setOptions(options);
```
#### Go (go-redis)
```go
client := redis.NewClient(&redis.Options{
Addr: "endpoint.cache.amazonaws.com:6379",
TLSConfig: &tls.Config{},
DialTimeout: 5 * time.Second, // TCP + TLS handshake
ReadTimeout: 2 * time.Second, // per-command read
WriteTimeout: 2 * time.Second, // per-command write
MaxRetries: 3,
MinRetryBackoff: 100 * time.Millisecond,
MaxRetryBackoff: 2 * time.Second,
})
```
### Timeout Tuning After Deployment
1. Check `SuccessfulReadRequestLatency` p99 in CloudWatch (units: microseconds). A p99 of 3000 = 3ms. This metric is calculated at the cache node level; for node-based caches, query it using the `CacheClusterId` dimension; for serverless caches, use `ServerlessCacheName`.
2. Set command timeout to at least 10x the observed p99 to accommodate tail latency and brief slowdowns.
3. If failovers are common (check `aws elasticache describe-events --source-type replication-group --duration 1440`), set connect timeout to at least 5s to survive DNS propagation.
4. For Lambda with VPC attachment, use a 10s connect timeout to handle cold start ENI attachment.
---
## Connection Pooling
Opening a new connection for each request adds 5-20ms of TLS handshake overhead. Connection pooling amortizes this cost across requests.
### Pool Sizing
A good starting point: **pool size = expected concurrent requests per application instance**. For most web applications, 10-50 connections per instance is sufficient.
Signs the pool is too small: commands queue waiting for a free connection, latency increases under load while `EngineCPUUtilization` stays low.
Signs the pool is too large: `CurrConnections` is high, `NewConnections` is high, and most connections sit idle.
### Per-Library Configuration
#### Python (redis-py / valkey-py)
```python
pool = redis.ConnectionPool(
host="endpoint.cache.amazonaws.com",
port=6379,
ssl=True,
max_connections=50,
)
r = redis.Redis(connection_pool=pool)
# In async frameworks (FastAPI, aiohttp), use redis.asyncio.ConnectionPool instead.
```
#### Node.js (ioredis)
```javascript
// ioredis manages a single persistent connection by default.
// For cluster mode, it opens one connection per node.
// For concurrency, ioredis pipelines commands over the single connection.
// If you need multiple connections (rare), use a manual pool or ioredis Cluster.
const cluster = new Redis.Cluster(
[{ host: "endpoint.cache.amazonaws.com", port: 6379 }],
{
slotsRefreshTimeout: 2000,
dnsLookup: (address, callback) => callback(null, address),
redisOptions: { tls: {} },
scaleReads: "slave", // read from replicas
}
);
```
#### Java (Lettuce)
```java
// Lettuce uses a single connection with pipelining by default.
// For thread-safe concurrent access, use StatefulRedisConnection (thread-safe)
// or GenericObjectPool for connection pooling:
GenericObjectPoolConfig<StatefulRedisConnection<String, String>> poolConfig =
new GenericObjectPoolConfig<>();
poolConfig.setMaxTotal(50);
poolConfig.setMaxIdle(20);
poolConfig.setMinIdle(5);
poolConfig.setTestOnBorrow(true);
```
#### Java (Lettuce) -- Cluster Mode Enabled
For cluster mode enabled, configure `ClusterTopologyRefreshOptions` and node filtering to handle topology changes during failovers:
```java
ClusterTopologyRefreshOptions topologyOptions = ClusterTopologyRefreshOptions.builder()
.enableAllAdaptiveRefreshTriggers()
.enablePeriodicRefresh()
.dynamicRefreshSources(true)
.build();
ClusterClientOptions clusterOptions = ClusterClientOptions.builder()
.topologyRefreshOptions(topologyOptions)
.nodeFilter(it ->
!(it.is(RedisClusterNode.NodeFlag.FAIL)
|| it.is(RedisClusterNode.NodeFlag.EVENTUAL_FAIL)
|| it.is(RedisClusterNode.NodeFlag.NOADDR)))
.validateClusterNodeMembership(false)
.build();
RedisClusterClient clusterClient = RedisClusterClient.create(clientResources, redisUri);
clusterClient.setOptions(clusterOptions);
```
#### Go (go-redis)
```go
client := redis.NewClient(&redis.Options{
Addr: "endpoint.cache.amazonaws.com:6379",
TLSConfig: &tls.Config{},
PoolSize: 50, // max connections in pool
MinIdleConns: 5, // keep warm connections ready
PoolTimeout: 3 * time.Second, // wait for free connection
ConnMaxIdleTime: 5 * time.Minute,
})
```
### Monitoring Pool Health
Check these CloudWatch metrics at the cache level:
- `CurrConnections` (Maximum): total open connections across all clients. Compare against expected (pool size x number of app instances).
- `NewConnections` (Sum, per minute): should be low after initial ramp-up. Sustained high values indicate connections are not being reused.
---
## TLS Connection Quick Reference
All ElastiCache serverless caches require TLS. Node-based caches require TLS if created with `--transit-encryption-enabled`, or if TLS is enabled later on an existing cluster using the two-step migration process (`transit-encryption-mode`: `preferred` then `required`). For tunnel-mode TLS settings (connecting through SSM to localhost), see `../setup/connection-guide.md`.
| Language | Library | TLS Setting |
|----------|---------|-------------|
| Python | redis-py / valkey-py | `ssl=True` in connection params |
| Node.js | ioredis | `tls: {}` in options, or use `rediss://` URL scheme |
| Java | Lettuce | `.withSsl(true)` on RedisURI, or `SslOptions.builder().build()` on ClientResources |
| Go | go-redis | `TLSConfig: &tls.Config{}` in Options |
| CLI | valkey-cli | `--tls` flag |
For common TLS error messages and their causes, see `../setup/connectivity-diagnostics.md`.
---
## Missing CloudWatch Metrics
When expected metrics do not appear in CloudWatch, work through this checklist in order.
### 1. No Traffic Yet
ElastiCache emits most metrics only after the cache receives client traffic. A newly created cache with zero commands will have no datapoints for latency, hit rate, or command-family metrics. Send a PING or a test SET/GET, wait 5-10 minutes, then check again.
Metrics that always emit regardless of traffic (node-based): `CurrConnections`, `EngineCPUUtilization`, `DatabaseMemoryUsagePercentage`, `FreeableMemory`.
Metrics that require traffic: `CacheHitRate`, `CacheMisses`, `CacheHits`, `SuccessfulReadRequestLatency`, `SuccessfulWriteRequestLatency`, command-family metrics.
### 2. Wrong Namespace or Dimensions
ElastiCache metrics live under the `AWS/ElastiCache` namespace (not `ElastiCache` or `aws/elasticache`; the capitalization and prefix matter).
Dimension reference:
| Deployment | Dimension Name | Dimension Value |
|-----------|---------------|----------------|
| Serverless | `ServerlessCacheName` | The cache name (e.g., `my-cache`) |
| Node-based (cluster-wide) | `ReplicationGroupId` | The replication group ID |
| Node-based (per-node) | `CacheClusterId` | The individual node ID (e.g., `my-cluster-001`) |
Common mistakes:
- Using `CacheClusterId` when the metric only publishes at the `ReplicationGroupId` level, or vice versa. For node-based caches, most metrics including `CacheHits` and `CacheMisses` are per-node (`CacheClusterId`). For serverless caches, `CacheHitRate` uses the `ServerlessCacheName` dimension.
- Using `ReplicationGroupId` for per-node metrics like `EngineCPUUtilization` when you need per-shard visibility.
- Using the cache name as the dimension value for a node-based cache instead of the replication group ID.
### 3. Verify via CLI
```bash
# List available metrics for a serverless cache
aws cloudwatch list-metrics \
--namespace AWS/ElastiCache \
--dimensions Name=ServerlessCacheName,Value=<cache-name> \
--region <region>
# List available metrics for a node-based cache
aws cloudwatch list-metrics \
--namespace AWS/ElastiCache \
--dimensions Name=ReplicationGroupId,Value=<replication-group-id> \
--region <region>
```
If the list is empty: confirm the cache exists and is in `available` status, confirm the region matches, and confirm traffic has been sent.
### 4. Console vs. CLI Mismatch
When metrics appear in the CLI but not in the CloudWatch console:
- Check the time range in the console. The default view may be too narrow to include the metric's retention period.
- Check the statistic selected. Some metrics only make sense with specific statistics (e.g., `ElastiCacheProcessingUnits` should use Sum, not Average).
- Check the region selector in the console matches the cache's region.
### 5. Metric Retention
CloudWatch retains ElastiCache metrics at these resolutions:
- 1-minute datapoints: 15 days
- 5-minute datapoints: 63 days
- 1-hour datapoints: 455 days
If investigating an issue older than 15 days, you must use 5-minute or 1-hour period in your query, or the datapoints will have already expired.
references/monitoring/cloudwatch-dashboards.md
# CloudWatch Dashboard Templates
Pre-built dashboard specifications for ElastiCache serverless and node-based deployments. Deploy via CloudFormation or use as a reference for manual dashboard creation.
## Serverless Dashboard
Metrics focused on consumption-based billing, throttling, latency, and connection health.
### Key Widgets
| Widget | Metric(s) | Statistic | Period | Purpose |
|--------|-----------|-----------|--------|---------|
| ECPU Consumption | ElastiCacheProcessingUnits | Sum | 1 min | Cost driver and usage trend |
| Throttled Commands | ThrottledCmds | Sum | 1 min | Indicates hitting usage limits |
| Cache Hit Rate | CacheHitRate | Average | 5 min | Data access efficiency |
| Read Latency | SuccessfulReadRequestLatency | p99, Average | 1 min | Client-perceived read performance |
| Write Latency | SuccessfulWriteRequestLatency | p99, Average | 1 min | Client-perceived write performance |
| Current Connections | CurrConnections | Maximum | 1 min | Connection pool health |
| New Connections | NewConnections | Sum | 1 min | Connection churn (high values suggest missing pooling) |
| Data Storage | BytesUsedForCache | Maximum | 5 min | Storage consumption vs. configured limit |
| Total Commands | TotalCmdsCount | Sum | 1 min | Overall command throughput |
| Cache Hits | CacheHits | Sum | 1 min | Successful key lookups |
| Cache Misses | CacheMisses | Sum | 1 min | Unsuccessful key lookups |
| Current Items | CurrItems | Maximum | 1 min | Number of items stored in cache |
| Volatile Items | CurrVolatileItems | Maximum | 1 min | Number of items with TTL set |
| Evictions | Evictions | Sum | 5 min | Keys evicted by the cache |
| Network In | NetworkBytesIn | Sum | 1 min | Bytes transferred into cache |
| Network Out | NetworkBytesOut | Sum | 1 min | Bytes transferred out of cache |
| Auth Failures | AuthenticationFailures | Sum | 1 min | Failed AUTH attempts (set alarm to detect unauthorized access) |
| Key Auth Failures | KeyAuthorizationFailures | Sum | 1 min | Failed key access attempts (set alarm to detect unauthorized access) |
| Command Auth Failures | CommandAuthorizationFailures | Sum | 1 min | Failed command authorization attempts (set alarm to detect unauthorized access) |
| IAM Auth Expirations | IamAuthenticationExpirations | Sum | 1 min | Expired IAM-authenticated connections |
| IAM Auth Throttling | IamAuthenticationThrottling | Sum | 1 min | Throttled IAM auth requests |
| String Commands | StringBasedCmds | Sum | 1 min | GET/SET workload volume |
| Hash Commands | HashBasedCmds | Sum | 1 min | Hash-based workload volume |
| Sorted Set Commands | SortedSetBasedCmds | Sum | 1 min | Leaderboard/ranking activity |
| List Commands | ListBasedCmds | Sum | 1 min | List-based workload volume |
| Set Commands | SetBasedCmds | Sum | 1 min | Set-based workload volume |
| Pub/Sub Commands | PubSubBasedCmds | Sum | 1 min | Real-time messaging activity |
| Key Commands | KeyBasedCmds | Sum | 1 min | Key management operations |
Serverless also supports `*ECPUs` companions for each command-family metric (e.g., `StringBasedCmdsECPUs`) to track ECPU consumption by command type.
Use `scripts/generate_dashboards.py` to produce the full CloudFormation template from these widget specifications.
### ECPU Cost Attribution by Command Type
For serverless caches, ElastiCache emits per-command-type ECPU metrics that are critical for understanding cost drivers in a consumption-based billing model:
| Widget | Metric(s) | Statistic | Period | Purpose |
|--------|-----------|-----------|--------|---------|
| Read ECPUs | GetTypeCmdsECPUs | Sum | 1 min | ECPUs consumed by read commands |
| Write ECPUs | SetTypeCmdsECPUs | Sum | 1 min | ECPUs consumed by write commands |
| Hash ECPUs | HashBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by hash commands |
| String ECPUs | StringBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by string commands |
| Sorted Set ECPUs | SortedSetBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by sorted set commands |
| Stream ECPUs | StreamBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by stream commands |
| List ECPUs | ListBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by list commands |
| Set ECPUs | SetBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by set commands |
| JSON ECPUs | JsonBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by JSON commands |
| PubSub ECPUs | PubSubBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by pub/sub commands |
| Key ECPUs | KeyBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by key commands |
| Eval ECPUs | EvalBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by eval commands |
| GeoSpatial ECPUs | GeoSpatialBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by geospatial commands |
| HyperLogLog ECPUs | HyperLogLogBasedCmdsECPUs | Sum | 1 min | ECPUs consumed by HyperLogLog commands |
| NonKey ECPUs | NonKeyTypeCmdsECPUs | Sum | 1 min | ECPUs consumed by non-key commands |
### Generate via Script
```bash
python3 scripts/generate_dashboards.py --serverless <cache-name> --region us-east-1 --output serverless-dashboard.json
```
## Node-Based Dashboard
Metrics focused on engine performance, memory pressure, replication health, and command workload distribution.
### Key Widgets
| Widget | Metric(s) | Statistic | Period | Purpose |
|--------|-----------|-----------|--------|---------|
| Engine CPU | EngineCPUUtilization | Maximum | 1 min | Single-threaded engine bottleneck (most critical for performance) |
| Host CPU | CPUUtilization | Average | 1 min | Overall host CPU including background tasks |
| Memory Usage | DatabaseMemoryUsagePercentage | Maximum | 1 min | Memory pressure; triggers evictions when high |
| Capacity Usage | DatabaseCapacityUsagePercentage | Maximum | 1 min | DatabaseCapacityUsagePercentage is available on all node-based clusters. On data-tiering instances (r6gd), the formula includes SSD storage; on all other instances, it is calculated as `used_memory/maxmemory`. |
| Cache Hit Rate | CacheHitRate | Average | 5 min | Data access efficiency. `CacheHitRate` is available for both serverless and node-based Valkey/Redis OSS clusters. |
| Replication Lag | ReplicationLag | Maximum | 1 min | Replica staleness (seconds); critical for read consistency |
| Current Connections | CurrConnections | Maximum | 1 min | Connection pool health |
| Evictions | Evictions | Sum | 5 min | Keys evicted due to memory pressure |
| Network I/O | NetworkBytesIn, NetworkBytesOut | Sum | 1 min | Bandwidth utilization |
| String Commands | StringBasedCmds | Sum | 1 min | GET/SET workload volume |
| Hash Commands | HashBasedCmds | Sum | 1 min | Hash-based workload volume |
| Sorted Set Commands | SortedSetBasedCmds | Sum | 1 min | Leaderboard/ranking activity |
| Stream Commands | StreamBasedCmds | Sum | 1 min | Event stream workload |
| Search Commands | SearchBasedCmds | Sum | 1 min | Search command activity (includes all Search read and write commands; populates only when Valkey Search is in use) |
| JSON Commands | JsonBasedCmds | Sum | 1 min | JSON document usage |
| Pub/Sub Commands | PubSubBasedCmds | Sum | 1 min | Real-time messaging activity |
Use `scripts/generate_dashboards.py` to produce the full CloudFormation template from these widget specifications.
### Generate via Script
```bash
python3 scripts/generate_dashboards.py --replication-group <cluster-id> --region us-east-1 --output node-dashboard.json
```
## Deploying Dashboards
After generating the CloudFormation template:
```bash
aws cloudformation deploy \
--template-file serverless-dashboard.json \
--stack-name my-cache-dashboard \
--parameter-overrides ServerlessCacheName=my-cache Region=us-east-1 \
--region us-east-1
```
Or for node-based:
```bash
aws cloudformation deploy \
--template-file node-dashboard.json \
--stack-name my-cluster-dashboard \
--parameter-overrides ReplicationGroupId=my-cluster Region=us-east-1 \
--region us-east-1
```
references/monitoring/cost-reporting.md
# Cost and Usage Reporting
Guide for understanding, monitoring, and optimizing ElastiCache costs across serverless and node-based deployments.
## Cost Spike Investigation
When a user reports unexpected cost increase, follow this sequence:
1. **Identify which component spiked.** Check Cost Explorer grouped by usage type:
```bash
aws ce get-cost-and-usage \
--time-period Start=<spike-start>,End=<today> \
--granularity DAILY \
--filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon ElastiCache"]}}' \
--metrics "UnblendedCost" \
--group-by Type=DIMENSION,Key=USAGE_TYPE
```
2. **Map the usage type to the cause:**
- `ElastiCache:ECPU` spiked: check `ElastiCacheProcessingUnits` metric (serverless only) for traffic increase or inefficient commands. For node-based clusters, use command-level metrics such as `GetTypeCmds`, `SetTypeCmds`, etc.
- `ElastiCache:DataStorage` spiked: check `BytesUsedForCache` for data growth without TTLs
- `ElastiCache:NodeUsage` spiked: new nodes added (scaling event or parameter change)
- `DataTransfer` spiked: cross-AZ traffic increased (new replicas, or app moved AZs)
- `CloudWatch:Logs` spiked: log delivery enabled recently (see log-delivery.md cost warning)
3. **Correlate with timeline.** Check `describe-events` and CloudTrail for changes around the spike start date.
## Serverless Cost Drivers
| Cost Component | Description | How to Monitor |
|---------------|-------------|----------------|
| ECPUs consumed | ElastiCache Processing Units measure compute consumption per command; charged per million ECPUs | CloudWatch `ElastiCacheProcessingUnits` metric; `ThrottledCmds` indicates hitting limits |
| Data storage (GB) | Billed per GB-hour of data stored in the cache | CloudWatch `BytesUsedForCache` metric |
| Network transfer | Standard AWS data transfer charges for cross-AZ and internet-bound traffic | AWS Cost Explorer with usage type filters |
**Serverless minimum cost:** Approximately $6/month for a Valkey serverless cache with minimal traffic in us-east-1. Serverless Valkey caches bill for a minimum of 100 MB storage even with zero stored data, resulting in a baseline cost when the cache exists (Memcached and Redis OSS serverless bill for a minimum of 1 GB).
**Minimum usage limits (pre-scaling):** When you set a minimum usage limit (for ECPUs/second or data storage), you are charged for that minimum even if your actual usage is lower. For example, setting a minimum of 100,000 ECPUs/second costs 360 million ECPUs/hour at the per-ECPU rate for your engine (Valkey rates are lower than Redis OSS). Use `scripts/price_calculator.py --engine valkey --mode serverless` to calculate the exact hourly cost for your region. Be aware of this when configuring pre-scaling minimums to avoid unexpected charges.
## Node-Based Cost Drivers
| Cost Component | Description | How to Monitor |
|---------------|-------------|----------------|
| Instance hours | Billed per node per hour based on instance type; primary + replicas each incur charges | AWS Cost Explorer filtered by ElastiCache service |
| Reserved node discounts | Reserved nodes save approximately 30-55% depending on term length and payment option (run `python3 scripts/price_calculator.py --mode node --node-type <type> --nodes <N> --show-ri-options` for current estimates) | Reserved node utilization in Cost Explorer |
| Snapshot storage | Automated and manual backups stored in S3; one free snapshot per node, additional snapshots charged per GB | Snapshot count and size via DescribeSnapshots |
| Data transfer | Cross-AZ replication traffic and internet-bound transfer | Cost Explorer usage type filters |
## Hidden Costs
These costs are easy to overlook during initial planning:
- **Log delivery charges**: Enabling slow log or engine log delivery incurs charges. CloudWatch Logs and Kinesis Data Firehose are mutually exclusive destinations (you choose one per log type, not both). When using CloudWatch Logs, standard ingestion and storage charges apply. When using Kinesis Data Firehose, Firehose delivery charges apply in addition to CloudWatch Logs vended-log charges. Monitor the relevant cost line items in Cost Explorer.
- **Cross-AZ data transfer for replicas**: Node-based clusters with replicas in different Availability Zones incur cross-AZ data transfer charges for replication traffic. This is typically small but scales with write throughput.
- **ENI costs for Lambda**: Lambda functions in a VPC use Elastic Network Interfaces. While ENIs are free, the VPC attachment adds cold start latency that can increase Lambda duration costs.
- **Secrets Manager rotation**: If using RBAC with Secrets Manager, rotation Lambda invocations and API calls add minor cost.
- **Extended Support charges**: Running Redis OSS engines past their end of standard support date automatically enrolls them in Extended Support at additional cost. Redis OSS v4 and v5 enter Extended Support on Feb 1, 2026; Redis OSS v6 on Feb 1, 2027. Extended Support has tiered Y1/Y2/Y3 premiums and lasts up to 3 years. Upgrade to a supported version or migrate to Valkey to avoid these charges.
## Cost Optimization Strategies
### Serverless
- **Set CacheUsageLimits** to cap spend: configure `DataStorage.Maximum` (GB) and `ECPUPerSecond.Maximum` in the cache configuration to prevent runaway costs. Commands exceeding the ECPU limit are throttled (monitor `ThrottledCmds`).
- **Monitor ECPU utilization patterns**: if ECPU consumption is consistently low and predictable, evaluate whether node-based with reserved instances would be cheaper. Run `scripts/price_calculator.py --mode serverless --data-gb <N> --ops-per-sec <N>` and `--mode node --node-type <type> --nodes <N> --show-ri-options` to compare.
- **Optimize command efficiency**: use pipelining to batch commands (reduces round trips and ECPU overhead), prefer MGET/MSET over individual GET/SET for bulk operations, and use appropriate data structures (HGETALL vs multiple GETs).
### Node-Based
- **Right-size instances**: if `EngineCPUUtilization` is consistently below 20% and `DatabaseMemoryUsagePercentage` is below 30%, the instance is over-provisioned. Scale down to a smaller node type.
- **Use reserved nodes for steady-state workloads**: Reserved nodes save approximately 30-55% depending on term length and payment option (run `python3 scripts/price_calculator.py --mode node --node-type <type> --nodes <N> --show-ri-options` for current estimates).
- **Enable data tiering for large datasets**: for r6gd instance types, data tiering moves less-frequently-accessed data to SSD, allowing larger datasets on fewer nodes.
- **Cluster mode enabled for horizontal scaling**: instead of scaling up to larger (more expensive) node types, scale out with more shards on smaller nodes.
### General
- **Review eviction metrics**: high evictions (`Evictions` > 0 sustained) may indicate the cache is undersized. Evictions force re-computation of cached values, adding load to the backing store. Sizing up or increasing TTL diversity can reduce evictions.
- **Check hit rate**: a low cache hit rate means the cache is not effectively serving its purpose. Use the `CacheHitRate` metric directly (available for both serverless and node-based). For node-based, you can also calculate hit rate from `CacheHits / (CacheHits + CacheMisses)`. Target > 80%. Investigate key design, TTL strategy, and whether the right data is being cached.
- **Use Valkey instead of Redis OSS**: Valkey is 33% cheaper for serverless and 20% cheaper for node-based, with API compatibility. The in-place upgrade to Valkey is zero-downtime for Multi-AZ replication groups with Redis OSS 5.0.6+. Single-node clusters must first be converted to a replication group before upgrading. Upgrading from earlier Redis OSS versions may experience brief unavailability during DNS propagation.
## Serverless vs Node-Based Cost Decision
Use actual usage data to decide:
- **Switch to node-based when:** ECPU consumption is steady and predictable, and `scripts/price_calculator.py --mode node --node-type <type> --nodes <N> --show-ri-options` shows node-based with 1-year reservation is 30%+ cheaper than current serverless spend.
- **Stay on serverless when:** Traffic is bursty or growing unpredictably, utilization swings by more than 3x between peak and trough, or the team cannot commit to capacity planning.
- **Switch to serverless when:** Node-based CPU is below 20% and memory below 30% sustained (over-provisioned), or traffic has become highly variable after an initially steady workload.
Run `scripts/price_calculator.py --mode serverless --data-gb <N> --ops-per-sec <N>` and `--mode node --node-type <type> --nodes <N> --show-ri-options` to get a concrete comparison.
## Valkey Savings Summary
| Deployment | Valkey Savings vs Redis OSS |
|------------|---------------------------|
| Serverless | 33% lower cost |
| Node-based | 20% lower cost |
For organizations running Redis OSS workloads, migrating to Valkey is the single highest-impact cost optimization. The migration is a zero-downtime in-place engine upgrade for Multi-AZ replication groups with Redis OSS 5.0.6+ (see the migration sub-skill).
## Cost Estimation Tool
Use the bundled price calculator for detailed estimates:
```bash
# Quick serverless estimate
python3 scripts/price_calculator.py --engine valkey --data-gb 5 --ops-per-sec 1000
# Node-based estimate with reserved pricing options
python3 scripts/price_calculator.py --mode node --engine valkey --node-type <type> --nodes <N> --show-ri-options
# Node-based estimate with specific instance type
python3 scripts/price_calculator.py --mode node --engine valkey --node-type cache.r7g.xlarge --nodes 3
# Interactive mode
python3 scripts/price_calculator.py --interactive
```
## Metrics Collection
Use the bundled metrics collector to gather CloudWatch metrics for cost analysis and capacity planning:
```bash
# Collect metrics for a cache (last 7 days by default)
# Positional args: <endpoint> [port] [output_prefix]
bash scripts/collect_metrics.sh <endpoint> 6379 <output-prefix>
# Example: collect from a serverless endpoint
bash scripts/collect_metrics.sh my-cache.serverless.use1.cache.amazonaws.com 6379 my-cache
# Example: collect from a node-based endpoint
bash scripts/collect_metrics.sh my-cluster.abc123.use1.cache.amazonaws.com 6379 my-cluster
```
## Serverless Migration Cost Estimation
Use the serverless estimator to project costs when evaluating a migration from node-based to serverless (or vice versa):
```bash
# Estimate serverless cost from a cluster inventory CSV
python3 scripts/serverless_estimator.py --input clusters.csv
# Estimate with per-command stats for higher accuracy
python3 scripts/serverless_estimator.py --input clusters.csv --commandstats stats.csv
```
For actual spend on existing resources, use the AWS CLI:
```bash
aws ce get-cost-and-usage \
--time-period Start=2026-04-01,End=2026-04-16 \
--granularity DAILY \
--filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon ElastiCache"]}}' \
--metrics "UnblendedCost" "UsageQuantity" \
--group-by Type=DIMENSION,Key=USAGE_TYPE
```
## Cost Allocation Tags
Use tags to attribute ElastiCache costs to teams, applications, and environments in AWS Cost Explorer.
| Tag Key | Example Value | Purpose |
|---------|--------------|---------|
| `aws:elasticache:serverlessCacheName` | my-cache | Auto-applied by AWS; identifies the serverless cache resource |
| `aws:elasticache:replicationGroupId` | my-cluster | Auto-applied by AWS; identifies the node-based cluster resource |
| `Environment` | dev / staging / prod | Cost allocation by environment |
| `Application` | order-service | Cost allocation by consuming application |
| `Owner` | team-platform | Accountability and chargeback |
| `managed_by` | aws-skills | Skill attribution; tracks resources created via this skill |
Activate these tags in the AWS Billing console under **Cost Allocation Tags** to make them available in Cost Explorer reports and budgets. The `aws:elasticache:*` tags are auto-applied by the service and only need activation, not manual tagging.
references/monitoring/event-routing.md
# Event Routing
How to capture, route, and act on ElastiCache operational events across node-based and serverless deployments.
## Event Sources Overview
| Event Source | What It Captures | Deployment Type | Destination |
|-------------|-----------------|-----------------|-------------|
| ElastiCache cluster events (SNS) | Failover, maintenance, scaling, snapshot, configuration changes | Node-based | SNS topic |
| DescribeEvents API | Same as SNS but pull-based; also supports serverless-cache and serverless-cache-snapshot source types | Both | API response |
| CloudTrail | All ElastiCache API calls (create, modify, delete, describe) | Both | S3, CloudWatch Logs |
| EventBridge (via CloudTrail) | CloudTrail-derived API events for rule-based routing | Both | Lambda, SNS, SQS, Step Functions, etc. |
| EventBridge (direct service events) | Cache lifecycle events: Cache Created, Cache Deleted, Cache Updated, Cache Limit Approaching, Snapshot Created, etc. | Serverless | Lambda, SNS, SQS, Step Functions, etc. |
## Node-Based Event Pipeline
Node-based clusters emit operational events via SNS and the DescribeEvents API. **Note:** Node-based clusters use SNS for real-time cluster event notifications and EventBridge for CloudTrail-derived API events. Native ElastiCache service events via EventBridge are serverless-only.
### SNS Notifications
ElastiCache publishes cluster events directly to an SNS topic. This is the fastest way to get notified about operational events.
**SNS topic constraints:**
- Only **one** SNS topic can be configured per cluster for ElastiCache notifications.
- The SNS topic **cannot be encrypted** (at-rest). Attaching an encrypted topic will cause its status to show as "inactive", effectively disassociating it from the cluster.
- The SNS topic must be in the **same Region** as the ElastiCache cluster.
- The AWS account owning the SNS topic must be the **same account** that owns the cluster.
> **Security note:** Because at-rest encryption is not supported on this topic, event data (cluster names, failover and maintenance details) is stored unencrypted in SNS. Restrict the topic access policy to authorized principals only, and use HTTPS-only subscription protocols to protect events in transit.
For Valkey/Redis OSS replication groups:
```bash
aws elasticache modify-replication-group \
--replication-group-id my-cluster \
--notification-topic-arn arn:aws:sns:us-east-1:123456789012:elasticache-events \
--notification-topic-status active \
--apply-immediately \
--region us-east-1
```
For Memcached clusters (which use cache clusters, not replication groups):
```bash
aws elasticache modify-cache-cluster \
--cache-cluster-id my-memcached-cluster \
--notification-topic-arn arn:aws:sns:us-east-1:123456789012:elasticache-events \
--notification-topic-status active \
--apply-immediately \
--region us-east-1
```
**Key events delivered via SNS:**
| Event Type | Description | Action |
|-----------|-------------|--------|
| `ElastiCache:FailoverComplete` | Failover completed (Valkey/Redis OSS) | Verify application reconnection; check for data loss |
| `ElastiCache:CacheClusterProvisioningComplete` | Cluster provisioning finished | Verify cluster health; begin application traffic |
| `ElastiCache:CacheClusterScalingComplete` | Node type change or shard modification finished | Verify performance post-scaling |
| `ElastiCache:CacheClusterScalingFailed` | Scaling operation failed | Investigate; retry or revert |
| `ElastiCache:SnapshotComplete` | Backup creation finished (Valkey/Redis OSS) | No action needed |
| `ElastiCache:SnapshotFailed` | Backup creation failed (Valkey/Redis OSS) | Investigate; check snapshot limits |
| `ElastiCache:NodeReplacementScheduled` | Upcoming node replacement (maintenance) | Plan maintenance window; inform team |
| `ElastiCache:ServiceUpdateAvailableForNode` | Patch or version update available | Review release notes; schedule application |
| `ElastiCache:CacheNodeReplaceStarted` | Node replacement in progress | Monitor; expect brief disruption |
| `ElastiCache:CacheNodeReplaceComplete` | Node replacement finished | Verify cluster health |
### DescribeEvents API (Pull-Based)
```bash
# For node-based clusters:
aws elasticache describe-events \
--source-type replication-group \
--source-identifier my-cluster \
--duration 1440 \
--region us-east-1
# For serverless caches:
aws elasticache describe-events \
--source-type serverless-cache \
--max-items 40 \
--region us-east-1
```
DescribeEvents supports both node-based and serverless source types. Valid `--source-type` values include: `cache-cluster`, `replication-group`, `serverless-cache`, `serverless-cache-snapshot`, `user`, `user-group`, and others. Events can be retrieved for up to 14 days.
## Serverless Event Pipeline
ElastiCache emits native service events directly to EventBridge for serverless caches (best-effort delivery). Serverless caches do not support SNS topic notifications; use EventBridge for serverless event routing.
### Native EventBridge Service Events
ElastiCache sends these events directly to EventBridge for serverless caches:
| Detail-Type | Category | Description |
|-------------|----------|-------------|
| `Cache Created` | notification | Serverless cache provisioning complete |
| `Cache Creation Failed` | notification | Serverless cache provisioning failed |
| `Cache Deleted` | notification | Serverless cache deleted |
| `Cache Updated` | notification | Serverless cache modification complete |
| `Cache Update Failed` | notification | Serverless cache modification failed |
| `Cache Limit Approaching` | notification | A slot is using more than X% of the 32 GB per-slot limit |
| `Snapshot Created` | notification | Snapshot creation complete |
| `Snapshot Creation Failed` | notification | Snapshot creation failed |
| `Snapshot Export Failed` | notification | Snapshot export to S3 failed |
| `Snapshot Copy Failed` | notification | Cross-region snapshot copy failed |
> **Note:** These native EventBridge service events apply to serverless caches only. Some events like `Cache Limit Approaching` are serverless-specific.
### EventBridge Rules for Service Events
Create EventBridge rules that match native ElastiCache service events:
```bash
aws events put-rule \
--name elasticache-serverless-events \
--event-pattern '{
"source": ["aws.elasticache"],
"detail-type": [
"Cache Created",
"Cache Creation Failed",
"Cache Deleted",
"Cache Updated",
"Cache Update Failed",
"Cache Limit Approaching",
"Snapshot Created",
"Snapshot Creation Failed",
"Snapshot Export Failed",
"Snapshot Copy Failed"
]
}' \
--region us-east-1
```
Add targets via `aws events put-targets` pointing to SNS, Lambda, SQS, or Step Functions.
For change management auditing, also create a CloudTrail-based rule:
```bash
aws events put-rule \
--name elasticache-serverless-api-audit \
--event-pattern '{
"source": ["aws.elasticache"],
"detail-type": ["AWS API Call via CloudTrail"],
"detail": {
"eventSource": ["elasticache.amazonaws.com"],
"eventName": [
"CreateServerlessCache",
"ModifyServerlessCache",
"DeleteServerlessCache"
]
}
}' \
--region us-east-1
```
### CloudTrail API Events
All ElastiCache API calls (including serverless operations) are recorded in CloudTrail:
- `CreateServerlessCache`
- `ModifyServerlessCache`
- `DeleteServerlessCache`
- `CreateUser`, `ModifyUser`, `DeleteUser`
- `CreateUserGroup`, `ModifyUserGroup`, `DeleteUserGroup`
## CloudTrail for Both Deployment Types
Ensure CloudTrail is enabled for ElastiCache API event recording.
Generate EventBridge rules and SNS topics via CloudFormation or CDK. The pattern matches `source: aws.elasticache` with `detail-type: AWS API Call via CloudTrail`.
## Recommended Event Pipeline by Deployment Type
### Node-Based Production
1. **SNS notifications** on the replication group for real-time operational alerts (failover, maintenance, scaling).
2. **EventBridge rule** matching CloudTrail API events for change management auditing.
3. **CloudTrail** enabled for compliance and post-incident investigation.
### Serverless Production
1. **EventBridge rule** matching native service events (`Cache Limit Approaching`, `Cache Update Failed`, etc.) for operational alerting.
2. **EventBridge rule** matching CloudTrail-derived API events for change management auditing.
3. **CloudWatch alarms** (see `alarm-packs.md`) for performance and throttling alerts (ECPU consumption, latency, throttled commands).
4. **CloudTrail** enabled for compliance and post-incident investigation.
### Development / Staging
1. **EventBridge rule** for destructive operations only (delete, modify) to catch accidental changes.
2. **CloudTrail** with short retention for debugging.
## Event-to-Action Mapping
| Event | Severity | Recommended Action |
|-------|----------|-------------------|
| Failover started | P1 (page) | Alert on-call; verify application reconnection |
| Failover complete | P2 (next hour) | Confirm replicas healthy; check for data loss |
| Maintenance scheduled | P3 (next day) | Inform team; verify maintenance window is acceptable |
| Service update available | P3 (next day) | Review release notes; schedule during maintenance window |
| DeleteServerlessCache / DeleteReplicationGroup | P1 (page) | Alert immediately; verify intentional (audit trail) |
| ModifyReplicationGroup (parameter change) | P2 (next hour) | Review change; verify no unexpected impact |
| Throttled commands (serverless) | P2 (next hour) | Handled by CloudWatch alarms; increase ECPU limits |
references/monitoring/hot-key-detection.md
# Hot Key and Hot Shard Detection
**When to use:** The user reports uneven shard load, one node pinned at high CPU while peers are idle, latency spikes correlated with specific request patterns, or suspected "hot key" problem.
**When not needed:** Broadly high engine CPU across all shards (that's capacity, not a hot key: see `troubleshooting.md` High CPU). Memory pressure with evictions is a different playbook (see Memory Pressure). Connection storms are not hot-key issues (see Connection Spikes).
## Objective
Given a cluster with a suspected hot-key problem, answer three questions in order:
1. Is there actually per-shard imbalance, or is the whole cluster hot?
2. If imbalanced, which shard (which slot range) is hot?
3. Within the hot shard, which specific key is hot?
Each step has stop conditions. Do not proceed to the next step unless the current step confirms the problem.
## Preconditions and Decision Tree
### Check 1: Deployment model
```
Serverless → Tier A (CloudWatch) plus limited client-side diagnostics.
Tiers B and C are not available.
See "Serverless-specific considerations" below for the full diagnostic order.
Node-based → All tiers available, proceed through Check 2.
```
### Check 2: Engine and version (node-based only)
```
Valkey 8.0+ cluster mode (node-based only) → Tier B (CLUSTER SLOT-STATS) is available and preferred.
Valkey 7.x, Redis OSS 7.x → Skip Tier B. Use Tier A to find the hot node, then Tier C.
Cluster mode disabled → Single shard. Skip Tier B. Tier A and Tier C only.
```
Check engine version (use `describe-cache-clusters` which exposes both fields):
```bash
aws elasticache describe-cache-clusters \
--cache-cluster-id <node-id> --region <region> \
--query 'CacheClusters[0].[CacheNodeType,EngineVersion,Engine]'
```
### Check 3: Eviction policy (gates Tier C only)
`OBJECT FREQ` requires an LFU policy (`allkeys-lfu` or `volatile-lfu`). `CONFIG GET` is restricted on all ElastiCache caches; check the parameter group instead:
```bash
aws elasticache describe-cache-parameters \
--cache-parameter-group-name <parameter-group-name> \
--query "Parameters[?ParameterName=='maxmemory-policy'].ParameterValue" \
--output text --region <region>
```
- **LFU policy set** → Tier C is available.
- **Non-LFU, can change** -> Change via parameter group; `maxmemory-policy` takes effect immediately (no restart needed), then Tier C.
- **Non-LFU, cannot change** → Skip Tier C. Fallback: Tier B if Valkey 8.0+ cluster mode, otherwise slow log analysis, then `valkey-cli --bigkeys`, then client-side instrumentation. Note: `valkey-cli --hotkeys` also requires LFU.
- **Serverless** → Skip Tier C entirely. Serverless blocks `OBJECT FREQ`.
## Tier A: Triage with CloudWatch
Always-available first step. Tells you whether the problem is real and narrows where to look.
### Step 1: Confirm per-shard imbalance (node-based)
Query `EngineCPUUtilization` per node using the `CacheClusterId` dimension. Do NOT query with `ReplicationGroupId`; the aggregate hides imbalance.
> **Note on CPU metrics:** `EngineCPUUtilization` measures only the main Valkey/Redis OSS engine thread. On nodes with 4+ vCPUs where enhanced I/O features are active, network I/O and TLS processing are offloaded to dedicated I/O threads not captured by `EngineCPUUtilization`, so it may appear lower than expected. For smaller node types with 2 vCPUs or less, use `CPUUtilization` instead. Also check `TrafficManagementActive`; a value of 1 indicates the node may be underscaled for the workload — ElastiCache is actively managing/throttling traffic to protect the engine.
```bash
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache \
--metric-name EngineCPUUtilization \
--dimensions Name=CacheClusterId,Value=<node-id> \
--start-time <1h-ago> --end-time <now> \
--period 60 --statistics Maximum \
--region <region>
```
Run for each node (`describe-cache-clusters --show-cache-node-info` to list nodes).
**Stop condition:** If all nodes are within 15 percentage points of each other, this is not a hot-key problem. Route to the High CPU playbook in `troubleshooting.md`.
If one node is at least 1.5x the cluster median, proceed.
### Step 2: Identify the data type driving traffic
Query command-family metrics on the hot node: `StringBasedCmds`, `HashBasedCmds`, `SetBasedCmds`, `SortedSetBasedCmds`, `ListBasedCmds`, `StreamBasedCmds`, `PubSubBasedCmds`, `JsonBasedCmds`, `SearchBasedCmds`.
The family with the highest rate on the hot node points to the data type of the hot key.
### Step 3: Check slow log (node-based only, with log delivery enabled)
Slow log delivery must be enabled via log delivery configuration (see `log-delivery.md` for setup). Search for repeated key patterns. A key appearing in many slow-log entries is a strong hot-key candidate, especially with `HGETALL`, `SMEMBERS`, `LRANGE 0 -1`, `ZRANGE 0 -1`, or `SORT`.
```bash
aws logs filter-log-events \
--log-group-name /aws/elasticache/<cluster-id>/slowlog \
--start-time <1h-ago-ms> \
--filter-pattern "<suspected-key-prefix>" \
--region <region>
```
### Step 4: Consult LATENCY commands (node-based only)
```bash
valkey-cli -h <endpoint> -p 6379 --tls LATENCY DOCTOR
valkey-cli -h <endpoint> -p 6379 --tls LATENCY LATEST
valkey-cli -h <endpoint> -p 6379 --tls LATENCY HISTORY <event-name>
```
Serverless: the entire `LATENCY *` family is blocked. Use `SuccessfulReadRequestLatency` and `SuccessfulWriteRequestLatency` CloudWatch metrics at p50/p99/p100 instead.
**Tier A summary (produce before proceeding):**
- Per-shard imbalance? (yes / no / single shard N/A)
- Which node is hot? (node-id or "aggregate only")
- Dominant data type? (from Step 2)
- Slow-log fingerprints? (top 3 keys, or N/A)
- Next action: Tier B, Tier C, or serverless diagnostic order
## Tier B: Slot-level diagnosis (Valkey 8.0+ cluster mode)
`CLUSTER SLOT-STATS` reports per-slot command counts and CPU usage. Not available in Redis OSS or earlier Valkey. No LFU requirement.
| Signal | Typical value |
|---|---|
| Runtime | Subsecond on any cluster size |
| Throughput impact | Negligible |
| Production safe | Yes, use `LIMIT 10` for bounded response |
### Step 1: Identify hot slots
```bash
valkey-cli -h <endpoint> -p 6379 --tls --cluster-yes \
CLUSTER SLOT-STATS ORDERBY CPU-USEC LIMIT 10
```
**Stop condition:** If top 10 slots are within 2x of each other, no single hot slot. Reconsider whether the problem is command-family cost rather than a hot key.
### Step 2: Enumerate candidate keys in the hot slot
```bash
valkey-cli -h <endpoint> -p 6379 --tls \
CLUSTER GETKEYSINSLOT <hot-slot-number> 200
```
Cross-reference with `MEMORY USAGE <key>` and command family from Tier A Step 2.
**Stop condition:** If Tier A already identified a specific key in slow log that hashes into this slot (verify via `CLUSTER KEYSLOT <key>`), skip enumeration.
### Step 3: Verify the key is hot
If LFU policy: proceed to Tier C on each candidate. If not LFU, use these alternatives in order:
1. **Client-side counters.** Instrument the application to log command and key for a 5-10 minute sampling window. Most effective non-LFU option.
2. **Slow-log analysis.** Run `aws logs filter-log-events`, group by key, rank by frequency.
3. **`MEMORY USAGE <key>` per candidate.** If one candidate is distinctly larger, that's a strong big+hot signal.
## Tier C: Per-key frequency (LFU policy required)
`OBJECT FREQ` reads the logarithmic frequency counter maintained under LFU policies. O(1) per call, safe to run at scale.
### Step 1: Confirm LFU policy
```bash
aws elasticache describe-cache-parameters \
--cache-parameter-group-name <parameter-group-name> \
--query "Parameters[?ParameterName=='maxmemory-policy'].ParameterValue" \
--output text --region <region>
```
If not LFU, stop. The counter is meaningless under LRU.
### Step 2: Sample candidate keys
```bash
valkey-cli -h <endpoint> -p 6379 --tls OBJECT FREQ <key>
```
Interpreting the counter (logarithmic):
- 0-3: rarely accessed
- 4-8: moderate access
- 9-15: high access, candidate
- 15+: capped maximum, very hot
### Step 3: Confirm and assess
Rank candidates by `OBJECT FREQ`. Retrieve sizes via `MEMORY USAGE <key>` to determine whether the problem is request rate (fix with client-side caching or key splitting) or value size (fix with value decomposition).
## MONITOR: do not use on production
**Serverless:** `MONITOR` is blocked. This section applies to node-based only.
`MONITOR` streams every command to the connected client. Do not use on production:
- 50%+ throughput reduction per valkey.io documentation; impact increases with multiple MONITOR clients
- Can saturate the client's network pipe
- No rate limit, cannot scope to a subset of keys
**If genuinely the only option** (no LFU, Tier B unavailable, client instrumentation not feasible):
- Replica node only, never the primary
- Short bounded window: `valkey-cli MONITOR | head -n 100000`
- Off-peak or maintenance window only
- Explicit user acknowledgement of the performance risk
## Serverless-specific considerations
### Blocked commands on serverless
`OBJECT *`, `MEMORY *`, `LATENCY *`, `SLOWLOG *`, `COMMANDLOG` (Valkey 8.1+; not available on serverless), `CONFIG *`, `MONITOR`, `valkey-cli --hotkeys`, `valkey-cli --memkeys`. `valkey-cli --bigkeys` works (uses SCAN + type-length commands).
### Serverless diagnostic order
1. **CloudWatch triage:** command-family metrics to identify the hot data type, latency metrics at p50/p99/p100 to confirm impact, `ThrottledCmds` to spot bursts.
2. **Narrow with `--bigkeys`:** client-side sample scan to identify large values. Run during off-peak.
3. **Confirm with application instrumentation:** log top-N command-and-key pairs for a bounded window.
4. **Remediate** per the options below.
## Remediation options
Ordered by preference. Pick the first that fits; layer in later options if needed.
1. **Client-side caching for read-heavy hot keys.** Cache the value in-process with a short TTL. Reduces cache-side request rate by an order of magnitude.
2. **Key splitting.** Replicate across N logical keys (`hot:1` through `hot:N`), load-balance reads. Use when client-side caching is not feasible.
3. **Add read replicas.** Distribute read load if the application uses reader endpoints.
4. **Value decomposition.** Break large hashes/lists so each request touches less data. Indicated when slow log shows `HGETALL` or `LRANGE 0 -1`.
5. **Finer-grained sharding.** Introduce a finer key space (shard by user-id suffix, time bucket, tenant id).
6. **TTL tuning.** Longer TTL on cache-aside entries reduces miss-driven reloads.
## Cross-links
- Cluster-wide high CPU (not a hot key): `troubleshooting.md` High CPU section
- Hot shard causing replication lag: `troubleshooting.md` Replication Lag section
- Alarm patterns for per-node imbalance: `alarm-packs.md`
- Slow log configuration: `log-delivery.md`
references/monitoring/instructions.md
# Monitoring & Cost
**When to use:** The user wants to view CloudWatch metrics, set up dashboards or alarms, analyze ElastiCache costs, troubleshoot performance issues, run a security audit, or optimize cache spending.
**When not needed:** The user is creating a new cache (use create-secure-cache.md), choosing an engine or deployment model, writing connection code, or designing data models.
CloudWatch metrics, cost analysis, performance troubleshooting, and cost optimization for ElastiCache.
## Loading
Read this file first. Other references in this folder load on demand when the current answer requires them. Scripts in `scripts/` run on demand (for example, `generate_dashboards.py`, `security_audit.py`). For ElastiCache-specific error codes, see `references/shared-ux/error-remediation.md`.
## On-demand references
| File | Load when |
|------|-----------|
| `troubleshooting.md` | User reports a performance issue, latency, errors, or operational problem |
| `hot-key-detection.md` | User suspects a hot key, uneven shard load, or single-node CPU spike while peers are idle |
| `big-key-hunter.md` | User suspects oversized values, memory growing faster than key count, or latency on HGETALL/LRANGE/SMEMBERS |
| `key-space-distribution-by-prefix.md` | User wants to know what is in their cache by prefix, cost attribution by tenant, or TTL audit |
| `slot-memory-imbalance-detection.md` | User reports one shard full while others are low, hash-tag concentration, or pre-resharding analysis (Valkey 8.0+ cluster mode) |
| `slow-log-cross-signal-diagnosis.md` | User reports intermittent latency spike and needs root-cause correlation (node-based with slow-log enabled) |
| `client-tuning-and-diagnostics.md` | User asks about client timeouts, connection pooling config, TLS setup per language, or missing CloudWatch metrics |
| `alarm-packs.md` | User wants CloudWatch alarms for their cache |
| `cloudwatch-dashboards.md` | User wants a CloudWatch dashboard |
| `event-routing.md` | User wants event notifications, EventBridge rules, or CloudTrail integration |
| `log-delivery.md` | User wants to enable slow log, engine log, or command log delivery |
| `cost-reporting.md` | User wants cost analysis, optimization advice, or spend reporting |
## Symptom-to-File Routing
| User describes | Load |
|----------------|------|
| Slow responses, high latency | `troubleshooting.md` (High Latency) |
| One node hotter than others, uneven load | `hot-key-detection.md` |
| Can't connect, connection errors | `troubleshooting.md` (Cannot Connect) |
| High CPU on one shard | `hot-key-detection.md` then `troubleshooting.md` (High CPU) |
| Cache not helping, low hit rate | `troubleshooting.md` (Low Hit Rate) |
| Memory growing fast, big values suspected | `big-key-hunter.md` |
| Which prefix uses most memory, cost attribution by tenant | `key-space-distribution-by-prefix.md` |
| One shard full, others low, slot imbalance | `slot-memory-imbalance-detection.md` |
| Intermittent latency spike, need root cause | `slow-log-cross-signal-diagnosis.md` |
| Keys without TTL, orphan key cleanup | `key-space-distribution-by-prefix.md` (TTL Audit) |
| Bill too high, cost spike | `cost-reporting.md` |
| Want dashboards or alarms | `cloudwatch-dashboards.md` or `alarm-packs.md` |
| Want event notifications | `event-routing.md` (note: serverless caches do not support SNS topic notifications; instead, ElastiCache sends service events (Cache Created, Cache Updated, Cache Deleted, Snapshot Created, etc.) directly to EventBridge, and API calls are also available via CloudTrail + EventBridge. Node-based clusters use SNS for cluster events. EventBridge receives CloudTrail-derived API events for both deployment models; native ElastiCache service events via EventBridge are serverless-only) |
| Want to enable logging | `log-delivery.md` |
| Metrics not appearing, empty CloudWatch graphs | `client-tuning-and-diagnostics.md` (Missing CloudWatch Metrics) |
| Client timeout tuning, connection pool sizing | `client-tuning-and-diagnostics.md` (Client Timeout / Connection Pooling) |
| TLS connection setup per language | `client-tuning-and-diagnostics.md` (TLS Connection Quick Reference) |
| Specific error code (MOVED, CROSSSLOT, CLUSTERDOWN, WRONGPASS, OOM, READONLY, LOADING, MULTI/EXEC failure, IAM token error) | `references/shared-ux/error-remediation.md` |
## Check for existing context (agent-facing)
Before fetching metrics, check if CloudWatch data or cluster details are already in the conversation context. If `.elasticache/requirements.json` exists, read `infrastructure.cache_name`, `deployment_model`, `engine`, and `engine_version` to avoid re-asking. The engine and version gate certain diagnostics (e.g., CLUSTER SLOT-STATS requires Valkey 8.0+, OBJECT FREQ requires LFU policy).
## Day-1 Observability Checklist
For a newly created cache, run these 5 commands to establish baseline observability:
```bash
# 1. Generate dashboard + alarms
python3 scripts/generate_dashboards.py --serverless <name> --sns-topic <arn> --output observability.json
# 2. Deploy the stack
aws cloudformation deploy --template-file observability.json --stack-name <name>-observability --region <region>
# 3. Enable slow log delivery (node-based only)
# Log delivery (slow log, engine log, command log) is a node-based feature.
# Serverless caches do not support log delivery via the API or console.
# For serverless, use CloudWatch metrics and client-side logging instead.
# For node-based clusters, use modify-replication-group with --apply-immediately
# (see log-delivery.md for details).
# 4. Verify metrics are flowing (wait 5-10 min after first successful command)
# NOTE: CacheHitRate will return empty datapoints until the cache has served
# real traffic. This is expected. If empty, wait and retry after sending test
# commands (e.g., SET/GET). For node-based caches, use CacheHits with the
# CacheClusterId dimension instead (CacheHitRate is available for both serverless and node-based).
aws cloudwatch get-metric-statistics --namespace AWS/ElastiCache --metric-name CacheHitRate \
--dimensions Name=ServerlessCacheName,Value=<name> --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
--end-time $(date -u +%Y-%m-%dT%H:%M:%S) --period 300 --statistics Average --region <region>
# macOS: replace -d '1 hour ago' with -v-1H
# 5. Confirm alarm state
aws cloudwatch describe-alarms --alarm-name-prefix <name> --region <region> --query 'MetricAlarms[].{Name:AlarmName,State:StateValue}'
```
For node-based, replace `--serverless <name>` with `--replication-group <id>` in step 1. Log delivery (step 3) applies to node-based clusters only.
## Workflow
1. Identify what the user needs: current metrics, cost data, security posture, or optimization advice
2. Use the AWS CLI or SDK to gather data
3. For collecting raw metrics from a running cache:
```bash
./scripts/collect_metrics.sh <endpoint> [port] [output_prefix]
```
4. If the user doesn't have dashboards/alarms yet, generate them:
```bash
python3 scripts/generate_dashboards.py --serverless <name> --output observability.json
python3 scripts/generate_dashboards.py --replication-group <name> --output observability.json
```
5. For security and operational posture, run the audit:
```bash
python3 scripts/security_audit.py --serverless <name>
python3 scripts/security_audit.py --replication-group <name>
```
6. Present results in a clean summary
7. Recommend optimizations if applicable
For metric names, dimensions, and query examples, see the CLI template in `troubleshooting.md`.
## Freshness disclaimer
When your response includes pricing, version constraints, or feature availability, include the freshness disclaimer per SKILL.md Global Rule #5: "For current pricing see https://aws.amazon.com/elasticache/pricing/. For current feature availability see https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/."
## Cost Analysis
Top 4 optimizations (highest impact first):
1. **Redis OSS to Valkey**: 33% cheaper serverless, 20% cheaper node-based. Zero-downtime in-place upgrade.
2. **Right-size nodes**: If CPU < 20% and memory < 30% sustained, scale down or switch to serverless.
3. **Reserved instances**: Reserved nodes save approximately 30-55% depending on term length and payment option (run `python3 scripts/price_calculator.py --mode node --node-type <type> --nodes <N> --show-ri-options` for current estimates). Node-based only; not applicable to serverless.
4. **Avoid Extended Support charges**: Redis OSS v4 and v5 enter Extended Support on Feb 1, 2026 (v6 on Feb 1, 2027) with escalating yearly premiums. Upgrade to Valkey or a supported version before the end-of-standard-support date to avoid these charges.
For full cost analysis, estimates, and hidden costs, load `cost-reporting.md`. For estimates, use `scripts/price_calculator.py --mode serverless --data-gb <N> --ops-per-sec <N>` and `--mode node --node-type <type> --nodes <N> --show-ri-options`.
references/monitoring/key-space-distribution-by-prefix.md
# Key-Space Distribution by Prefix
**When to use:** User wants to understand what is in their cache grouped by key prefix. Typical questions: "which prefix is using the most memory?", "how many keys per tenant?", "is session:*growing faster than cache:*?", "who owns orphan keys?". Also used for cost attribution, tenant onboarding/offboarding, and pre-migration audits.
**When not needed:** Already knows which specific keys are big or hot (use big-key-hunter.md or hot-key-detection.md). Wants a full memory map including overhead (requires MEMORY USAGE, node-based only).
## Preconditions
| Deployment | Available Tiers |
|---|---|
| Serverless | Tier A + Tier B only. MEMORY USAGE blocked; estimate from cardinality x element size. |
| Node-based | All tiers. Must fan out SCAN across all primary endpoints. |
### Command availability
| Command | Node-based | Serverless | Purpose |
|---|---|---|---|
| SCAN \<cursor\> MATCH \<pattern\> COUNT \<n\> | Yes | Yes | Cursor-based keyspace iteration |
| TYPE \<key\> | Yes | Yes | Data type per key |
| STRLEN, HLEN, LLEN, ZCARD, SCARD, XLEN | Yes | Yes | Per-type cardinality |
| DBSIZE | Yes | Yes | Fast total count |
| CLUSTER SHARDS | Yes (Redis OSS 7.0+ / Valkey 7.2+). For Redis OSS 6.x and earlier, use CLUSTER SLOTS. | Yes (virtual shard) | Topology discovery |
| MEMORY USAGE \<key\> | Yes | Blocked | Exact byte footprint |
| KEYS \<pattern\> | Yes (never use) | Blocked | Blocking enumeration. Use SCAN. |
---
## Tier A: Triage with CloudWatch and scope the scan
### Step 1: CloudWatch signal check
Query `BytesUsedForCache` and `CurrItems` trend. If both flat and within range, analysis may not be urgent. If memory-per-key is rising or both climbing together, proceed.
**Stop condition:** If CloudWatch shows no growth and concern is theoretical, consider a sampled scan only.
### Step 2: Get a size estimate
```bash
valkey-cli -h <endpoint> -p 6379 --tls DBSIZE
```
Node-based cluster mode: run DBSIZE against each primary, sum results.
### Step 3: Decide scan strategy
| Total DBSIZE | Strategy |
|---|---|
| < 1M keys | Full scan, 1-3 minutes, low impact |
| 1M-10M keys | Full scan with COUNT 500, off-peak or against replica |
| > 10M keys | Sampled scan (stop after 100k keys) or offline RDB analysis |
### Step 4: Identify delimiter and prefix depth
Common conventions: `:` (app:module:id), `/` (URL-shaped), `|` (legacy). Start with depth 1.
**Stop condition:** If a sample of 100 keys shows no consistent separator, aggregate by first N characters and flag "no structured naming" as output.
---
## Tier B: Narrow with client-side sampling
### Minimal shell pattern (prefix-depth 1, `:` delimiter)
```bash
valkey-cli -h <endpoint> -p 6379 --tls --scan --pattern '*' \
| awk -F: '{ prefix=$1; count[prefix]++ } END { for (p in count) print count[p]"\t"p }' \
| sort -rn | head -50
```
For richer aggregation (per-type counts, cardinality sum), use a Python or Bash script that calls TYPE per sampled key with proper key escaping. The minimal awk pipeline above is safe for prefix counting; per-type aggregation with shell pipelines breaks on keys containing spaces, quotes, or special characters.
Run against a replica endpoint on node-based clusters.
### Cost and impact
| Signal | Typical value |
|---|---|
| Runtime on 1M keys | 2-4 minutes |
| Runtime on 10M keys | 20-40 minutes |
| Throughput impact | 0.5% to 2% |
| EngineCPU impact | Minimal (reads only) |
Same safety class as `valkey-cli --bigkeys`: cursor-bounded, read-only, stoppable. Stop if EngineCPU rises >10 percentage points.
**Serverless:** run off-peak. Every sampled key consumes ECPUs.
### Interpret the output
| Prefix | Count | Sum cardinality | Dominant type | Max single-key |
|---|---|---|---|---|
| session:user: | 1,245,000 | 4,980,000 | hash (5 fields avg) | hash 142 fields |
| cache:query: | 234,000 | 234,000 | string (avg 12KB) | string 4MB |
| _tmp: | 89,000 | 89,000 | string (avg 200B) | string 8KB |
- Count = volume. Over a million small session keys is normal. Tens of thousands of untracked `_tmp:` keys with no TTL is a leak.
- Sum cardinality approximates memory load.
- Max catches big-key outliers hiding in an otherwise normal prefix (route to big-key-hunter.md).
**Stop condition:** If distribution is uniform (within an order of magnitude, no type surprises, no unexpected prefixes), the keyspace is healthy.
---
## Tier C: Verify with MEMORY USAGE (node-based only)
### Step 1: Sample keys from the hot prefix
```bash
valkey-cli -h <endpoint> -p 6379 --tls --scan --pattern '<prefix>*' | head -100 > keys-sample.txt
```
### Step 2: Measure byte footprint
```bash
valkey-cli -h <endpoint> -p 6379 --tls MEMORY USAGE <key>
```
### Step 3: Extrapolate
If 100 sampled keys from a 1,245,000-key prefix average 4,200 bytes, the prefix uses ~5GB. Validate against the `BytesUsedForCacheItems` CloudWatch metric (node-based).
---
## TTL Audit Extension
Run alongside Tier B in the same SCAN loop. Adds ~10-15% runtime. TTL and EXPIRETIME are O(1).
| Command | Purpose | Notes |
|---|---|---|
| TTL \<key\> | Seconds until expiry | -1 = no TTL, -2 = key missing |
| EXPIRETIME \<key\> | Absolute Unix timestamp | Valkey 7.2+ / Redis OSS 7.0+. Prefer when available. |
### Augmented output
| Prefix | Count | Keys without TTL | Avg TTL (sec) |
|---|---|---|---|
| session:user: | 1,245,000 | 0 (good) | 1,800 |
| cache:query: | 234,000 | 12,400 | 3,600 |
| _tmp: | 89,000 | 89,000 (orphan suspects) | -1 |
### TTL hygiene signals
- Keys without TTL in a prefix that should be volatile: application lost track
- All keys without TTL in unknown prefix: strong orphan signal (deprecated feature, crashed job)
- Bimodal TTL distribution: two writers with inconsistent policy
### Remediation for TTL problems
1. **Application-level fix** (default): add EX to the SET/HSET/SADD that writes without TTL
2. **Backfill TTL**: run batched EXPIRE during off-peak
3. **Enforce conventions**: document TTL policy per prefix, review in code review
4. **Bulk cleanup**: UNLINK orphan prefixes in batches (non-blocking)
---
## Remediation for prefix distribution problems
1. **TTL hygiene** (default): shorten or enforce TTL on the bloated prefix
2. **Application cleanup**: UNLINK deprecated/orphan keys in batches
3. **Move to different tier**: large blobs that don't benefit from cache semantics belong in S3/DynamoDB
4. **Split to separate cache**: when one tenant/prefix dominates (>30% of memory). If resharding within a cluster instead, note that ElastiCache cannot migrate slots containing items with serialized size larger than 256 MB.
5. **Compression**: gzip/zstd for text-heavy values at application layer
---
## Cross-links
- Specific key in a hot prefix is oversized: `big-key-hunter.md`
- Per-shard imbalance (not prefix-based): `slot-memory-imbalance-detection.md`
- Eviction has started: `troubleshooting.md` Memory Pressure
- Alarm patterns: `alarm-packs.md`
references/monitoring/log-delivery.md
# Log Delivery
ElastiCache log types, destinations, configuration, and cost considerations.
## Log Types
| Log Type | What It Captures | When to Enable | Engine Support |
|----------|-----------------|----------------|----------------|
| Slow log | Commands that exceed the `slowlog-log-slower-than` threshold (default 10,000 microseconds). **Note:** A fixed number of slow log entries are retrieved periodically; depending on `slowlog-max-len`, some entries may not be delivered. | Always recommended for production. Essential for identifying slow commands that degrade performance. | Valkey 7.x+, Redis OSS 6.0+ |
| Engine log | Internal engine events: startup, shutdown, configuration changes, replication events, cluster events | Recommended for production. Useful for debugging replication issues, failover events, and configuration drift. | Valkey 7.x+, Redis OSS 6.2+ |
| Command log (COMMANDLOG) | Records slow executions, large requests, and large replies. Separate from slow log. | Use selectively for debugging or auditing. High-volume workloads generate significant log data. | Valkey 8.1+ only |
## Log Destinations
| Destination | Strengths | Considerations |
|-------------|-----------|----------------|
| CloudWatch Logs | Native integration with CloudWatch Insights, alarms, metric filters, and dashboards. Simple to set up. | CloudWatch Logs charges apply per GB ingested and stored. For high-throughput caches, costs can be significant. |
| Kinesis Data Firehose | Stream to S3, OpenSearch, Redshift, or third-party tools. Good for high-volume log pipelines. | Firehose delivery charges apply in addition to CloudWatch Logs vended-log charges (AWS charges for vended logs even when delivered directly to Firehose). |
### Cost Warning
When using CloudWatch Logs as the destination, standard CloudWatch Logs ingestion and storage charges apply. When using Kinesis Data Firehose as the destination, Firehose delivery charges apply in addition to CloudWatch Logs vended-log charges (AWS charges for vended logs even when delivered directly to Firehose). Factor log delivery costs into your operational budget, especially for:
- High-throughput caches (>10,000 ops/sec) with command logging enabled.
- Large clusters with many nodes, each emitting engine logs.
- Development environments where logs may not be actively monitored (consider disabling or sampling).
## Configuration via CLI
Use the following JSON structure for `--log-delivery-configurations`:
```json
{
"LogType": "<LogType>",
"DestinationType": "<DestinationType>",
"DestinationDetails": {
"<DestinationDetailsKey>": {
"<TargetKey>": "<TargetValue>"
}
},
"LogFormat": "json",
"Enabled": true
}
```
### LogType Values
| Log Type | LogType Value | API Status |
|----------|--------------|------------|
| Slow log | `slow-log` | Available |
| Engine log | `engine-log` | Available |
**⚠️ Command log (`command-log`):** The ElastiCache API accepts only `slow-log` and `engine-log` as LogType values; `command-log` is not a valid LogType. The COMMANDLOG feature (Valkey 8.1+) may be exposed via the `engine-log` LogType or require a separate API enum value for `command-log`. Check the latest API reference before attempting to configure command log delivery.
### DestinationType Values
| Destination | DestinationType | DestinationDetails Key | Target Key |
|-------------|----------------|------------------------|------------|
| CloudWatch Logs | `cloudwatch-logs` | `CloudWatchLogsDetails` | `LogGroup` |
| Kinesis Data Firehose | `kinesis-firehose` | `KinesisFirehoseDetails` | `DeliveryStream` |
Pass to `create-replication-group` (at creation time, without `--apply-immediately`) or `modify-replication-group` / `modify-cache-cluster` (for existing clusters, with `--apply-immediately`). You **must** set the `--apply-immediately` parameter when modifying log delivery on existing clusters.
> **Node-based only.** Log delivery is available for node-based clusters only. Serverless caches do not support log delivery (slow log, engine log, or command log) via the API or console. For serverless observability, use CloudWatch metrics (latency, ECPU, throttling) and client-side logging.
To disable, set `"Enabled": false` (only `LogType` and `Enabled` are required).
### Required IAM Permissions
To configure log delivery, your IAM user/role must have the following permissions:
**For CloudWatch Logs destination:**
- `logs:CreateLogDelivery`
- `logs:UpdateLogDelivery`
- `logs:DeleteLogDelivery`
- `logs:GetLogDelivery`
- `logs:ListLogDeliveries`
**For Kinesis Data Firehose destination:**
The same `logs:*` permissions listed above are required (ElastiCache uses CloudWatch Logs vended-log infrastructure for Firehose delivery). Ensure the Firehose delivery stream resource policy allows the `logs.amazonaws.com` service principal to deliver to it.
## Log Format
ElastiCache supports two log formats:
| Format | Use Case |
|--------|----------|
| `json` | Recommended. Structured, easy to parse with CloudWatch Insights, Athena, or log processing pipelines. |
| `text` | Human-readable. Useful for quick manual inspection via `valkey-cli` slow log format. |
### Sample Slow Log Entry (JSON)
```json
{
"CacheClusterId": "my-cluster-001",
"CacheNodeId": "0001",
"Id": 42,
"Timestamp": 1700000000,
"Duration (us)": 15230,
"Command": "SORT ... (8 more arguments)",
"ClientAddress": "10.0.1.50:54321",
"ClientName": "app-worker-3"
}
```
**Note:** ElastiCache replaces actual key names and values with `(N more arguments)` to avoid exposing sensitive data.
**Encrypt logs at rest.** Slow log and engine log entries still include client IP addresses, client names, and command verbs. Encrypt the target CloudWatch Logs log group with a KMS key (`aws logs associate-kms-key --log-group-name <name> --kms-key-id <arn>`) to protect this operational data at rest.
## Interpreting Slow Log Entries
The `Duration (us)` field is in microseconds.
| Duration | Severity | Action |
|----------|----------|--------|
| < 1,000 (1ms) | Normal | No action needed |
| 1,000 - 10,000 (1-10ms) | Warning | Investigate if frequent; may indicate suboptimal data access |
| 10,000 - 100,000 (10-100ms) | High | Identify the command and key; likely O(N) operation on a large collection |
| > 100,000 (100ms+) | Critical | Blocking the engine thread; fix immediately |
Common slow commands: `KEYS *` (scan all keys), `SMEMBERS` on large sets, `HGETALL` on large hashes, `SORT` without LIMIT, `LRANGE 0 -1` on long lists, `ZRANGEBYSCORE` returning thousands of members.
## Querying Logs
### CloudWatch Logs Insights
Find the slowest commands in the last 24 hours:
```
fields @timestamp, `Duration (us)`, Command
| filter `Duration (us)` > 10000
| sort `Duration (us)` desc
| limit 20
```
Find commands from a specific client:
```
fields @timestamp, `Duration (us)`, Command, ClientAddress
| filter ClientAddress like "10.0.1.50"
| sort @timestamp desc
| limit 50
```
Top 10 slowest command types in the last 24 hours:
```
fields Command, `Duration (us)`
| parse Command /^(?<cmd>\S+)/
| stats avg(`Duration (us)`) as avg_duration, max(`Duration (us)`) as max_duration, count(*) as invocations by cmd
| sort max_duration desc
| limit 10
```
Hourly slow command distribution (spot time-of-day patterns):
```
fields @timestamp, `Duration (us)`
| stats count(*) as slow_commands by bin(1h)
| sort bin asc
```
Per-node comparison (find the hot node):
```
fields CacheClusterId, `Duration (us)`, Command
| stats count(*) as slow_count, avg(`Duration (us)`) as avg_duration by CacheClusterId
| sort slow_count desc
```
## Recommendations
| Environment | Slow Log | Engine Log | Command Log |
|-------------|----------|------------|-------------|
| Production | Enable (JSON, CloudWatch Logs, 30-day retention) | Enable (JSON, CloudWatch Logs, 30-day retention) | Disable unless actively debugging |
| Staging | Enable | Enable | Enable selectively for integration testing |
| Development | Optional | Optional | Optional |
references/monitoring/slot-memory-imbalance-detection.md
# Slot Memory Imbalance Detection
**When to use:** Uneven memory across cluster nodes (one node near DatabaseMemoryUsagePercentage ceiling while siblings are well below), suspected hash-tag bundling, or planning resharding and want to identify which slots carry the most memory. Valkey 8.0+ cluster mode enabled only.
**When not needed:** Cluster-wide uniform memory pressure (troubleshooting.md Memory Pressure). Per-node CPU imbalance without memory skew (hot-key-detection.md). Serverless (abstracts slot topology).
## Preconditions
| Deployment | Applicability |
|---|---|
| Serverless | Does not apply. Slot topology is abstracted. |
| Node-based, cluster mode disabled | Does not apply (single-shard). Use big-key-hunter.md + key-space-distribution-by-prefix.md. |
| Node-based, cluster mode enabled | Full playbook. |
### Engine version gate
CLUSTER SLOT-STATS with MEMORY-BYTES requires **Valkey 8.0+** (this is a Valkey-only feature and is not available on any Redis OSS engine version). Verify:
```bash
valkey-cli -h <endpoint> -p 6379 --tls INFO server | grep -E 'redis_version|valkey_version'
```
**Pre-Valkey 8.0 degraded workflow:** 1) Per-shard CloudWatch triage (Tier A). 2) `valkey-cli --bigkeys` against the hot shard's primary. 3) `CLUSTER KEYSLOT <key>` per big-key candidate to find the heavy slot. 4) `MEMORY USAGE <key>` on top candidates for byte confirmation. Takes 15-30 minutes on a 10M-key shard vs. subsecond with SLOT-STATS.
### Feature flag
`cluster-slot-stats-enabled` must be `yes` in the parameter group. This is a Valkey 8.0+ parameter; verify it is available in your cache parameter group before relying on it. Note: This parameter may not appear in ElastiCache parameter groups documentation. It may be enabled by default or not exposed as a configurable parameter. `CONFIG GET` is restricted on all ElastiCache caches; check via the AWS API:
```bash
aws elasticache describe-cache-parameters \
--cache-parameter-group-name <parameter-group-name> \
--query "Parameters[?ParameterName=='cluster-slot-stats-enabled'].ParameterValue" \
--output text --region <region>
```
If missing or `no`, enable via parameter group modification (zero-downtime on most configs).
### Command availability
| Command | Supported | Notes |
|---|---|---|
| CLUSTER SLOT-STATS ORDERBY MEMORY-BYTES | Valkey 8.0+ cluster mode, NB only. Requires `cluster-slot-stats-enabled` parameter set to `yes`. Availability on ElastiCache depends on engine version and parameter group support. | Primary Tier B tool |
| CLUSTER SHARDS | NB (Valkey 7.2+ / Redis OSS 7.0+), SL (virtual shard) | Topology discovery |
| CLUSTER COUNTKEYSINSLOT \<slot\> | NB all versions | Per-slot key count |
| CLUSTER GETKEYSINSLOT \<slot\> \<count\> | NB all versions | Enumerate keys in a slot |
| CLUSTER KEYSLOT \<key\> | NB all versions | Hash key to slot number |
| MEMORY USAGE \<key\> | NB only, blocked on SL | Tier C byte confirmation |
---
## Tier A: Triage with CloudWatch
### Step 1: Per-node memory utilization
```bash
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache \
--metric-name DatabaseMemoryUsagePercentage \
--dimensions Name=CacheClusterId,Value=<node-id> Name=CacheNodeId,Value=0001 \
--start-time <24h-ago> --end-time <now> \
--period 300 --statistics Maximum --region <region>
```
Run for each primary (discover with `CLUSTER SHARDS`).
**Stop condition:** If all primaries are within 10 percentage points, memory is distributed uniformly. No slot-imbalance problem. Route to troubleshooting.md Memory Pressure if overall memory is high.
Proceed if spread is >15 percentage points (e.g., one node at 85%, another at 45%).
### Step 2: Confirm persistence
Query `BytesUsedForCache` per node over 24h (node-based, using `CacheClusterId` and `CacheNodeId` dimensions). For serverless, use `BytesUsedForCache` with the `ServerlessCacheName` dimension. A stable gap over hours is real. A 10-minute transient is not.
---
## Tier B: Narrow with CLUSTER SLOT-STATS
### Step 1: Identify memory-heavy slots
```bash
valkey-cli -h <endpoint> -p 6379 --tls \
CLUSTER SLOT-STATS ORDERBY MEMORY-BYTES DESC LIMIT 20
```
A single dominant slot (order of magnitude above peers) is a memory hotspot.
### Cost and impact
Subsecond, negligible impact. Reads stats Valkey already maintains. Safe at any time on production.
### Step 2: Map hot slots to shards
Use `CLUSTER SHARDS` output to map each hot slot to its primary. Hot slot on the already-hot node confirms the story.
### Step 3: Correlate with key count
```bash
valkey-cli -h <endpoint> -p 6379 --tls CLUSTER COUNTKEYSINSLOT <hot-slot>
```
| Pattern | Cause | Next step |
|---|---|---|
| Few keys, large memory | Big-key(s) in this slot | big-key-hunter.md |
| Many keys, large memory | Hash-tag concentration | Continue to Tier C |
**Stop condition:** If top 20 slots are within 2x of each other, no single hot slot. Consider rebalancing slot assignments rather than application-level remediation.
---
## Tier C: Verify with key-level inspection
### Step 1: Sample keys from the hot slot
```bash
valkey-cli -h <endpoint> -p 6379 --tls \
CLUSTER GETKEYSINSLOT <hot-slot> 200
```
### Step 2: Identify hash-tag pattern
Look for common `{...}` hash-tag prefix in returned keys. If most keys are `{tenant_42}:session:*`, `{tenant_42}:cache:*`, tenant 42 has concentrated its keys into a single slot.
### Step 3: Measure top keys
```bash
valkey-cli -h <endpoint> -p 6379 --tls MEMORY USAGE <key>
```
If one key dominates bytes, this is a big-key problem (route to big-key-hunter.md). If memory is evenly distributed, hash-tag concentration is the cause.
---
## Remediation
Pick based on Tier C classification:
| Cause | Remediation |
|---|---|
| **Hash-tag concentration** (default) | Redesign hash-tag: `{tenant_id}:{shard_id}` where shard_id is a random 4-bit prefix. Spreads keys across 16 slots while keeping intra-tenant MULTI/LUA local. Application change required. |
| **Oversized tenant** | Split to dedicated cache. For tenants using >30% of cluster memory. |
| **Big key in hot slot** | big-key-hunter.md remediation (decomposition, compression, move to S3). |
| **Slot-count skew** | Manual slot rebalance via online resharding (`modify-replication-group-shard-configuration`). |
| **Orphan keys** | Bulk UNLINK after confirming via key-space-distribution-by-prefix.md TTL audit. |
| **Scale up** (last resort) | Larger node type. Only masks the symptom. |
---
## Cross-links
- Hot slot also has high CPU: `hot-key-detection.md` Tier B (same SLOT-STATS, ORDERBY CPU-USEC axis)
- Specific key in slot is oversized: `big-key-hunter.md`
- Concentration is prefix-based, not slot-based: `key-space-distribution-by-prefix.md`
- Alarm patterns for memory imbalance: `alarm-packs.md`
references/monitoring/slow-log-cross-signal-diagnosis.md
# Slow Log Cross-Signal Diagnosis
**When to use:** Intermittent latency spikes, customer-facing timeouts, or paged alerts where EngineCPUUtilization rose briefly on one node. Wants to identify which commands caused the spike, from which clients, against which keys. Node-based clusters with slow-log delivery enabled.
**When not needed:** Sustained latency (troubleshooting.md High Latency). Slow-log not enabled (set up via log-delivery.md). Serverless (slow log not available; use troubleshooting.md Throttling + CloudWatch latency metrics).
## Preconditions
| Deployment | Applicability |
|---|---|
| Serverless | Does not apply. SLOWLOG, CLIENT LIST, INFO commandstats all blocked. |
| Node-based | Full playbook. Slow-log delivery must be enabled (to CloudWatch Logs or Amazon Data Firehose). This playbook assumes CloudWatch Logs as the destination; if using Firehose, adapt the log-query steps accordingly. |
Verify slow-log delivery is configured (the log group name is user-specified, not a fixed path):
```bash
aws elasticache describe-replication-groups \
--replication-group-id <replication-group-id> \
--region <region> \
--query 'ReplicationGroups[0].LogDeliveryConfigurations'
```
Look for an entry with `LogType: slow-log` and note the `DestinationDetails` (CloudWatch Logs log group name or Firehose stream name). If the destination is Firehose rather than CloudWatch Logs, the `aws logs filter-log-events` commands in Tier B will not apply; query your Firehose destination (e.g., S3, OpenSearch) instead.
**TLS note:** All `valkey-cli` examples in this playbook include `--tls`. Omit `--tls` if your cluster does not have transit encryption enabled.
### Command availability (node-based only)
| Command | Notes |
|---|---|
| SLOWLOG GET [count] | Read slow-log entries. Without a count argument, returns the latest 10 entries by default. Use `SLOWLOG GET -1` to retrieve all entries up to `slowlog-max-len` (default 128). |
| SLOWLOG LEN | Total entries stored. |
| COMMANDLOG GET \<count\> | Valkey 8.1+ only. Richer categorization (slow-command, large-reply, large-request). |
| INFO commandstats | Per-command call count, total microseconds, microseconds-per-call. |
| CLIENT LIST [TYPE \<type\>] | Enumerate clients. Parse for cmd=, addr=, age=, idle=, name=. |
---
## Tier A: Triage with CloudWatch
Pin the incident window before pulling logs. CloudWatch only, zero cluster impact.
### Step 1: Confirm the spike window
Query `EngineCPUUtilization` per node at 1-minute resolution:
```bash
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache --metric-name EngineCPUUtilization \
--dimensions Name=CacheClusterId,Value=<node-id> \
--start-time <incident-start> --end-time <incident-end> \
--period 60 --statistics Maximum --region <region>
```
Identify the node and exact 5-minute window where CPU peaked.
**Stop condition:** If no node shows a CPU peak during the reported time range, the latency is not engine-thread-bound. Route to troubleshooting.md High Latency or Connection Spikes.
### Step 2: Confirm latency correlation
Query `SuccessfulReadRequestLatency` at p99 over the same window:
```bash
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache --metric-name SuccessfulReadRequestLatency \
--dimensions Name=CacheClusterId,Value=<node-id> \
--start-time <window-start> --end-time <window-end> \
--period 60 --extended-statistics p99 --region <region>
```
P99 spike aligned with CPU spike confirms engine-bound nature.
### Step 3: Identify command-family signature
```bash
for family in StringBasedCmds HashBasedCmds ListBasedCmds SetBasedCmds SortedSetBasedCmds StreamBasedCmds; do
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache --metric-name $family \
--dimensions Name=CacheClusterId,Value=<node-id> \
--start-time <window-start> --end-time <window-end> \
--period 60 --statistics Sum --region <region>
done
```
The family that spiked most sharply points to the data type of the slow command.
---
## Tier B: Narrow with slow-log correlation
### Step 1: Pull slow-log entries for the window
```bash
aws logs filter-log-events \
--log-group-name <your-log-group-name> \
--log-stream-name-prefix elasticache/<engine-name>/<cache-cluster-id> \
--start-time <window-start-ms> --end-time <window-end-ms> \
--region <region>
```
Replace `<your-log-group-name>` with the log group from the precondition check. Log stream names follow the format `elasticache/${engine-name}/${cache-cluster-id}/${cache-node-id}/${log-type}`.
Each entry includes: timestamp, duration (microseconds), client address (`ClientAddress`), and command name. Note: in delivered logs (CloudWatch Logs / Firehose), ElastiCache redacts key names and values, replacing them with `(N more arguments)` to avoid exposing sensitive data. To see full command arguments including keys, use `SLOWLOG GET` directly on the node. Sort by duration descending.
### Step 2: Cross-reference with INFO commandstats
```bash
valkey-cli -h <node-endpoint> -p 6379 --tls INFO commandstats
```
A command that appears repeatedly in the slow-log window AND has high usec_per_call in commandstats is the culprit.
**Stop condition:** If no command exceeds both signals, the spike is not from a recurring slow command. Route to hot-key-detection.md (single-key spike) or troubleshooting.md High Latency.
### Step 3: Identify the key pattern
In delivered logs (CloudWatch Logs / Firehose), key names are redacted. To identify key patterns, use `SLOWLOG GET` directly on the node, which returns full command arguments including keys:
```bash
valkey-cli -h <node-endpoint> -p 6379 --tls SLOWLOG GET 128
```
The first argument of most entries is the key. Group by first-argument prefix. A key or prefix appearing repeatedly is the target.
### Step 4: COMMANDLOG (Valkey 8.1+ only)
Check engine version first. COMMANDLOG is Valkey 8.1+ only. On older engines, skip this step.
```bash
valkey-cli -h <node-endpoint> -p 6379 --tls COMMANDLOG GET 128
```
Categories: `slow-command` (runtime), `large-reply` (oversized response), `large-request` (oversized argument). Tells you WHY a command was slow, not just that it was.
### Cost and impact
All Tier B operations are read-only and subsecond. 2-5 minutes of active querying per incident. Safe on production.
---
## Tier C: Verify with CLIENT LIST and key inspection
### Step 1: Identify the client
Slow-log entries include a `ClientAddress` field (e.g., `"ClientAddress": "10.0.1.42:50234"`). Match against CLIENT LIST:
```bash
valkey-cli -h <node-endpoint> -p 6379 --tls CLIENT LIST
```
Parse for matching `addr=`. The `name=` field identifies the app instance if CLIENT SETNAME is used. Recommend `CLIENT SETNAME <app-instance-id>` on every connection as a convention.
**Stop condition:** If no matching connection (already closed), fall back to VPC flow logs keyed to source IP:port. Do not spend more than 10 minutes on CLIENT LIST attribution alone.
### Step 2: Inspect the target key
```bash
valkey-cli -h <node-endpoint> -p 6379 --tls TYPE <key>
valkey-cli -h <node-endpoint> -p 6379 --tls MEMORY USAGE <key>
```
Cross-reference with big-key-hunter.md if the key is large.
### Step 3: Classify the root cause
| Slow-log signature | Tier C findings | Root cause |
|---|---|---|
| HGETALL key 50x in window | 5MB hash, client is batch job | Big-key read by batch job |
| KEYS pattern:* once | Script or admin operation | Blocking enumeration |
| LRANGE key 0 -1 30x | 100K-item list | Big-list without pagination |
| SORT key | Large sortable collection | O(N log N) sort on prod path |
| EVAL script 100x | High per-call microseconds | Lua script complexity |
| SUBSCRIBE + blocked client | cmd=subscribe age>3600 | Long-held pub/sub subscriber |
---
## Remediation
| Root cause | Fix |
|---|---|
| Big-key read by batch job | Move batch to read replica, paginate (HSCAN/LRANGE with bounds), split value |
| Blocking enumeration (KEYS, SMEMBERS on large sets) | Replace with SCAN/SSCAN/HSCAN/ZSCAN |
| Big-list O(N) reads | Paginate with cursor or bounded LRANGE |
| Expensive Lua script | Profile, reduce allocations, use EVALSHA, move logic to app |
| Long-held blocking commands | Adjust timeout, pool blocked connections separately |
| Operator mistake | Document anti-pattern in team runbooks |
---
## Cross-links
- Root cause is a big key: `big-key-hunter.md`
- Root cause is a hot key (same key, high access): `hot-key-detection.md`
- Concentrated on one shard: `slot-memory-imbalance-detection.md` or `hot-key-detection.md`
- Enabling slow-log delivery: `log-delivery.md`
- Alarm patterns for catching spikes: `alarm-packs.md`
references/monitoring/troubleshooting.md
# Troubleshooting Playbooks
Symptom-based runbooks for diagnosing and resolving common ElastiCache operational issues.
## Diagnostic Quick Reference
| Symptom | First Metric to Check | First Action | Deployment |
|---------|----------------------|--------------|------------|
| Slow responses | `EngineCPUUtilization` | Check slow log for expensive commands | Both |
| Low hit rate | `CacheHitRate`, `Evictions` | Verify TTLs and eviction policy | Both |
| Connection errors | `CurrConnections` | Check security groups and connection pooling | Both |
| High CPU | `EngineCPUUtilization` per node | Look for hot keys or expensive commands | Node-based |
| OOM / write failures | `DatabaseMemoryUsagePercentage` | Check eviction policy and key TTLs | Node-based |
| Stale reads | `ReplicationLag` | Check write volume and replica capacity | Node-based |
| Throttled (serverless) | `ThrottledCmds`, `ElastiCacheProcessingUnits` | Increase ECPU limit | Serverless |
| Connection drops | `describe-events` | Check for failover events, verify retry logic | Both |
---
## CLI Template for Metric Checks
All investigation steps below reference metrics by name. Use this template, substituting the metric name and dimensions.
```bash
aws cloudwatch get-metric-statistics \
--namespace AWS/ElastiCache --metric-name <metric-name> \
--dimensions Name=<dimension-name>,Value=<dimension-value> \
--start-time <start> --end-time <now> --period <seconds> --statistics <stat> \
--region <region>
```
Common dimensions: `ReplicationGroupId` (cluster-wide), `CacheClusterId` (per node), `ServerlessCacheName` (serverless).
---
## Metric Units Quick Reference
| Metric | Deployment | Unit | Common Mistake |
|--------|------------|------|----------------|
| ReplicationLag | Node-based | Seconds | Not milliseconds. Threshold of 1 = 1 second. |
| SuccessfulReadRequestLatency | Both | Microseconds | Not milliseconds. 5000 = 5ms. |
| SuccessfulWriteRequestLatency | Both | Microseconds | Not milliseconds. 5000 = 5ms. |
| BytesUsedForCache | Both | Bytes | Not GB. 1 GB = 1,073,741,824 bytes. |
| ElastiCacheProcessingUnits | Serverless | Count (ECPUs) | Sum over period, not rate. |
| ThrottledCmds | Serverless | Count | Sum over period. |
| DatabaseMemoryUsagePercentage | Node-based | Percent | Not a ratio. 80 = 80%. |
| DatabaseCapacityUsagePercentage | Node-based | Percent | Available on all node-based clusters. On data-tiering instances (r6gd), the formula includes SSD storage; on all other instances, it is calculated as `used_memory/maxmemory`. |
| EngineCPUUtilization | Node-based | Percent | Single-threaded engine; can hit 100% on one core while host CPU is low. |
| CacheHitRate | Both | Percent | Empty until cache serves real traffic. |
| Slow log duration | Both | Microseconds | Not milliseconds. 10000 = 10ms. |
## Metrics Return Empty
New caches take 5-10 minutes to emit their first CloudWatch datapoints. Serverless caches with zero traffic emit no metrics at all (this is normal, not broken). After confirming the cache exists with `describe-serverless-caches` or `describe-replication-groups`, wait for traffic before investigating further.
---
## High Latency
**Applies to:** Both serverless and node-based. **Severity:** P1 (fix now)
| Aspect | Detail |
|--------|--------|
| Symptoms | `SuccessfulReadRequestLatency` or `SuccessfulWriteRequestLatency` elevated |
| Key metrics | `EngineCPUUtilization`, `SuccessfulReadRequestLatency`, `SuccessfulWriteRequestLatency`, `CurrConnections` |
### Likely Causes
1. **Slow commands**: `KEYS *`, `SMEMBERS` on large sets, `HGETALL` on large hashes, `SORT` on large lists
2. **Network latency**: cross-AZ traffic, VPC peering hops, insufficient bandwidth
3. **TLS overhead**: handshake on every new connection (mitigated by connection reuse)
4. **Insufficient capacity**: node CPU/memory saturated, or serverless ECPU limit reached
5. **Hot key**: single key bottlenecking a single shard
### Investigation Steps
Check: `EngineCPUUtilization` (per node, Maximum), `CurrConnections` (Maximum). For slow log:
```bash
aws logs filter-log-events --log-group-name <your-slow-log-group> \
--region <region>
```
**Stop condition:** If `EngineCPUUtilization` is below 60% and `CurrConnections` is stable, this is not a capacity or connection problem. Check slow log for expensive commands next.
### Resolution
- **Slow commands**: replace `KEYS` with `SCAN`, use `SSCAN`/`HSCAN`/`ZSCAN` for large data structures
- **Network**: colocate app and cache in same AZ, use reader endpoints for read-heavy workloads
- **TLS**: enable connection pooling and reuse connections
- **Insufficient capacity**: scale up node type, add shards, or increase serverless ECPU limits
- **Hot key**: distribute across multiple logical keys with client-side sharding, or add read replicas. For detailed diagnosis, see `hot-key-detection.md`. For intermittent spikes with slow-log correlation, see `slow-log-cross-signal-diagnosis.md`.
## Low Hit Rate
**Applies to:** Both serverless and node-based. **Severity:** P2 (fix next business day)
| Aspect | Detail |
|--------|--------|
| Symptoms | `CacheHitRate` below 80%, backing store load not decreasing |
| Key metrics | `CacheHits`, `CacheMisses`, `CacheHitRate`, `Evictions`, `DatabaseMemoryUsagePercentage` |
### Likely Causes
1. **TTL too short**: keys expire before reuse
2. **Wrong eviction policy**: keys evicted before natural expiry
3. **Cold cache**: recently created or restarted, not yet populated
4. **Key space mismatch**: different key naming between write and read paths
5. **Cache stampede**: concurrent requests miss simultaneously, overwhelming the backend
### Investigation Steps
Check: `Evictions` (Sum), `DatabaseMemoryUsagePercentage` (Maximum), `CacheHitRate` (Average). All at ReplicationGroupId level.
**Stop condition:** If `Evictions` is zero and `DatabaseMemoryUsagePercentage` is below 70%, eviction is not the cause. Focus on key naming and TTL mismatch.
### Resolution
- **TTL too short**: increase TTL. Start with 5-15 min for frequently changing data, 1-24 hours for reference data.
- **Wrong eviction policy** (node-based only): use `allkeys-lru` for general caching. Use `volatile-lru` only if some keys must never be evicted. **Note:** Serverless caches use `volatile-lru` and this is not configurable; ensure all keys have TTLs set to enable eviction on serverless.
- **Cold cache**: implement cache warming on deployment or restart.
- **Key space mismatch**: audit key naming in write and read paths. Ensure derivation is deterministic and identical.
- **Cache stampede**: implement locking (SETNX-based) so only one request populates on miss, or use "stale while revalidate" TTLs.
## Connection Spikes
**Applies to:** Both serverless and node-based. **Severity:** P1 (fix now)
| Aspect | Detail |
|--------|--------|
| Symptoms | `CurrConnections` spikes, `NewConnections` elevated, connection errors in app logs |
| Key metrics | `CurrConnections`, `NewConnections`, `EngineCPUUtilization` |
### Likely Causes
1. **Connection storm**: app restart causing all instances to connect simultaneously
2. **Missing connection pooling**: each request opens a new connection
3. **Lambda cold starts**: each cold start establishes a new connection
4. **Connection leaks**: connections not returned to pool
### Investigation Steps
Check: `CurrConnections` (Maximum, period 60s), `NewConnections` (Sum, period 60s). Compare per-node to find imbalance.
### Resolution
- **Connection storm**: implement backoff with jitter on startup
- **Missing pooling**: use connection pools (`max_connections` in client library)
- **Lambda cold starts**: initialize connection outside handler. Consider provisioned concurrency or serverless cache.
- **Connection leaks**: add health checks, set idle timeout, implement cleanup in `finally` blocks
## High CPU
**Applies to:** Node-based only. For serverless, see Throttling. **Severity:** P1 (fix now)
| Aspect | Detail |
|--------|--------|
| Symptoms | `EngineCPUUtilization` above 90%, increased latency |
| Key metrics | `EngineCPUUtilization`, `CPUUtilization`, `CurrConnections` |
### Important: CPUUtilization vs EngineCPUUtilization
`EngineCPUUtilization` measures only the main Redis/Valkey engine thread and is the recommended metric for monitoring engine capacity. `CPUUtilization` reflects aggregate CPU usage across all cores, including dedicated I/O threads from Enhanced I/O features, and is **not** a reliable indicator of engine capacity for nodes with 4+ vCPUs. For nodes with 2 or fewer vCPUs, use `CPUUtilization` with a threshold of 90 divided by the number of cores (e.g., 45% for 2-core nodes). For nodes with 4+ vCPUs, use `EngineCPUUtilization` with a 90% threshold.
### Likely Causes
1. **Hot key**: one key receiving disproportionate traffic on a single shard
2. **Expensive commands**: `KEYS`, `SORT`, `SUNIONSTORE` on large datasets, complex Lua scripts
3. **Insufficient shards**: workload concentrated on too few shards
4. **High connection churn**: TLS handshake overhead with many short-lived connections
### Investigation Steps
Check: `EngineCPUUtilization` (per node via CacheClusterId, Maximum, period 60s). Compare across nodes to find hot shards. For slow log:
```bash
aws logs filter-log-events --log-group-name <your-slow-log-group> \
--region <region>
```
Also check command-family metrics (`StringBasedCmds`, `HashBasedCmds`, etc.) on the hot node to identify the data type driving load before scanning slow logs.
### Resolution
- **Hot key**: use client-side caching, replicate key with suffixes (`hotkey:1`, `hotkey:2`) and load-balance reads
- **Expensive commands**: replace `KEYS` with `SCAN`, use `SSCAN`/`HSCAN`/`ZSCAN`, limit Lua complexity
- **Insufficient shards**: add shards (`modify-replication-group-shard-configuration`), rebalance slots
- **Connection churn**: implement connection pooling, use keep-alive
## Memory Pressure
**Applies to:** Node-based primarily. Serverless auto-scales storage but can hit configured limits. **Severity:** P1 (fix now)
| Aspect | Detail |
|--------|--------|
| Symptoms | `DatabaseMemoryUsagePercentage` above 80%, `Evictions` increasing, OOM errors |
| Key metrics | `DatabaseMemoryUsagePercentage`, `BytesUsedForCache`, `Evictions`, `CurrItems`. For data-tiering instances (r6gd), also check `DatabaseCapacityUsagePercentage` which covers both memory and SSD tiers. |
### Likely Causes
1. **No eviction policy**: `maxmemory-policy` set to `noeviction`, rejects writes when full
2. **Large keys**: individual keys in the MB range consuming disproportionate memory
3. **Memory fragmentation**: high `MemoryFragmentationRatio`
4. **TTL not set**: keys persist indefinitely
### Investigation Steps
Check: `DatabaseMemoryUsagePercentage` (Maximum, period 3600s over 7d), `Evictions` (Sum, period 3600s over 7d). To check eviction policy:
```bash
aws elasticache describe-cache-parameters \
--cache-parameter-group-name <parameter-group> \
--region <region> \
--query "Parameters[?ParameterName=='maxmemory-policy']"
```
**Stop condition:** If `Evictions` is zero and memory is rising slowly, this may be normal growth. Only act if approaching 80% or evictions are sustained.
### Resolution
- **No eviction policy**: set `maxmemory-policy` to `allkeys-lru`
- **Large keys**: break into smaller keys, compress values, use hashes instead of large JSON blobs. For detailed big-key diagnosis, see `big-key-hunter.md`.
- **Memory fragmentation**: restart during maintenance window. For node-based, set `activedefrag` to `yes` in the parameter group (CONFIG SET is restricted on ElastiCache).
- **TTL not set**: audit key creation code, add TTLs. Use `OBJECT IDLETIME` to find stale keys.
- **Reserved memory**: set `reserved-memory-percent` to 25 (for Valkey and Redis OSS 2.8.22+) or 50 (for older Redis OSS versions) via a custom parameter group. The default is 25% for all parameter group families (available since March 2017). For accounts created before March 16, 2017, older parameter families may default to 0. A value of 0 allows the engine to consume all of `maxmemory` with data, leaving insufficient memory for background write processes (BGSAVE, replication sync), which can cause snapshot and sync failures.
- **Scale up**: larger node type or add shards to distribute data
## Replication Lag
**Applies to:** Node-based only. Serverless handles replication internally. **Severity:** P2 (fix next business day, P1 if > 5s)
| Aspect | Detail |
|--------|--------|
| Symptoms | `ReplicationLag` above 1s, stale reads from replicas, failover risk |
| Key metrics | `ReplicationLag`, `ReplicationBytes`, `NetworkBytesOut` (primary), `SaveInProgress` |
### Likely Causes
1. **Write-heavy workload**: primary generating data faster than replicas can consume
2. **Insufficient replica capacity**: replica node type too small
3. **Network bandwidth**: replication traffic saturating the interface
4. **Background save**: `BGSAVE`/snapshot operations competing for I/O
### Investigation Steps
Check: `ReplicationLag` (per replica via CacheClusterId, Maximum, period 60s), `SaveInProgress` (per primary, Maximum, period 60s).
**Stop condition:** If `SaveInProgress` is 1 and lag correlates with snapshot timing, this is snapshot-induced. Schedule snapshots during off-peak rather than scaling.
### Resolution
- **Write-heavy workload**: add shards to distribute writes, or batch writes using pipelining
- **Insufficient replica capacity**: scale up replica node type (must match primary in most configs)
- **Network bandwidth**: move to a larger node type
- **Background save**: schedule snapshots during low-traffic periods
## Throttling (Serverless)
**Applies to:** Serverless only. **Severity:** P1 (fix now)
| Aspect | Detail |
|--------|--------|
| Symptoms | `ThrottledCmds` increasing, throttle errors in clients, `ElastiCacheProcessingUnits` near max |
| Key metrics | `ThrottledCmds`, `ElastiCacheProcessingUnits` |
### Likely Causes
1. **ECPU limit reached**: `CacheUsageLimits.ECPUPerSecond.Maximum` too low for current workload
2. **Burst traffic**: sudden spike exceeding configured maximum
3. **Expensive commands**: single commands consuming many ECPUs (e.g., `SORT`, `ZRANGEBYSCORE` returning thousands)
### Investigation Steps
Check: `ElastiCacheProcessingUnits` (Sum, period 60s), `ThrottledCmds` (Sum, period 60s). Use ServerlessCacheName dimension.
### Resolution
- **Increase ECPU limit**:
```bash
aws elasticache modify-serverless-cache \
--serverless-cache-name <name> \
--cache-usage-limits '{"ECPUPerSecond": {"Maximum": <higher-value>}}' \
--region <region>
```
- **Optimize commands**: replace expensive operations with efficient alternatives, use pipelining
- **Cost concern**: consider node-based for steady-state high-throughput with reserved pricing
- **Add client-side retry**: exponential backoff for throttled commands
## Failover Events
**Applies to:** Node-based primarily. Serverless handles failover transparently. **Severity:** P1 (fix now)
| Aspect | Detail |
|--------|--------|
| Symptoms | Brief connection interruption, DNS endpoint change |
| Key metrics | `ReplicationLag` (pre-failover), events from `describe-events` |
### Likely Causes
1. **Node failure**: hardware or software issue on the primary
2. **AZ failure**: entire Availability Zone unavailable
3. **Maintenance**: service update applied during maintenance window
4. **Manual failover**: operator-initiated via `test-failover`
### Investigation Steps
Check recent events:
```bash
aws elasticache describe-events \
--source-type replication-group \
--source-identifier <replication-group-id> \
--duration 1440 \
--region <region>
```
Also check `ReplicationLag` (Maximum) before the event; high lag means potential data loss.
### Resolution
- **Ensure Multi-AZ is enabled**: automatic failover; failover time varies depending on cluster size and configuration
- **Application resilience**: retry with backoff. Most client libraries handle failover with cluster-aware config.
- **DNS caching**: set app DNS TTL to 5 seconds or less
- **Post-failover**: verify replication is healthy, confirm new primary is in expected AZ
- **Prevent data loss**: keep `ReplicationLag` low. High lag before failover means unreplicated writes are lost.
## Cannot Connect
**Applies to:** Both serverless and node-based. **Severity:** P1 (fix now)
| Aspect | Detail |
|--------|--------|
| Symptoms | Connection timeout, connection refused, TLS handshake failure |
| Key metrics | `CurrConnections` (if any succeed), `describe-events` for failover |
### Decision Fork
- **Never worked (new setup):** Security group, subnet, or endpoint misconfiguration. Check the setup sub-skill's connectivity-diagnostics.
- **Was working, now broken:** Recent change or failover event.
### Likely Causes (was working, now broken)
1. **Security group change**: inbound rule removed or modified
2. **Failover event**: DNS endpoint changed, client caching old IP
3. **Auth credential rotation**: password or IAM token expired
4. **TLS certificate mismatch**: client missing CA bundle or wrong ssl_cert_reqs setting
5. **Network change**: VPC peering route removed, subnet NACL modified
6. **maxclients reached**: some connections succeed but new ones are rejected. Check `CurrConnections` against node's `maxclients` (default 65,000 for most instance types; exceptions: t2.micro/small/medium and t3.micro = 20,000; t3.small/medium = 46,000; t4g.micro = 20,000)
### Investigation Steps
Check `describe-events` for recent failover. Verify security group allows inbound on port 6379 from the client's security group. Test with:
```bash
valkey-cli -h <endpoint> -p 6379 --tls PING
```
If TLS fails, try without TLS to isolate (note: serverless always requires TLS).
### Resolution
- **Security group**: add inbound rule for port 6379 from client SG
- **Failover DNS**: set client DNS TTL to 5s or less; restart application to pick up new IP
- **Auth expired**: rotate credentials in Secrets Manager or refresh IAM token
- **TLS**: ensure client uses `ssl=True` and the correct CA bundle (Amazon root CA)
- **Never worked**: route to `setup/connectivity-diagnostics.md`
references/requirements/instructions.md
# Solution Fit
Gather enough context to recommend the right ElastiCache configuration, then hand off to the appropriate sub-skill.
## Loading
Read this file first. Load `scripts/price_calculator.py` when estimating cost.
## Check for existing context
Before starting, check if `.elasticache/requirements.json` exists from a previous session. If it does, read it, present the values to the user, and ask if anything has changed. If confirmed accurate, skip directly to **Hand off**.
## Fast path (skip the interview)
If the user already knows what they want, don't interview them. Run the workspace scan silently, then route:
- User names a specific product or engine ("create a Valkey serverless cache") -> skip to **Summarize and hand off**. Ask only for missing critical values (region, VPC). Serverless supports Valkey 7.2, 8.0, and 8.1. If user requests Redis OSS, note that Valkey is the recommended forward path. Redis OSS remains available (highest version on ElastiCache: 7.1), but Valkey receives new feature investments (vector search, Bloom filters, memory-efficient hash table, COMMANDLOG, and more). If the user needs vector search (semantic caching, agentic memory, RAG), note that Valkey 8.2+ is required and is available on node-based clusters only (not available on Serverless or data-tiering r6gd instances).
- User names a specific pattern ("session store", "rate limit my API", "semantic cache for Bedrock") -> confirm inferences from scan, skip to **Summarize and hand off**.
- User names a specific task ("migrate from Redis to Valkey", "check my cache costs") -> route directly to matching sub-skill. Do not run this flow.
## Full flow (for unclear or exploratory requests)
### 1. Workspace Scan (always run first, silently)
Scan the user's workspace for: language/framework, compute/deployment model, networking/region, cache commands, and existing caching solutions. If workspace has docker-compose with a Redis image, redis.conf, .env with Redis URLs, or ElastiCache in existing IaC, route to `migration`. Map discovered commands to patterns:
| Commands in code | Likely pattern |
|-----------------|----------------|
| INCR, EXPIRE, DECR | Rate limiting or counters |
| HSET, HGET, HMSET, HMGET | Sessions, carts, or profile/state |
| ZADD, ZRANGE, ZRANGEBYSCORE | Leaderboards or ranking |
| XADD, XREADGROUP, XACK | Streams or durable queues |
| PUBLISH, SUBSCRIBE | Real-time messaging / pub/sub |
| GET, SET, SETEX with serialized values | Cache-aside / query caching |
| Embedding generation, vector similarity | GenAI (route to `genai`) |
### 2. Present Inferences
Present what you found in a single confirmation block with evidence from file paths.
Inference safety hierarchy:
- **Low risk** (use as default, mention but don't ask): Language, framework, region
- **Medium risk** (present and ask to confirm): Use case pattern, deployment model
- **High risk** (always confirm explicitly): Engine choice, security posture, multi-region
If the workspace scan reveals nothing, skip to step 3.
### 3. Evaluate fit
If caching with ElastiCache is clearly the right approach, skip to step 4.
If the user is unsure, evaluate:
**When ElastiCache is NOT the right fit:**
| Symptom | Better alternative | Can ElastiCache complement it? |
|---------|-------------------|-------------------------------|
| Need complex queries, joins, ACID transactions | RDS, Aurora | Yes, as a read cache in front of it |
| Need a DynamoDB-specific transparent cache | DAX | Yes, as a general cross-service cache alongside DAX |
| Need CDN / static content delivery | CloudFront | Yes, as an app-layer cache behind CloudFront |
| Need a durable message queue with exactly-once delivery | SQS | Yes, for rate limiting in front of SQS |
| Need an event bus with routing rules | EventBridge | No |
| Need search over massive archival datasets | OpenSearch | Yes, as a fast real-time layer alongside OpenSearch |
| Slow DB reads but every query is unique | DB indexing, read replicas, query optimization | No |
| Need a durable primary database with Redis/Valkey API compatibility | MemoryDB | No, use MemoryDB instead of ElastiCache when the workload requires a durable primary database with microsecond reads and single-digit ms writes |
**Common "use both" patterns:**
- ElastiCache + RDS/Aurora: Cache-aside for read acceleration
- ElastiCache + DynamoDB: General cross-service cache layer
- ElastiCache + Bedrock: Semantic cache to reduce LLM cost and latency
- ElastiCache + OpenSearch: Fast real-time search layer with OpenSearch for archival analytics
- ElastiCache + SQS: Rate limiting and deduplication in front of a queue
**Shared or application-local?** (only when the role is cache layer)
| Factor | Application-local sufficient | Shared (ElastiCache) needed |
|--------|------------------------------|----------------------------|
| Multiple app instances need the same data | No | Yes |
| Data must survive app restarts | No | Yes |
| Atomic operations across instances | No | Yes |
| Rich data structures (sorted sets, streams, hashes) | No | Yes |
| Single instance, simple memoization | Yes | Overkill |
If application-local is sufficient, recommend it and stop.
### 4. Route or ask
If the scan + user's message reveal the job, route immediately. Otherwise ask remaining questions 2-3 at a time, skipping anything the scan already answered.
**Questions (only what remains unknown):**
| Category | Question |
|----------|----------|
| Goal | Build new, migrate existing, or troubleshoot? |
| Data source | What is your primary data store? |
| Multi-region | Single-region or multi-region? |
| AI workload | Working with embeddings, LLM inference, or agent memory? |
| Compliance | Any regulatory requirements (HIPAA, PCI DSS, SOC 2, FedRAMP)? |
| Traffic | Expected request rate? Steady or spiky? |
| Connections | Concurrent connections? |
| Data size | GB of cached data? |
| Latency | Sub-millisecond critical, or single-digit ms acceptable? |
| Staleness | How stale can the data be? |
| Budget | Cost sensitive? |
**Signal-to-route mapping:**
| User signal | Route to |
|------------|----------|
| Slow database reads | `setup` + `data-modeling` (cache-aside) |
| Session storage across instances | `setup` + `data-modeling` (session) |
| Real-time rankings or scoring | `setup` + `data-modeling` (leaderboard) |
| API protection or throttling | `setup` + `data-modeling` (rate limiter) |
| Reducing LLM costs or latency | `setup` + `genai` (semantic cache); server-side path requires Valkey 8.2 or above, app-side works on any version |
| AI agent memory across sessions | `setup` + `genai` (conversational memory), requires Valkey 8.2+ |
| Search or recommendations | `setup` + `genai` (vector search / RAG), requires Valkey 8.2+ (node-based clusters only) |
| Real-time event distribution | `setup` + `data-modeling` (pub/sub or streams) |
**Compliance-to-configuration mapping:**
If the user mentions a compliance framework, these ElastiCache settings are non-negotiable:
| Framework | Required configuration |
|---|---|
| HIPAA | In-transit encryption (TLS), at-rest encryption, RBAC or IAM auth, Multi-AZ, slow log + engine log delivery enabled, CloudTrail API logging, VPC-only access |
| PCI DSS | In-transit encryption (TLS), at-rest encryption, RBAC or IAM auth, VPC-only access, no public endpoints, slow log + engine log delivery enabled, CloudTrail API logging, key rotation via Secrets Manager |
| SOC 2 | In-transit encryption (TLS), at-rest encryption, RBAC or IAM auth, CloudTrail enabled, security-relevant metric alarms (e.g., AuthenticationFailures, NewConnections) + CloudTrail |
| FedRAMP | All HIPAA requirements plus: GovCloud region, FIPS endpoints, Config rules for drift detection |
When compliance is flagged, pass the requirement to `setup` so it enforces encryption and auth from the start. At-rest encryption cannot be added after cluster creation; getting this wrong requires a full cluster recreation.
**AWS-stack translation table:**
| AWS context | Default ElastiCache choice |
|---|---|
| RDS or Aurora read-heavy app | Query caching / cache-aside |
| DynamoDB read-heavy app | ElastiCache cache-aside. Mention DAX only if user wants DynamoDB-specific transparent cache. |
| Lambda | VPC connectivity required. Default serverless unless node-based requirement applies. |
| ECS, EKS, or EC2 | Pick pattern per signal above. |
| Bedrock or AgentCore | Route to `genai`, classify Mode 1/2/3. |
| Global app footprint (multi-region) | Node-based with Global Datastore. Limited to specific instance families (M5, M6g, M7g, R5, R6g, R6gd, R7g, C7gn in size large+), max 2 secondary regions, no autofailover across regions (manual promotion only), no IPv6, no Local Zones. |
## Summarize and hand off
Summarize what you learned. Mark inferred values as "(inferred)" so the user can correct them.
### Save requirements artifact
After the user confirms, write to `.elasticache/requirements.json` in the project root. Create `.elasticache/` if needed. Use `null` for undetermined values. If the file exists with an `infrastructure` section, preserve it.
```json
{
"use_case": "cache-aside for Aurora read acceleration",
"patterns": ["cache-aside"],
"runtime": { "language": "python", "framework": "fastapi", "compute": "ecs" },
"region": "us-east-1",
"engine": "valkey",
"deployment_model": null,
"data_source": "aurora-postgresql",
"multi_region": false,
"ai_workload": false,
"next_steps": ["setup", "data-modeling"],
"infrastructure": null
}
```
### Hand off
1. `setup` to create and connect
2. `data-modeling` or `genai` to implement the pattern
If the user's request spans multiple patterns, note that both can run on the same cache and plan the data model accordingly.
## Freshness disclaimer
When your response includes pricing, version constraints, or feature availability, include the freshness disclaimer per SKILL.md Global Rule #5: "For current pricing see https://aws.amazon.com/elasticache/pricing/. For current feature availability see https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/."
references/setup/auth-model-selector.md
# Auth Model Selector
Decision guide for choosing the right authentication model for ElastiCache.
## Auth Model Comparison
| Auth Model | Where It Fits | Strengths | Constraints | Recommendation |
|------------|--------------|-----------|-------------|----------------|
| IAM auth + RBAC | Valkey 7.2+ / Redis OSS 7.0+ with cloud-native clients (Lambda, ECS, EKS) | Least-privilege; no password distribution; short-lived credentials; integrates with IAM roles | 15-minute token validity; 12-hour max connection lifecycle; no IAM re-auth inside MULTI/EXEC; requires IAM SDK integration in client | **Preferred for new builds** when clients support IAM token generation |
| RBAC + passwords in Secrets Manager | Valkey 7.2+ / Redis OSS 6.0+; libraries without IAM-token support; third-party integrations | Per-user ACLs via access strings; rotation via Secrets Manager with a custom Lambda function; works with any client library | Requires Secrets Manager setup and a custom Lambda rotation function; longer-lived credentials than IAM tokens | **Preferred alternative** when IAM auth is impractical |
| Legacy AUTH token | Node-based clusters only; existing clusters with a single shared password | Simple to configure; supported on older engine versions | Single shared password (up to two tokens during rotation); no per-user ACLs; not available on serverless; no built-in rotation | **Never for new builds.** Migration path to RBAC only. |
## Hard Rules
- **Serverless caches**: AUTH tokens are not supported. Use IAM auth or RBAC with passwords.
- **TLS required**: IAM auth and AUTH tokens require TLS. RBAC with passwords strongly recommends TLS but does not require it on node-based clusters. Serverless caches always have TLS enabled.
- **IAM auth**: Requires Valkey 7.2+ or Redis OSS 7.0+. Requires TLS (in-transit encryption). Available on both serverless and node-based. The IAM user ID and username must be identical; if they differ, authentication will fail.
- **IAM auth cache name**: Cache names are lowercased at creation time. Authenticating code must supply the cache name in lowercase in the presigned URL, or authentication will fail.
- **Legacy AUTH token**: Node-based only. Do not recommend for new deployments.
## Decision Flowchart
1. **Is this a serverless cache?**
- Yes: AUTH tokens are not supported. Go to step 2.
- No: All three models are available. Go to step 2.
2. **Does the client library support IAM token generation?**
- Yes: Use IAM auth + RBAC.
- No: Use RBAC with passwords stored in Secrets Manager with rotation enabled.
3. **Is this an existing cluster using a legacy AUTH token?**
- Yes: Plan migration to RBAC. Create RBAC users, then use `modify-replication-group --auth-token-update-strategy SET --auth-token <token>` to switch from AUTH-only to RBAC-compatible mode. Test RBAC users, then remove the AUTH token.
- No: Do not configure AUTH tokens.
## Access String Examples
| Use Case | Access String | Notes |
|----------|--------------|-------|
| Full access (app) | `on ~* +@all` | All commands, all keys |
| Read-only | `on ~* +@read -@write -@admin` | No writes, no admin |
| Scoped to key prefix | `on ~app:* +@read +@write` | Only keys starting with `app:` |
| Admin disabled (default user) | `off ~* -@all` | Locks out the default user |
| Read-only on specific prefix | `on ~cache:* +@read` | Read-only, scoped keys |
references/setup/cluster-topology.md
# Cluster Topology
Guidance for cluster-mode selection, shard count, replica count, and IP address planning. Applies only to node-based deployments.
## When to Shard (Cluster-Mode-Enabled)
Use cluster-mode-enabled (multiple shards) when:
- **Data exceeds single-node memory**: A single node maxes out at the largest available instance type. Sharding distributes data across multiple nodes.
- **Write throughput exceeds single-primary capacity**: Each shard has its own primary. More shards means more write capacity.
- **Key space is naturally partitionable**: Hash-tag based routing works well when multi-key operations target the same logical entity.
Use single-shard when:
- Data fits comfortably in a single node's memory.
- Write throughput is within a single primary's capacity.
- The application relies heavily on multi-key operations across unrelated keys (CROSSSLOT errors are a concern with cluster mode).
- Simplicity is preferred and the workload does not need horizontal scaling.
## Cluster-Mode Decision Table
| Factor | Single-Shard | Cluster-Mode-Enabled |
|--------|-------------|---------------------|
| Max data capacity | Limited by largest node type | Scales horizontally across shards |
| Max write throughput | Single primary | One primary per shard |
| Multi-key operations | No slot restrictions | Must use hash tags or limit to same slot |
| Operational complexity | Lower | Higher (shard management, rebalancing) |
| Failover blast radius | Entire dataset | One shard only |
| Online resharding | Not applicable | Supported (add/remove shards live) |
### Node and Shard Limits
The default quota is **90 nodes per cluster** (cluster mode enabled). This can be increased to a maximum of **500 nodes per cluster** for Valkey (all versions) or Redis OSS 5.0.6+ (for versions below 5.0.6, the limit is 250). The regional limit is **300 non-reserved nodes per region** across all clusters. To increase these limits, submit a service limit increase request via the AWS Service Quotas console.
## Replica Count Guidance
| Environment | Recommended Replicas per Shard | Rationale |
|-------------|-------------------------------|-----------|
| Development | 0 | Cost savings; no HA needed (note: cluster-mode-enabled always has automatic failover on, which requires at least 1 replica for HA; Multi-AZ with cluster mode disabled also requires at least 1 replica) |
| Staging | 1 | Mimics production topology without excess cost |
| Production | 2-3 | AWS Well-Architected recommends a minimum of 2 replicas per shard for HA workloads; 3 replicas for read-heavy or higher durability requirements |
| Production (read-heavy) | 2-5 | Scale reads across replicas; balance against replication overhead |
## IP Address Planning
Each node (primary or replica) in a node-based cluster consumes one IP address per subnet. Plan subnet CIDR ranges accordingly:
| Topology | Nodes per Shard | Total Nodes (example: 3 shards) | IPs Needed |
|----------|----------------|--------------------------------|------------|
| 1 shard, 1 replica | 2 | 2 | 2 per AZ used |
| 3 shards, 1 replica | 2 per shard | 6 | 6 distributed across AZs |
| 3 shards, 2 replicas | 3 per shard | 9 | 9 distributed across AZs |
Ensure each subnet in the subnet group has enough available IPs for the planned node count plus headroom for scaling. A /24 subnet (251 usable IPs) is sufficient for most ElastiCache deployments, but tightly-sized /28 subnets (11 usable IPs) can be exhausted quickly if multiple clusters share the same subnets.
## Data Tiering
Data tiering (SSD-backed memory extension) requires the r6gd instance family and Valkey 7.2+ or Redis OSS 6.2+. Node-based only, not available on serverless.
references/setup/connection-guide.md
# Connecting to ElastiCache
ElastiCache-specific connection constraints, tunnel patterns, and runtime gotchas.
## VPC Access Only
ElastiCache runs inside a VPC. You cannot connect from the public internet. Options:
1. Connect from compute in the same VPC (EC2, Lambda, ECS, EKS)
2. SSM port forwarding (no SSH keys needed, use `scripts/start_tunnel.py` and `scripts/find_tunnel_host.py`)
3. Jump host with SSH tunnel
## Jump Host / SSM Tunnel
Primary path (bundled scripts, boto3-based):
```bash
python3 scripts/find_tunnel_host.py --vpc-id <vpc-id> --region <region>
python3 scripts/start_tunnel.py --instance-id <id> --cache-host <endpoint> --region <region>
```
`find_tunnel_host.py` locates an SSM-managed instance in the VPC. `start_tunnel.py` opens the SSM port-forwarding tunnel through it.
After tunnel is established, clean up: stop or terminate the EC2 instance if the skill created one. If `find_tunnel_host.py` reused an existing SSM-managed instance, stopping the tunnel is enough.
## Tunnel Mode TLS Gotcha
When connecting through an SSM tunnel to `127.0.0.1`, the TLS certificate is issued for `*.cache.amazonaws.com`, not localhost. You must disable hostname verification while keeping CA certificate validation.
| Language | Setting |
|----------|---------|
| Python (valkey-py) | `ssl_check_hostname=False` (keep `ssl_cert_reqs="required"`) |
| Node.js (ioredis) | `tls: { checkServerIdentity: () => undefined }` |
| Java (Lettuce) | `redisURI.setVerifyPeer(SslVerifyMode.CA)` |
| Go (go-redis) | `TLSConfig: &tls.Config{InsecureSkipVerify: true, VerifyPeerCertificate: customCACertVerifier}` (use a custom `VerifyPeerCertificate` func to validate the CA chain) |
| CLI | `valkey-cli --tls --insecure` (note: `--insecure` disables both CA and hostname verification) |
**These settings are for local development only. Never use in production.**
## IAM Auth Token Generation
ElastiCache IAM auth uses SigV4 presigned URLs (not a dedicated SDK method):
```python
from botocore.signers import RequestSigner
from botocore.model import ServiceId
import boto3
def generate_iam_auth_token(cache_name: str, user_id: str, region: str, is_serverless: bool = True) -> str:
import botocore.session
session = boto3.Session()
creds = session.get_credentials().get_frozen_credentials()
signer = RequestSigner(
ServiceId("elasticache"), region, "elasticache", "v4",
creds, botocore.session.get_session().get_component("event_emitter"),
)
query = f"Action=connect&User={user_id}"
if is_serverless:
query += "&ResourceType=ServerlessCache"
url = signer.generate_presigned_url(
{"method": "GET", "url": f"https://{cache_name}/?{query}",
"body": {}, "headers": {}, "context": {}},
operation_name="connect", expires_in=900, region_name=region,
)
return url[len("https://"):] if url.startswith("https://") else url
token = generate_iam_auth_token(
"my-cache", "my-iam-user", "us-east-1", is_serverless=True
)
# Pass token as password, user ID as username, ssl=True
```
Constraints:
- Token valid for 15 minutes (for initial AUTH/HELLO). An authenticated connection remains valid for up to 12 hours; re-AUTH with a new token within 12 hours to extend. Connections that are not re-authenticated within 12 hours are terminated.
- IAM auth does NOT work inside MULTI/EXEC transactions
- Requires Valkey 7.2+ or Redis OSS 7.0+
## Endpoint Formats
| Deployment | Primary | Reader |
|-----------|---------|--------|
| Serverless | `name-xxxxx.serverless.use1.cache.amazonaws.com:6379` | `name-xxxxx.serverless.use1.cache.amazonaws.com:6380` |
| Node-based (cluster mode) | `name.xxxxx.clustercfg.use1.cache.amazonaws.com:6379` (configuration endpoint) | *(cluster-aware clients discover shards via the configuration endpoint; use `CLUSTER SLOTS` or `CLUSTER SHARDS` for shard-aware routing)* |
| Node-based (single shard) | `name.xxxxx.ng.0001.use1.cache.amazonaws.com:6379` | (reader endpoint from `describe-replication-groups`) |
> **Note:** Always retrieve endpoints from `describe-serverless-caches` or `describe-replication-groups` rather than constructing them manually. Serverless caches operate in cluster-mode-enabled only; clients must use a cluster-aware driver (e.g., `ValkeyCluster` / `RedisCluster` in Python, `new Redis.Cluster()` in ioredis). For TLS-enabled cluster mode clusters, discovery commands (`cluster slots`, `cluster shards`, `cluster nodes`) return hostnames instead of IPs. The IP Discovery parameter has no effect on TLS-enabled clusters; the IP protocol used is determined by the client's DNS resolution preference.
## TLS Rules
- **Serverless**: TLS always enabled, cannot be disabled. Always use `ssl=True` / `tls: {}`.
- **Node-based**: TLS is optional but recommended. Can be enabled at creation time, or enabled on existing clusters using a two-step migration: first set `transit-encryption-mode` to `preferred` (allows both encrypted and unencrypted connections), then set to `required` (encrypted only).
- Serverless security groups need both port 6379 (primary) and 6380 (reader).
## Authentication by Deployment
| Auth Method | Serverless | Node-Based |
|-------------|-----------|------------|
| RBAC + IAM auth | Yes (preferred) | Yes (preferred) |
| RBAC + password | Yes | Yes |
| AUTH token (legacy) | **NOT supported** | Yes (legacy only) |
## Runtime-Specific Gotchas
### Lambda
- Requires VPC attachment (`VpcConfig` with subnets and security group)
- Needs `ec2:CreateNetworkInterface`, `ec2:DescribeNetworkInterfaces`, `ec2:DeleteNetworkInterface` permissions
- Since the September 2019 Hyperplane ENI improvements, VPC cold starts add only tens to low-hundreds of milliseconds (not the 1-2s of pre-2019 behavior). For IAM-authenticated connections, expect approximately 50-100ms additional overhead (estimate; actual overhead depends on SigV4 signing and network conditions). Reuse connections across invocations via module-level client
- Use IAM auth to avoid managing secrets in environment variables
### ECS
- Use `awsvpc` network mode for per-task ENIs in the same VPC
- Task security group must allow outbound to cache security group on port 6379
- Reuse connections across requests (connection pooling)
### EKS
- VPC CNI plugin affects pod-to-cache reachability; pods must have IPs in the VPC CIDR
- Use Kubernetes secrets or AWS Secrets Manager CSI driver for credential injection
- Pod service account with IAM role for IAM auth (IRSA or EKS Pod Identity)
references/setup/connectivity-diagnostics.md
# Connectivity Diagnostics for ElastiCache
Decision-tree diagnostic for "I can't connect to my cache." Work through steps in order; fix at the step where it fails.
## Step 0: Are You in the VPC?
ElastiCache has no public endpoints. Determine where the client runs before diagnosing.
| Client location | Action |
|----------------|--------|
| EC2/Lambda/ECS/EKS in same VPC | Proceed to Step 1 |
| EC2 in different VPC | Verify peering/transit gateway routes, then Step 1 |
| Local laptop / CI runner | Set up SSM tunnel first, then diagnose against `127.0.0.1` with `--tunnel-mode` |
Tunnel setup:
```bash
python3 scripts/find_tunnel_host.py --vpc-id <vpc-id> --region <region>
python3 scripts/start_tunnel.py --instance-id <id> --cache-host <endpoint> --region <region>
python3 scripts/test_connection.py 127.0.0.1 --port 6379 --tunnel-mode --server-name <endpoint>
```
If tunnel fails: check SSM agent status, `AmazonSSMManagedInstanceCore` policy, and routes to SSM endpoints (NAT or VPC endpoints for `ssm`, `ssmmessages`, `ec2messages`).
## Step 1: DNS Resolution
Retrieve the correct endpoint:
```bash
# Serverless
aws elasticache describe-serverless-caches --serverless-cache-name <name> \
--query 'ServerlessCaches[0].Endpoint'
# Node-based (cluster-mode enabled) -- uses ConfigurationEndpoint
aws elasticache describe-replication-groups --replication-group-id <name> \
--query 'ReplicationGroups[0].ConfigurationEndpoint'
# Node-based (cluster-mode disabled) -- uses NodeGroups[0].PrimaryEndpoint
aws elasticache describe-replication-groups --replication-group-id <name> \
--query 'ReplicationGroups[0].NodeGroups[0].PrimaryEndpoint'
```
Note: `ConfigurationEndpoint` is null for cluster-mode-disabled clusters. Use `PrimaryEndpoint` and `ReaderEndpoint` from `NodeGroups[0]` instead.
Common mistakes:
* Using `ConfigurationEndpoint` for cluster-mode-disabled clusters (it will be null)
* Using a node endpoint instead of the configuration endpoint for cluster-mode-enabled caches
* Including the port in the hostname (`endpoint:6379` instead of just `endpoint`)
If DNS times out: verify VPC has `enableDnsSupport` and `enableDnsHostnames` set to `true`.
## Step 2: TCP / Port Reachability
**Serverless caches need both ports open**: 6379 (primary) AND 6380 (reader) in the security group.
Find the cache's security group:
```bash
aws elasticache describe-serverless-caches \
--serverless-cache-name <name> \
--query 'ServerlessCaches[0].SecurityGroupIds'
```
Failure causes:
* **Timeout**: security group inbound rule missing for TCP 6379 (and 6380 for serverless) from client's SG or CIDR
* **Refused**: cache not in `available` status, or wrong port
* **Lambda**: must have VPC attachment. Verify with `aws lambda get-function-configuration --function-name <name> --query 'VpcConfig'`
## Step 3: TLS Handshake
Serverless always requires TLS. Node-based requires TLS only if created with `--transit-encryption-enabled`, or if TLS was enabled later via the preferred→required migration path (available for Redis OSS 7.0+/Valkey 7.2+).
ElastiCache-specific failures:
| Symptom | Cause | Fix |
|---------|-------|-----|
| "wrong version number" on serverless | Client connecting without TLS | Set `ssl=True` / `tls: {}` in client |
| "wrong version number" on node-based | Cache created without TLS, client sending TLS | For Redis OSS 7.0+/Valkey 7.2+, enable TLS on existing cluster via `modify-replication-group` with `--transit-encryption-enabled` and `--transit-encryption-mode preferred`, then `required`. For older versions, recreate cache with TLS. |
| Hostname mismatch through tunnel | Cert is for `*.cache.amazonaws.com`, not `127.0.0.1` | `ssl_check_hostname=False` (Python), `checkServerIdentity: () => undefined` (Node), `--insecure` (CLI) |
| Verify code 20 | Missing Amazon Trust Services root CA | Update system CA bundle |
## Step 4: Authentication
Test with CLI:
```bash
# RBAC with password
valkey-cli -h <endpoint> -p 6379 --tls --user <username> --pass <password> PING
# IAM auth -- generate token via Python (no CLI equivalent exists)
# python3 -c "
# import boto3; from botocore.signers import RequestSigner; from botocore.model import ServiceId
# s = boto3.Session(); c = s.get_credentials().get_frozen_credentials()
# sr = RequestSigner(ServiceId('elasticache'), '<region>', 'elasticache', 'v4', c, s._session.get_component('event_emitter'))
# For serverless caches, add ResourceType=ServerlessCache:
# url = sr.generate_presigned_url({'method':'GET','url':'https://<endpoint>/?Action=connect&User=<user-id>&ResourceType=ServerlessCache','body':{},'headers':{},'context':{}}, operation_name='connect', expires_in=900, region_name='<region>')
# For node-based caches, omit ResourceType:
# url = sr.generate_presigned_url({'method':'GET','url':'https://<endpoint>/?Action=connect&User=<user-id>','body':{},'headers':{},'context':{}}, operation_name='connect', expires_in=900, region_name='<region>')
# print(url[len('https://'):])
# "
# Then pass the output as TOKEN:
valkey-cli -h <endpoint> -p 6379 --tls --user <user-id> --pass "$TOKEN" PING
```
ElastiCache-specific auth failures:
| Error | Cause | Fix |
|-------|-------|-----|
| `NOAUTH Authentication required` | Client not sending credentials | Configure username/password in client connection params |
| `WRONGPASS` (RBAC) | Wrong password or user not in cache's user group | Verify user is in the user group linked to the cache |
| `ERR AUTH invalid-iam-credentials` | Expired token (15 min validity) | Regenerate immediately before connecting |
| `ERR AUTH invalid-iam-credentials` | Missing `elasticache:Connect` on both cache AND user ARNs | Add permission on both resource ARNs |
| `ERR AUTH invalid-iam-credentials` | `--user-id` in token generation doesn't match AUTH username | Must be identical |
| `WRONGPASS` (user disabled) | Access string starts with `off` | Change to `on` via `modify-user` |
IAM permission check:
```bash
aws iam simulate-principal-policy --policy-source-arn <role-arn> \
--action-names elasticache:Connect \
--resource-arns \
arn:aws:elasticache:<region>:<account>:serverlesscache:<cache-name> \
arn:aws:elasticache:<region>:<account>:user:<user-id>
```
## Step 5: Command Permissions (RBAC ACL)
| Error | Cause | Fix |
|-------|-------|-----|
| `NOPERM ... run the 'set' command` | Access string missing command category | Add `+@write` or specific `+set` |
| `NOPERM No permissions to access a key` | Key doesn't match allowed pattern | Update access string pattern or fix app key prefix |
| `NOPERM` on pub/sub | Missing channel permissions | Add `&*` (all channels) or `&prefix:*` |
Commands requiring special categories:
| Command | Required | Notes |
|---------|----------|-------|
| SUBSCRIBE, PUBLISH | `+@pubsub` + channel pattern (`&*`) | |
| EVAL, EVALSHA | `+@scripting` | |
| MULTI, EXEC | `+@transaction` | **IAM re-auth (AUTH/HELLO) cannot be used inside MULTI/EXEC blocks** |
Access string changes take effect on new connections only. Existing connections retain old permissions until reconnect.
references/setup/create-secure-cache.md
# Create Secure Cache
End-to-end workflow from intent to a production-safe ElastiCache cache. Use when the user wants to create a new cache and you have enough context (engine, region, VPC). If context is missing, run requirements sub-skill first.
## Preflight Checks
**Do not create a new VPC if one already exists.** Use the VPC where the user's compute (Lambda, ECS, EKS, EC2) already runs. Find it from existing IaC, task definitions, or ask the user. If no VPC is found, confirm with the user before creating one.
Before provisioning, verify:
| Check | Pass criteria | Severity |
|-------|--------------|----------|
| Subnets span 2+ AZs | At least 2 subnets in different AZs | FAIL |
| Available IPs | Each subnet has 10+ free IPs (serverless needs ENIs) | WARN |
| Security group allows port 6379 | Inbound from app SG on TCP 6379 | FAIL |
| Security group allows port 6380 (serverless only) | Inbound from app SG on TCP 6380 (reader endpoint) | FAIL |
| Service quotas | Nodes per region (L-DFE45DF3), nodes per cluster CME (L-AF354865), or serverless caches per region (L-BBCDAECC) have headroom | FAIL |
| Caller permissions | `elasticache:Create*`, `ec2:CreateNetworkInterface` (serverless) | FAIL |
| Node type available (node-based only) | Offerings exist for the node type in the target region | FAIL |
| Subnet group exists (node-based only) | Named subnet group found | FAIL |
## Product Rules (check before proceeding)
- Vector search needed? Force node-based Valkey 8.2 or above (recommend 9.0; not available on data-tiering nodes; t2/t3/t4g instances require increased memory reserve). Note: T2 is transitioning to previous-generation. While still available, prefer T3 or T4g for new clusters.
- Global Datastore needed? Force node-based
- Serverless? AUTH tokens not supported. RBAC required. Clients must support cluster mode enabled.
- Node-based? Must explicitly enable: TLS, at-rest encryption, Multi-AZ, automatic failover, daily backups. At-rest encryption is immutable after creation. TLS can be enabled post-creation via a two-step process (set transit-encryption-mode to `preferred`, then `required`).
## Create the Cache
### Serverless
Key differences from node-based:
- Deploys in under a minute
- TLS, encryption at rest, Multi-AZ: all automatic (no flags needed)
- Snapshots are supported but must be explicitly configured via `SnapshotRetentionLimit` > 0 and optionally `DailySnapshotTime`. They are not enabled by default.
- Subnet IDs go directly (no subnet group resource needed)
- Must set `--cache-usage-limits` for cost controls (DataStorage in GB + ECPUPerSecond)
- Primary: `aws elasticache create-serverless-cache ...` (or boto3 `create_serverless_cache`, or CloudFormation `AWS::ElastiCache::ServerlessCache`)
### Node-Based
Key differences from serverless:
- Takes 5-15 minutes to provision
- Requires explicit security flags: `--transit-encryption-enabled`, `--at-rest-encryption-enabled`, `--automatic-failover-enabled`, `--multi-az-enabled`, `--snapshot-retention-limit 7`
- Requires a subnet group (separate resource)
- For property names per IaC tool, see `iac-reference.md`
- Primary: `aws elasticache create-replication-group ...` (Valkey/Redis with replication; or boto3 `create_replication_group`, or CloudFormation `AWS::ElastiCache::ReplicationGroup`). Use `create-cache-cluster` only for Memcached.
### VPC-Only Access Notice (mandatory after every creation)
Always tell the user after creating:
> Your cache is VPC-only. ElastiCache has no public endpoints. If you are on a local machine, you need a tunnel to connect.
>
> 1. **Reuse an existing EC2 instance ($0):**
> - Primary: `python3 scripts/find_tunnel_host.py --vpc-id <vpc-id> --region <region>` (locates an SSM-managed instance via boto3)
> 2. **Create a new jump host (approximately $3/month, t4g.nano in us-east-1, pricing subject to change; verify current rates, only if no existing instance found):**
> - Primary: launch a t4g.nano with `aws ec2 run-instances` and attach the `AmazonSSMManagedInstanceCore` IAM policy so SSM can manage it. Then open the tunnel with `python3 scripts/start_tunnel.py`. Bills until stopped/terminated.
> 3. **Skip** if your app runs inside the VPC (Lambda, ECS, EKS, EC2)
>
> Use the bundled scripts (`scripts/find_tunnel_host.py`, `scripts/start_tunnel.py`) as the primary path. Always try option 1 before option 2.
Tunnel startup:
```bash
python3 scripts/start_tunnel.py --instance-id <id> --cache-host <endpoint> --region <region>
python3 scripts/test_connection.py 127.0.0.1 --port 6379 --tunnel-mode --server-name <endpoint>
```
## Set Up Authentication
### RBAC sequence (both deployment models)
1. Create RBAC user: `create-user --user-id <name>-appuser --user-name <name>-appuser --engine <valkey|redis> --access-string "on ~* +@all" --authentication-mode Type=iam`
2. Create user group: `create-user-group --user-group-id <name>-usergroup --engine <valkey|redis> --user-ids default <name>-appuser`
Use the engine value that matches your cache engine (`valkey` for Valkey caches, `redis` for Redis OSS caches).
3. Attach to cache:
- Serverless: `modify-serverless-cache --serverless-cache-name <name> --user-group-id <name>-usergroup`
- Node-based: set `--user-group-ids` at creation time
**Note:** The `--engine` value for user and user-group accepts `valkey` or `redis`. Use the value that matches your cache engine.
### Authentication mode
| Mode | Flag | When to use |
|------|------|-------------|
| IAM auth (recommended) | `--authentication-mode Type=iam` | Cloud-native clients (Lambda, ECS, EKS). Requires Valkey 7.2+ or Redis OSS 7.0+ and TLS. |
| Password auth | `--authentication-mode Type=password,Passwords=<pw>` | Clients without IAM token support. Store password in Secrets Manager with rotation enabled. |
### Lock down the default user
Always disable the default user: `modify-user --user-id default --access-string "off ~* -@all"`
## Restore from Snapshot
Restore creates a new cache (cannot restore to an existing one). Constraints:
- Backups are cross-compatible between deployment models for Valkey and Redis OSS: node-based snapshots can be restored into serverless caches, and serverless snapshots can be restored into node-based clusters. Memcached serverless snapshots can only be restored into Memcached serverless caches. Serverless restore requires RDB files compatible with Valkey 7.2+ or Redis OSS 5.0+. Data-tiering (r6gd) backups can only restore to r6gd node types.
<!-- Source: https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/backups-restoring.html -->
- CLI commands differ: `create-serverless-cache --snapshot-arns-to-restore` vs `create-replication-group --snapshot-name`
- Final snapshot on delete: `--final-snapshot-name` (serverless) vs `--final-snapshot-identifier` (node-based). Memcached has no snapshot support; `--final-snapshot-identifier` does not apply to Memcached.
- Security settings (SGs, encryption, RBAC user groups) are NOT inherited. Must re-specify on the new cache.
- Cross-region snapshot copy: Native cross-region copy API is node-based only. Serverless snapshots can be exported to S3 via `export-serverless-cache-snapshot` and then copied cross-region through S3.
- S3 export for node-based snapshots uses `copy-snapshot --target-bucket`. For serverless snapshots, use the separate `export-serverless-cache-snapshot --serverless-cache-snapshot-name <name> --s3-bucket-name <bucket>` command. Available for Valkey and Redis OSS only (not Serverless Memcached). Bucket needs `elasticache.amazonaws.com` service principal in its policy.
<!-- Source: https://docs.aws.amazon.com/cli/latest/reference/elasticache/export-serverless-cache-snapshot.html -->
## Validate and Finish
1. Run `python3 scripts/test_connection.py <endpoint>` (or with `--tunnel-mode` if local)
2. If validation fails, see `connectivity-diagnostics.md`
3. Run `python3 scripts/security_audit.py --serverless <name> --region <region>` to verify security posture
4. Optionally generate observability: `python3 scripts/generate_dashboards.py --serverless <name> --region <region>`
5. Optionally generate IaC: see `iac-reference.md`
## Attribution Tags
Apply to all resources created by the skill:
| Tag Key | Value |
|---------|-------|
| `managed_by` | `aws-skills` |
| `skill` | `elasticache` |
| `skill_version` | `1.0.0` |
| `created_by` | `elasticache-skill` |
| `generation_model` | model ID of the Claude instance that ran the skill (e.g. `claude-sonnet-4-20250514`) |
| `Environment` | dev / staging / prod |
references/setup/engine-selection.md
# Engine Selection Guide
## Quick Decision
| Factor | Valkey | Redis OSS | Memcached |
|--------|--------|-----------|-----------|
| Default recommendation | **Yes** | Only if Redis-specific compat needed | Only for simple key-value |
| Serverless pricing | **Lowest (33% less than Redis OSS)** | Higher | Available |
| Node-based pricing | **20% less than Redis OSS** | Baseline | Similar |
| Vector search | Yes (8.2+, node-based only; not on data tiering nodes) | No | No |
| IAM auth | Yes (7.2+) | Yes (7+) | No |
| Global Datastore | Yes (node-based only) | Yes (node-based only) | No |
| JSON native data type | Yes (7.2+) | Yes (6.2+) | No |
> Vector search is available with Valkey 8.2 or above on node-based clusters in all AWS Regions at no additional cost. Not supported on data-tiering instances (r6gd) or serverless caches. ElastiCache Serverless regional availability may vary; check the AWS Region Table before provisioning.
## When to recommend each
### Valkey (default for all new workloads)
- Open source, backed by Linux Foundation and 40+ companies
- API-compatible with Redis OSS 7.2
- Best price-performance ratio across both serverless and node-based
- Cheapest engine option on ElastiCache for every deployment model
- Zero-downtime upgrade path from existing Redis OSS clusters
### Redis OSS
- Only when the user has a hard dependency on a Redis OSS feature not yet in Valkey
- Or when they have existing Redis OSS clusters and don't want to migrate yet
- Redis OSS 7.1 is the highest Redis OSS version available on ElastiCache. Recommend Valkey 9.0 for new builds. Verify the latest available versions at https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/.
- In-place upgrade to Valkey is zero-downtime and reduces cost immediately
### Memcached
- Only when the user explicitly wants Memcached or has an existing Memcached client they cannot change
- No persistence for node-based clusters (serverless supports backup/restore), no replication for node-based clusters (serverless stores data redundantly across 3 AZs), no IAM/RBAC auth (TLS supported from 1.6.12; always enabled on serverless), simple data types (strings and objects)
references/setup/iac-best-practices.md
# IaC Deployment Best Practices
Operational guidance for deploying ElastiCache via CloudFormation, Terraform, and CDK. For resource names, property mappings, and endpoint attributes, see `iac-reference.md`.
## API Rate Limiting During Deployment
ElastiCache control-plane APIs enforce throttle limits. Deploying multiple caches in a single stack or account can trigger `ThrottlingException`.
**Mitigation strategies:**
* **DependsOn chains**: When a single CFN stack creates multiple caches, add explicit `DependsOn` between them so they provision sequentially rather than in parallel.
* **StackSets with MaxConcurrentCount**: For multi-account or multi-region rollouts, set `MaxConcurrentCount: 1` (or a low value) in StackSet operation preferences to stagger deployments.
* **Terraform parallelism**: Use `terraform apply -parallelism=1` or add `depends_on` between cache resources to serialize creation.
* **Retry with backoff**: If using custom scripts or CI/CD wrappers, implement exponential backoff on throttle errors.
## Common CloudFormation Failures
### Stack stuck in UPDATE_ROLLBACK_FAILED
A stack can get stuck when a rollback itself fails (for example, a resource that was modified outside of CFN).
**Resolution:**
1. Open the CloudFormation console or run `aws cloudformation describe-stack-events` to identify the failed resource.
2. Run `aws cloudformation continue-update-rollback --stack-name <name>` to retry.
3. If a specific resource cannot roll back, skip it: `aws cloudformation continue-update-rollback --stack-name <name> --resources-to-skip <logical-id>`.
4. After the rollback completes, fix the template and update again.
### Deletion fails due to VPC endpoints
Serverless caches create VPC endpoints in the associated subnets. If these endpoints still exist when the stack tries to delete the cache or its networking resources, deletion fails.
**Resolution:**
1. List endpoints: `aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=<vpc-id>` and identify ElastiCache-related endpoints.
2. Delete them: `aws ec2 delete-vpc-endpoints --vpc-endpoint-ids <id1> <id2>`.
3. Retry the stack deletion.
**Prevention:** In CFN templates, place the cache resource with an explicit `DependsOn` on the VPC/subnet resources so CFN deletes the cache (and its endpoints) before attempting to remove networking resources.
### CREATE_FAILED on ReplicationGroup
Common causes:
* Subnet group references subnets in a single AZ (Multi-AZ requires at least 2 AZs).
* Security group does not allow inbound on port 6379.
* Insufficient IP addresses in the subnet for the requested node count.
## Template Structure Best Practices
### Separate parameter groups as their own resources
Define `AWS::ElastiCache::ParameterGroup` (CFN) or `aws_elasticache_parameter_group` (Terraform) as standalone resources rather than relying on defaults. This allows parameter changes without replacing the cache.
### Export endpoints via Outputs
Always export cache endpoints so dependent stacks or applications can reference them:
```yaml
Outputs:
CacheEndpoint:
# For AWS::ElastiCache::ServerlessCache: !GetAtt Cache.Endpoint.Address
# For AWS::ElastiCache::ReplicationGroup (CME): !GetAtt Cache.ConfigurationEndPoint.Address
# For AWS::ElastiCache::ReplicationGroup (CMD): !GetAtt Cache.PrimaryEndPoint.Address
Value: !GetAtt Cache.PrimaryEndPoint.Address
Export:
Name: !Sub "${AWS::StackName}-cache-endpoint"
```
### Multi-environment templates with Conditions
Use `Conditions` and `Mappings` to manage dev/staging/prod from a single template:
```yaml
Parameters:
Environment:
Type: String
AllowedValues: [dev, staging, prod]
Conditions:
IsProd: !Equals [!Ref Environment, prod]
```
Then use `!If [IsProd, ...]` to vary node types, replica counts, snapshot retention, and cost limits by environment. For serverless, vary `CacheUsageLimits` (lower maximums in dev to control cost).
### Auto Scaling policies via CloudFormation
For node-based replication groups, define auto-scaling policies directly in your templates using `AWS::ApplicationAutoScaling::ScalableTarget` and `AWS::ApplicationAutoScaling::ScalingPolicy`. You can scale replicas (`elasticache:replication-group:Replicas`) or shards (`elasticache:replication-group:NodeGroups`). For example:
```yaml
ScalingTarget:
Type: 'AWS::ApplicationAutoScaling::ScalableTarget'
Properties:
MaxCapacity: 5
MinCapacity: 1
ResourceId: !Sub replication-group/${MyReplicationGroup}
ScalableDimension: 'elasticache:replication-group:Replicas'
ServiceNamespace: elasticache
ScalingPolicy:
Type: 'AWS::ApplicationAutoScaling::ScalingPolicy'
Properties:
ScalingTargetId: !Ref ScalingTarget
ServiceNamespace: elasticache
PolicyName: target-tracking-cpu
PolicyType: TargetTrackingScaling
ScalableDimension: 'elasticache:replication-group:Replicas'
TargetTrackingScalingPolicyConfiguration:
PredefinedMetricSpecification:
PredefinedMetricType: ElastiCacheReplicaEngineCPUUtilization
TargetValue: 60
```
> **Note:** The `ScalableTarget` supports an optional `RoleARN` property. If omitted, Application Auto Scaling uses the service-linked role `AWSServiceRoleForApplicationAutoScaling_ElastiCacheRG`, which is created automatically. If the service-linked role does not yet exist in your account, either add `RoleARN` explicitly or ensure the deploying principal has `iam:CreateServiceLinkedRole` permission.
### Terraform workspaces
Use Terraform workspaces or tfvars files per environment. Define cache configuration as variables with environment-specific defaults:
```hcl
variable "node_type" {
default = "cache.t4g.micro" # overridden in prod.tfvars
}
```
## Pre-Deployment Checklist
* Subnets span at least 2 AZs (required for Multi-AZ on node-based)
* Security group allows inbound TCP 6379 (and 6380 for serverless read port on the same endpoint hostname)
* IAM role used by CFN/Terraform has least-privilege ElastiCache actions scoped to provisioning: `elasticache:Create*`, `elasticache:Modify*`, `elasticache:Delete*`, `elasticache:Describe*`, `elasticache:List*`, `elasticache:AddTagsToResource`, `elasticache:ListTagsForResource`, `elasticache:CopySnapshot`, and `elasticache:TestFailover`, plus `ec2:CreateVpcEndpoint`, `ec2:DeleteVpcEndpoints`, and `iam:CreateServiceLinkedRole` (with condition `iam:AWSServiceName: elasticache.amazonaws.com`). This action set covers every create/modify/delete/describe call this skill issues. See the Provisioning profile in `iam-policies.md` for the canonical list. `elasticache:*` is acceptable only for initial prototyping; do not use it in production. The `iam:CreateServiceLinkedRole` permission is required for first-time deployments or when the ElastiCache service-linked role does not yet exist in the account (see IAM.IdentityBasedPolicies Example 4). Note: for serverless caches, VPC endpoint lifecycle is managed by the ElastiCache service-linked role, but the deploying role still needs `ec2:CreateVpcEndpoint` for the initial creation call.
* For node-based: verify subnet IP capacity covers (1 primary + replicas per shard) x (shards). Each node consumes one IP address regardless of AZ placement. Spread across subnets, so ensure each AZ's subnet has enough IPs for the nodes placed there.
* For serverless: `CacheUsageLimits` set to prevent unexpected cost in non-production environments
references/setup/iac-reference.md
# IaC Reference for ElastiCache
ElastiCache-specific gotchas, resource/construct names, and property mappings across all IaC tools. The model can generate full templates/stacks given these constraints.
## Gotchas (apply to all IaC tools)
- **Engine value for RBAC**: User and UserGroup resources accept `valkey` or `redis` for the engine parameter. Use the value that matches your cache engine. Older Redis OSS clusters continue to use `redis`.
- **Subnet handling differs by deployment**: Serverless accepts subnet IDs directly. Node-based requires a separate subnet group resource.
- **Dependency ordering**: The cache resource must depend on its user group. Without explicit dependency, the cache may be created before RBAC is ready.
- **Serverless uses two ports**: Port 6379 (read/write) and port 6380 (reader). Security groups must allow both.
- **Encryption defaults are hard to change**: At-rest encryption cannot be changed after creation. Transit encryption can be enabled on existing node-based clusters via a two-step migration (`preferred` → `required` mode), but cannot be disabled once required. For serverless, TLS is always on. Always set encryption at creation time when possible.
- **CacheName restrictions**: Serverless cache names must be lowercase, start with a letter, letters/numbers/hyphens only, max 40 characters.
- **CDK L2 constructs**: ElastiCache has only L1 constructs in CDK (`CfnServerlessCache`, `CfnReplicationGroup`, `CfnCacheCluster`). No stable L2 construct module is available.
## Resource/Construct Names
| Resource | CloudFormation | CDK (L1) | Terraform |
|----------|---------------|----------|-----------|
| Serverless cache | `AWS::ElastiCache::ServerlessCache` | `elasticache.CfnServerlessCache` | `aws_elasticache_serverless_cache` |
| Node-based cluster | `AWS::ElastiCache::ReplicationGroup` | `elasticache.CfnReplicationGroup` | `aws_elasticache_replication_group` |
| Subnet group | `AWS::ElastiCache::SubnetGroup` | `elasticache.CfnSubnetGroup` | `aws_elasticache_subnet_group` |
| User (RBAC) | `AWS::ElastiCache::User` | `elasticache.CfnUser` | `aws_elasticache_user` |
| User group | `AWS::ElastiCache::UserGroup` | `elasticache.CfnUserGroup` | `aws_elasticache_user_group` |
## Output/Endpoint Attribute Paths
| Deployment | Value | CloudFormation | CDK | Terraform |
|-----------|-------|---------------|-----|-----------|
| Serverless | Primary endpoint | `!GetAtt Cache.Endpoint.Address` | `.attrEndpointAddress` | `.endpoint[0].address` |
| Serverless | Reader endpoint | `!GetAtt Cache.ReaderEndpoint.Address` | `.attrReaderEndpointAddress` | `.reader_endpoint[0].address` |
| Serverless | Port | `!GetAtt Cache.Endpoint.Port` | `.attrEndpointPort` | `.endpoint[0].port` |
| Serverless | ARN | `!GetAtt Cache.ARN` | `.attrArn` | `.arn` |
| Node-based (CME) | Config endpoint | `!GetAtt RG.ConfigurationEndPoint.Address` | `.attrConfigurationEndPointAddress` | `.configuration_endpoint_address` |
| Node-based (CMD) | Primary endpoint | `!GetAtt RG.PrimaryEndPoint.Address` | `.attrPrimaryEndPointAddress` | `.primary_endpoint_address` |
| Node-based (CMD) | Reader endpoint | `!GetAtt RG.ReaderEndPoint.Address` | `.attrReaderEndPointAddress` | `.reader_endpoint_address` |
| Node-based | Port | `!GetAtt RG.ConfigurationEndPoint.Port` | `.attrConfigurationEndPointPort` | (use 6379) |
## Serverless Cost Controls Structure
| Tool | Syntax |
|------|--------|
| CloudFormation | `CacheUsageLimits: { DataStorage: { Minimum: 5, Maximum: 10, Unit: GB }, ECPUPerSecond: { Minimum: 1000, Maximum: 15000 } }` |
| CDK | `cacheUsageLimits: { dataStorage: { minimum: 5, maximum: 10, unit: 'GB' }, ecpuPerSecond: { minimum: 1000, maximum: 15000 } }` |
| Terraform | `cache_usage_limits { data_storage { minimum = 5 / maximum = 10 / unit = "GB" } ecpu_per_second { minimum = 1000 / maximum = 15000 } }` (each `/` is a newline in HCL; commas are not used between block attributes) |
| CLI | `--cache-usage-limits 'DataStorage={Minimum=5,Maximum=10,Unit=GB},ECPUPerSecond={Minimum=1000,Maximum=15000}'` |
> **Note:** `Minimum` values enable pre-scaling (pre-warming) for anticipated traffic spikes. You are charged for the minimum even if actual usage is lower. Set minimums at least 60 minutes before peak events.
## Node-Based Required Properties for Production
| Property | CFN/CDK | Terraform | CLI flag |
|----------|---------|-----------|----------|
| Engine | `Engine: valkey` | `engine = "valkey"` | `--engine valkey` |
| Version | `EngineVersion: '9.0'` | `engine_version = "9.0"` | `--engine-version 9.0` |
| Node type | `CacheNodeType: cache.r7g.large` | `node_type = "cache.r7g.large"` | `--cache-node-type cache.r7g.large` |
| Shards | `NumNodeGroups: 2` | `num_node_groups = 2` | `--num-node-groups 2` |
| Replicas | `ReplicasPerNodeGroup: 1` | `replicas_per_node_group = 1` | `--replicas-per-node-group 1` |
| TLS | `TransitEncryptionEnabled: true` | `transit_encryption_enabled = true` | `--transit-encryption-enabled` |
| At-rest encryption | `AtRestEncryptionEnabled: true` | `at_rest_encryption_enabled = true` | `--at-rest-encryption-enabled` |
| Failover | `AutomaticFailoverEnabled: true` | `automatic_failover_enabled = true` | `--automatic-failover-enabled` |
| Multi-AZ | `MultiAZEnabled: true` | `multi_az_enabled = true` | `--multi-az-enabled` |
| Snapshots | `SnapshotRetentionLimit: 7` | `snapshot_retention_limit = 7` | `--snapshot-retention-limit 7` |
## Secure Defaults Checklist
- Transit encryption enabled (node-based) / always-on (serverless)
- At-rest encryption enabled with optional KMS key
- Multi-AZ with automatic failover (node-based)
- RBAC user group with restricted default user (`off ~* -@all`). Note: for Redis OSS user groups, the `default` user ID must be included in every user group; omitting it causes `CreateUserGroup` to fail with `DefaultUserRequired`. Valkey user groups do not have this requirement — the default user is automatically disabled when a Valkey user group is attached.
- Serverless caches operate in cluster-mode-enabled only; clients must support cluster protocol
- IAM auth for application users (Valkey 7.2+ or Redis OSS 7.0+); RBAC available from Redis OSS 6.0+
- Private subnets only
- Security group scoped to application SG, never `0.0.0.0/0`
- Snapshot retention configured (node-based)
- Cost controls via CacheUsageLimits (serverless)
references/setup/iam-policies.md
# IAM Policies for ElastiCache
Which permissions to generate based on what the user is doing.
## Profile Routing
| Activity | Profile | Key actions |
|----------|---------|-------------|
| Connecting from app (Lambda, ECS, EKS) | Connectivity | `elasticache:Connect` |
| Creating/modifying caches and RBAC | Provisioning | `elasticache:Create*`, `Modify*`, `Delete*` |
| Monitoring, alarms, cost review | Monitoring | `elasticache:Describe*`, `cloudwatch:PutMetricAlarm`, `ce:GetCostAndUsage` |
| Read-only inspection, debugging | Discovery | `elasticache:Describe*`, `elasticache:List*` |
| Password rotation Lambda | Rotation | `elasticache:ModifyUser`, `elasticache:DescribeUsers`, `secretsmanager:GetSecretValue`, `secretsmanager:PutSecretValue`, `secretsmanager:DescribeSecret`, `secretsmanager:UpdateSecretVersionStage`, `secretsmanager:GetRandomPassword` |
## ElastiCache-Specific Gotchas
**`elasticache:Connect` requires two ARNs.** The Resource list must include both the cache AND the user. Without both, connection is denied:
```
arn:aws:elasticache:<region>:<account-id>:serverlesscache:<cache-name>
arn:aws:elasticache:<region>:<account-id>:user:<user-id>
```
For node-based, replace `serverlesscache` with `replicationgroup`.
**KMS condition for at-rest encryption** (only if using customer-managed key):
```
"Condition": { "StringEquals": { "kms:ViaService": "elasticache.<region>.amazonaws.com" } }
```
Required KMS actions: `kms:CreateGrant`, `kms:DescribeKey`, `kms:GenerateDataKey`, `kms:Decrypt`.
**Service-linked role:** `AWSServiceRoleForElastiCache`. First-time account setup needs `iam:CreateServiceLinkedRole` with condition `"iam:AWSServiceName": "elasticache.amazonaws.com"`. Once created, remove.
**SSM tunnel document ARN:** `arn:aws:ssm:<region>::document/AWS-StartPortForwardingSessionToRemoteHost`. Include in connectivity profile only if using SSM port forwarding.
**Rotation Lambda** must call `elasticache.modify_user(UserId=..., Passwords=[new_password])`. Note: `Passwords` is a top-level parameter, not nested inside `AuthenticationMode`. The rotation Lambda also needs `secretsmanager:GetRandomPassword` on `Resource: "*"` (not scoped to a specific secret ARN).
## Combining Profiles
| Persona | Profiles |
|---------|----------|
| Developer | Discovery |
| Platform engineer | Provisioning + Discovery |
| SRE / on-call | Monitoring + Discovery |
| Application runtime | Connectivity only |
## Lambda VPC Connectivity
Any Lambda function connecting to ElastiCache in a VPC needs the `AWSLambdaVPCAccessExecutionRole` managed policy (or equivalent permissions: `ec2:CreateNetworkInterface`, `ec2:DescribeNetworkInterfaces`, `ec2:DeleteNetworkInterface`). Without these, the Lambda will time out connecting to ElastiCache.
references/setup/instructions.md
# Setup & Connect
Create ElastiCache caches and connect applications to them.
## Loading
Read this file first. Load files referenced in the workflow steps on demand.
## Check for existing context
Before starting, read `.elasticache/requirements.json` if it exists. Use the values (engine, region, compute, use case, deployment model) as inputs rather than re-asking. If the file doesn't exist and the user skipped requirements, gather the minimum needed: engine (default Valkey), region, VPC, and subnets.
## Preflight (run before any provisioning)
Before creating or modifying anything, verify the execution path works:
1. **Credentials check (primary):** Run `aws sts get-caller-identity`. If it works, use AWS CLI, boto3 SDK, CloudFormation, and the bundled scripts for all control-plane operations.
2. **If credentials fail:** Stop. Tell the user to configure AWS CLI credentials (`aws configure`).
3. **Region:** Confirm region is set (from `.elasticache/requirements.json`, workspace scan, or ask the user).
4. **Permissions:** Run a read-only call to confirm the caller has ElastiCache access. Try `aws elasticache describe-serverless-caches --region <region>` first. If that fails with AccessDenied (the caller may only have node-based permissions), fall back to `aws elasticache describe-replication-groups --region <region>`. Either succeeding confirms ElastiCache access.
If all checks pass, proceed. If any fail, surface the specific error and suggest a fix before continuing.
## Workflow
1. **Select engine** -- skip if already set in `.elasticache/requirements.json`. Default Valkey. See `references/setup/engine-selection.md`.
2. **Select deployment model** -- skip if already set in JSON. Default serverless. See `references/setup/serverless-vs-node.md`.
3. **Estimate cost** -- skip if requirements already ran it. Run `scripts/price_calculator.py --mode serverless --data-gb <N> --ops-per-sec <N>` and `--mode node --node-type <type> --nodes <N> --show-ri-options`.
4. **Create the cache and set up auth** -- see `references/setup/create-secure-cache.md`. Covers provisioning, RBAC, tunnel setup, and restore. AWS CLI/SDK and bundled scripts are the primary path. For auth model decision: see `references/setup/auth-model-selector.md`.
5. **Generate IAM policies** -- see `references/setup/iam-policies.md` for which permissions the caller/app needs.
6. **Validate connectivity** -- run `scripts/test_connection.py <cache-endpoint>`. See `references/setup/connection-guide.md` for TLS gotchas, endpoint formats, and tunnel mode.
For local development access, use the tunnel scripts:
- `python3 scripts/find_tunnel_host.py --vpc-id <vpc-id> --region <region>` to locate an SSM-managed instance
- `python3 scripts/start_tunnel.py --instance-id <id> --cache-host <endpoint> --region <region>` to start the tunnel
7. **Verify security posture** -- run `python3 scripts/security_audit.py --serverless <cache-name>` (or `--replication-group <replication-group-id>`) to verify encryption, auth, network, and backup settings meet best practices.
8. **Emit IaC** -- optional. See `references/setup/iac-reference.md`.
## After provisioning
### Update requirements artifact (single source of truth)
After the cache is created and connectivity is validated, update `.elasticache/requirements.json`. Setup owns the `infrastructure` section and top-level infra fields. Read the existing file first, merge your updates, then write it back. Do not overwrite fields owned by requirements (use_case, patterns, runtime, data_source, etc.).
Update these top-level fields:
- `deployment_model`: "serverless" or "node-based"
- `vpc_id`, `subnet_ids`, `security_group_ids`
Add or update the `infrastructure` section:
```json
{
"infrastructure": {
"cache_name": "myapp-cache",
"resource_id": "myapp-cache",
"engine_version": "9.0",
"topology": null,
"endpoint": "myapp-cache.serverless.use1.cache.amazonaws.com",
"port": 6379,
"reader_port": 6380,
"auth_model": "iam",
"tls": true,
"client_library": "valkey-py",
"execution_path": "cli",
"access_mode": "tunnel",
"tunnel_instance_id": "i-0abc123def456",
"embedding_provider": null,
"embedding_model": null,
"embedding_dim": null,
"embedding_module": null
}
}
```
For node-based, `topology` should be: `{ "shards": <n>, "replicas_per_shard": <n> }`. For serverless, use `null`.
`access_mode` is `"tunnel"` (local dev via SSM) or `"direct"` (app runs in VPC). When `access_mode` is `"tunnel"`, proactively start the tunnel before any cache operation instead of waiting for connection failure.
`embedding_provider`, `embedding_model`, `embedding_dim`, and `embedding_module` are set by the genai sub-skill when the user chooses an embedding provider. Examples: `"bedrock"` / `"amazon.titan-embed-text-v2:0"` / `1024` / `"utils/embeddings.py"`. These are `null` until the user works with vector search. Once set, all sub-skills import from `embedding_module` instead of re-asking or regenerating embedding code.
The genai sub-skill also owns a `genai` section:
```json
{
"genai": {
"mode": "2",
"mode_2_path": "server-side",
"framework": "mem0"
}
}
```
`mode` is `"1"`, `"2"`, or `"3"` (plain cache, semantic response cache, full vector search). **Important:** Modes 2 (server-side path) and 3 require Valkey 8.2+ on node-based clusters for vector search (FT.CREATE/FT.SEARCH). Serverless does not support vector search. If the user selects mode 2 or 3, verify the engine version supports it before provisioning. `mode_2_path` is `"app-side"` or `"server-side"` (only set when mode is `"2"`). `framework` is the user's chosen framework (`"strands"`, `"mem0"`, `"langchain"`, or `null` for raw client). These are `null` until the user works with genai patterns. Once set, the genai sub-skill skips routing questions on return visits.
## Scaffold Missing Files
When the user's workspace does not contain expected files (e.g., no `requirements.txt`, no `utils/` directory, no `.elasticache/` folder), generate them rather than refusing. The skill should be able to bootstrap a project from scratch:
- If `.elasticache/requirements.json` does not exist, create it with defaults after gathering minimum inputs. If the project has a `.gitignore`, add `.elasticache/` to it. The file may contain VPC IDs, subnet IDs, and endpoint URLs that should not be committed to source control.
- If the user's project has no dependency file (`requirements.txt`, `package.json`, etc.), create one with the needed client library (`valkey`, `redis`, `ioredis`, `lettuce`, etc.).
- If a utility file referenced by `infrastructure.embedding_module` does not exist yet, generate it when the user first works with embeddings.
- If the user asks for working code but has an empty project directory, scaffold the minimum structure: connection utility, main application file, and dependency file. Do not require pre-existing project structure.
## Additional references
These files are not part of the main workflow but load on demand when the situation requires them:
- `references/shared-ux/production-readiness.md` -- when the user asks "is my cache ready for production?" or wants a pre-production gate checklist
- `references/setup/connectivity-diagnostics.md` -- when connection validation fails
- `references/setup/iac-best-practices.md` -- when the user asks about CloudFormation, Terraform, CDK, IaC deployment issues, stack failures, or deploying multiple caches
- `references/setup/cluster-topology.md` -- when configuring node-based cluster mode, shard count, replica count, or subnet IP planning
- `references/setup/service-quotas.md` -- when the user asks about quotas, limits, capacity errors ("maximum number of nodes"), or how to request a limit increase
- `references/shared-foundation/attribution.md` -- when generating CLI commands, SDK code, or IaC
- `references/shared-runtime/lambda.md` -- when connecting from Lambda (cold start, IAM auth, lazy init)
- `references/shared-runtime/ecs.md` -- when connecting from ECS (SIGTERM, task definition, pool drain)
- `references/shared-runtime/eks.md` -- when connecting from EKS (IRSA, service mesh, SecurityGroupPolicy)
- `references/shared-runtime/rds-acceleration.md` -- when caching RDS/Aurora queries (thundering herd, invalidation)
- `references/shared-security/encryption-defaults.md` -- when adding encryption to an existing unencrypted cluster
## Freshness disclaimer
When your response includes pricing, version constraints, or feature availability, include the freshness disclaimer per SKILL.md Global Rule #5: "For current pricing see https://aws.amazon.com/elasticache/pricing/. For current feature availability see https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/."
references/setup/serverless-vs-node.md
# Serverless vs Node-Based
## Hard Routing Rules (non-negotiable)
These constraints override all other guidance. Never recommend serverless for these workloads.
1. **Vector search MUST use node-based Valkey 8.2 or above (recommend 9.0).** Serverless does not support vector search. Any workload requiring vector similarity (semantic cache, RAG, conversational memory with vector retrieval, recommendations via embeddings, semantic search) MUST be deployed on a node-based Valkey 8.2 or above cluster (recommend 9.0). Do NOT suggest serverless as a fallback or future option for vector search.
2. **Global Datastore MUST use node-based clusters.** Multi-Region replication via Global Datastore is not available on serverless. Any workload requiring cross-Region reads or disaster recovery via Global Datastore MUST use a node-based deployment.
> ElastiCache Serverless supports Valkey 7.2+, Redis OSS 7.1+, and Memcached 1.6.22+. Regional availability may vary; check the [ElastiCache pricing page](https://aws.amazon.com/elasticache/pricing/) or the AWS Region Table before selecting a deployment model.
## Quick Decision
**Default to serverless** unless one of the hard routing rules or node-based requirements below applies.
| Factor | Serverless | Node-Based |
|--------|-----------|---------------------------|
| Setup time | 1-5 minutes | 5-15 minutes |
| Capacity planning | Automatic | Manual (choose node type, count) |
| Scaling | Instant, automatic | Manual or auto-scaling policies |
| Maintenance windows | None (zero downtime) | Required for patching |
| Pricing model | Pay per GB stored + ECPUs consumed | Pay per node-hour (reserved or on-demand) |
| Minimum cost | ~$6/month (Valkey) | Depends on node type |
| In-transit encryption (TLS) | Always on (cannot disable) | Optional (recommended) |
| Authentication | RBAC/IAM only (no AUTH tokens) | RBAC/IAM and AUTH tokens (legacy) |
| Best for | Variable/unpredictable traffic, new projects, getting started | Predictable high-throughput, cost optimization at scale |
## When serverless wins
- New projects or prototypes
- Variable or spiky traffic patterns
- Teams that don't want to manage infrastructure
- When time-to-value matters (under 1 minute to create)
- Workloads under ~100 GB or with unpredictable growth
## When node-based is required or preferred
- **Vector search** -- requires Valkey 8.2 or above on node-based clusters (recommend 9.0). Available in all AWS Regions at no additional cost. Not supported on data-tiering instances (r6gd) or serverless.
- **Data Tiering** -- the r6gd instance family (SSD-backed data tiering) is only available on node-based clusters.
- **Global Datastore** -- multi-Region replication requires node-based clusters. Global Datastore has specific engine version requirements and is available only in regions that support multi-Region replication. Check the [ElastiCache documentation](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/GlobalDatastore.html) for supported regions and engine versions.
- **Reserved instances** -- reserved nodes save approximately 30-55% depending on term length and payment option (run `python3 scripts/price_calculator.py --mode node --node-type <type> --nodes <N> --show-ri-options` for current estimates)
- **Fine-grained parameter tuning** -- custom parameter groups for advanced optimization
- Predictable, steady-state high throughput with strict sub-millisecond latency at 50k+ connections
- Large datasets where reserved instances save significant cost
- Compliance requirements needing specific instance families
## Serverless Constraints and Limits
These constraints are inherent to the serverless deployment model and cannot be changed. Verify each one against the workload requirements before choosing serverless.
### Encryption and TLS
- **TLS is always on.** In-transit encryption cannot be disabled on serverless caches. All clients must connect with TLS enabled (`ssl=True` in valkey-py, `tls: {}` in ioredis, `--tls` in valkey-cli). If the application cannot use TLS, use node-based instead.
- **At-rest encryption is always on.** Cannot be disabled.
### Authentication
- **RBAC or IAM only.** AUTH token (password-only) authentication is not supported on serverless. Applications using legacy AUTH tokens must switch to RBAC users or IAM-based authentication before migrating to serverless.
### Connections
- **65,000 max concurrent connections.** Beyond this threshold, additional connections may or may not succeed depending on current load. If the workload consistently approaches this limit, use node-based with multiple nodes, or implement connection pooling.
- Connection pooling is strongly recommended regardless of deployment model.
### Storage and Throughput Limits
- **Maximum storage: 5,000 GiB.** Configurable via `CacheUsageLimits.DataStorage.Maximum`. Maximum per-slot storage is 32 GiB.
- **Maximum ECPU/s: 15,000,000 ECPU/s.** This is the hard platform maximum. `CacheUsageLimits.ECPUPerSecond.Maximum` lets you set a lower ceiling for cost control, but cannot exceed 15,000,000. Per-slot throughput ranges from 30,000 to 90,000 ECPU/s.
- For workloads requiring storage beyond 5 TB or predictable high throughput, evaluate node-based for cost efficiency.
### Feature Restrictions
- **No vector search.** Vector search requires Valkey 8.2 or above on node-based clusters (recommend 9.0).
- **No Global Datastore.** Multi-region replication requires node-based clusters.
- **Cluster-mode-enabled only.** All serverless caches operate in cluster mode. Clients must support cluster protocol (MOVED/ASK redirects). Non-cluster clients will fail to connect properly.
- **No keyspace notifications.** Keyspace events are not supported on serverless caches. Workloads relying on keyspace notifications must use node-based.
- **Eviction policy is volatile-lru (not configurable).** Only keys with a TTL set are candidates for eviction. Keys without TTL are never evicted. If storage reaches the maximum and no keys can be evicted, writes receive OOM errors. Always set a TTL on serverless.
- **No custom parameter groups.** Parameter tuning is not available on serverless.
- **Manual snapshots supported.** Use `CreateServerlessCacheSnapshot` for on-demand snapshots. Restore via `create-serverless-cache --snapshot-arns-to-restore`. Automatic daily snapshots are not enabled by default; set `SnapshotRetentionLimit` > 0 to enable them.
- **SELECT command not supported.** Multiple logical databases (SELECT 0, SELECT 1, etc.) are not available. All data lives in a single logical namespace. Use key prefixes for logical separation.
- **Lua scripting restrictions.** Scripts that access keys across multiple slots may fail. Keep scripts single-key or use hash tags to colocate keys.
### Operations
- **No maintenance windows.** Patching is applied transparently with zero downtime. No operator action required.
- **Provisioning time: 1-5 minutes.** While the console may say "under a minute," actual creation takes 1-5 minutes in practice.
### Networking
- **Port 6379 and 6380 on the same hostname.** Serverless uses port 6379 for writes and port 6380 for reads, both on the same endpoint hostname. Security groups must allow inbound TCP on both ports. Many clients connect to both ports automatically. Since TLS is mandatory, ensure no intermediate proxies strip TLS.
- **VPC required.** Serverless caches are always deployed inside a VPC.
### Latency
- **Single-digit millisecond latency.** Serverless does not guarantee sub-millisecond latency. For workloads requiring consistent sub-millisecond response times at high concurrency (50,000+ connections), use node-based.
## Serverless Performance Tuning
### ECPU consumption model
Serverless bills in ElastiCache Processing Units (ECPUs). How commands are charged:
- **Fixed commands** (GET, SET, HGET, INCR, EXISTS, PING, DEL, UNLINK, and other O(1) operations): each request consumes at least 1 ECPU, with 1 ECPU per KB of data transferred. A GET returning 3.2 KB of data consumes 3.2 ECPUs (see [ElastiCache pricing](https://aws.amazon.com/elasticache/pricing/)).
- **Non-fixed commands** (EVAL, SORT, MGET, HGETALL, and other variable-cost operations): consume ECPUs based on the higher of vCPU time or data transferred. For example, an HMGET that takes 3x the vCPU time of a GET and transfers 3.2 KB consumes 3.2 ECPUs (data wins). If it transfers only 2 KB, it consumes 3 ECPUs (vCPU wins). See the [ElastiCache Serverless pricing blog](https://aws.amazon.com/blogs/database/unlock-on-demand-cost-optimized-performance-with-amazon-elasticache-serverless/).
- **Pipelined commands** are charged individually (each command in the pipeline costs its own ECPUs), but pipelining still reduces round-trip latency and network overhead.
- **Non-key commands** (ACL, SELECT, ECHO, TIME, etc.): these consume ECPUs as fixed commands. INFO and PUBLISH are exceptions — they are not metered.
- **Pub/sub commands** (SUBSCRIBE, PUBLISH, etc.): SUBSCRIBE and UNSUBSCRIBE are not metered. SPUBLISH, SSUBSCRIBE, and SUNSUBSCRIBE are metered.
- **Metadata commands** (AUTH, MULTI, EXEC, CONFIG): these are not metered.
- **Replication/cluster commands** (REPLCONF, PSYNC, CLUSTER): these are internal to the serverless infrastructure and are not metered.
Use `scripts/command_classifier.py` with `INFO commandstats` output for ECPU estimation of an existing workload. Note: the classifier uses heuristic approximations; actual ECPU consumption may differ. For precise billing, use CloudWatch ECPU metrics on a running serverless cache.
### Key CloudWatch metrics for serverless
| Metric | What it tells you |
|--------|-------------------|
| `ElastiCacheProcessingUnits` (Sum) | Total ECPUs consumed per period. Primary cost driver. |
| `ThrottledCmds` (Sum) | Commands rejected due to ECPU limit. Any value > 0 needs action. |
| `SuccessfulReadRequestLatency` (p99) | Read latency. Baseline is single-digit ms; spikes suggest hot keys or large payloads. |
| `SuccessfulWriteRequestLatency` (p99) | Write latency. Same baseline expectations as reads. |
| `BytesUsedForCache` | Storage consumed. Drives the storage component of the bill. |
| `CurrConnections` | Active connections against the 65K limit. |
### When to increase ECPU limits
If `ThrottledCmds > 0` sustained across multiple minutes, raise the ceiling:
```bash
aws elasticache modify-serverless-cache \
--serverless-cache-name <name> \
--cache-usage-limits '{"ECPUPerSecond": {"Maximum": <higher-value>}}' \
--region <region>
```
Before increasing, check whether expensive commands (SORT, large LRANGE, unoptimized Lua scripts) are inflating ECPU consumption. Optimizing command patterns is cheaper than raising limits.
### Client-side optimization
- Keep payloads small. Every KB transferred adds ECPUs. Compress values over 1 KB (gzip or lz4) when the workload is read-heavy.
- Use pipelining to batch commands and reduce round trips. Each command still costs ECPUs, but latency drops significantly.
- Prefer MGET/MSET over loops of GET/SET for bulk operations.
- Use hash tags `{tag}` to colocate related keys on the same slot when using multi-key commands.
## Serverless Connection Management
Proper connection pooling is critical given the 65K per-cache connection limit (documented above). For per-library pool configuration and code examples, see `../monitoring/client-tuning-and-diagnostics.md` (Connection Pooling section).
Key serverless-specific guidance:
- Monitor `CurrConnections` in CloudWatch. Alert at 50K to leave headroom for spikes.
- Lambda and short-lived compute: reuse connections across invocations (module-level client). Each new TLS handshake adds ~5 ms latency and a connection slot.
## Serverless Backup Behavior
- **Automatic daily snapshots**: serverless caches can take one automated snapshot per day, but this must be explicitly enabled by setting `SnapshotRetentionLimit` > 0. The default is 0 (disabled).
- **On-demand snapshots**: use `CreateServerlessCacheSnapshot` for on-demand logical snapshots of serverless caches.
- **Retention**: `SnapshotRetentionLimit` controls how many days automatic snapshots are kept (0 disables, max 35 days). Set via `--snapshot-retention-limit` on `create-serverless-cache` or `modify-serverless-cache`.
- **Cross-region snapshot copy**: not available natively for serverless. As a workaround, use the `export-serverless-cache-snapshot` API to export a snapshot to S3, then copy the S3 object to the target region. For long-term retention beyond 35 days, consider node-based clusters which support `copy-snapshot --target-bucket` for S3 export.
## Cost comparison tip
For workloads over ~50 GB with steady traffic, run `scripts/price_calculator.py --mode serverless --data-gb <N> --ops-per-sec <N>` and `--mode node --node-type <type> --nodes <N> --show-ri-options` to compare. Reserved node-based instances can be cheaper at scale, but serverless eliminates operational overhead.
references/setup/service-quotas.md
# Service Quotas and Limits
Default account-level quotas for ElastiCache. Most are adjustable via the Service Quotas console or CLI without filing a support ticket.
## Default Quotas
| Quota | Default | Adjustable | Quota Code |
|-------|---------|------------|------------|
| Nodes per region | 300 | Yes | L-DFE45DF3 |
| Nodes per cluster (cluster mode enabled) | 90 (max 500 for Valkey 7.2+ or Redis OSS 5.0.6+) | Yes | L-AF354865 (verify in your account via `list-service-quotas`) |
| Nodes per cluster (Memcached) | 60 | Yes | L-8C334AD1 |
| Nodes per shard (architectural limit) | 6 | No | — |
| Parameter groups per region | 300 | Yes | L-3F15A733 |
| Serverless caches per region | 40 | Yes | L-BBCDAECC |
| Serverless snapshots per day per cache | 24 | Yes | L-75A7B5A4 |
| Subnet groups per region | 300 | Yes | L-3E7F7726 |
| Subnets per subnet group | 20 | Yes | L-A87EE522 |
| User groups per region | 200 | Yes | L-AD484FC5 |
| Users per region | 2,000 | Yes | L-80E085C7 |
| Users per user group | 100 | Yes | L-943F0F1C |
Nodes per replication group breaks down as: shards x (1 primary + replicas per shard). For example, 6 shards x (1 primary + 5 replicas = 6 nodes/shard) = 36 nodes per group, well under the 90-node default.
Per-cache serverless limits (storage, ECPU) are adjusted directly via `modify-serverless-cache --cache-usage-limits`, not through Service Quotas.
## Checking Current Quotas
**CLI:**
```bash
aws service-quotas list-service-quotas --service-code elasticache --region <region>
```
**Console:** Service Quotas > AWS services > Amazon ElastiCache.
To check applied (effective) values vs defaults:
```bash
aws service-quotas get-service-quota --service-code elasticache --quota-code <quota-code>
```
## Requesting Increases
**Self-service (most quotas):**
```bash
aws service-quotas request-service-quota-increase \
--service-code elasticache \
--quota-code <quota-code> \
--desired-value <value>
```
Find the quota code from `list-service-quotas` output or the table above. Increases are typically approved within minutes for standard quotas.
**Serverless storage/ECPU:** Adjust directly, no quota request needed:
```bash
aws elasticache modify-serverless-cache \
--serverless-cache-name <name> \
--cache-usage-limits '{
"DataStorage": {"Minimum": 1, "Maximum": 10, "Unit": "GB"},
"ECPUPerSecond": {"Minimum": 1000, "Maximum": 30000}
}'
```
## Common Quota-Related Errors
| Error pattern | Likely quota | Action |
|---------------|-------------|--------|
| "Maximum number of nodes" | Nodes per region (300, L-DFE45DF3) | Request increase via Service Quotas |
| "Maximum number of cache parameter groups" | Parameter groups (300, L-3F15A733) | Delete unused groups or request increase |
| "Maximum number of subnet groups" | Subnet groups (300, L-3E7F7726) | Delete unused groups or request increase |
| Serverless creation fails at region limit | Serverless caches per region (40, L-BBCDAECC) | Request increase via Service Quotas |
references/shared-foundation/architecture-diagrams.md
# Architecture Diagrams
Mermaid diagrams for common ElastiCache deployment patterns.
---
## 1. Default Application Caching (Cache-Aside Pattern)
Standard cache-aside pattern with ElastiCache Serverless Valkey in front of a relational database. The application checks the cache first; on a miss, it queries the database and populates the cache.
```mermaid
flowchart LR
subgraph VPC
App["Application<br/>(Lambda / ECS / EKS / EC2)"]
subgraph ElastiCache["ElastiCache Serverless Valkey"]
Cache["Valkey<br/>TLS + Auth"]
end
subgraph Database["RDS / Aurora"]
DB["PostgreSQL / MySQL"]
end
end
App -->|"1. GET key<br/>(TLS, auth)"| Cache
Cache -->|"2a. Cache HIT<br/>return value"| App
App -->|"2b. Cache MISS<br/>query DB"| DB
DB -->|"3. Return rows"| App
App -->|"4. SET key + TTL<br/>populate cache"| Cache
style Cache fill:#2563eb,color:#fff
style DB fill:#7c3aed,color:#fff
style App fill:#059669,color:#fff
```
**Key characteristics:**
- Serverless Valkey is the default cache layer (deploys in under a minute)
- TLS is always on for all serverless caches (Valkey, Redis OSS, and Memcached)
- Application handles cache-miss logic (lazy loading)
- TTL controls staleness tolerance
- Write-through on database updates for active invalidation (optional)
---
## 2. RDS/Aurora Read Acceleration
ElastiCache sits between the application and the database, accelerating reads with optional write-through invalidation.
```mermaid
flowchart TB
subgraph VPC
subgraph AppLayer["Application Layer"]
App["Application<br/>(ECS / EKS)"]
end
subgraph CacheLayer["Cache Layer"]
Writer["Cache Writer<br/>(write-through)"]
Reader["Cache Reader<br/>(read-through)"]
Cache["ElastiCache<br/>Serverless Valkey"]
end
subgraph DataLayer["Data Layer"]
Primary["RDS Aurora<br/>Primary (writes)"]
Replica["RDS Aurora<br/>Read Replica"]
end
end
App -->|"Read request"| Reader
Reader -->|"1. Check cache"| Cache
Cache -->|"2a. HIT"| Reader
Reader -->|"2b. MISS<br/>query replica"| Replica
Replica -->|"3. Return data"| Reader
Reader -->|"4. Populate cache"| Cache
Reader -->|"Return to app"| App
App -->|"Write request"| Writer
Writer -->|"1. Write to DB"| Primary
Primary -->|"2. Confirm"| Writer
Writer -->|"3. Invalidate/update cache"| Cache
Primary -.->|"Replication"| Replica
style Cache fill:#2563eb,color:#fff
style Primary fill:#7c3aed,color:#fff
style Replica fill:#7c3aed,color:#fff
style App fill:#059669,color:#fff
style Writer fill:#d97706,color:#fff
style Reader fill:#059669,color:#fff
```
**Key characteristics:**
- Reads go through cache first (significantly faster than direct DB queries; per AWS published benchmarks, results vary by workload)
- Writes go to the database primary, then invalidate or update the cache
- Aurora read replicas handle cache misses and complex queries
- Reduces Aurora read replica load and RDS costs significantly (per AWS published benchmarks, results vary by workload)
- TTL prevents stale data from persisting if invalidation fails
---
## 3. AI/GenAI Architecture (Semantic Cache + Vector Search)
ElastiCache Valkey 8.2 or above (node-based) provides semantic caching and vector search for LLM applications, reducing Bedrock inference costs and latency.
```mermaid
flowchart TB
subgraph VPC
App["AI Application<br/>(Lambda / ECS)"]
subgraph ElastiCache["ElastiCache Node-Based<br/>Valkey 8.2+"]
SemCache["Semantic Cache<br/>(prompt embeddings)"]
VecSearch["Vector Search Index<br/>(knowledge base)"]
Memory["Agent Memory<br/>(conversation history)"]
end
subgraph AI["AI Services"]
Bedrock["Amazon Bedrock<br/>(Claude / Titan)"]
EmbModel["Embedding Model<br/>(Titan Embeddings)"]
end
subgraph Data["Data Sources"]
S3["S3<br/>(documents)"]
DB["RDS / Aurora<br/>(structured data)"]
end
end
App -->|"1. User prompt"| SemCache
SemCache -->|"2a. Semantic HIT<br/>return cached response"| App
SemCache -->|"2b. Semantic MISS"| EmbModel
EmbModel -->|"3. Prompt embedding"| VecSearch
VecSearch -->|"4. Relevant context<br/>(RAG retrieval)"| App
App -->|"5. Retrieve conversation<br/>history"| Memory
Memory -->|"6. Relevant past turns"| App
App -->|"7. Prompt + context +<br/>memory -> LLM"| Bedrock
Bedrock -->|"8. Response"| App
App -->|"9a. Cache response<br/>with embedding"| SemCache
App -->|"9b. Store conversation<br/>turn"| Memory
S3 -->|"Ingest + embed"| VecSearch
DB -->|"Ingest + embed"| VecSearch
style SemCache fill:#2563eb,color:#fff
style VecSearch fill:#2563eb,color:#fff
style Memory fill:#2563eb,color:#fff
style Bedrock fill:#f59e0b,color:#000
style EmbModel fill:#f59e0b,color:#000
style App fill:#059669,color:#fff
style S3 fill:#7c3aed,color:#fff
style DB fill:#7c3aed,color:#fff
```
**Key characteristics:**
- **Semantic cache**: Stores prompt embeddings and LLM responses. On a new prompt, computes embedding similarity to find cached answers. Significantly reduces inference cost and latency (per AWS published benchmarks, results vary by workload).
- **Vector search**: Stores document chunk embeddings for RAG. Retrieves semantically relevant context to ground LLM responses and reduce hallucinations.
- **Agent memory**: Stores conversation turns as vectors. Retrieves only relevant past interactions per LLM invocation to avoid context window overflow.
- **Deployment**: Requires node-based Valkey 8.2 or above (recommend 9.0; vector search is not available on serverless or on data tiering instances such as r6gd node types).
- **Data flow**: Documents are embedded and stored during ingestion. At query time, the embedding model converts the prompt to a vector, which is used for both semantic cache lookup and RAG retrieval.
---
## 4. Network Architecture (VPC Layout)
Common VPC layout showing how ElastiCache integrates with compute resources and security boundaries.
```mermaid
flowchart TB
subgraph VPC["VPC (10.0.0.0/16)"]
subgraph PublicSubnets["Public Subnets"]
ALB["Application<br/>Load Balancer"]
NAT["NAT Gateway"]
end
subgraph PrivateSubnetsA["Private Subnet AZ-a"]
ECS_A["ECS Tasks /<br/>EKS Pods"]
Lambda_A["Lambda<br/>ENIs"]
end
subgraph PrivateSubnetsB["Private Subnet AZ-b"]
ECS_B["ECS Tasks /<br/>EKS Pods"]
Lambda_B["Lambda<br/>ENIs"]
end
subgraph CacheSubnets["Cache Subnets (private)"]
Cache_A["ElastiCache<br/>Node AZ-a"]
Cache_B["ElastiCache<br/>Node AZ-b"]
end
SG_App["SG: App<br/>(outbound 6379/6380)"]
SG_Cache["SG: Cache<br/>(inbound 6379/6380<br/>from SG: App)"]
end
ALB --> ECS_A
ALB --> ECS_B
ECS_A -->|"TLS :6379"| Cache_A
ECS_B -->|"TLS :6379"| Cache_B
Lambda_A -->|"TLS :6379"| Cache_A
Lambda_B -->|"TLS :6379"| Cache_B
Cache_A <-.->|"Replication"| Cache_B
SG_App -.-> ECS_A
SG_App -.-> ECS_B
SG_App -.-> Lambda_A
SG_App -.-> Lambda_B
SG_Cache -.-> Cache_A
SG_Cache -.-> Cache_B
style Cache_A fill:#2563eb,color:#fff
style Cache_B fill:#2563eb,color:#fff
style ECS_A fill:#059669,color:#fff
style ECS_B fill:#059669,color:#fff
style Lambda_A fill:#059669,color:#fff
style Lambda_B fill:#059669,color:#fff
style ALB fill:#d97706,color:#fff
```
**Key characteristics:**
- ElastiCache runs in private subnets only (no public internet access)
- Security groups restrict inbound to port 6379 (and 6380 for serverless reader endpoint) from the app security group
- Multi-AZ deployment with nodes/endpoints in at least 2 availability zones
- Lambda requires VPC attachment and ENI capacity in the private subnets
- For local development, use SSM port forwarding or a jump host (no direct access)
references/shared-foundation/attribution.md
# Attribution
How to identify and track resources created or managed by the ElastiCache skill. This file is loaded on demand when generating CLI commands, SDK code, or IaC templates (see SKILL.md "Reference loading").
## Core Rule
Every AWS API call the skill generates -- whether via CLI, SDK, or script -- must include the `AWSSkill-ElastiCache` user-agent identifier. Every resource the skill creates must carry the attribution tags defined below. These two mechanisms work together: user-agent covers ephemeral actions (describe, modify), tags cover persistent resources.
## App ID / User-Agent
The `AWS_SDK_UA_APP_ID` environment variable makes all SDK and CLI calls include the skill identifier in the user-agent string.
**Environment variable:**
```
AWS_SDK_UA_APP_ID=AWSSkill-ElastiCache
```
### How It Works
- The AWS SDK (all languages) reads `AWS_SDK_UA_APP_ID` and appends it to the `User-Agent` HTTP header on every API call.
- CloudTrail records the `userAgent` field in every event log entry.
- CloudTrail Lake can query events by user-agent to find all actions taken by the skill.
### Where It Is Set (Primary Path)
- **AWS CLI / SDK**: The identifier is set via the AWS CLI and SDK as the primary path, through the `AWS_SDK_UA_APP_ID` environment variable or the SDK user-agent config. See "Where the Agent Must Set It" below.
### Where the Agent Must Set It
- **CLI commands**: Prefix any `aws elasticache` command block with `export AWS_SDK_UA_APP_ID=AWSSkill-ElastiCache`.
- **SDK code (control-plane)**: Set the app ID on the client. The config key varies by SDK:
- Python (boto3): `botocore.config.Config(user_agent_appid="AWSSkill-ElastiCache")`
- Node.js (SDK v3): Set `AWS_SDK_UA_APP_ID=AWSSkill-ElastiCache` environment variable (preferred), or programmatically via `new ElastiCacheClient({ appId: "AWSSkill-ElastiCache" })`
- Java (SDK v2): `.overrideConfiguration(ClientOverrideConfiguration.builder().appId("AWSSkill-ElastiCache").build())` or JVM system property `-Dsdk.ua.appId=AWSSkill-ElastiCache` (Note: the Kotlin SDK uses `-Daws.userAgentAppId` instead). Note: Java SDK v2 does not support the `AWS_SDK_UA_APP_ID` environment variable.
- Go (SDK v2): `config.WithAppID("AWSSkill-ElastiCache")`
- **Scripts**: Bundled scripts that use boto3 should include the `user_agent_appid` config when constructing clients.
### Where It Does NOT Apply
- **CloudFormation / CDK deploys**: CloudFormation does not support custom user-agent strings. Use resource tags for attribution instead (see below).
- **Data-plane client connections** (redis-py, ioredis, Lettuce, go-redis connecting to the cache): These are not AWS API calls. Do not add `AWS_SDK_UA_APP_ID` to data-plane client code.
## Resource Tags
Tag schema (`managed_by`, `skill`, `skill_version`, `created_by`, `generation_model`) and examples are in `setup/create-secure-cache.md`. This section covers only the gotchas.
### Enterprise Tag Policy Compatibility
If the target AWS account uses AWS Organizations tag policies that restrict allowed tag keys, the skill's default tags (`managed_by`, `skill`, `skill_version`, `created_by`, `generation_model`) may be rejected. If tag application fails due to an Organization tag policy, remove the conflicting tags and proceed. Attribution tags are best-effort, not a gate -- cache creation should never fail because of attribution.
## What NOT to Use
- **Do not** use key prefixes in cached data for attribution. This pollutes the data namespace and adds overhead to every operation.
- **Do not** use synthetic commands or hidden keys to mark the cache as skill-managed. This creates operational confusion.
- **Do not** use outbound beacons or telemetry calls. The skill does not phone home.
references/shared-foundation/boundary-doc.md
# ElastiCache Skill -- Ownership Boundary Document
This document defines what the ElastiCache skill owns directly and what it defers to AWS documentation.
## What the Skill Owns Directly
The skill is the authoritative source for the following capabilities. It produces these outputs inline, without requiring external tool calls.
* **Intent routing.** Classifying user requests across the six sub-skills (requirements, setup, data-modeling, genai, monitoring, migration) and selecting the correct pipeline.
* **Architecture guidance.** Recommending engine (Valkey, Redis OSS, Memcached), deployment model (serverless vs node-based), and topology (standalone, cluster mode, Global Datastore) based on workload requirements.
* **Hard routing enforcement.** Ensuring vector search always routes to node-based Valkey 8.2 or later (serverless caches do not support vector search; data-tiering node types are also excluded from vector search), Global Datastore always routes to node-based (requires M5, M6g, M7g, R5, R6g, R6gd, R7g, or C7gn instance families, size large and above), serverless auth never includes AUTH tokens, and online migration targets do not have encryption in-transit enabled, sources do not have AUTH enabled, and targets have Multi-AZ enabled. Online migration has additional prerequisites; see `references/migration/topology-validation.md` for the full preflight checklist.
* **IaC generation.** Producing CloudFormation, CDK, and Terraform templates for cache creation, security groups, IAM policies, and RBAC user setup.
* **Connection recipes.** Generating SDK connection snippets (Python, Java, Node.js, Go, CLI) with correct TLS, auth, and VPC configuration for both serverless and node-based deployments.
* **Observability bootstrap.** Generating CloudWatch dashboard and alarm definitions via `scripts/generate_dashboards.py`, and providing metric-driven troubleshooting guidance.
* **Cost estimation.** Running `scripts/price_calculator.py` to compare serverless vs node-based pricing for a given workload profile.
* **Security audit.** Running `scripts/security_audit.py` to validate post-creation security posture.
* **Migration planning.** Producing migration runbooks, preflight checks (via `scripts/migration_preflight.py`), validation steps, and rollback procedures.
* **Data modeling patterns.** Recommending data structures, key schemas, TTL strategies, and invalidation approaches for common use cases (cache-aside, session store, rate limiting, leaderboards, etc.).
* **GenAI pattern design.** Designing semantic cache, conversational memory, RAG, recommendation, and vector search implementations using Valkey.
## Execution Paths
The skill generates instructions, code, or parameters for each operation. The primary execution paths are AWS CLI, SDK, and valkey-py code.
* **Control-plane operations** (creating, modifying, and deleting serverless caches and replication groups; creating and managing RBAC users and user groups; describing cache clusters, endpoints, and configuration; creating and managing snapshots and backups; setting up Global Datastore; creating jump hosts and generating SSH tunnel commands): the skill generates AWS CLI commands, boto3 SDK calls, or CloudFormation/CDK templates.
* **Data-plane operations** (Valkey or Memcached commands, vector search, JSON operations against a live endpoint): the skill generates valkey-py (Python) code that runs commands via `execute_command()`.
## What the Skill Defers to AWS Documentation
The skill does not attempt to be exhaustive on edge cases, tuning parameters, or service limits. It links to official AWS docs for the following.
* **Parameter group tuning.** The skill recommends starting defaults but defers to the ElastiCache User Guide for the full parameter reference and advanced optimization.
* **Service limits and quotas.** Current limits for nodes per cluster, connections, memory, ECPUs, and other quotas change over time and live in the AWS Service Quotas console and documentation.
* **Version-specific release notes.** Detailed engine release notes, patch contents, and deprecation timelines are maintained in the ElastiCache User Guide.
* **Edge-case configurations.** Unusual topologies (e.g., multi-AZ with read replicas in specific failure modes), niche parameter interactions, and rarely used features are covered in the User Guide.
* **Compliance and certification details.** Specific compliance programs (HIPAA, PCI, FedRAMP) and their ElastiCache coverage are documented in AWS compliance resources.
* **Pricing changes.** The skill uses `scripts/price_calculator.py` for estimation but defers to the official AWS ElastiCache Pricing page for authoritative, current pricing.
* **API Reference details.** Full API request/response schemas, error codes, and throttling behavior are in the ElastiCache API Reference.
## AI-Generated Output Disclaimer
All code, configurations, CLI commands, and recommendations produced by this skill are AI-generated. Review all outputs before deploying to production environments.
references/shared-runtime/api-gateway.md
# API Gateway + ElastiCache Integration
There is no direct integration path from Amazon API Gateway to ElastiCache. API Gateway cannot connect to a VPC-hosted cache endpoint on its own. All access goes through a compute layer that bridges the public API surface to the private VPC where ElastiCache runs.
## Integration Patterns
### Pattern 1: API Gateway -> Lambda -> ElastiCache (most common)
The simplest and most widely used path. API Gateway invokes a Lambda function that reads from or writes to ElastiCache.
```
Client -> API Gateway -> Lambda (VPC-attached) -> ElastiCache (VPC)
```
Requirements:
- Lambda must be attached to the VPC where ElastiCache resides
- Lambda execution role needs ENI permissions (`ec2:CreateNetworkInterface`, `ec2:DescribeNetworkInterfaces`, `ec2:DeleteNetworkInterface`, `ec2:DescribeSubnets`, `ec2:AssignPrivateIpAddresses`, `ec2:UnassignPrivateIpAddresses`). Alternatively, use the managed policy `AWSLambdaVPCAccessExecutionRole` which includes all required permissions.
- Lambda security group must have outbound access to the cache security group on port 6379 (and also port 6380 if using the serverless read-optimized endpoint for lower-latency eventually-consistent reads; routing is handled by the endpoint itself, not by issuing the READONLY command)
- For IAM auth: Lambda execution role needs `elasticache:Connect` permission on both the cache resource ARN (`arn:aws:elasticache:<region>:<account-id>:serverlesscache:<cache-name>` or `arn:aws:elasticache:<region>:<account-id>:replicationgroup:<rg-id>`) and the user resource ARN (`arn:aws:elasticache:<region>:<account-id>:user:<user-id>`). IAM auth requires Redis OSS 7.0+ or Valkey 7.2+, and the ElastiCache user-id and user-name must be identical.
- Default to serverless cache to match Lambda's variable invocation pattern. Note that serverless caches are cluster-mode-enabled only, so your client library must use a cluster-aware client (e.g., `ValkeyCluster`, `RedisCluster`).
Best for: REST APIs, HTTP APIs, WebSocket APIs with caching needs, low-to-medium throughput workloads.
### Pattern 2: API Gateway -> VPC Link -> NLB/ALB -> ECS/EKS -> ElastiCache
For container-based backends that already run in a VPC. API Gateway connects through a VPC Link to a load balancer (NLB for REST APIs, or ALB/NLB for HTTP APIs) in front of ECS tasks or EKS pods.
```
Client -> API Gateway -> VPC Link -> NLB/ALB -> ECS/EKS -> ElastiCache (same VPC)
```
Requirements:
- REST API VPC Links (V1) support NLB only. REST APIs also support VPC Link V2 with ALB as a target. HTTP API VPC Links (V2) support ALB, NLB, or Cloud Map.
- Load balancer must be in the same VPC as ElastiCache
- ECS tasks use `awsvpc` network mode; EKS pods use VPC CNI
- Task or pod security group must allow outbound to cache security group on port 6379
Best for: high-throughput APIs, long-lived connections with connection pooling, workloads already running on containers.
## Distributed Rate Limiting with ElastiCache
Use ElastiCache as the backing store for rate limiting behind API Gateway. This pattern supplements API Gateway's built-in throttling with application-level granularity.
```
Client -> API Gateway -> Lambda/Container -> ElastiCache (rate limit check) -> Backend
```
How it works:
1. API Gateway forwards the request to the compute layer
2. The compute layer increments a counter in ElastiCache keyed by user, API key, or IP
3. If the counter exceeds the limit, the compute layer returns HTTP 429 immediately
4. If within limits, the request proceeds to the backend logic
This enables:
- Per-user or per-tenant rate limits (API Gateway throttling is per-stage or per-key)
- Sliding-window rate limiting using sorted sets
- Shared rate limit state across multiple Lambda functions or container instances
- Custom rate limit rules that change dynamically without redeploying
See `references/data-modeling/common-patterns.md` for the rate limiting data structure pattern.
## Caching Layers: API Gateway Stage Cache vs ElastiCache
These are complementary, not competing, layers.
| Aspect | API Gateway Stage Cache | ElastiCache |
|--------|------------------------|-------------|
| Layer | HTTP response cache at the gateway (REST APIs only, not HTTP APIs) | Application-level cache inside VPC |
| Granularity | Full HTTP response by URL + query params | Any data structure, any key schema |
| Scope | Single API Gateway stage | Shared across all compute in the VPC |
| TTL control | Per-method or per-stage | Per-key, per-pattern |
| Invalidation | Cache flush or TTL expiry | Fine-grained key invalidation, write-through |
| Cost | Included in API Gateway caching charge | Separate ElastiCache pricing |
| Use case | Cache entire API responses for read-heavy public endpoints | Cache database queries, sessions, rate limits, computed results |
When to use both:
- API Gateway stage cache for static or semi-static GET responses that rarely change
- ElastiCache for dynamic, fine-grained caching behind the API (database queries, session lookups, computed aggregations)
- The two layers do not conflict; API Gateway cache reduces calls to Lambda/containers, and ElastiCache reduces calls to databases and external services within those functions
When to skip API Gateway cache:
- POST/PUT/DELETE requests (not cacheable at the gateway)
- Personalized responses that vary per user
- Responses that need sub-second invalidation
- When ElastiCache already handles the caching and the API response is always computed fresh from cached data
references/shared-runtime/ecs.md
# ECS Integration with ElastiCache
## Network Mode: awsvpc
Use `awsvpc` network mode for ECS tasks. Each task gets its own Elastic Network Interface (ENI) with a private IP in the VPC, enabling direct communication with ElastiCache endpoints.
## Security Group Configuration
The ECS task security group must allow outbound traffic to the cache security group:
| Direction | Protocol | Port | Target | Purpose |
|-----------|----------|------|--------|---------|
| Task SG outbound | TCP | 6379 | Cache SG | Primary endpoint |
| Task SG outbound | TCP | 6380 | Cache SG | Reader endpoint (serverless only) |
| Cache SG inbound | TCP | 6379 | Task SG | Allow from ECS tasks |
| Cache SG inbound | TCP | 6380 | Task SG | Allow from ECS tasks (serverless) |
## Task Definition Configuration
### Environment variables
```json
{
"containerDefinitions": [
{
"name": "app",
"environment": [
{ "name": "CACHE_ENDPOINT", "value": "my-cache.serverless.use1.cache.amazonaws.com" },
{ "name": "CACHE_PORT", "value": "6379" }
],
"secrets": [
{
"name": "CACHE_PASSWORD",
"valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:elasticache/myapp/appuser:password::"
},
{
"name": "CACHE_USERNAME",
"valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:elasticache/myapp/appuser:username::"
}
]
}
]
}
```
> **`valueFrom` pattern:** The format is `<secret-arn>:<json-key>:<version-stage>:<version-id>`. The trailing `::` leaves version-stage and version-id empty, which resolves to the latest version (AWSCURRENT).
When using IAM auth instead of passwords, omit the `secrets` block and configure the task role with `elasticache:Connect` permissions.
> **IAM auth note:** The ElastiCache user-id and user-name must be identical when using IAM-authenticated users. If they differ, the IAM auth token signing will not match and the connection will be rejected.
### Task role permissions (IAM auth)
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ElastiCacheConnect",
"Effect": "Allow",
"Action": "elasticache:Connect",
"Resource": [
"arn:aws:elasticache:<region>:<account-id>:serverlesscache:<cache-name>",
"arn:aws:elasticache:<region>:<account-id>:replicationgroup:<rg-id>",
"arn:aws:elasticache:<region>:<account-id>:user:<user-id>"
]
}
]
}
```
Use `serverlesscache:` for serverless caches, `replicationgroup:` for node-based clusters. Include both if the policy covers multiple deployment types.
### Task execution role permissions (Secrets Manager)
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SecretsAccess",
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:<region>:<account-id>:secret:elasticache/*"
}
]
}
```
> **KMS note:** If the secret is encrypted with a customer-managed KMS key (not the default `aws/secretsmanager` key), the execution role also needs `kms:Decrypt` permission on that KMS key ARN. The default AWS-managed key does not require this additional permission.
## Connection Pooling
ECS tasks are long-lived (unlike Lambda). Use connection pools to manage connections efficiently:
- **Pool size**: Match to expected concurrent command usage per task. Start with 10-20 connections.
- **Keep-alive**: Enable TCP keepalive to prevent idle connection drops by NAT gateways or load balancers.
- **Health checks**: Periodically ping the connection to detect stale connections early.
- **Graceful shutdown**: Drain connections on task stop (handle SIGTERM).
## Python Connection (valkey-py with TLS and connection pool)
```python
import os
import valkey
# Connection pool: shared across all requests within this task
# For ElastiCache Serverless (cluster mode enabled), use ValkeyCluster:
_client = valkey.cluster.ValkeyCluster(
host=os.environ['CACHE_ENDPOINT'],
port=int(os.environ.get('CACHE_PORT', '6379')),
username=os.environ.get('CACHE_USERNAME', 'appuser'),
password=os.environ.get('CACHE_PASSWORD', ''),
ssl=True,
ssl_cert_reqs='required',
decode_responses=True,
max_connections=20,
socket_connect_timeout=5,
socket_timeout=5,
socket_keepalive=True,
retry_on_timeout=True,
health_check_interval=30,
)
# For node-based cluster-mode-disabled deployments, use instead:
# _pool = valkey.ConnectionPool(
# host=os.environ['CACHE_ENDPOINT'],
# port=int(os.environ.get('CACHE_PORT', '6379')),
# ...
# )
# def get_cache_client() -> valkey.Valkey:
# return valkey.Valkey(connection_pool=_pool)
def get_cache_client() -> valkey.cluster.ValkeyCluster:
"""Return the shared cluster-aware client."""
return _client
# Usage in a request handler (e.g., Flask, FastAPI)
def handle_request(item_id: str) -> dict:
cache = get_cache_client()
key = f"item:{item_id}"
cached = cache.get(key)
if cached:
return {'data': cached, 'source': 'cache'}
result = fetch_from_database(item_id)
cache.set(key, result, ex=300)
return {'data': result, 'source': 'database'}
# Graceful shutdown on SIGTERM
import signal
import sys
def shutdown_handler(signum, frame):
_client.close()
sys.exit(0)
signal.signal(signal.SIGTERM, shutdown_handler)
```
## Node.js Connection (iovalkey with TLS and connection pool)
```javascript
const { Cluster } = require('iovalkey');
// Cluster client with built-in connection pooling
const client = new Cluster(
[{ host: process.env.CACHE_ENDPOINT, port: parseInt(process.env.CACHE_PORT || '6379', 10) }],
{
dnsLookup: (address, callback) => callback(null, address),
slotsRefreshTimeout: 2000,
redisOptions: {
tls: {},
username: process.env.CACHE_USERNAME || 'appuser',
password: process.env.CACHE_PASSWORD,
connectTimeout: 5000,
commandTimeout: 5000,
keepAlive: 30000,
},
// Retry strategy for transient failures
clusterRetryStrategy: (times) => Math.min(times * 100, 3000),
}
);
client.on('error', (err) => {
console.error('Cache connection error:', err.message);
});
// Usage in an Express/Fastify route
async function handleRequest(req, res) {
const key = `item:${req.params.itemId}`;
const cached = await client.get(key);
if (cached) {
return res.json({ data: cached, source: 'cache' });
}
const result = await fetchFromDatabase(req.params.itemId);
await client.set(key, result, 'EX', 300);
return res.json({ data: result, source: 'database' });
}
// Graceful shutdown
process.on('SIGTERM', async () => {
await client.quit();
process.exit(0);
});
```
## CloudFormation snippet: ECS service with cache access
```yaml
ECSTaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
NetworkMode: awsvpc
TaskRoleArn: !GetAtt TaskRole.Arn
ExecutionRoleArn: !GetAtt ExecutionRole.Arn
ContainerDefinitions:
- Name: app
Image: !Ref AppImage
Environment:
- Name: CACHE_ENDPOINT
Value: !ImportValue cache-stack-Endpoint
- Name: CACHE_PORT
Value: '6379'
Secrets:
- Name: CACHE_PASSWORD
ValueFrom: !Sub 'arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:elasticache/myapp/appuser:password::'
ECSService:
Type: AWS::ECS::Service
Properties:
TaskDefinition: !Ref ECSTaskDefinition
NetworkConfiguration:
AwsvpcConfiguration:
Subnets: !Ref PrivateSubnetIds
SecurityGroups:
- !Ref TaskSecurityGroup
```
references/shared-runtime/eks.md
# EKS Integration with ElastiCache
## VPC CNI: Pods Get VPC IPs
With the Amazon VPC CNI plugin (default on EKS), each pod receives a VPC IP address. This means pods can reach ElastiCache endpoints directly without NAT or proxies, as long as security groups allow the traffic.
## Security Group Configuration
| Approach | How it works |
|----------|-------------|
| **Node security group** (default) | All pods on a node share the node's SG. Add an outbound rule from the node SG to the cache SG on port 6379. |
| **Security groups for pods** (recommended for production) | Pods get their own SGs via the `SecurityGroupPolicy` CRD. Allows per-workload least-privilege access to the cache. |
### Node SG approach (simpler)
Add to the EKS node security group:
| Direction | Protocol | Port | Target | Purpose |
|-----------|----------|------|--------|---------|
| Node SG outbound | TCP | 6379 | Cache SG | Cache endpoint (primary and reader) |
| Node SG outbound | TCP | 6380 | Cache SG | Reader endpoint (serverless only) |
| Cache SG inbound | TCP | 6379 | Node SG | Allow from EKS nodes |
| Cache SG inbound | TCP | 6380 | Node SG | Allow from EKS nodes (serverless) |
### Security groups for pods (per-workload isolation)
First, attach the `AmazonEKSVPCResourceController` managed IAM policy to your EKS cluster role (required prerequisite). Then enable the VPC CNI `ENABLE_POD_ENI` setting and create a `SecurityGroupPolicy`:
```bash
# Attach the required managed policy to the cluster role
aws iam attach-role-policy \
--role-name <your-eks-cluster-role> \
--policy-arn arn:aws:iam::aws:policy/AmazonEKSVPCResourceController
# Enable pod ENI on the VPC CNI
kubectl set env daemonset aws-node -n kube-system ENABLE_POD_ENI=true
```
Then create a `SecurityGroupPolicy`:
```yaml
apiVersion: vpcresources.k8s.aws/v1beta1
kind: SecurityGroupPolicy
metadata:
name: cache-access
namespace: myapp
spec:
podSelector:
matchLabels:
cache-access: "true"
securityGroups:
groupIds:
- sg-0123456789abcdef0 # Pod SG that allows outbound to cache
```
Add the label `cache-access: "true"` to pods that need cache access.
## Secret Wiring
### Recommended: IAM Roles for Service Accounts (IRSA) with IAM Auth
For IAM auth (no passwords needed), configure IRSA so the pod's service account can call `elasticache:Connect`. This is the preferred approach for Valkey 7.2+ / Redis OSS 7.0+ because it eliminates password management entirely.
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: myapp-sa
namespace: myapp
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/myapp-cache-role
```
The IAM role trust policy must allow the OIDC provider, and the role must have:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "elasticache:Connect",
"Resource": [
"arn:aws:elasticache:<region>:<account-id>:serverlesscache:<cache-name>",
"arn:aws:elasticache:<region>:<account-id>:replicationgroup:<rg-id>",
"arn:aws:elasticache:<region>:<account-id>:user:<user-id>"
]
}
]
}
```
Use `serverlesscache:` for serverless caches, `replicationgroup:` for node-based clusters. Include both if the policy covers multiple deployment types.
### Alternative: EKS Pod Identity with IAM Auth
EKS Pod Identity (available on EKS 1.28+ with platform version eks.4+) is a newer alternative to IRSA with simpler setup. Both work for ElastiCache IAM auth. See AWS documentation for Pod Identity configuration.
### Alternative: External Secrets Operator (ESO) with RBAC Passwords
Use this when IAM auth is not available (older engine versions) or when the client library does not support IAM token generation. The External Secrets Operator syncs secrets from AWS Secrets Manager into Kubernetes secrets automatically.
Install ESO, then create an `ExternalSecret`:
```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: cache-credentials
namespace: myapp
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: cache-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: elasticache/myapp/appuser
property: username
- secretKey: password
remoteRef:
key: elasticache/myapp/appuser
property: password
```
### Alternative: AWS Secrets and Configuration Provider (ASCP) with RBAC Passwords
Mount secrets directly as files in the pod using the CSI Secrets Store driver with the AWS provider:
```yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: cache-credentials
namespace: myapp
spec:
provider: aws
parameters:
objects: |
- objectName: "elasticache/myapp/appuser"
objectType: "secretsmanager"
jmesPath:
- path: username
objectAlias: cache-username
- path: password
objectAlias: cache-password
secretObjects:
- secretName: cache-credentials
type: Opaque
data:
- objectName: cache-username
key: username
- objectName: cache-password
key: password
```
## Pod Deployment with Cache Access
> **Auth paths:** The example below shows env vars for both IAM auth (via IRSA service account) and RBAC password auth (via Kubernetes secrets). In practice, choose one path: use `serviceAccountName` with IAM token generation and omit the `CACHE_USERNAME`/`CACHE_PASSWORD` secret refs for IAM auth, or use the secret refs and omit the IRSA service account annotation for RBAC password auth.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
namespace: myapp
spec:
replicas: 3
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
cache-access: "true" # For SecurityGroupPolicy
spec:
serviceAccountName: myapp-sa # For IRSA (IAM auth) -- omit for RBAC password auth
containers:
- name: app
image: myapp:latest
env:
- name: CACHE_ENDPOINT
value: "my-cache.serverless.use1.cache.amazonaws.com"
- name: CACHE_PORT
value: "6379"
# The following are for RBAC password auth only.
# For IAM auth, generate a SigV4 token at runtime instead (see IAM Auth Token Generation below).
- name: CACHE_USERNAME
valueFrom:
secretKeyRef:
name: cache-credentials
key: username
- name: CACHE_PASSWORD
valueFrom:
secretKeyRef:
name: cache-credentials
key: password
```
## IAM Auth Token Generation
When using IRSA or EKS Pod Identity for IAM auth, the pod's service account provides AWS credentials automatically. Use them to generate a SigV4 presigned URL token:
```python
import os
import botocore.session
from botocore.signers import RequestSigner
from botocore.model import ServiceId
def get_iam_auth_token(cache_name: str, user_id: str, is_serverless: bool = True) -> str:
region = os.environ.get('AWS_REGION', 'us-east-1')
session = botocore.session.get_session()
creds = session.get_credentials().get_frozen_credentials()
signer = RequestSigner(
ServiceId("elasticache"), region, "elasticache", "v4",
creds, session.get_component("event_emitter"),
)
query = f"Action=connect&User={user_id}"
if is_serverless:
query += "&ResourceType=ServerlessCache"
url = signer.generate_presigned_url(
{"method": "GET", "url": f"https://{cache_name}/?{query}",
"body": {}, "headers": {}, "context": {}},
operation_name="connect", expires_in=900, region_name=region,
)
return url[len("https://"):] if url.startswith("https://") else url
```
Pass the returned token as the `password` parameter when connecting, with `user_id` as `username`.
## Connection Snippets
Connection code is the same as ECS. The key differences are in how secrets are injected (Kubernetes secrets vs ECS secrets) and how network access is configured (pod SG vs task SG).
### Python (redis-py/valkey-py with TLS and RBAC password auth)
> **Note:** The AWS-validated Python client is [redis-py](https://github.com/redis/redis-py) v4.1.2 (AWS-validated). The `valkey-py` package is the Valkey-specific fork with an API-compatible interface. Either can be used; the examples below use `valkey-py` but you can substitute `import redis` and `redis.RedisCluster` if preferred.
```python
import os
import valkey
# Serverless caches operate in cluster mode enabled only.
# You MUST use a cluster-aware client.
cache = valkey.ValkeyCluster(
host=os.environ['CACHE_ENDPOINT'],
port=int(os.environ.get('CACHE_PORT', '6379')),
username=os.environ.get('CACHE_USERNAME', 'appuser'),
password=os.environ.get('CACHE_PASSWORD', ''),
ssl=True,
ssl_cert_reqs='required',
decode_responses=True,
max_connections=20,
socket_connect_timeout=5,
socket_timeout=5,
socket_keepalive=True,
retry_on_timeout=True,
health_check_interval=30,
)
```
### Node.js (iovalkey with TLS)
```javascript
const { Cluster } = require('iovalkey');
const client = new Cluster(
[{ host: process.env.CACHE_ENDPOINT, port: parseInt(process.env.CACHE_PORT || '6379', 10) }],
{
dnsLookup: (address, callback) => callback(null, address),
slotsRefreshTimeout: 2000,
redisOptions: {
tls: {},
username: process.env.CACHE_USERNAME || 'appuser',
password: process.env.CACHE_PASSWORD,
connectTimeout: 5000,
commandTimeout: 5000,
keepAlive: 30000,
},
}
);
```
## Service Mesh Considerations
If using AWS App Mesh or Istio:
- **Envoy intercepts all outbound TCP by default**. Envoy sidecar proxies intercept all outbound TCP traffic by default via iptables rules. ElastiCache/Redis traffic is passed through as opaque TCP (not HTTP-routed). This can cause issues with MOVED redirects in cluster mode and TLS termination. Consider excluding ElastiCache ports from sidecar interception if you experience connectivity issues.
- **mTLS**: The mesh's mTLS does not apply to cache traffic. ElastiCache has its own TLS (always-on for serverless). Do not double-TLS.
- **Timeouts and retries**: Configure these in your application client, not in the mesh, since the mesh does not manage the cache protocol.
- **Port exclusion**: If you experience connectivity issues, add an outbound exclusion for the cache endpoint IP range or port 6379 (and port 6380 for serverless reader endpoint) in the mesh configuration.
references/shared-runtime/lambda.md
# Lambda Integration with ElastiCache
## VPC Attachment Requirement
Lambda must be deployed in the same VPC as the ElastiCache cache. Without VPC attachment, Lambda cannot reach ElastiCache endpoints.
Configure VPC in the Lambda function:
- Assign at least two private subnets (same ones the cache uses, or subnets that can route to them)
- Assign a security group that allows outbound traffic to the cache security group on port 6379 (and 6380 for serverless reader endpoint)
## IAM Permissions
The Lambda execution role needs two sets of permissions:
### VPC networking (required)
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "LambdaVPCAccess",
"Effect": "Allow",
"Action": [
"ec2:CreateNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface",
"ec2:DescribeSubnets",
"ec2:AssignPrivateIpAddresses",
"ec2:UnassignPrivateIpAddresses"
],
"Resource": "*"
}
]
}
```
Alternatively, attach the AWS managed policy `AWSLambdaVPCAccessExecutionRole`.
> **Note:** The `AWSLambdaVPCAccessExecutionRole` managed policy also includes CloudWatch Logs permissions (`logs:CreateLogGroup`, `logs:CreateLogStream`, `logs:PutLogEvents`). If using the custom policy above instead, attach `AWSLambdaBasicExecutionRole` alongside it to retain CloudWatch logging.
### ElastiCache Connect (required for IAM auth)
```json
{
"Sid": "ElastiCacheConnect",
"Effect": "Allow",
"Action": "elasticache:Connect",
"Resource": [
"arn:aws:elasticache:<region>:<account-id>:serverlesscache:<cache-name>",
"arn:aws:elasticache:<region>:<account-id>:replicationgroup:<rg-id>",
"arn:aws:elasticache:<region>:<account-id>:user:<user-id>"
]
}
```
Use `serverlesscache:` for serverless caches, `replicationgroup:` for node-based clusters. Include both if the policy covers multiple deployment types.
### Secrets Manager access (required for RBAC password auth)
```json
{
"Sid": "SecretsAccess",
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "arn:aws:secretsmanager:<region>:<account-id>:secret:elasticache/*"
}
```
## Environment Variables
Set these on the Lambda function configuration:
| Variable | Example | Purpose |
|----------|---------|---------|
| `CACHE_ENDPOINT` | `my-cache-abc123.serverless.use1.cache.amazonaws.com` | Cache hostname |
| `CACHE_PORT` | `6379` | Cache port |
| `CACHE_NAME` | `my-cache` | Cache name (used for IAM token generation) |
For RBAC password auth, reference a Secrets Manager secret rather than storing credentials in environment variables.
## Python Connection (valkey-py with IAM auth)
```python
import os
import boto3
import valkey
from valkey import AuthenticationError
# Module-level: connection is reused across warm invocations
_client = None
def _get_iam_auth_token(cache_name: str, user_id: str, is_serverless: bool = True) -> str:
"""Generate IAM auth token via SigV4 presigned URL."""
import botocore.session
from botocore.signers import RequestSigner
from botocore.model import ServiceId
region = os.environ.get('AWS_REGION', 'us-east-1')
session = boto3.Session()
creds = session.get_credentials().get_frozen_credentials()
signer = RequestSigner(
ServiceId("elasticache"), region, "elasticache", "v4",
creds, botocore.session.get_session().get_component("event_emitter"),
)
query = f"Action=connect&User={user_id}"
if is_serverless:
query += "&ResourceType=ServerlessCache"
url = signer.generate_presigned_url(
{"method": "GET", "url": f"https://{cache_name}/?{query}",
"body": {}, "headers": {}, "context": {}},
operation_name="connect", expires_in=900, region_name=region,
)
return url[len("https://"):] if url.startswith("https://") else url
def get_cache_client():
"""Lazy-init connection with IAM auth. Reuses across warm invocations."""
global _client
if _client is not None:
try:
_client.ping()
return _client
except Exception:
_client = None
endpoint = os.environ['CACHE_ENDPOINT']
port = int(os.environ.get('CACHE_PORT', '6379'))
user_id = os.environ.get('CACHE_USER_ID', 'appuser')
cache_name = os.environ.get('CACHE_NAME', 'cache-01')
is_serverless = os.environ.get('CACHE_IS_SERVERLESS', 'true').lower() == 'true'
token = _get_iam_auth_token(cache_name, user_id, is_serverless)
conn_kwargs = dict(
host=endpoint,
port=port,
username=user_id,
password=token,
ssl=True,
ssl_cert_reqs='required',
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=5,
retry_on_timeout=True,
)
if is_serverless:
# Serverless requires cluster-mode client
_client = valkey.ValkeyCluster(**conn_kwargs)
else:
_client = valkey.Valkey(**conn_kwargs)
return _client
def handler(event, context):
cache = get_cache_client()
# Example: cache-aside pattern
key = f"item:{event.get('item_id')}"
cached = cache.get(key)
if cached:
return {'statusCode': 200, 'body': cached, 'source': 'cache'}
# On miss: fetch from origin, store with TTL
result = fetch_from_database(event['item_id'])
cache.set(key, result, ex=300) # 5 minute TTL
return {'statusCode': 200, 'body': result, 'source': 'database'}
```
Key patterns in this snippet:
- **Module-level `_client`**: Connection persists across warm Lambda invocations, avoiding cold-start reconnection overhead
- **Lazy init with health check**: `ping()` validates the connection; recreates on failure
- **IAM auth token**: Generated fresh on each new connection (tokens valid 15 minutes, connections are automatically disconnected after 12 hours unless prolonged by re-authenticating with a new token). For long-lived connections, consider using a Valkey or Redis OSS client that supports a credentials provider interface to auto-generate tokens.
- **TLS enabled**: `ssl=True` is mandatory for serverless caches
- **Timeouts**: Explicit connect and socket timeouts prevent Lambda from hanging
## Python Connection (valkey-py with RBAC password from Secrets Manager)
```python
import os
import json
import valkey
import boto3
_client = None
_secrets_client = boto3.client('secretsmanager')
def _get_credentials() -> tuple[str, str]:
"""Retrieve RBAC credentials from Secrets Manager."""
secret_name = os.environ.get('CACHE_SECRET_NAME', 'elasticache/myapp/appuser')
response = _secrets_client.get_secret_value(SecretId=secret_name)
secret = json.loads(response['SecretString'])
return secret['username'], secret['password']
def get_cache_client() -> valkey.Valkey:
"""Lazy-init connection with RBAC password auth."""
global _client
if _client is not None:
try:
_client.ping()
return _client
except Exception:
_client = None
endpoint = os.environ['CACHE_ENDPOINT']
port = int(os.environ.get('CACHE_PORT', '6379'))
username, password = _get_credentials()
_client = valkey.Valkey(
host=endpoint,
port=port,
username=username,
password=password,
ssl=True,
ssl_cert_reqs='required',
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=5,
retry_on_timeout=True,
)
return _client
```
## Node.js Connection (iovalkey with TLS)
**Note:** This example reads credentials from environment variables for simplicity. In production, use IAM authentication or retrieve credentials from Secrets Manager instead of storing passwords in plaintext environment variables.
```javascript
const { Cluster } = require('iovalkey');
// Module-level: reused across warm invocations
let client = null;
function getCacheClient() {
if (client) return client;
const endpoint = process.env.CACHE_ENDPOINT;
const port = parseInt(process.env.CACHE_PORT || '6379', 10);
// For serverless or cluster-mode-enabled caches
client = new Cluster(
[{ host: endpoint, port }],
{
dnsLookup: (address, callback) => callback(null, address),
slotsRefreshTimeout: 2000,
redisOptions: {
tls: {},
username: process.env.CACHE_USERNAME || 'appuser',
password: process.env.CACHE_PASSWORD,
connectTimeout: 5000,
commandTimeout: 5000,
},
}
);
client.on('error', (err) => {
console.error('Cache connection error:', err.message);
client = null;
});
return client;
}
exports.handler = async (event) => {
const cache = getCacheClient();
const key = `item:${event.itemId}`;
const cached = await cache.get(key);
if (cached) {
return { statusCode: 200, body: cached, source: 'cache' };
}
const result = await fetchFromDatabase(event.itemId);
await cache.set(key, result, 'EX', 300);
return { statusCode: 200, body: result, source: 'database' };
};
```
## Secret Management
| Method | When to use | How |
|--------|------------|-----|
| **IAM auth (preferred)** | Valkey 7.2+ or Redis OSS 7.0+ serverless or node-based | No secrets to manage. Token generated from IAM credentials. |
| **Secrets Manager** | RBAC with password auth | Store `{"username":"...","password":"..."}` in Secrets Manager. Reference via `CACHE_SECRET_NAME` env var. Enable rotation. |
| **Never in env vars** | Passwords should not be stored as plaintext Lambda environment variables | Use Secrets Manager or IAM auth instead. |
## Cold Start Considerations
- Since the September 2019 Hyperplane ENI improvements, VPC attachment no longer adds significant cold-start latency (significantly reduced from the previous ~14 seconds to under 1 second, per the AWS blog on Hyperplane ENI improvements). This is a one-time cost per execution environment.
- Use provisioned concurrency for latency-sensitive workloads to keep Lambda warm.
- The lazy connection pattern above ensures the cache client is only created when first needed, not during module import (which would add to cold start time if the connection fails).
- IAM auth token generation adds ~50-100ms on first connection (approximate; not an AWS-documented figure). The token is reused for the lifetime of the connection.
references/shared-runtime/rds-acceleration.md
# RDS/Aurora Acceleration with ElastiCache
## Caching Patterns
### Cache-Aside (Lazy Loading)
The application manages the cache explicitly. On read, check cache first. On miss, read from RDS and populate cache.
```
Read path: App -> Cache? -> hit -> return cached data
-> miss -> RDS -> write to cache -> return data
Write path: App -> RDS -> invalidate/update cache
```
**Best for**: Most read-heavy workloads. Simple to implement. Application controls exactly what gets cached and for how long.
**Trade-off**: First request for any key is always a cache miss (cold start). Stale data possible between TTL expiry and next read.
### Read-Through
A cache layer handles RDS reads transparently. The application always reads from the cache layer, which fetches from RDS on a miss.
```
Read path: App -> Cache Layer -> hit -> return cached data
-> miss -> RDS -> store in cache -> return data
```
**Best for**: When you want to encapsulate caching logic in a middleware layer rather than scattering it across the application.
**Trade-off**: Requires a wrapper layer. More complexity than cache-aside but cleaner application code.
### Write-Through
The application writes to both cache and RDS simultaneously (or through a layer that does both). Every write updates the cache, so reads are always fresh.
```
Write path: App -> Cache Layer -> write to cache + write to RDS
Read path: App -> Cache -> always a hit (for previously written keys)
```
**Best for**: Workloads where read-after-write consistency is critical. Eliminates stale data for writes you control.
**Trade-off**: Higher write latency (two writes per operation). Cache fills with data that may never be read.
### Reading from Replicas
If you are using ElastiCache Serverless or have provisioned read replicas (node-based clusters), direct reads to replicas to achieve better scalability and/or lower latency. Reads from replicas are eventually consistent with the primary. In ElastiCache Serverless, reading from the replica port (6380) will direct reads to the client's local availability zone when possible, reducing retrieval latency.
> **Security group / NACL note:** If using the serverless reader port (6380), ensure your security groups and NACLs allow traffic on both port 6379 (primary) and port 6380 (reader). Missing rules for 6380 will cause reader-port connections to fail silently.
> **Best practice:** For node-based clusters with multiple read replicas, distribute read traffic across replicas rather than pinning all reads to a single replica. Cluster-aware clients handle this automatically; for non-clustered (CMD) deployments, use the reader endpoint, which DNS-load-balances across replicas.
## Invalidation Strategies
### TTL-based (simplest)
Set a time-to-live on every cached entry. Data becomes stale for at most TTL seconds.
```python
cache.set(f"user:{user_id}", json.dumps(user_data), ex=300) # 5 minute TTL
```
- **Pros**: Zero coordination. Works with any write pattern.
- **Cons**: Stale data window equals the TTL. Must balance freshness vs cache hit rate.
### Event-driven (RDS events to Lambda to cache invalidation)
Use application-level events, database triggers, or change data capture (CDC) tools like AWS DMS to detect data changes and trigger cache invalidation. For simpler cases, time-based TTL expiry provides eventual consistency without additional infrastructure.
**Note**: RDS and ElastiCache SNS/EventBridge event notifications are for operational events (failovers, backups, configuration changes), not row-level data changes. Data-change-driven invalidation requires application-level mechanisms.
```
App write -> Application event / CDC -> Lambda -> cache.delete(key)
```
- **Pros**: Near-real-time invalidation. No stale window.
- **Cons**: More infrastructure (event source, Lambda). Eventual consistency between event publish and cache delete.
### Application-level invalidation (most common)
Invalidate or update the cache in the same code path as the database write.
```python
def update_user(user_id: str, data: dict):
db.execute("UPDATE users SET ... WHERE id = %s", (user_id,))
cache.delete(f"user:{user_id}") # Invalidate on write
```
- **Pros**: Precise. No extra infrastructure. Immediate.
- **Cons**: Must be applied consistently in every write path. Easy to miss an invalidation.
## "Create cache from RDS settings" path
ElastiCache can inherit VPC, subnet group, and security group settings from an existing RDS instance. When creating a cache via the AWS Console, use "Create ElastiCache cache" from the RDS console and it will pre-populate network settings from your database. For IaC, reference the same VPC, subnets, and create a security group that allows traffic from the application SG.
## Python Example: Cache-Aside with valkey-py and psycopg2
```python
import os
import json
import valkey
import psycopg2
from psycopg2.extras import RealDictCursor
# Connection pool for cache (reuse across requests)
cache_pool = valkey.ConnectionPool(
host=os.environ['CACHE_ENDPOINT'],
port=int(os.environ.get('CACHE_PORT', '6379')),
username=os.environ.get('CACHE_USERNAME', 'appuser'),
password=os.environ.get('CACHE_PASSWORD', ''),
ssl=True,
ssl_cert_reqs='required', # Use 'required' for production; 'none' disables certificate verification
decode_responses=True,
max_connections=20,
socket_connect_timeout=5,
socket_timeout=5,
retry_on_timeout=True,
)
cache = valkey.Valkey(connection_pool=cache_pool)
# Database connection
db = psycopg2.connect(
host=os.environ['DB_HOST'],
dbname=os.environ['DB_NAME'],
user=os.environ['DB_USER'],
password=os.environ['DB_PASSWORD'],
sslmode='require',
)
DEFAULT_TTL = 300 # 5 minutes
def get_user(user_id: str) -> dict:
"""Cache-aside read: check cache first, fall back to RDS."""
cache_key = f"user:{user_id}"
# Step 1: Try cache
try:
cached = cache.get(cache_key)
if cached:
return json.loads(cached)
except valkey.ValkeyError as e:
# Cache failure should not break the application
print(f"Cache read error: {e}")
# Step 2: Cache miss - read from RDS
with db.cursor(cursor_factory=RealDictCursor) as cur:
cur.execute("SELECT id, name, email, plan FROM users WHERE id = %s", (user_id,))
row = cur.fetchone()
if row is None:
return None
user_data = dict(row)
# Step 3: Populate cache with TTL
try:
cache.set(cache_key, json.dumps(user_data, default=str), ex=DEFAULT_TTL)
except valkey.ValkeyError as e:
print(f"Cache write error: {e}")
return user_data
def update_user(user_id: str, name: str, email: str) -> None:
"""Write to RDS, then invalidate cache."""
with db.cursor() as cur:
cur.execute(
"UPDATE users SET name = %s, email = %s WHERE id = %s",
(name, email, user_id),
)
db.commit()
# Invalidate cache entry
try:
cache.delete(f"user:{user_id}")
except valkey.ValkeyError as e:
print(f"Cache invalidation error: {e}")
def get_user_orders(user_id: str, limit: int = 20) -> list:
"""Cache-aside for a query result set."""
cache_key = f"user:{user_id}:orders:limit:{limit}"
try:
cached = cache.get(cache_key)
if cached:
return json.loads(cached)
except valkey.ValkeyError:
pass
with db.cursor(cursor_factory=RealDictCursor) as cur:
cur.execute(
"SELECT id, total, status, created_at FROM orders WHERE user_id = %s ORDER BY created_at DESC LIMIT %s",
(user_id, limit),
)
rows = [dict(r) for r in cur.fetchall()]
try:
cache.set(cache_key, json.dumps(rows, default=str), ex=60) # Short TTL for order data
except valkey.ValkeyError:
pass
return rows
```
Key patterns in this example:
- **Cache failures are non-fatal**: Every cache operation is wrapped in try/except. The application falls back to RDS if cache is unavailable.
- **TTL on every key**: Prevents unbounded cache growth and limits stale data window.
- **Invalidate on write**: `update_user` deletes the cache entry after a successful database write.
- **Query result caching**: `get_user_orders` caches full query results with shorter TTL for frequently changing data.
## Python Example: Cache-Aside with valkey-py and PyMySQL
Replace psycopg2 with PyMySQL for MySQL/Aurora MySQL:
```python
import pymysql
db = pymysql.connect(
host=os.environ['DB_HOST'],
database=os.environ['DB_NAME'],
user=os.environ['DB_USER'],
password=os.environ['DB_PASSWORD'],
ssl={'ssl': True},
cursorclass=pymysql.cursors.DictCursor,
)
```
The caching logic is identical. Only the database driver changes.
## Java: Automatic Query Caching with the AWS Advanced JDBC Wrapper
For Java applications using JDBC with PostgreSQL, MySQL, or MariaDB, the [AWS Advanced JDBC Wrapper](https://github.com/aws/aws-advanced-jdbc-wrapper) provides a Remote Query Cache Plugin that automatically caches query results in ElastiCache with minimal code changes. Instead of implementing cache-aside logic manually, you annotate queries with SQL comment hints:
```java
// The SQL comment hint tells the plugin to cache this query for 300 seconds
ResultSet rs = stmt.executeQuery(
"/* CACHE_PARAM(ttl=300s) */ SELECT product_name, price FROM products WHERE category = 'electronics'"
);
```
The plugin handles cache lookups, misses, and population transparently. Prerequisites include AWS Advanced JDBC Wrapper 3.3.0+, Apache Commons Pool 2.11.1+, and Valkey Glide 2.3.0+. Both ElastiCache Serverless and node-based caches are supported.
Query caching is not recommended for queries where strong consistency is required, or for queries inside multi-statement transactions that require read-after-write consistency.
## Invalidation Gotchas
### Stale reads after write
**Problem**: Between a database write and cache invalidation (or TTL expiry), reads return stale data.
**Mitigation**: Use application-level invalidation (delete cache key immediately after DB write). For strict consistency, use write-through instead of cache-aside.
### Thundering herd
**Problem**: When a popular cache key expires, many concurrent requests all miss the cache and hit RDS simultaneously, overloading the database.
**Mitigation**: Use cache stampede protection. Only one request fetches from RDS while others wait:
```python
import time
import uuid
def get_with_stampede_protection(cache_key: str, fetch_fn, ttl: int = 300, lock_ttl: int = 10) -> str:
"""Cache-aside with distributed lock to prevent thundering herd."""
cached = cache.get(cache_key)
if cached:
return cached
# Try to acquire a lock
lock_key = f"lock:{cache_key}"
lock_value = str(uuid.uuid4())
acquired = cache.set(lock_key, lock_value, nx=True, ex=lock_ttl)
if acquired:
# This request won the lock: fetch from DB and populate cache
try:
result = fetch_fn()
cache.set(cache_key, result, ex=ttl)
return result
finally:
# Release lock (only if we still hold it)
if cache.get(lock_key) == lock_value:
cache.delete(lock_key)
else:
# Another request is fetching. Wait briefly, then retry cache.
for _ in range(10):
time.sleep(0.1)
cached = cache.get(cache_key)
if cached:
return cached
# Fallback: fetch from DB directly
return fetch_fn()
```
### Cache stampede on hot keys
**Problem**: A key with extremely high read rate expires, and even with lock protection, the single fetch may be too slow.
**Mitigation**: Use "early recompute" -- refresh the cache before TTL expires. Set a logical TTL shorter than the actual TTL:
```python
import json
import time
def set_with_early_recompute(cache_key: str, value: str, ttl: int = 300, buffer: int = 30):
"""Store value with metadata for early recompute."""
payload = json.dumps({'value': value, 'expires_at': time.time() + ttl - buffer})
cache.set(cache_key, payload, ex=ttl)
def get_with_early_recompute(cache_key: str, fetch_fn, ttl: int = 300, buffer: int = 30):
"""Return cached value, trigger background refresh if near expiry."""
cached = cache.get(cache_key)
if cached:
data = json.loads(cached)
if time.time() < data['expires_at']:
return data['value']
# Near expiry: refresh in background (or synchronously)
result = fetch_fn()
set_with_early_recompute(cache_key, result, ttl, buffer)
return result
result = fetch_fn()
set_with_early_recompute(cache_key, result, ttl, buffer)
return result
```
references/shared-runtime/secret-injection.md
# Secret Injection Patterns for ElastiCache Credentials
Standardized patterns for injecting cache credentials into Lambda, ECS, and EKS workloads. The endpoint hostname and port are not secrets and can be passed as plain environment variables. Auth credentials (RBAC passwords or IAM tokens) require secure handling.
## Principles
- Never hardcode credentials in application code or container images
- Use IAM auth when possible to eliminate static secrets entirely
- For RBAC password auth, store credentials in AWS Secrets Manager with rotation enabled
- Inject the endpoint (non-secret) as a plain environment variable
- Inject auth credentials through the most secure path available for each compute platform
- IAM auth requires TLS (in-transit encryption). Serverless caches always have TLS enabled; for node-based clusters, TLS must be explicitly enabled before using IAM or RBAC auth.
## Lambda
### Option 1: IAM Auth (preferred)
No secrets to manage. The Lambda execution role generates a short-lived token at runtime.
IAM policy on the Lambda execution role:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "elasticache:Connect",
"Resource": [
"arn:aws:elasticache:<region>:<account-id>:serverlesscache:<cache-name>",
"arn:aws:elasticache:<region>:<account-id>:replicationgroup:<rg-id>",
"arn:aws:elasticache:<region>:<account-id>:user:<user-id>"
]
}
]
}
```
Use `serverlesscache:` for serverless caches, `replicationgroup:` for node-based clusters. Include both if the policy covers multiple deployment types.
Application code (Python):
```python
import os
import boto3
import valkey
from botocore.signers import RequestSigner
from botocore.model import ServiceId
# Non-secret config from environment variables
ENDPOINT = os.environ["CACHE_ENDPOINT"]
PORT = int(os.environ.get("CACHE_PORT", "6379"))
REGION = os.environ["AWS_REGION"]
USER_ID = os.environ.get("CACHE_USER_ID", "appuser")
# Generate IAM token at init (outside handler, reused across warm invocations)
CACHE_NAME = os.environ.get("CACHE_NAME", "cache-01")
CACHE_IS_SERVERLESS = os.environ.get("CACHE_IS_SERVERLESS", "true").lower() == "true"
def generate_token():
import botocore.session
session = boto3.Session()
creds = session.get_credentials().get_frozen_credentials()
signer = RequestSigner(
ServiceId("elasticache"), REGION, "elasticache", "v4",
creds, botocore.session.get_session().get_component("event_emitter"),
)
query = f"Action=connect&User={USER_ID}"
if CACHE_IS_SERVERLESS:
query += "&ResourceType=ServerlessCache"
url = signer.generate_presigned_url(
{"method": "GET", "url": f"https://{CACHE_NAME}/?{query}",
"body": {}, "headers": {}, "context": {}},
operation_name="connect", expires_in=900, region_name=REGION,
)
return url[len("https://"):] if url.startswith("https://") else url
_client = None
def get_client():
"""Lazy-init connection with IAM auth. Reuses across warm invocations."""
global _client
if _client is not None:
try:
_client.ping()
return _client
except Exception:
_client = None
token = generate_token()
conn_kwargs = dict(
host=ENDPOINT, port=PORT, ssl=True,
username=USER_ID, password=token,
decode_responses=True, socket_connect_timeout=5,
socket_timeout=5, retry_on_timeout=True,
)
if CACHE_IS_SERVERLESS:
# Serverless requires cluster-mode client
_client = valkey.ValkeyCluster(**conn_kwargs)
else:
_client = valkey.Valkey(**conn_kwargs)
return _client
def handler(event, context):
# Note: IAM tokens are valid for 15 min; connections auto-disconnect after 12 hours.
# For long-lived connections, send AUTH with a new token to extend the connection.
# Lambda execution environments are typically recycled before the 12-hour limit.
client = get_client()
return client.get("key")
```
Environment variables in the Lambda configuration:
```
CACHE_ENDPOINT = your-endpoint.cache.amazonaws.com
CACHE_PORT = 6379
CACHE_USER_ID = appuser
CACHE_NAME = cache-01
CACHE_IS_SERVERLESS = true
```
### Option 2: Secrets Manager (for RBAC password auth)
Store the RBAC password in Secrets Manager and retrieve it at Lambda init.
```python
import os
import json
import boto3
import valkey
ENDPOINT = os.environ["CACHE_ENDPOINT"]
SECRET_ARN = os.environ["CACHE_SECRET_ARN"]
# Retrieve secret at init (outside handler)
sm_client = boto3.client("secretsmanager")
secret = json.loads(
sm_client.get_secret_value(SecretId=SECRET_ARN)["SecretString"]
)
client = valkey.Valkey(
host=ENDPOINT, port=6379, ssl=True,
username=secret["username"], password=secret["password"],
decode_responses=True,
)
def handler(event, context):
return client.get("key")
```
Lambda execution role needs:
```json
{
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "<secret-arn>"
}
```
### Option 3: AWS Parameters and Secrets Lambda Extension
Use the `AWS-Parameters-and-Secrets-Lambda-Extension` layer to cache Secrets Manager values locally within the Lambda execution environment, reducing Secrets Manager API calls and latency.
```python
import os
import json
import urllib.request
import valkey
ENDPOINT = os.environ["CACHE_ENDPOINT"]
SECRET_ARN = os.environ["CACHE_SECRET_ARN"]
SECRETS_EXTENSION_PORT = os.environ.get("PARAMETERS_SECRETS_EXTENSION_HTTP_PORT", "2773")
# Retrieve via the local extension HTTP endpoint (cached automatically)
headers = {"X-Aws-Parameters-Secrets-Token": os.environ["AWS_SESSION_TOKEN"]}
url = f"http://localhost:{SECRETS_EXTENSION_PORT}/secretsmanager/get?secretId={SECRET_ARN}"
req = urllib.request.Request(url, headers=headers)
secret = json.loads(urllib.request.urlopen(req).read())
client = valkey.Valkey(
host=ENDPOINT, port=6379, ssl=True,
username=secret["username"], password=secret["password"],
decode_responses=True,
)
```
### Environment Variables (non-secret only)
Safe to pass as plain environment variables:
- `CACHE_ENDPOINT` -- the cache hostname
- `CACHE_PORT` -- 6379 or 6380
- `CACHE_USER_ID` -- the RBAC user ID (not the password)
Never place passwords or auth tokens in Lambda environment variables.
## ECS
### Option 1: IAM Auth (preferred)
Attach `elasticache:Connect` to the ECS task role. The application generates IAM tokens at runtime.
Task role policy (for serverless caches):
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "elasticache:Connect",
"Resource": [
"arn:aws:elasticache:<region>:<account-id>:serverlesscache:<cache-name>",
"arn:aws:elasticache:<region>:<account-id>:replicationgroup:<rg-id>",
"arn:aws:elasticache:<region>:<account-id>:user:<user-id>"
]
}
]
}
```
For node-based clusters (replication groups), use the `replicationgroup` resource type instead of `serverlesscache` in the Resource ARN.
Task definition (environment variables for non-secret config):
```json
{
"containerDefinitions": [
{
"name": "app",
"image": "your-app-image",
"environment": [
{ "name": "CACHE_ENDPOINT", "value": "your-endpoint.cache.amazonaws.com" },
{ "name": "CACHE_PORT", "value": "6379" },
{ "name": "CACHE_USER_ID", "value": "appuser" },
{ "name": "AWS_REGION", "value": "us-east-1" }
]
}
]
}
```
### Option 2: Secrets Manager via Task Definition
Use the `secrets` block in the ECS task definition to inject Secrets Manager values as environment variables. ECS resolves the secret at task launch.
```json
{
"containerDefinitions": [
{
"name": "app",
"image": "your-app-image",
"environment": [
{ "name": "CACHE_ENDPOINT", "value": "your-endpoint.cache.amazonaws.com" }
],
"secrets": [
{
"name": "CACHE_USERNAME",
"valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:<secret-name>:username::"
},
{
"name": "CACHE_PASSWORD",
"valueFrom": "arn:aws:secretsmanager:<region>:<account-id>:secret:<secret-name>:password::"
}
]
}
]
}
```
Task execution role needs:
```json
{
"Effect": "Allow",
"Action": "secretsmanager:GetSecretValue",
"Resource": "<secret-arn>"
}
```
Note: this resolves the secret at task launch. If the secret is rotated, the running task still uses the old value. For rotation-aware behavior, use the init container pattern or SDK-based refresh.
### Option 3: Init Container for Rotation-Aware Caching
Use an init container to fetch and refresh credentials, writing them to a shared volume that the application container reads.
```json
{
"containerDefinitions": [
{
"name": "secret-init",
"image": "amazon/aws-cli",
"essential": false,
"command": [
"sh", "-c",
"aws secretsmanager get-secret-value --secret-id $SECRET_ARN --query SecretString --output text > /secrets/cache-creds.json"
],
"environment": [
{ "name": "SECRET_ARN", "value": "<secret-arn>" }
],
"mountPoints": [
{ "sourceVolume": "secrets", "containerPath": "/secrets" }
]
},
{
"name": "app",
"image": "your-app-image",
"dependsOn": [
{ "containerName": "secret-init", "condition": "SUCCESS" }
],
"mountPoints": [
{ "sourceVolume": "secrets", "containerPath": "/secrets", "readOnly": true }
]
}
],
"volumes": [
{ "name": "secrets" }
]
}
```
## EKS
### Option 1: IAM Auth via IRSA (preferred)
Use IAM Roles for Service Accounts (IRSA) to grant the pod's service account `elasticache:Connect` permission. No secrets stored in Kubernetes.
1. Create an IAM role with `elasticache:Connect` and a trust policy for the EKS OIDC provider.
2. Annotate the Kubernetes service account:
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: cache-app
namespace: default
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/CacheAppRole
```
1. The pod uses the service account and generates IAM auth tokens at runtime using the injected AWS credentials.
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: cache-app
spec:
template:
spec:
serviceAccountName: cache-app
containers:
- name: app
image: your-app-image
env:
- name: CACHE_ENDPOINT
value: "your-endpoint.cache.amazonaws.com"
- name: CACHE_PORT
value: "6379"
- name: CACHE_USER_ID
value: "appuser"
```
### Option 2: External Secrets Operator (for RBAC password auth)
Sync Secrets Manager secrets into Kubernetes Secrets using the External Secrets Operator (ESO).
```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: cache-creds
namespace: default
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: cache-creds
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: elasticache/myapp/appuser
property: username
- secretKey: password
remoteRef:
key: elasticache/myapp/appuser
property: password
```
Reference in the pod:
```yaml
containers:
- name: app
image: your-app-image
env:
- name: CACHE_ENDPOINT
value: "your-endpoint.cache.amazonaws.com"
- name: CACHE_USERNAME
valueFrom:
secretKeyRef:
name: cache-creds
key: username
- name: CACHE_PASSWORD
valueFrom:
secretKeyRef:
name: cache-creds
key: password
```
The `refreshInterval` ensures rotated secrets are picked up automatically. ESO creates a standard Kubernetes Secret that stays in sync with Secrets Manager.
### Option 3: AWS Secrets and Config Provider (ASCP) for CSI Driver
Mount Secrets Manager secrets as files in the pod filesystem using the AWS Secrets Store CSI Driver.
```yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: cache-creds
spec:
provider: aws
parameters:
objects: |
- objectName: "elasticache/myapp/appuser"
objectType: "secretsmanager"
jmesPath:
- path: username
objectAlias: cache-username
- path: password
objectAlias: cache-password
secretObjects:
- secretName: cache-creds-k8s
type: Opaque
data:
- objectName: cache-username
key: username
- objectName: cache-password
key: password
```
Pod spec:
```yaml
spec:
serviceAccountName: cache-app
containers:
- name: app
image: your-app-image
volumeMounts:
- name: secrets
mountPath: /mnt/secrets
readOnly: true
env:
- name: CACHE_ENDPOINT
value: "your-endpoint.cache.amazonaws.com"
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: cache-creds
```
The ASCP driver rotates mounted secrets automatically based on the configured rotation interval.
## Decision Guide
| Factor | IAM Auth | Secrets Manager (injected) | Secrets Manager (CSI/ESO) |
|--------|----------|---------------------------|---------------------------|
| No static secrets | Yes | No | No |
| Works with serverless caches | Yes | Yes | Yes |
| IAM re-auth inside MULTI/EXEC | Not supported (MULTI/EXEC data commands still work on IAM-authenticated connections) | N/A | N/A |
| Rotation handling | Automatic (token refresh) | Manual restart or SDK refresh | Automatic (ESO/ASCP) |
| Lambda | Preferred | Good | Not applicable |
| ECS | Preferred | Good (task definition secrets) | Not applicable |
| EKS | Preferred (IRSA) | Good (ESO) | Good (ASCP) |
Default recommendation: use IAM auth when the workload runs on Valkey 7.2+ or Redis OSS 7.0+ with TLS enabled. Note that IAM re-authentication cannot occur inside `MULTI`/`EXEC` blocks, but MULTI/EXEC commands themselves work on IAM-authenticated connections. Fall back to Secrets Manager with rotation for all other cases.
references/shared-security/config-guardrails.md
# AWS Config Guardrails for ElastiCache
Use AWS Config rules to continuously audit ElastiCache deployments for security and operational compliance. These rules detect configuration drift and non-compliant resources.
## Managed Rules
**Note:** These managed Config rules evaluate only node-based replication groups (`AWS::ElastiCache::ReplicationGroup`). They do not apply to ElastiCache Serverless caches. Serverless caches always have encryption (at-rest and in-transit) enabled by default and have built-in Multi-AZ with automatic failover, so these rules are not needed for serverless deployments.
### elasticache-repl-grp-auto-failover-enabled
**What it checks**: Node-based replication groups have automatic failover enabled.
**Why it matters**: Without automatic failover, a primary node failure requires manual intervention to promote a replica, causing extended downtime.
**Remediation**:
```bash
aws elasticache modify-replication-group \
--replication-group-id <cluster-name> \
--automatic-failover-enabled \
--multi-az-enabled \
--apply-immediately \
--region <region>
```
Note: Automatic failover requires at least one replica per shard. If the cluster has no replicas, add replicas first. Also enable Multi-AZ (`--multi-az-enabled`) for automatic failover to provide meaningful HA; without Multi-AZ, the primary and replica may be in the same AZ, reducing fault tolerance.
### elasticache-repl-grp-encrypted-at-rest
**What it checks**: Node-based replication groups have at-rest encryption enabled.
**Why it matters**: Without at-rest encryption, data on disk (snapshots, swap files) is stored in plaintext. This is a compliance finding for most security frameworks (SOC 2, HIPAA, PCI DSS).
**Remediation**: At-rest encryption cannot be enabled on an existing cluster. The remediation path is:
1. Create a new replication group with `AtRestEncryptionEnabled: true`
2. Migrate data from the old cluster to the new one (snapshot-restore or dual-write)
3. Decommission the old cluster
For new clusters, always set `AtRestEncryptionEnabled: true` in the template.
### elasticache-repl-grp-encrypted-in-transit
**What it checks**: Node-based replication groups have in-transit encryption (TLS) enabled.
**Why it matters**: Without TLS, data travels in plaintext between clients and the cache, and between primary and replica nodes. Susceptible to eavesdropping.
**Remediation**: For clusters created without TLS, the remediation depends on engine version:
**Redis OSS 7.0+ / Valkey 7.2+:** In-place TLS enablement is supported. Migrate in three steps:
1. Set `TransitEncryptionEnabled: true` with `TransitEncryptionMode: preferred` (allows both TLS and plaintext clients)
2. Update all application clients to use TLS (`ssl=True` / `tls: {}`)
3. Set `TransitEncryptionMode: required` to enforce TLS-only
**Older engine versions (Redis OSS < 7.0):** Cluster recreation is required:
1. Create a new replication group with `TransitEncryptionEnabled: true`
2. Update application clients to use TLS
3. Migrate data and cut over
4. Decommission the old cluster
## Custom Rules
### RBAC Auth Enabled (custom rule concept)
**What it checks**: Replication groups or serverless caches have a user group associated (not relying solely on the default user with no password).
**Implementation**: Custom Config rule Lambda. Logic: call `describe_replication_groups` or `describe_serverless_caches`, check `UserGroupIds` (node-based) or `UserGroupId` (serverless) is non-empty. For node-based clusters, also check `AuthTokenEnabled`; clusters using AUTH tokens have a valid authentication method even without user groups. Note: `AuthTokenEnabled` is only available in the `describe_replication_groups` response; the `describe_serverless_caches` API does not return this field, so for serverless the check is limited to user group association. COMPLIANT if a user group is associated or AUTH token is enabled (node-based only), NON_COMPLIANT otherwise.
**Remediation**:
```bash
aws elasticache modify-serverless-cache \
--serverless-cache-name <cache-name> \
--user-group-id <user-group-id> --region <region>
aws elasticache modify-replication-group \
--replication-group-id <cluster-name> \
--user-group-ids-to-add <user-group-id> --region <region>
```
### Engine Version Compliance (custom rule concept)
**What it checks**: ElastiCache clusters run a minimum required engine version (e.g., Valkey 7.2+ or Redis OSS 7.0+ for IAM auth, Valkey 8.2 or above for vector search on node-based clusters).
**Implementation**: Custom Config rule Lambda. Logic: call `describe_replication_groups`, get a member cluster via `describe_cache_clusters`, compare `EngineVersion` against a `minimumVersion` rule parameter (default `7.2`). Use `packaging.version.parse` for comparison.
**Remediation**:
```bash
# For clusters already running Valkey, upgrade the version:
aws elasticache modify-replication-group \
--replication-group-id <cluster-name> \
--engine-version 7.2 \
--apply-immediately --region <region>
# For clusters running Redis OSS, specify --engine valkey to switch engines.
# WARNING: This performs an engine migration from Redis OSS to Valkey, not just a
# version upgrade. Test in a non-production environment first. See valkey-migration-guide.md.
aws elasticache modify-replication-group \
--replication-group-id <cluster-name> \
--engine valkey --engine-version 7.2 \
--apply-immediately --region <region>
```
## Deploying Config Rules
### CloudFormation for managed rules
```yaml
Resources:
AutoFailoverRule:
Type: AWS::Config::ConfigRule
Properties:
ConfigRuleName: elasticache-auto-failover-enabled
Source:
Owner: AWS
SourceIdentifier: ELASTICACHE_REPL_GRP_AUTO_FAILOVER_ENABLED
Scope:
ComplianceResourceTypes:
- AWS::ElastiCache::ReplicationGroup
EncryptedAtRestRule:
Type: AWS::Config::ConfigRule
Properties:
ConfigRuleName: elasticache-encrypted-at-rest
Source:
Owner: AWS
SourceIdentifier: ELASTICACHE_REPL_GRP_ENCRYPTED_AT_REST
Scope:
ComplianceResourceTypes:
- AWS::ElastiCache::ReplicationGroup
EncryptedInTransitRule:
Type: AWS::Config::ConfigRule
Properties:
ConfigRuleName: elasticache-encrypted-in-transit
Source:
Owner: AWS
SourceIdentifier: ELASTICACHE_REPL_GRP_ENCRYPTED_IN_TRANSIT
Scope:
ComplianceResourceTypes:
- AWS::ElastiCache::ReplicationGroup
```
### CloudFormation for custom rules
> **Note:** The `RBACCheckFunction` Lambda is not defined in the snippet below. You must define it separately as an `AWS::Lambda::Function` resource (or import its ARN) with the custom Config rule evaluation logic described in the "RBAC Auth Enabled" section above. Grant Config permission to invoke it via an `AWS::Lambda::Permission` resource with `Principal: config.amazonaws.com`.
```yaml
Resources:
RBACAuthRule:
Type: AWS::Config::ConfigRule
Properties:
ConfigRuleName: elasticache-rbac-auth-enabled
Source:
Owner: CUSTOM_LAMBDA
SourceIdentifier: !GetAtt RBACCheckFunction.Arn
SourceDetails:
- EventSource: aws.config
MessageType: ConfigurationItemChangeNotification
Scope:
ComplianceResourceTypes:
- AWS::ElastiCache::ReplicationGroup
- AWS::ElastiCache::ServerlessCache
```
## Summary of Rules
| Rule | Type | Checks | Remediation complexity |
|------|------|--------|----------------------|
| `elasticache-repl-grp-auto-failover-enabled` | Managed | Automatic failover on | Low (modify in place) |
| `elasticache-repl-grp-encrypted-at-rest` | Managed | At-rest encryption on | High (requires cluster recreation) |
| `elasticache-repl-grp-encrypted-in-transit` | Managed | TLS on | Medium (in-place for Redis OSS 7.0+/Valkey 7.2+; recreation for older versions) |
| `elasticache-rbac-auth-enabled` | Custom | RBAC user group associated | Medium (create and associate user group) |
| Engine version compliance | Custom | Minimum engine version | Medium (in-place version upgrade) |
references/shared-security/encryption-defaults.md
# Encryption Defaults
Node-based encryption gotchas and migration paths. Serverless encryption is always-on and requires no configuration (covered in create-secure-cache.md).
## Node-Based In-Transit Encryption (TLS)
- Best practice is to enable TLS at creation time via `TransitEncryptionEnabled: true`.
- For Redis OSS 7.0+ / Valkey 7.2+, TLS can be enabled post-creation via `modify-replication-group` with `--transit-encryption-enabled` and `--transit-encryption-mode preferred` (then switch to `required`). This is a three-step migration: (1) enable TLS in preferred mode, (2) update all clients to use TLS, (3) switch to required mode.
- For older engine versions, TLS cannot be added post-creation; the only option is snapshot-restore to a new cluster with TLS enabled.
- **Recommendation**: Always enable TLS at creation time to avoid the complexity of post-creation migration.
## Node-Based At-Rest Encryption (KMS)
- Enable via `AtRestEncryptionEnabled: true` at creation time.
- Cannot be changed after creation. A cluster created without at-rest encryption cannot have it added later.
- Uses default service-managed encryption at rest. To use a customer-managed KMS key, specify `KmsKeyId` at creation.
- **Warning**: If you delete or disable the KMS key (or revoke its grants) used to encrypt a cache, the cache becomes **irrecoverable**. AWS KMS deletes root keys only after a waiting period of at least seven days.
- **Recommendation**: Enable at creation time. Use a customer-managed KMS key if your compliance requirements mandate key control or key rotation policies beyond the service-managed defaults.
> **Valkey default:** For Valkey caches, `AtRestEncryptionEnabled` defaults to `true` if not explicitly specified. Redis OSS caches require it to be explicitly set.
## Migration Path for Unencrypted Node-Based Clusters
If an existing node-based cluster was created without encryption, the migration path depends on which encryption is missing.
### Adding In-Transit Encryption (TLS) to an Existing Cluster
For supported engine versions (Redis OSS 7.0+, Valkey 7.2+), you can enable TLS in three steps:
**Step 1: Enable TLS in preferred mode** (accepts both TLS and non-TLS connections)
```bash
aws elasticache modify-replication-group \
--replication-group-id my-cluster \
--transit-encryption-enabled \
--transit-encryption-mode preferred \
--apply-immediately \
--region us-east-1
```
Wait for the modification to complete. During this phase, both encrypted and unencrypted connections are accepted.
> **Important**: DNS endpoints change during TLS migration. When switching to `preferred`, new TLS endpoints are generated. When switching to `required`, old non-TLS endpoints are deleted. Do not hardcode endpoints in your application. After each migration step completes, use `describe-replication-group` to retrieve the current endpoints.
#### Step 2: Update all clients to use TLS connections
Verify all application clients are configured with TLS before proceeding.
**Step 3: Switch to required mode** (only TLS connections accepted)
```bash
aws elasticache modify-replication-group \
--replication-group-id my-cluster \
--transit-encryption-mode required \
--apply-immediately \
--region us-east-1
```
For older engine versions that do not support `modify-replication-group` with transit encryption changes, the only option is:
1. Create a snapshot of the existing cluster.
2. Create a new cluster from the snapshot with `TransitEncryptionEnabled: true`.
3. Update application endpoints to point to the new cluster.
4. Delete the old cluster after validation.
### Adding At-Rest Encryption to an Existing Cluster
At-rest encryption cannot be enabled on an existing cluster. The migration path is:
1. Create a snapshot of the existing cluster.
2. Create a new cluster from the snapshot with `AtRestEncryptionEnabled: true`.
3. Update application endpoints to point to the new cluster.
4. Validate data integrity and application connectivity.
5. Delete the old cluster.
This requires a brief cutover window. Plan for:
- Snapshot creation time (depends on data size).
- New cluster creation time from snapshot.
- DNS or application configuration update.
- Validation period before decommissioning the old cluster.
references/shared-security/vpc-patterns.md
# VPC Patterns for ElastiCache
ElastiCache-specific networking rules. Generic VPC knowledge is omitted (the model already knows how subnets and security groups work).
## Port Requirements
| Deployment Type | Port | Purpose |
|----------------|------|---------|
| Serverless | 6379 | Primary endpoint (read/write) |
| Serverless | 6380 | Reader endpoint (same DNS name as primary, port 6380 for read-optimized access; node-based reader endpoints use a separate DNS name on port 6379) |
| Node-based | 6379 | Valkey/Redis OSS data port (both primary and reader endpoints) |
| Node-based (cluster mode) | 16379 | Cluster bus port (node-to-node, auto-managed) |
| Node-based (Memcached) | 11211 | Memcached data port |
| Serverless (Memcached) | 11211 | Memcached serverless endpoint (TLS mandatory) |
> **TLS note:** TLS is mandatory for all serverless caches (Valkey/Redis OSS and Memcached). There is no option to disable in-transit encryption on serverless deployments.
## Security Group Anti-Patterns
- Do not open `0.0.0.0/0` on any port. ElastiCache is VPC-internal only.
- Do not use IP-based rules when security-group-based rules are possible. SG references survive IP changes.
- Do not allow port ranges (e.g., 6379-6400). Use specific ports only.
- Do not attach the cache to a default security group that allows all inbound from itself.
- For serverless (Valkey/Redis OSS): open ports 6379 (primary) and 6380 (reader) from the app security group. The primary and reader endpoints use the same DNS name on different ports (6379 for primary, 6380 for read-optimized).
- For serverless (Memcached): open port 11211 from the app security group.
## Subnet Group Requirements
- **Node-based clusters** require a cache subnet group. ElastiCache uses the subnet group to select subnets and assign IP addresses to cache nodes.
- **Serverless caches** do not use a subnet group resource. Instead, pass a list of subnet IDs directly during creation.
## Subnet IP Address Capacity
- CIDR blocks for each subnet must be large enough to provide spare IP addresses for ElastiCache to use during maintenance activities.
- Common pitfalls: subnets in the subnet group have too small a CIDR range, or subnets are shared and heavily used by other clusters.
- For large cluster-mode-enabled deployments (up to 500 nodes), ensure sufficient available IP addresses to accommodate scaling.
## PrivateLink and Cross-VPC Access
- **PrivateLink (VPC Endpoints)** covers ElastiCache control-plane APIs only (e.g., `CreateCacheCluster`, `DescribeReplicationGroups`). It does not provide data-plane connectivity to cache endpoints.
- **Cross-VPC data access** requires VPC Peering, Transit Gateway (TGW), AWS Direct Connect, or site-to-site VPN. Use security group references (preferred, when peering supports it) or CIDR-based rules to allow traffic between the application VPC and the cache VPC.
references/shared-ux/action-safety.md
# Action Safety Guide
Safety semantics for destructive and high-impact ElastiCache operations. Every action that can cause data loss, downtime, or irreversible changes must follow the safeguards defined here.
## Safety Levels
| Risk Level | Definition | Required Safeguards |
|-----------|------------|---------------------|
| High | Data loss or significant downtime likely. Irreversible. | Explicit user confirmation. Final snapshot. Impact explanation. |
| Medium | Possible disruption or partial data impact. May be reversible with effort. | User confirmation. Recommend snapshot. Explain impact. |
| Low | Minimal risk. Fully reversible or non-destructive. | Inform the user. Proceed with standard confirmation. |
## Action Safety Matrix
### High Risk Actions
#### DELETE Cache (Serverless or Node-Based)
**Risk level:** High
**Reversibility:** Irreversible. All data is permanently deleted unless a final snapshot is taken.
**Required safeguards:**
1. **Always suggest a final snapshot before deletion.** Ensure the caller has `elasticache:CreateSnapshot` permission (or `elasticache:CreateServerlessCacheSnapshot` for serverless); without it, the API call will fail with an `Access Denied` exception.
2. Require explicit user confirmation with the cache name.
3. Warn that all data, connections, and endpoints will be destroyed.
4. Note that manual snapshots are retained after deletion, but **automatic cache snapshots are NOT retained**. A final snapshot or existing manual snapshot is the only recovery path.
5. Confirm the cache is not referenced by active applications.
**Implementation:**
> **Option A (recommended):** Use `--final-snapshot-identifier` on the delete call. This atomically creates a snapshot during deletion in a single step. Option B below creates a separate snapshot first, which is useful if you want to verify the snapshot before proceeding with deletion.
```bash
# Option A: Delete with atomic final snapshot (single step)
aws elasticache delete-replication-group \
--replication-group-id my-cluster \
--final-snapshot-identifier my-cluster-final-$(date +%Y%m%d-%H%M%S) \
--region us-east-1
```
```bash
# Option B: Create snapshot first, verify, then delete (two steps)
# Step 1: Create and verify snapshot
aws elasticache create-snapshot \
--replication-group-id my-cluster \
--snapshot-name my-cluster-final-$(date +%Y%m%d-%H%M%S) \
--region us-east-1
aws elasticache describe-snapshots \
--snapshot-name my-cluster-final-<timestamp> \
--region us-east-1
# Step 2: Delete without final snapshot (already taken above)
aws elasticache delete-replication-group \
--replication-group-id my-cluster \
--region us-east-1
```
For serverless:
```bash
aws elasticache delete-serverless-cache \
--serverless-cache-name my-cache \
--final-snapshot-name my-cache-final-$(date +%Y%m%d-%H%M%S) \
--region us-east-1
```
---
#### FLUSHALL / FLUSHDB
**Risk level:** High
**Reversibility:** Irreversible. All keys in the database (FLUSHDB) or all databases (FLUSHALL) are permanently deleted.
**Required safeguards:**
1. **Warn about complete data loss.**
2. Suggest taking a snapshot before flushing.
3. Require explicit user confirmation.
4. Confirm this is intentional and not a cache invalidation scenario (where targeted `DEL` or TTL-based expiry is more appropriate).
**Guidance to present to the user:**
- FLUSHALL removes ALL data from ALL databases on the cache.
- FLUSHDB removes ALL data from the currently selected database.
- There is no undo. A snapshot taken before the operation is the only recovery path.
- For cache invalidation, consider using `DEL` for specific keys, `UNLINK` for async deletion, or TTL-based expiry instead.
---
#### Engine Upgrade (Major Version)
**Risk level:** High
**Reversibility:** Difficult. Downgrade is not supported in-place for major version changes, with one exception: Valkey 7.2 can be rolled back in-place to Redis OSS 7.1 (this is the documented cross-engine rollback path).
**Required safeguards:**
1. Flag compatibility concerns before proceeding.
2. Recommend testing the new engine version in a staging environment first.
3. Create a snapshot before the upgrade.
4. Check for deprecated commands or behavior changes in the target version.
5. Verify client library compatibility with the new engine version.
6. For Redis OSS to Valkey migration, versions 5.0.6+ support zero-downtime migration with Multi-AZ enabled; earlier versions are supported but may experience 30–60 seconds of failover during DNS propagation. Still recommend a snapshot.
**Implementation:**
```bash
# Step 1: Snapshot
aws elasticache create-snapshot \
--replication-group-id my-cluster \
--snapshot-name pre-upgrade-$(date +%Y%m%d) \
--region us-east-1
# Step 2: Modify engine version (cross-engine upgrade to Valkey example)
aws elasticache modify-replication-group \
--replication-group-id my-cluster \
--engine valkey \
--engine-version 8.0 \
--cache-parameter-group-name my-valkey8-param-group \
--apply-immediately \
--region us-east-1
```
---
### Medium Risk Actions
#### Failover (Manual or Test)
**Risk level:** Medium
**Reversibility:** Reversible (fail back). Brief connectivity disruption during promotion.
**Required safeguards:**
1. Explain the impact: the primary will be demoted, a replica will be promoted, and clients will experience a brief disconnection (typically seconds).
2. Suggest performing during a maintenance window or low-traffic period.
3. Verify automatic failover is enabled and replicas are healthy before testing.
4. Confirm the application handles reconnection gracefully (cluster-aware clients, retry logic).
**Implementation:**
```bash
aws elasticache test-failover \
--replication-group-id my-cluster \
--node-group-id 0001 \
--region us-east-1
```
---
#### Modify That Causes Downtime
**Risk level:** Medium
**Reversibility:** Varies by modification type.
Operations that may cause downtime or brief disruption:
- Changing node type (vertical scaling).
- Changing number of shards (horizontal scaling).
- Enabling cluster mode (migration from disabled to enabled via compatible mode; can be reverted from "compatible" back to "disabled", but cannot be reverted once set to fully "enabled").
- Changing engine version (minor or major).
- Changing at-rest encryption (requires creating a new cluster).
- Enabling in-transit encryption (can be done in-place via a two-step process: set transit encryption mode to `preferred`, then to `required`; no new cluster needed).
**Required safeguards:**
1. Flag that the modification may cause brief downtime or failover.
2. Explain the specific impact (e.g., "node type change triggers a rolling replacement with brief failover per shard").
3. Recommend applying during a maintenance window: `--no-apply-immediately` for non-urgent changes.
4. For production caches, suggest testing the modification in staging first.
---
#### Scale-Down (Node Type or Shard Count)
**Risk level:** Medium
**Reversibility:** Reversible (scale back up), but data loss is possible with Memcached.
**Required safeguards:**
1. **For Memcached**: Warn about data loss. Memcached has no persistence or replication. Scaling down removes nodes and their data permanently.
2. **For Valkey/Redis OSS node-based**: Data is redistributed during shard removal. Brief disruption during rebalancing. Verify sufficient memory on remaining shards.
3. **For serverless**: Scaling is automatic; manual scale-down is not applicable.
4. Verify the remaining capacity can handle the current data size and throughput.
---
#### Modify Security Group Rules
**Risk level:** Medium
**Reversibility:** Reversible by restoring previous rules.
**Required safeguards:**
1. Warn that removing inbound rules may immediately disconnect active clients.
2. Verify that any new rules maintain connectivity for all application clients.
3. Review changes before applying: removing a source security group reference will disconnect all clients in that security group.
---
### Low Risk Actions
#### Create Snapshot
**Risk level:** Low
**Reversibility:** Snapshot can be deleted. No impact on the running cache.
**Safeguards:**
- For node-based caches, snapshot creation uses the fork mechanism (or forkless save on newer engine versions such as Valkey 8.0+) and may cause brief memory spike on fork-based engines. Ensure `reserved-memory-percent` allows headroom.
- Snapshots incur S3 storage costs.
---
#### Modify Tags
**Risk level:** Low
**Reversibility:** Fully reversible. Tags can be added, changed, or removed at any time.
**Safeguards:**
- Inform the user that tag changes may affect cost allocation reports and IAM tag-based policies.
---
#### Describe / List Operations
**Risk level:** Low
**Reversibility:** Read-only. No state change.
**Safeguards:**
- None required. These are safe to run at any time.
---
#### Add Replicas
**Risk level:** Low
**Reversibility:** Replicas can be removed later.
**Safeguards:**
- Adding replicas triggers a sync from the primary. Initial sync may briefly impact primary performance for large datasets.
- Verify sufficient subnet IPs for new replicas.
---
#### Enable Log Delivery
**Risk level:** Low
**Reversibility:** Can be disabled at any time.
**Safeguards:**
- Warn about log delivery costs. Slow log and engine log output can be delivered to CloudWatch Logs or Kinesis Data Firehose (mutually exclusive destinations per log type). CloudWatch Logs ingestion costs can be significant for high-throughput caches.
## Never-Auto-Execute List
The following operations must **never** run without direct user confirmation in the current conversation turn, regardless of context, automation pipelines, scripted workflows, or IDE mode:
- Cache deletion (any type)
- FLUSHALL / FLUSHDB
- Snapshot deletion (especially the last/only snapshot for a cache)
- Credential or AUTH token changes on production caches
- Security group rule removal
- User or user group deletion
- Migration cutover execution
- Manual failover on production caches
references/shared-ux/error-remediation.md
# Error Remediation Guide
**Scope:** Valkey and Redis OSS engines only. Memcached uses SASL authentication (1.6.12+) and does not support RBAC user groups or IAM auth. Memcached error patterns differ and are not covered here.
ElastiCache-specific errors that require non-obvious remediation. Generic errors (AUTH required, TLS handshake, DNS resolution, timeouts, replication lag, throttling, slow commands) are omitted; see monitoring/troubleshooting.md for metric-based diagnosis of those.
## 1. Connection Refused
**Symptom:** Client gets `Connection refused` or `ECONNREFUSED` when connecting to the cache endpoint.
**Causes:**
- Security group does not allow inbound TCP on port 6379 (or 6380 for serverless read port) from the client's security group or IP.
- The cache endpoint is incorrect or the cache is not in `available` state.
- The client is outside the VPC (ElastiCache has no public endpoints).
- Subnet routing issue: the client's subnet cannot reach the cache's subnet.
**Next steps:**
1. Verify the cache is in `available` status: `aws elasticache describe-serverless-caches` or `describe-replication-groups`.
2. Check the security group attached to the cache allows inbound from the client's security group on port 6379.
3. Confirm the client is in the same VPC or has a valid network path (VPC peering, transit gateway, or tunnel).
4. For serverless, verify both ports 6379 (primary port for reads and writes) and 6380 (read port for eventually-consistent reads via READONLY) are allowed; both use the same hostname.
5. Run: `python3 scripts/test_connection.py <endpoint>`
## 2. WRONGPASS / Invalid Credentials
**Symptom:** `WRONGPASS invalid username-password pair` or `ERR invalid password`.
**Causes:**
- Incorrect password or expired IAM auth token.
- RBAC user password was rotated but the client is using the old password.
- IAM auth token expired (tokens are valid for 15 minutes).
- Wrong username specified.
**Next steps:**
1. For IAM auth: regenerate the token. Tokens expire after 15 minutes for new AUTH/HELLO; an already-authenticated connection remains valid for up to 12 hours. Sending AUTH (or HELLO with auth) using a new IAM token resets the 12-hour disconnect timer.
2. For password auth: retrieve the current password from Secrets Manager.
3. Verify the username matches an active RBAC user associated with the cache's user group.
4. If the password is correct but commands still fail, check for NOPERM errors (section 7), which indicate access-string restrictions rather than credential issues.
## 3. MOVED Error
**Symptom:** `MOVED <slot> <ip>:<port>` response from the server.
**Causes:**
- The client sent a command to a node that does not own the hash slot for the key. This happens with cluster-mode-enabled deployments when the client is not using a cluster-aware driver.
**Next steps:**
1. Use a cluster-aware client library (e.g., `redis-py` with `RedisCluster`, `ioredis` with `Cluster` mode, Lettuce with cluster topology refresh).
2. Do not use a standalone client to connect to a cluster-mode-enabled cache.
3. Verify the client is connected to the configuration endpoint, not an individual node endpoint.
## 4. ASK Error
**Symptom:** `ASK <slot> <ip>:<port>` response during a resharding operation.
**Causes:**
- The cluster is in the middle of a slot migration (resharding). The target node holds the key temporarily.
**Next steps:**
1. This is transient during resharding. Cluster-aware clients handle ASK redirects automatically.
2. Ensure the client library supports ASK redirect handling.
3. If resharding is not expected, check for ongoing maintenance or scaling operations.
## 5. OOM / Maxmemory Reached
**Symptom:** `OOM command not allowed when used memory > 'maxmemory'` or `ERR command not allowed when maxmemory is set and the server is currently unable to free memory`.
**Causes:**
- The cache has reached its configured `maxmemory` limit and the current eviction policy (`maxmemory-policy`) does not allow the write.
- `noeviction` policy is set, preventing automatic key eviction.
**Next steps:**
1. Check `DatabaseMemoryUsagePercentage` in CloudWatch.
2. If the eviction policy is `noeviction`, consider changing to `allkeys-lru` or `volatile-lru` depending on the use case.
3. Scale up the node type for more memory, or add shards to distribute data.
4. Review TTLs on keys; ensure transient data has appropriate expiry.
5. Use `MEMORY USAGE <key>` to identify large keys consuming disproportionate memory.
## 6. CROSSSLOT Error
**Symptom:** `CROSSSLOT Keys in request don't hash to the same slot`.
**Causes:**
- A multi-key command (MGET, MSET, pipeline with multi-key operations, Lua script with multiple keys) targets keys that hash to different slots in a cluster-mode-enabled deployment.
**Next steps:**
1. Use hash tags to force related keys to the same slot: `{user:123}:profile`, `{user:123}:sessions`.
2. Split multi-key operations into per-slot batches.
3. If the workload cannot use hash tags, consider a single-shard deployment (if data fits).
4. Review the client library's support for automatic slot-aware batching.
## 7. Permission Denied (RBAC ACL)
**Symptom:** `NOPERM this user has no permissions to run the '<command>' command` or `NOPERM... on key '<key>'`.
**Causes:**
- The RBAC user's access string does not permit the command category or key pattern.
- The user is restricted to specific key prefixes and the application is accessing keys outside that prefix.
**Next steps:**
1. Check the user's access string: `aws elasticache describe-users --user-id <user-id>`.
2. Update the access string to include the required command categories and key patterns.
3. Common fix: change `on ~app:* +@read` to `on ~app:* +@read +@write` if writes are needed.
4. Access string changes (via `aws elasticache modify-user`) take effect immediately on **all** existing connections authenticated as that user — not just new ones. However, setting a user to `off` only prevents new `AUTH` attempts (it does not disconnect existing connections). To delete a user, use `aws elasticache delete-user`; ElastiCache does not support the `ACL DELUSER` command. Deleting a user via the API removes them from all associated user groups.
## 8. READONLY Error
**Symptom:** `READONLY You can't write against a read only replica`.
**Causes:**
- The client is sending write commands to a read replica instead of the primary.
- The client is connected to the reader endpoint instead of the primary endpoint.
- After a failover, the client's cached topology is stale.
**Next steps:**
1. Ensure writes go to the primary endpoint and reads go to the reader endpoint.
2. For cluster-mode, verify the client refreshes topology after failover.
3. Check if a failover recently occurred and the client reconnected to the wrong node.
## 9. LOADING Error
**Symptom:** `LOADING Redis is loading the dataset in memory`.
**Causes:**
- The node is starting up and loading data from a snapshot (RDB) or AOF file.
- Occurs after a restart, failover, or restore from backup.
**Next steps:**
1. Wait for the loading to complete. Duration depends on data size.
2. Monitor the node status in the console or via `describe-replication-groups`.
3. If loading takes excessively long, check the snapshot size and node type (more memory = faster load).
## 10. CLUSTERDOWN Error
**Symptom:** `CLUSTERDOWN The cluster is down` or `CLUSTERDOWN Hash slot not served`.
**Causes:**
- One or more hash slots are not covered (a shard is unavailable).
- Cluster is in a degraded state after multiple node failures.
**Next steps:**
1. Check cluster health: `describe-replication-groups` for node status.
2. If a shard is down, automatic failover should promote a replica. Verify failover status.
3. If no replica was available for the failed shard, manual intervention may be needed.
4. Check the `cluster-allow-reads-when-down` parameter if read availability during partial failure is important.
## 11. Max Clients Reached
**Symptom:** `ERR max number of clients reached`.
**Causes:**
- The node has reached its maximum connection limit.
- Connection leaks in the application (connections opened but never closed).
- Missing connection pooling.
**Next steps:**
1. Check `CurrConnections` metric in CloudWatch.
2. Implement connection pooling in the application.
3. For node-based: scale up the node type (larger nodes support more connections). For serverless: connection scaling is automatic; max-clients errors indicate per-connection ECPU starvation or client-side connection leaks, not server capacity.
4. Investigate and fix connection leaks (common in Lambda without reuse across invocations).
5. Check for idle connections that can be closed with `timeout` parameter.
## 12. Snapshot/Backup Failure
**Symptom:** Snapshot creation fails or takes excessively long.
**Causes:**
- Insufficient memory for the fork operation (background save requires memory overhead).
- Node type with limited resources.
**Next steps:**
1. Ensure `reserved-memory-percent` leaves enough headroom for fork operations. The default is 25 for accounts created after March 16, 2017. Older accounts may default to 0. AWS recommends 25% for all deployments.
2. Schedule snapshots during low-traffic periods.
3. Scale up the node type if memory is tight during snapshot creation.
## 13. Parameter Group Incompatibility
**Symptom:** `InvalidParameterCombination` when creating or modifying a cluster.
**Causes:**
- Parameter group family does not match the engine and version.
- Conflicting parameters specified.
**Next steps:**
1. Verify the parameter group family matches the engine. Documented valid values include: `memcached1.4`, `memcached1.5`, `memcached1.6`, `redis2.6`, `redis2.8`, `redis3.2`, `redis4.0`, `redis5.0`, `redis6.x`, `redis6.2`, `redis7`, `valkey7`, `valkey8`. Use `describe-engine-default-parameters` to confirm the correct family name for your engine version.
2. Check `describe-engine-default-parameters` for valid parameter names and ranges.
3. Create a new parameter group from the correct family.
## 14. IAM Auth Token Generation Failure
**Symptom:** Client fails to generate or use an IAM auth token. `ElastiCacheSigningError` or similar.
**Causes:**
- The IAM role/user does not have `elasticache:Connect` permission.
- The IAM auth token was generated for the wrong cache or user.
- Clock skew on the client machine (IAM tokens are time-sensitive).
**Next steps:**
1. Verify the IAM policy includes `elasticache:Connect` with the correct resource ARNs.
2. Ensure the token is generated for the correct `--replication-group-id` or `--serverless-cache-name`.
3. Check system clock synchronization.
4. Use the IAM policy simulator to validate permissions.
## 15. MULTI/EXEC Failure with IAM Auth
**Symptom:** `NOPERM` or authentication error during a `MULTI`/`EXEC` transaction on an IAM-authenticated connection.
**Causes:**
- The IAM auth token expired during the transaction and the client attempted to re-AUTH (call `AUTH` again) inside the `MULTI` block. Re-authentication inside a transaction is not supported.
- MULTI/EXEC itself works correctly on IAM-authenticated connections. The limitation is that token refresh cannot happen between `MULTI` and `EXEC`.
**Next steps:**
1. Refresh the IAM auth token **before** issuing `MULTI`, not inside the transaction. Ensure the token will remain valid for the duration of the transaction.
2. Use a connection pool that rotates IAM tokens on idle reconnect or before checkout, so connections handed to application code always have a fresh token.
3. If token expiry during long transactions is unavoidable, establish a new connection with a fresh token and retry the transaction.
## 16. Cluster Creation Timeout
**Symptom:** Cluster creation takes longer than expected or appears stuck in `creating` status.
**Causes:**
- Large node types with many shards take longer to provision.
- Region capacity constraints.
- Dependent resource issues (subnet group, security group, KMS key permissions).
**Next steps:**
1. For serverless: creation should take under 1 minute. If stuck, check VPC/subnet/SG configuration.
2. For node-based: creation can take 5-15 minutes. Wait and check status periodically.
3. Verify the KMS key (if specified) grants ElastiCache permission to use it.
4. Check AWS Service Health Dashboard for regional issues.
references/shared-ux/production-readiness.md
# Production Readiness Checklist
Gate checklist before promoting an ElastiCache cache to production. Run through each section. Items marked REQUIRED are non-negotiable. Items marked RECOMMENDED are best practice but may be deferred with documented justification.
## Security
- [ ] **REQUIRED** In-transit encryption (TLS) enabled. Serverless: automatic. Node-based: set at creation (`TransitEncryptionEnabled: true`), or enabled on existing clusters using the `preferred`->`required` transit encryption mode migration via `modify-replication-group`.
- [ ] **REQUIRED** At-rest encryption enabled. Serverless: automatic. Node-based: must be set at creation (`AtRestEncryptionEnabled: true`). Cannot be added later.
- [ ] **REQUIRED** RBAC or IAM auth configured. No default user active (`modify-user --user-id default --access-string "off ~* -@all"`).
- [ ] **REQUIRED** Security group allows only necessary ports (6379, and 6380 for serverless reader) from application security groups only. No `0.0.0.0/0`.
- [ ] **RECOMMENDED** Security audit passed: `python3 scripts/security_audit.py --serverless <name>` or `--replication-group <name>`.
- [ ] **RECOMMENDED** Credentials stored in Secrets Manager with rotation enabled (if using password auth).
## Availability
- [ ] **REQUIRED** Multi-AZ enabled (node-based). Serverless: automatic.
- [ ] **REQUIRED** Automatic failover enabled (node-based). Serverless: automatic.
- [ ] **REQUIRED** At least 1 replica per shard (node-based). Serverless: automatic.
- [ ] **RECOMMENDED** Subnets span 3 AZs for maximum fault tolerance.
## Backup and Recovery
- [ ] **REQUIRED** Daily backups configured. Node-based: `--snapshot-retention-limit 7` (recommended; valid range is 1-35, 0 disables backups). Serverless: must explicitly set `DailySnapshotTime` and `SnapshotRetentionLimit` (no automatic daily snapshots by default).
- [ ] **RECOMMENDED** Manual pre-deployment snapshot taken and verified.
- [ ] **RECOMMENDED** Restore procedure tested at least once (restore snapshot to a test cluster, validate data).
## Observability
- [ ] **REQUIRED** CloudWatch alarms deployed for critical metrics. Run: `python3 scripts/generate_dashboards.py --serverless <name> --sns-topic <arn> --output observability.json`
- [ ] **REQUIRED** Alarm notification routing configured (SNS to Slack/PagerDuty/email).
- [ ] **RECOMMENDED** Slow log delivery enabled to CloudWatch Logs (JSON format, 30-day retention).
- [ ] **RECOMMENDED** CloudWatch dashboard deployed for visual monitoring.
- [ ] **RECOMMENDED** Baseline metrics observed for 1-2 weeks; alarm thresholds tuned from baseline.
## Application Resilience
- [ ] **REQUIRED** Connection pooling configured in the application. See `monitoring/client-tuning-and-diagnostics.md`.
- [ ] **REQUIRED** Retry logic with exponential backoff implemented (3 retries, 100-200ms base).
- [ ] **REQUIRED** Cache failure graceful degradation tested. Application must function (slower) when cache is unavailable. Cache calls wrapped in try/except.
- [ ] **RECOMMENDED** Client timeouts configured (connect: 2-5s, command: 1-2s). See `monitoring/client-tuning-and-diagnostics.md`.
- [ ] **RECOMMENDED** DNS TTL set to 5 seconds or less in the application to handle failover endpoint changes.
- [ ] **RECOMMENDED** Failover tested with `test-failover` command (node-based) during a maintenance window.
## Cost Controls
- [ ] **REQUIRED** (Serverless) CacheUsageLimits set for both DataStorage and ECPUPerSecond to prevent unbounded cost growth.
- [ ] **RECOMMENDED** Cost allocation tags applied (Environment, Application, Owner).
- [ ] **RECOMMENDED** Cost baseline established. Run `python3 scripts/price_calculator.py --mode serverless --data-gb <N> --ops-per-sec <N>` and `--mode node --node-type <type> --nodes <N> --show-ri-options` to validate deployment model choice.
## Data Integrity
- [ ] **REQUIRED** TTL strategy defined for all key prefixes. No unbounded key growth.
- [ ] **REQUIRED** Eviction policy set appropriately. Serverless: fixed at `volatile-lru` (not configurable; all keys must have TTLs set to be evictable). Node-based: set via parameter group (`allkeys-lru` for general caching, `volatile-lru` if some keys must never be evicted).
- [ ] **RECOMMENDED** Key naming convention documented and consistent across the application.
## Quick Validation Commands
```bash
# 1. Verify security posture
python3 scripts/security_audit.py --serverless <name> --region <region>
# 2. Deploy alarms + dashboard
python3 scripts/generate_dashboards.py --serverless <name> --sns-topic <arn> --output observability.json
aws cloudformation deploy --template-file observability.json --stack-name <name>-observability --region <region>
# 3. Enable slow log (node-based only; serverless does not support log delivery --
# use CloudWatch metrics and client-side logging for serverless caches)
aws elasticache modify-replication-group --replication-group-id <id> \
--apply-immediately \
--log-delivery-configurations '[{"LogType":"slow-log","DestinationType":"cloudwatch-logs","DestinationDetails":{"CloudWatchLogsDetails":{"LogGroup":"/aws/elasticache/<name>/slowlog"}},"LogFormat":"json"}]'
# 4. Verify connectivity from application
python3 scripts/test_connection.py <endpoint>
# 5. Confirm alarms are in OK state
aws cloudwatch describe-alarms --alarm-name-prefix <name> --region <region> --query 'MetricAlarms[].{Name:AlarmName,State:StateValue}'
```
For node-based, replace `--serverless <name>` with `--replication-group <id>`. Log delivery (step 3) applies to node-based clusters only.
scripts/collect_metrics.sh
#!/usr/bin/env bash
# ============================================================
# ElastiCache Metrics Collection Script
#
# Run this on a machine that can connect to your ElastiCache
# cluster endpoints. It collects the data needed for the
# serverless cost estimator.
#
# Usage:
# ./collect_metrics.sh <endpoint> [port] [output_prefix]
#
# Example:
# ./collect_metrics.sh my-cluster.abc123.use1.cache.amazonaws.com 6379 my-cluster
#
# NOTE: For Redis OSS (cluster mode enabled) clusters, the INFO command only
# returns data for the node handling the connection, not the entire cluster.
# To get complete metrics for a multi-shard cluster, run this script against
# each shard's primary node individually and aggregate the results. You can
# find per-shard endpoints using:
# aws elasticache describe-replication-groups --replication-group-id <id>
#
# Requirements:
# - valkey-cli installed (or redis-cli)
# - bc (for floating-point math in summary output)
# - Network access to the ElastiCache endpoint (must run from within the VPC
# or a bastion/VPN with security group access on port 6379. Running from a
# local laptop will timeout unless VPN/tunnel is configured.)
# - For AUTH-enabled clusters, set REDIS_PASSWORD env var
# - For RBAC (ACL) auth, also set REDIS_USER env var
# - For non-TLS clusters, set NO_TLS=1
# - If valkey-cli was compiled without TLS support, set NO_TLS=1
# ============================================================
set -e
ENDPOINT="${1:?Usage: $0 <endpoint> [port] [output_prefix]}"
PORT="${2:-6379}"
PREFIX="${3:-cluster}"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
# Detect CLI
CLI="valkey-cli"
if ! command -v valkey-cli &>/dev/null; then
if command -v redis-cli &>/dev/null; then
CLI="redis-cli"
else
echo "Error: valkey-cli (or redis-cli) not found. Install with: brew install valkey" >&2
exit 1
fi
fi
if [ -n "$REDIS_PASSWORD" ]; then
export REDISCLI_AUTH="$REDIS_PASSWORD"
fi
# RBAC user support
USER_ARGS=""
if [ -n "${REDIS_USER:-}" ]; then
USER_ARGS="--user $REDIS_USER"
fi
# TLS support: check if NO_TLS is set, or if the CLI lacks TLS support.
# NOTE: The "$CLI --tls --help" probe below is a best-effort heuristic. Some
# CLI versions may return non-zero even when TLS is supported, or succeed
# without actual TLS support. If this detection gives wrong results, set
# NO_TLS=1 (for non-TLS endpoints) or leave it unset (for TLS endpoints)
# explicitly to bypass the probe.
TLS_ARGS="--tls"
if [ "${NO_TLS:-0}" = "1" ]; then
TLS_ARGS=""
elif ! $CLI --tls --help &>/dev/null 2>&1; then
echo "Warning: $CLI does not appear to support --tls (compiled without SSL?)." >&2
echo " This check ('$CLI --tls --help') can be unreliable across CLI versions." >&2
echo " Set NO_TLS=1 to suppress this check, or rebuild with TLS support." >&2
echo " Proceeding without --tls." >&2
TLS_ARGS=""
fi
if ! command -v bc &>/dev/null; then
echo "Warning: 'bc' not found. Summary calculations will be skipped." >&2
NO_BC=1
fi
echo "Collecting metrics from ${ENDPOINT}:${PORT}..."
echo "Using: ${CLI}"
# 1. Memory info
echo " Collecting memory info..."
$CLI -h "$ENDPOINT" -p "$PORT" $TLS_ARGS $USER_ARGS INFO memory > "${PREFIX}_memory_${TIMESTAMP}.txt"
# 2. Command stats
echo " Collecting commandstats..."
$CLI -h "$ENDPOINT" -p "$PORT" $TLS_ARGS $USER_ARGS INFO commandstats > "${PREFIX}_commandstats_${TIMESTAMP}.txt"
# 3. Replication info (to identify primary vs replica)
echo " Collecting replication info..."
$CLI -h "$ENDPOINT" -p "$PORT" $TLS_ARGS $USER_ARGS INFO replication > "${PREFIX}_replication_${TIMESTAMP}.txt"
# 4. Server info (engine version, uptime)
echo " Collecting server info..."
$CLI -h "$ENDPOINT" -p "$PORT" $TLS_ARGS $USER_ARGS INFO server > "${PREFIX}_server_${TIMESTAMP}.txt"
echo ""
echo "Done. Files created:"
echo " ${PREFIX}_memory_${TIMESTAMP}.txt"
echo " ${PREFIX}_commandstats_${TIMESTAMP}.txt"
echo " ${PREFIX}_replication_${TIMESTAMP}.txt"
echo " ${PREFIX}_server_${TIMESTAMP}.txt"
echo ""
echo "Key values to extract:"
echo ""
# Extract key metrics
DATASET=$(grep "used_memory_dataset:" "${PREFIX}_memory_${TIMESTAMP}.txt" | cut -d: -f2 | tr -d '[:space:]' || true)
ROLE=$(grep "role:" "${PREFIX}_replication_${TIMESTAMP}.txt" | cut -d: -f2 | tr -d '[:space:]' || true)
UPTIME=$(grep "uptime_in_seconds:" "${PREFIX}_server_${TIMESTAMP}.txt" | cut -d: -f2 | tr -d '[:space:]' || true)
if [ -z "${NO_BC:-}" ]; then
if [ -n "$DATASET" ]; then
DATASET_GB=$(echo "scale=4; $DATASET / 1073741824" | bc)
echo " used_memory_dataset: ${DATASET} bytes (${DATASET_GB} GB)"
fi
if [ -n "$ROLE" ]; then
echo " role: ${ROLE}"
fi
if [ -n "$UPTIME" ]; then
UPTIME_DAYS=$(echo "scale=1; $UPTIME / 86400" | bc)
echo " uptime: ${UPTIME_DAYS} days"
echo ""
echo "To get daily commands, divide each command's 'calls' by ${UPTIME_DAYS}"
fi
fi
echo ""
echo "Next steps:"
echo " 1. If this is a primary node, use avg_memory_gb = ${DATASET_GB:-<value>}"
echo " 2. Parse commandstats into CSV format for detailed estimation"
echo " 3. Run: python serverless_estimator.py --input clusters.csv --commandstats commandstats.csv"
scripts/command_classifier.py
"""
Command classification for ElastiCache Serverless ECPU estimation.
Serverless bills ECPUs based on both vCPU time and data transferred:
Simple commands (e.g., GET, SET, HGET):
1 ECPU per kilobyte (KB) of data transferred, minimum 1 ECPU.
For example, a GET returning 3.2 KB costs 3.2 ECPUs.
These are O(1) operations with predictable cost.
Complex commands (e.g., EVAL, SORT, MGET, HGETALL):
The HIGHER of two dimensions: vCPU time (relative to a baseline
GET/SET) or data transferred in KB. The two dimensions are NOT
additive. For example, a SORT at 3x vCPU time transferring 2 KB
costs 3 ECPUs (vCPU dimension wins).
NOTE: This module uses a heuristic approximation for the vCPU
dimension. AWS documentation does not publish the exact ECPU
calculation formula. AWS states that "the number of ECPUs consumed
by your requests depends on the vCPU time taken and the amount of
data transferred" and that the higher dimension determines the cost.
Actual ECPU consumption may differ from this estimate.
Internal/service commands (e.g., REPLCONF, PSYNC, CLUSTER):
Excluded from estimates. Includes commands that are not metered
(AUTH, MULTI, EXEC, SUBSCRIBE, UNSUBSCRIBE, CONFIG, CLUSTER),
replication internals (REPLCONF, PSYNC), and commands ignored by
the metering system (INFO, PUBLISH). If your workload issues a
high volume of commands not covered by this classifier, actual
ECPU costs may differ from the estimate. Use CloudWatch ECPU
metrics for precise serverless billing.
IMPORTANT: The simple/complex classification and specific ECPU formulas used
in this module are heuristic approximations based on publicly available AWS
documentation and pricing examples. AWS does not publish exact ECPU calculation
formulas. Per AWS docs, "the number of ECPUs consumed by your requests depends
on the vCPU time taken and the amount of data transferred." For accurate
billing data, use CloudWatch ElastiCacheProcessingUnits and per-command ECPU
metrics on serverless caches.
Reference: https://aws.amazon.com/elasticache/pricing/
"""
# Simple (fixed) commands - 1 ECPU per KB transferred (minimum 1 ECPU).
# Based on ElastiCache Serverless pricing documentation.
# Any command not listed here or in INTERNAL_COMMANDS falls through to
# nonfixed (complex), which uses MAX(calls, usec/3) as a conservative estimate.
FIXED_COMMANDS = frozenset([
"get", "set", "hget", "hset",
"incr", "decr", "incrby", "decrby", "incrbyfloat",
"expire", "pexpire", "pexpireat", "expireat", "persist",
"exists", "ttl", "pttl", "type", "strlen",
"scard", "zcard", "llen", "xlen",
"sismember", "hexists", "hlen", "hsetnx",
"hincrby", "hincrbyfloat",
"getbit", "setbit", "setnx", "setex", "psetex",
"zscore",
"ping",
"del", "unlink",
"select", "echo", "time", "quit", "reset",
"watch", "unwatch", "move", "asking",
"readonly", "readwrite",
"acl", "client", "command",
])
# Commands not metered by serverless - excluded from ECPU estimates.
# Includes free commands (AUTH, MULTI, EXEC, pub/sub) and ignored
# commands (INFO, PUBLISH). Also includes replication/cluster internals
# and commands not available on serverless.
INTERNAL_COMMANDS = frozenset([
"auth", "multi", "exec", "hello", "discard",
"subscribe", "unsubscribe", "psubscribe", "punsubscribe",
"publish", "pubsub",
"info", "config", "cluster",
"replconf", "psync", "replicaof",
"slowlog", "dbsize", "wait",
"object", "debug", "memory", "latency",
"module", "function", "swapdb",
])
def classify_command(cmd_name: str) -> str:
"""Classify a Redis/Valkey command.
Returns:
'fixed' - 1 ECPU per call (simple O(1) commands)
'nonfixed' - estimated as MAX(calls, usec/3) ECPUs (approximation)
'internal' - not metered, exclude from estimate
"""
cmd = cmd_name.lower().strip()
if cmd in INTERNAL_COMMANDS:
return "internal"
if cmd in FIXED_COMMANDS:
return "fixed"
return "nonfixed"
def estimate_ecpus_from_commandstats(commandstats: dict) -> dict:
"""Estimate ECPUs from Redis/Valkey INFO commandstats output.
Args:
commandstats: Dict mapping command name -> {calls: int, usec: int}
Example:
{"get": {"calls": 1000000, "usec": 1500000},
"eval": {"calls": 50000, "usec": 2000000}}
Returns:
Dict with:
total_ecpus: Estimated total ECPUs
fixed_ecpus: ECPUs from fixed-price commands
nonfixed_ecpus: ECPUs from non-fixed commands
internal_calls: Calls excluded (internal commands)
command_breakdown: Per-command detail list
"""
fixed_ecpus = 0
nonfixed_ecpus = 0
internal_calls = 0
breakdown = []
for cmd, stats in commandstats.items():
calls = stats.get("calls", 0)
usec = stats.get("usec", 0)
classification = classify_command(cmd)
if classification == "internal":
internal_calls += calls
breakdown.append({
"command": cmd, "type": "internal",
"calls": calls, "usec": usec, "ecpus": 0,
})
elif classification == "fixed":
ecpus = calls
fixed_ecpus += ecpus
breakdown.append({
"command": cmd, "type": "fixed",
"calls": calls, "usec": usec, "ecpus": ecpus,
})
else:
# Non-fixed: MAX(calls, usec/3) - approximation based on the
# assumption that 1 ECPU ~ 3 microseconds of vCPU time (derived
# from the AWS pricing blog example). Actual ECPU consumption
# may differ. See module docstring for details.
ecpus = max(calls, usec / 3.0)
nonfixed_ecpus += ecpus
breakdown.append({
"command": cmd, "type": "nonfixed",
"calls": calls, "usec": usec, "ecpus": round(ecpus),
})
return {
"total_ecpus": round(fixed_ecpus + nonfixed_ecpus),
"fixed_ecpus": round(fixed_ecpus),
"nonfixed_ecpus": round(nonfixed_ecpus),
"internal_calls": internal_calls,
"command_breakdown": sorted(
breakdown, key=lambda x: x["ecpus"], reverse=True
),
}
scripts/find_tunnel_host.py
#!/usr/bin/env python3
"""
Find an existing SSM-managed EC2 instance in a VPC to use as a tunnel target.
This is the zero-cost path for local development: reuse an instance the
customer already has instead of creating a new jump host.
Dependencies:
pip install boto3
Usage:
python find_tunnel_host.py --vpc-id vpc-abc123
python find_tunnel_host.py --vpc-id vpc-abc123 --region us-west-2
python find_tunnel_host.py --vpc-id vpc-abc123 --profile my-profile
"""
from __future__ import annotations
import argparse
import os
import sys
try:
import boto3
except ImportError:
print(
"Error: the 'boto3' package is required.\n"
"Install it with:\n"
" pip install boto3"
)
sys.exit(2)
def find_tunnel_hosts(vpc_id: str, region: str, profile: str | None) -> list[dict]:
session = boto3.Session(profile_name=profile, region_name=region)
ec2 = session.client("ec2")
ssm = session.client("ssm")
print(f"Scanning VPC {vpc_id} in {region} for SSM-managed instances ...\n")
paginator = ec2.get_paginator("describe_instances")
pages = paginator.paginate(
Filters=[
{"Name": "vpc-id", "Values": [vpc_id]},
{"Name": "instance-state-name", "Values": ["running"]},
]
)
instances = []
for page in pages:
for reservation in page["Reservations"]:
for inst in reservation["Instances"]:
name = ""
for tag in inst.get("Tags", []):
if tag["Key"] == "Name":
name = tag["Value"]
break
instances.append(
{
"instance_id": inst["InstanceId"],
"name": name,
"type": inst["InstanceType"],
"az": inst["Placement"]["AvailabilityZone"],
"private_ip": inst.get("PrivateIpAddress", ""),
}
)
if not instances:
print("No running EC2 instances found in this VPC.\n")
print("Options:")
print(" 1. Create a minimal jump host: t4g.nano (~$3/month)")
print(" Launch one with: aws ec2 run-instances (attach the AmazonSSMManagedInstanceCore IAM policy), then run scripts/start_tunnel.py to connect")
print(" 2. Deploy your app to Lambda/ECS/EKS in the VPC (no tunnel needed)")
return []
ssm_managed = set()
try:
instance_ids = [inst["instance_id"] for inst in instances]
ssm_paginator = ssm.get_paginator("describe_instance_information")
for page in ssm_paginator.paginate(
Filters=[{"Key": "InstanceIds", "Values": instance_ids}]
):
for info in page["InstanceInformationList"]:
ssm_managed.add(info["InstanceId"])
except Exception as exc:
error_code = ""
if hasattr(exc, "response"):
error_code = exc.response.get("Error", {}).get("Code", "")
if error_code in ("AccessDeniedException", "AccessDenied", "UnauthorizedAccess"):
print(f"ERROR: Access denied querying SSM. Ensure your IAM role/user has "
f"ssm:DescribeInstanceInformation permission.")
print("Listing all instances -- SSM status unknown.\n")
else:
print(f"Warning: could not query SSM: {exc}")
print("Listing all instances -- SSM status unknown.\n")
results = []
for inst in instances:
inst["ssm"] = inst["instance_id"] in ssm_managed
results.append(inst)
results.sort(key=lambda x: (not x["ssm"], x["type"]))
return results
def main() -> None:
parser = argparse.ArgumentParser(
description="Find SSM-managed EC2 instances in a VPC for tunnel use.",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument("--vpc-id", required=True, help="VPC ID to scan")
_default_region = (
os.environ.get("AWS_REGION")
or os.environ.get("AWS_DEFAULT_REGION")
or "us-east-1"
)
parser.add_argument(
"--region", default=_default_region, help=f"AWS region (default: {_default_region})"
)
parser.add_argument("--profile", default=None, help="AWS profile name")
args = parser.parse_args()
try:
results = find_tunnel_hosts(args.vpc_id, args.region, args.profile)
except Exception as e:
error_code = ""
if hasattr(e, "response"):
error_code = e.response.get("Error", {}).get("Code", "")
if error_code in ("AccessDeniedException", "AccessDenied", "UnauthorizedAccess"):
print(f"ERROR: Access denied. Ensure your IAM role/user has ec2:DescribeInstances "
f"and ssm:DescribeInstanceInformation permissions.")
else:
print(f"ERROR: {e}")
sys.exit(1)
if not results:
sys.exit(1)
ssm_ready = [r for r in results if r["ssm"]]
not_ssm = [r for r in results if not r["ssm"]]
if ssm_ready:
print(f"Found {len(ssm_ready)} SSM-managed instance(s) (ready for tunnel, $0 extra cost):\n")
print(f" {'Instance ID':<22} {'Name':<25} {'Type':<14} {'AZ':<16} {'Private IP'}")
print(f" {'-'*22} {'-'*25} {'-'*14} {'-'*16} {'-'*15}")
for r in ssm_ready:
print(f" {r['instance_id']:<22} {r['name']:<25} {r['type']:<14} {r['az']:<16} {r['private_ip']}")
best = ssm_ready[0]
print(f"\nRecommended: {best['instance_id']} ({best['name'] or best['type']})")
print(f"\nNext step: start a tunnel with:")
print(f" python scripts/start_tunnel.py \\")
print(f" --instance-id {best['instance_id']} \\")
print(f" --cache-host <your-cache-endpoint> \\")
print(f" --region {args.region}")
print(f"\n For ElastiCache Serverless, you must also tunnel port 6380:")
print(f" python scripts/start_tunnel.py \\")
print(f" --instance-id {best['instance_id']} \\")
print(f" --cache-host <your-cache-endpoint> \\")
print(f" --cache-port 6380 \\")
print(f" --local-port 6380 \\")
print(f" --region {args.region}")
print(f"\n Note: ElastiCache Serverless requires TLS. When connecting through")
print(f" the tunnel, use --tls with --sni <your-cache-endpoint> to pass the")
print(f" the real cache hostname for reference.")
else:
print(f"Found {len(not_ssm)} instance(s) but none have SSM agent:\n")
print(f" {'Instance ID':<22} {'Name':<25} {'Type':<14}")
print(f" {'-'*22} {'-'*25} {'-'*14}")
for r in not_ssm:
print(f" {r['instance_id']:<22} {r['name']:<25} {r['type']:<14}")
print("\nOptions:")
print(" 1. Install SSM agent on one of these instances (free):")
print(" Attach the AmazonSSMManagedInstanceCore IAM policy to the instance role")
print(" 2. Create a minimal jump host (~$3/month):")
print(" Launch one with: aws ec2 run-instances (attach the AmazonSSMManagedInstanceCore IAM policy), then run scripts/start_tunnel.py to connect")
if not_ssm and ssm_ready:
print(f"\n({len(not_ssm)} additional instance(s) without SSM agent not shown)")
if __name__ == "__main__":
main()
scripts/generate_dashboards.py
#!/usr/bin/env python3
"""
ElastiCache CloudWatch Dashboard and Alarm Generator
Emits CloudFormation JSON for dashboards and alarms, parameterized by
deployment type (serverless vs node-based) and cache identifier.
Usage:
# Serverless dashboard + alarms
python generate_dashboards.py --serverless my-cache --region us-east-1
# Node-based dashboard + alarms
python generate_dashboards.py --replication-group my-cluster --region us-east-1
# Dashboard only (no alarms)
python generate_dashboards.py --serverless my-cache --no-alarms
# Alarms only (no dashboard)
python generate_dashboards.py --serverless my-cache --no-dashboard
# Write to file
python generate_dashboards.py --serverless my-cache --output elasticache-observability.json
# Set SNS topic for alarm notifications
python generate_dashboards.py --serverless my-cache --sns-topic arn:aws:sns:us-east-1:123456789012:alerts
"""
import argparse
import json
import re
import sys
# ---------------------------------------------------------------------------
# Dependency check
# This script uses only the Python standard library (no pip install needed).
# It generates CloudFormation JSON locally and does not call AWS APIs.
# ---------------------------------------------------------------------------
def serverless_dashboard_body(cache_name, region):
"""CloudWatch dashboard body for a serverless cache.
Note: CacheHitRate and some command-family metrics are only available for
Valkey/Redis OSS serverless caches, not Memcached serverless caches.
"""
return {
"widgets": [
_metric_widget("Cache Hit Rate", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "CacheHitRate", "ServerlessCacheName", cache_name]],
period=300),
_metric_widget("ECPU Consumption", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "ElastiCacheProcessingUnits", "ServerlessCacheName", cache_name]],
period=60),
_metric_widget("Throttled Commands", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "ThrottledCmds", "ServerlessCacheName", cache_name]],
period=60, stat="Sum"),
_metric_widget("Read / Write Latency", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "SuccessfulReadRequestLatency", "ServerlessCacheName", cache_name],
["AWS/ElastiCache", "SuccessfulWriteRequestLatency", "ServerlessCacheName", cache_name]],
period=60),
_metric_widget("Current Connections", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "CurrConnections", "ServerlessCacheName", cache_name]],
period=300),
_metric_widget("New Connections", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "NewConnections", "ServerlessCacheName", cache_name]],
period=300, stat="Sum"),
_metric_widget("Bytes Used For Cache", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "BytesUsedForCache", "ServerlessCacheName", cache_name]],
period=300),
_metric_widget("Command-Family Breakdown", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "StringBasedCmds", "ServerlessCacheName", cache_name],
["AWS/ElastiCache", "HashBasedCmds", "ServerlessCacheName", cache_name],
["AWS/ElastiCache", "SortedSetBasedCmds", "ServerlessCacheName", cache_name],
["AWS/ElastiCache", "StreamBasedCmds", "ServerlessCacheName", cache_name],
["AWS/ElastiCache", "PubSubBasedCmds", "ServerlessCacheName", cache_name]],
period=300, stat="Sum"),
_metric_widget("Total Commands Count", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "TotalCmdsCount", "ServerlessCacheName", cache_name]],
period=300, stat="Sum"),
_metric_widget("Item Counts", region, "ServerlessCacheName", cache_name,
[["AWS/ElastiCache", "CurrVolatileItems", "ServerlessCacheName", cache_name],
["AWS/ElastiCache", "CurrItems", "ServerlessCacheName", cache_name]],
period=300),
]
}
def node_based_dashboard_body(rg_id, region, node_ids=None):
"""CloudWatch dashboard body for a node-based replication group.
Args:
node_ids: List of CacheClusterId values (e.g., ["my-cluster-001", "my-cluster-002"]).
If provided, per-node metrics are shown. If None, falls back to ReplicationGroupId
which only works for a subset of metrics.
"""
if node_ids:
# Per-node metrics using CacheClusterId dimension
def _node_metrics(metric_name):
return [["AWS/ElastiCache", metric_name, "CacheClusterId", nid] for nid in node_ids]
return {
"widgets": [
_metric_widget("Cache Hit Rate", region, "CacheClusterId", node_ids[0],
_node_metrics("CacheHitRate"), period=300),
_metric_widget("Engine CPU Utilization", region, "CacheClusterId", node_ids[0],
_node_metrics("EngineCPUUtilization") + _node_metrics("CPUUtilization"),
period=60),
_metric_widget("Database Memory Usage", region, "CacheClusterId", node_ids[0],
_node_metrics("DatabaseMemoryUsagePercentage"), period=300),
_metric_widget("Read / Write Latency", region, "CacheClusterId", node_ids[0],
_node_metrics("SuccessfulReadRequestLatency") + _node_metrics("SuccessfulWriteRequestLatency"),
period=60),
_metric_widget("Current Connections", region, "CacheClusterId", node_ids[0],
_node_metrics("CurrConnections"), period=300),
_metric_widget("Replication Lag", region, "CacheClusterId", node_ids[0],
_node_metrics("ReplicationLag"), period=60),
_metric_widget("Network Bytes In/Out", region, "CacheClusterId", node_ids[0],
_node_metrics("NetworkBytesIn") + _node_metrics("NetworkBytesOut"),
period=300, stat="Sum"),
_metric_widget("Evictions", region, "CacheClusterId", node_ids[0],
_node_metrics("Evictions"), period=300, stat="Sum"),
_metric_widget("Command-Family Breakdown", region, "CacheClusterId", node_ids[0],
_node_metrics("StringBasedCmds") + _node_metrics("HashBasedCmds") +
_node_metrics("SortedSetBasedCmds") + _node_metrics("StreamBasedCmds") +
_node_metrics("SearchBasedCmds"),
period=300, stat="Sum"),
]
}
else:
# Fallback: ReplicationGroupId (limited metrics available)
dim_name = "ReplicationGroupId"
return {
"widgets": [
_metric_widget("Cache Hit Rate", region, dim_name, rg_id,
[["AWS/ElastiCache", "CacheHitRate", dim_name, rg_id]], period=300),
_metric_widget("Engine CPU Utilization", region, dim_name, rg_id,
[["AWS/ElastiCache", "EngineCPUUtilization", dim_name, rg_id],
["AWS/ElastiCache", "CPUUtilization", dim_name, rg_id]], period=60),
_metric_widget("Database Memory Usage", region, dim_name, rg_id,
[["AWS/ElastiCache", "DatabaseMemoryUsagePercentage", dim_name, rg_id]], period=300),
_metric_widget("Read / Write Latency", region, dim_name, rg_id,
[["AWS/ElastiCache", "SuccessfulReadRequestLatency", dim_name, rg_id],
["AWS/ElastiCache", "SuccessfulWriteRequestLatency", dim_name, rg_id]], period=60),
_metric_widget("Current Connections", region, dim_name, rg_id,
[["AWS/ElastiCache", "CurrConnections", dim_name, rg_id]], period=300),
_metric_widget("Replication Lag", region, dim_name, rg_id,
[["AWS/ElastiCache", "ReplicationLag", dim_name, rg_id]], period=60),
_metric_widget("Evictions", region, dim_name, rg_id,
[["AWS/ElastiCache", "Evictions", dim_name, rg_id]], period=300, stat="Sum"),
]
}
def serverless_alarms(cache_name, sns_topic=None, max_storage_gb=None,
storage_alarm_pct=80,
hit_rate_threshold=80, throttle_threshold=None,
read_latency_threshold=None, write_latency_threshold=None,
ecpu_threshold=None, evictions_threshold=None):
"""CloudFormation alarm resources for a serverless cache.
Alarms are controlled via threshold parameters. Pass None to disable
an alarm. CacheHitRate is enabled by default at 80%.
"""
alarms = {}
dim = [{"Name": "ServerlessCacheName", "Value": cache_name}]
if hit_rate_threshold is not None:
alarms["CacheHitRateAlarm"] = _alarm(
f"{cache_name}-low-hit-rate",
f"Cache hit rate below {hit_rate_threshold}% for {cache_name}",
"AWS/ElastiCache", "CacheHitRate", dim,
threshold=hit_rate_threshold, comparison="LessThanThreshold",
period=300, eval_periods=6, datapoints_to_alarm=4, stat="Average",
sns_topic=sns_topic
)
if throttle_threshold is not None:
alarms["ThrottledCmdsAlarm"] = _alarm(
f"{cache_name}-throttled-cmds",
f"Throttled commands above {throttle_threshold} on {cache_name}",
"AWS/ElastiCache", "ThrottledCmds", dim,
threshold=throttle_threshold, comparison="GreaterThanThreshold",
period=60, eval_periods=3, datapoints_to_alarm=2, stat="Sum",
sns_topic=sns_topic
)
if max_storage_gb is not None:
storage_threshold_bytes = int(max_storage_gb * (storage_alarm_pct / 100) * 1073741824)
alarms["StorageLimitAlarm"] = _alarm(
f"{cache_name}-storage-approaching-limit",
f"Storage approaching {storage_alarm_pct}% of configured limit ({max_storage_gb} GB) on {cache_name}",
"AWS/ElastiCache", "BytesUsedForCache", dim,
threshold=storage_threshold_bytes, comparison="GreaterThanThreshold",
period=300, eval_periods=3, datapoints_to_alarm=2, stat="Maximum",
sns_topic=sns_topic
)
if read_latency_threshold is not None:
alarms["ReadLatencyAlarm"] = _alarm(
f"{cache_name}-high-read-latency",
f"Read latency above {read_latency_threshold}us on {cache_name}",
"AWS/ElastiCache", "SuccessfulReadRequestLatency", dim,
threshold=read_latency_threshold, comparison="GreaterThanThreshold",
period=60, eval_periods=5, datapoints_to_alarm=3, stat="Average",
sns_topic=sns_topic
)
if write_latency_threshold is not None:
alarms["WriteLatencyAlarm"] = _alarm(
f"{cache_name}-high-write-latency",
f"Write latency above {write_latency_threshold}us on {cache_name}",
"AWS/ElastiCache", "SuccessfulWriteRequestLatency", dim,
threshold=write_latency_threshold, comparison="GreaterThanThreshold",
period=60, eval_periods=5, datapoints_to_alarm=3, stat="Average",
sns_topic=sns_topic
)
if ecpu_threshold is not None:
alarms["ECPUSpikeAlarm"] = _alarm(
f"{cache_name}-ecpu-spike",
f"ECPU consumption above {ecpu_threshold} on {cache_name}",
"AWS/ElastiCache", "ElastiCacheProcessingUnits", dim,
threshold=ecpu_threshold, comparison="GreaterThanThreshold",
period=300, eval_periods=3, datapoints_to_alarm=2, stat="Sum",
sns_topic=sns_topic
)
if evictions_threshold is not None:
alarms["EvictionsAlarm"] = _alarm(
f"{cache_name}-evictions",
f"Evictions above {evictions_threshold} on {cache_name}",
"AWS/ElastiCache", "Evictions", dim,
threshold=evictions_threshold, comparison="GreaterThanThreshold",
period=300, eval_periods=3, datapoints_to_alarm=2, stat="Sum",
sns_topic=sns_topic
)
return alarms
def node_based_alarms(rg_id, sns_topic=None, engine_cpu_threshold=90,
memory_threshold=80, hit_rate_threshold=80,
replication_lag_threshold=None,
read_latency_threshold=None, write_latency_threshold=None,
evictions_threshold=None, new_connections_threshold=None,
node_ids=None):
"""CloudFormation alarm resources for a node-based replication group.
Alarms are controlled via threshold parameters. Pass None to disable
an alarm. EngineCPU (90%), memory (80%), and hit rate (80%) are enabled
by default. If node_ids are provided, alarms are created for each node
using CacheClusterId. Otherwise falls back to ReplicationGroupId
(limited metric availability).
"""
alarms = {}
if node_ids:
targets = [("CacheClusterId", nid) for nid in node_ids]
else:
targets = [("ReplicationGroupId", rg_id)]
for dim_name, dim_value in targets:
dim = [{"Name": dim_name, "Value": dim_value}]
prefix = dim_value
safe = re.sub(r'[^a-zA-Z0-9]', '', dim_value)
if hit_rate_threshold is not None:
alarms[f"CacheHitRateAlarm{safe}"] = _alarm(
f"{prefix}-low-hit-rate",
f"Cache hit rate below {hit_rate_threshold}% on {dim_value}",
"AWS/ElastiCache", "CacheHitRate", dim,
threshold=hit_rate_threshold, comparison="LessThanThreshold",
period=300, eval_periods=6, datapoints_to_alarm=4, stat="Average",
sns_topic=sns_topic
)
if engine_cpu_threshold is not None:
alarms[f"EngineCPUAlarm{safe}"] = _alarm(
f"{prefix}-high-engine-cpu",
f"Engine CPU above {engine_cpu_threshold}% on {dim_value}",
"AWS/ElastiCache", "EngineCPUUtilization", dim,
threshold=engine_cpu_threshold, comparison="GreaterThanThreshold",
period=60, eval_periods=5, datapoints_to_alarm=3, stat="Maximum",
sns_topic=sns_topic
)
if memory_threshold is not None:
alarms[f"MemoryAlarm{safe}"] = _alarm(
f"{prefix}-high-memory",
f"Memory usage above {memory_threshold}% on {dim_value}",
"AWS/ElastiCache", "DatabaseMemoryUsagePercentage", dim,
threshold=memory_threshold, comparison="GreaterThanThreshold",
period=60, eval_periods=5, datapoints_to_alarm=3, stat="Maximum",
sns_topic=sns_topic
)
if replication_lag_threshold is not None:
alarms[f"ReplicationLagAlarm{safe}"] = _alarm(
f"{prefix}-high-replication-lag",
f"Replication lag above {replication_lag_threshold}s on {dim_value}",
"AWS/ElastiCache", "ReplicationLag", dim,
threshold=replication_lag_threshold, comparison="GreaterThanThreshold",
period=60, eval_periods=5, datapoints_to_alarm=3, stat="Maximum",
sns_topic=sns_topic
)
if read_latency_threshold is not None:
alarms[f"ReadLatencyAlarm{safe}"] = _alarm(
f"{prefix}-high-read-latency",
f"Read latency above {read_latency_threshold}us on {dim_value}",
"AWS/ElastiCache", "SuccessfulReadRequestLatency", dim,
threshold=read_latency_threshold, comparison="GreaterThanThreshold",
period=60, eval_periods=5, datapoints_to_alarm=3, stat="Average",
sns_topic=sns_topic
)
if write_latency_threshold is not None:
alarms[f"WriteLatencyAlarm{safe}"] = _alarm(
f"{prefix}-high-write-latency",
f"Write latency above {write_latency_threshold}us on {dim_value}",
"AWS/ElastiCache", "SuccessfulWriteRequestLatency", dim,
threshold=write_latency_threshold, comparison="GreaterThanThreshold",
period=60, eval_periods=5, datapoints_to_alarm=3, stat="Average",
sns_topic=sns_topic
)
if evictions_threshold is not None:
alarms[f"EvictionsAlarm{safe}"] = _alarm(
f"{prefix}-evictions",
f"Evictions above {evictions_threshold} on {dim_value}",
"AWS/ElastiCache", "Evictions", dim,
threshold=evictions_threshold, comparison="GreaterThanThreshold",
period=300, eval_periods=3, datapoints_to_alarm=2, stat="Sum",
sns_topic=sns_topic
)
if new_connections_threshold is not None:
alarms[f"NewConnectionsAlarm{safe}"] = _alarm(
f"{prefix}-new-connections",
f"New connections above {new_connections_threshold}/min on {dim_value}",
"AWS/ElastiCache", "NewConnections", dim,
threshold=new_connections_threshold, comparison="GreaterThanThreshold",
period=60, eval_periods=3, datapoints_to_alarm=2, stat="Sum",
sns_topic=sns_topic
)
return alarms
def _metric_widget(title, region, dim_name, dim_value, metrics, period=300, stat="Average"):
"""Build a CloudWatch dashboard metric widget."""
return {
"type": "metric",
"properties": {
"title": title,
"region": region,
"metrics": metrics,
"period": period,
"stat": stat,
"view": "timeSeries",
},
"width": 12,
"height": 6,
}
def _alarm(name, description, namespace, metric, dimensions, threshold,
comparison, period, eval_periods, stat, sns_topic=None,
datapoints_to_alarm=None):
"""Build a CloudFormation alarm resource."""
alarm = {
"Type": "AWS::CloudWatch::Alarm",
"Properties": {
"AlarmName": name,
"AlarmDescription": description,
"Namespace": namespace,
"MetricName": metric,
"Dimensions": dimensions,
"Threshold": threshold,
"ComparisonOperator": comparison,
"Period": period,
"EvaluationPeriods": eval_periods,
"DatapointsToAlarm": datapoints_to_alarm if datapoints_to_alarm else eval_periods,
"Statistic": stat,
"TreatMissingData": "notBreaching",
}
}
if sns_topic:
props = alarm["Properties"]
assert isinstance(props, dict)
props["AlarmActions"] = [{"Ref": "SNSTopicARN"}]
props["OKActions"] = [{"Ref": "SNSTopicARN"}]
return alarm
def generate_template(cache_type, identifier, region, sns_topic=None,
include_dashboard=True, include_alarms=True,
max_storage_gb=None, storage_alarm_pct=80, memory_threshold=80,
engine_cpu_threshold=90,
replication_lag_threshold=None, hit_rate_threshold=80,
throttle_threshold=None,
read_latency_threshold=None, write_latency_threshold=None,
ecpu_threshold=None, evictions_threshold=None,
new_connections_threshold=None, node_ids=None):
"""Generate a complete CloudFormation template."""
resources = {}
if include_dashboard:
if cache_type == "serverless":
body = serverless_dashboard_body(identifier, region)
else:
body = node_based_dashboard_body(identifier, region, node_ids=node_ids)
safe_name = re.sub(r'[^a-zA-Z0-9]', '', identifier)
resources[f"{safe_name}Dashboard"] = {
"Type": "AWS::CloudWatch::Dashboard",
"Properties": {
"DashboardName": f"ElastiCache-{identifier}",
"DashboardBody": json.dumps(body),
}
}
if include_alarms:
if cache_type == "serverless":
alarms = serverless_alarms(identifier, sns_topic,
max_storage_gb=max_storage_gb,
storage_alarm_pct=storage_alarm_pct,
hit_rate_threshold=hit_rate_threshold,
throttle_threshold=throttle_threshold,
read_latency_threshold=read_latency_threshold,
write_latency_threshold=write_latency_threshold,
ecpu_threshold=ecpu_threshold,
evictions_threshold=evictions_threshold)
else:
alarms = node_based_alarms(identifier, sns_topic,
memory_threshold=memory_threshold,
engine_cpu_threshold=engine_cpu_threshold,
replication_lag_threshold=replication_lag_threshold,
hit_rate_threshold=hit_rate_threshold,
read_latency_threshold=read_latency_threshold,
write_latency_threshold=write_latency_threshold,
evictions_threshold=evictions_threshold,
new_connections_threshold=new_connections_threshold,
node_ids=node_ids)
resources.update(alarms)
template = {
"AWSTemplateFormatVersion": "2010-09-09",
"Description": f"ElastiCache observability stack for {identifier}",
"Resources": resources,
}
if sns_topic:
template["Parameters"] = {
"SNSTopicARN": {
"Type": "String",
"Default": sns_topic,
"Description": "SNS topic ARN for alarm notifications"
}
}
return template
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="ElastiCache Dashboard & Alarm Generator")
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("--serverless", metavar="CACHE_NAME", help="Generate for a serverless cache")
group.add_argument("--replication-group", metavar="RG_ID", help="Generate for a node-based replication group")
parser.add_argument("--node-ids", default=None,
help="Comma-separated CacheClusterIds for per-node metrics (e.g., my-cluster-001,my-cluster-002). "
"Required for node-based dashboards to show data.")
parser.add_argument("--region", default="us-east-1", help="AWS region for dashboard widgets")
parser.add_argument("--sns-topic", default=None, help="SNS topic ARN for alarm notifications")
parser.add_argument("--no-dashboard", action="store_true", help="Skip dashboard generation")
parser.add_argument("--no-alarms", action="store_true", help="Skip alarm generation")
parser.add_argument("--max-storage-gb", type=float, default=None,
help="Serverless MaxDataStorageGB for storage alarm. If omitted, alarm is skipped.")
parser.add_argument("--storage-alarm-pct", type=float, default=80,
help="Percentage of max-storage-gb to alarm at (default: 80)")
parser.add_argument("--memory-threshold", type=float, default=80,
help="Node-based memory usage alarm threshold in percent (default: 80)")
parser.add_argument("--engine-cpu-threshold", type=float, default=90,
help="Node-based engine CPU alarm threshold in percent (default: 90)")
parser.add_argument("--replication-lag-threshold", type=float, default=None,
help="Node-based replication lag alarm threshold in seconds. If omitted, alarm is skipped.")
parser.add_argument("--hit-rate-threshold", type=float, default=80,
help="Cache hit rate alarm threshold in percent (default: 80)")
parser.add_argument("--throttle-threshold", type=float, default=None,
help="Serverless throttled commands alarm threshold (off by default)")
parser.add_argument("--read-latency-threshold", type=float, default=None,
help="Read latency alarm threshold in microseconds (off by default)")
parser.add_argument("--write-latency-threshold", type=float, default=None,
help="Write latency alarm threshold in microseconds (off by default)")
parser.add_argument("--ecpu-threshold", type=float, default=None,
help="Serverless ECPU consumption alarm threshold (off by default)")
parser.add_argument("--evictions-threshold", type=float, default=None,
help="Evictions alarm threshold per 5-min period (off by default)")
parser.add_argument("--new-connections-threshold", type=float, default=None,
help="Node-based new connections alarm threshold per minute (off by default)")
parser.add_argument("--output", "-o", default=None, help="Write to file instead of stdout")
args = parser.parse_args()
if args.serverless:
cache_type = "serverless"
identifier = args.serverless
else:
cache_type = "node-based"
identifier = args.replication_group
node_ids = [nid.strip() for nid in args.node_ids.split(",") if nid.strip()] if args.node_ids else None
template = generate_template(
cache_type, identifier, args.region,
sns_topic=args.sns_topic,
include_dashboard=not args.no_dashboard,
include_alarms=not args.no_alarms,
max_storage_gb=args.max_storage_gb,
storage_alarm_pct=args.storage_alarm_pct,
memory_threshold=args.memory_threshold,
engine_cpu_threshold=args.engine_cpu_threshold,
replication_lag_threshold=args.replication_lag_threshold,
hit_rate_threshold=args.hit_rate_threshold,
throttle_threshold=args.throttle_threshold,
read_latency_threshold=args.read_latency_threshold,
write_latency_threshold=args.write_latency_threshold,
ecpu_threshold=args.ecpu_threshold,
evictions_threshold=args.evictions_threshold,
new_connections_threshold=args.new_connections_threshold,
node_ids=node_ids,
)
output = json.dumps(template, indent=2)
if args.output:
with open(args.output, "w") as f:
f.write(output)
print(f"Written to {args.output}")
print(f"\nDeploy with:")
print(f" aws cloudformation deploy --template-file {args.output} --stack-name {identifier}-observability")
else:
print(output)
scripts/input_validator.py
#!/usr/bin/env python3
"""
ElastiCache Input Validator
Validates and sanitizes user-provided values before they are interpolated into
CLI commands. Prevents shell injection and ensures identifiers conform to AWS
naming rules.
Usage (as a library):
from input_validator import validate_cache_name, sanitize_for_cli, validate_all
ok, err = validate_cache_name("my-cache")
safe = sanitize_for_cli("my-cache; rm -rf /")
errors = validate_all({"cache_name": "my-cache", "region": "us-east-1"})
Usage (standalone -- validates a set of key=value pairs):
python input_validator.py cache_name=my-cache region=us-east-1 subnet_id=subnet-abc123
"""
from __future__ import annotations
import re
import sys
from typing import Optional
# ---------------------------------------------------------------------------
# Individual validators
# ---------------------------------------------------------------------------
# Each function returns (is_valid: bool, error_message: str | None).
# A None error_message means the value is valid.
# ---------------------------------------------------------------------------
def validate_cache_name(name: str) -> tuple[bool, Optional[str]]:
"""Validate an ElastiCache cache or cluster name.
Rules:
- 1 to 40 characters
- Alphanumeric characters and hyphens only
- Must start with a letter
- No consecutive hyphens
- Must not end with a hyphen
Note: Serverless cache names must be lowercase. Node-based cluster names
are case-insensitive (AWS lowercases them). This validator enforces
lowercase to ensure compatibility with both deployment types.
"""
if not name:
return False, "Cache name must not be empty"
if len(name) > 40:
return False, f"Cache name must be 1-40 characters (got {len(name)})"
if not re.match(r'^[a-z]', name):
return False, "Cache name must start with a lowercase letter (serverless requires lowercase; node-based is case-insensitive but lowercased by AWS)"
if not re.match(r'^[a-z0-9-]+$', name):
return False, "Cache name may only contain lowercase alphanumeric characters and hyphens"
if '--' in name:
return False, "Cache name must not contain consecutive hyphens"
if name.endswith('-'):
return False, "Cache name must not end with a hyphen"
return True, None
def validate_replication_group_id(rg_id: str) -> tuple[bool, Optional[str]]:
"""Validate a replication group ID.
Same rules as cache name: 1-40 chars, alphanumeric + hyphens, starts with
a letter, no consecutive hyphens, no trailing hyphen.
"""
if not rg_id:
return False, "Replication group ID must not be empty"
if len(rg_id) > 40:
return False, f"Replication group ID must be 1-40 characters (got {len(rg_id)})"
if not re.match(r'^[a-zA-Z]', rg_id):
return False, "Replication group ID must start with a letter"
if not re.match(r'^[a-zA-Z0-9-]+$', rg_id):
return False, "Replication group ID may only contain alphanumeric characters and hyphens"
if '--' in rg_id:
return False, "Replication group ID must not contain consecutive hyphens"
if rg_id.endswith('-'):
return False, "Replication group ID must not end with a hyphen"
return True, None
def validate_subnet_id(subnet_id: str) -> tuple[bool, Optional[str]]:
"""Validate an AWS subnet ID (subnet-<8-17 hex chars>)."""
if not subnet_id:
return False, "Subnet ID must not be empty"
if not re.match(r'^subnet-[0-9a-f]{8,17}$', subnet_id):
return False, "Subnet ID must match pattern subnet-<8-17 hex characters> (e.g. subnet-0abc1234def56789a)"
return True, None
def validate_security_group_id(sg_id: str) -> tuple[bool, Optional[str]]:
"""Validate an AWS security group ID (sg-<8-17 hex chars>)."""
if not sg_id:
return False, "Security group ID must not be empty"
if not re.match(r'^sg-[0-9a-f]{8,17}$', sg_id):
return False, "Security group ID must match pattern sg-<8-17 hex characters> (e.g. sg-0abc1234def56789a)"
return True, None
def validate_region(region: str) -> tuple[bool, Optional[str]]:
"""Validate an AWS region code (e.g. us-east-1, eu-west-2, ap-southeast-1)."""
if not region:
return False, "Region must not be empty"
if not re.match(r'^[a-z]{2}-[a-z]+-\d+$', region):
return False, "Region must match AWS region pattern (e.g. us-east-1, eu-west-2)"
return True, None
def validate_engine_version(version: str) -> tuple[bool, Optional[str]]:
r"""Validate an engine version string.
Accepted formats:
- Major only: "7" (used by serverless MajorEngineVersion)
- Major.minor: "7.1", "8.2"
- Major.minor.patch: "7.0.4", "7.1.0"
"""
if not version:
return False, "Engine version must not be empty"
if not re.match(r'^\d+(\.\d+){0,2}$', version):
return False, "Engine version must match pattern <major>[.<minor>[.<patch>]] (e.g. 7, 7.1, 7.0.4)"
return True, None
def validate_vpc_id(vpc_id: str) -> tuple[bool, Optional[str]]:
"""Validate an AWS VPC ID (vpc-<8-17 hex chars>)."""
if not vpc_id:
return False, "VPC ID must not be empty"
if not re.match(r'^vpc-[0-9a-f]{8,17}$', vpc_id):
return False, "VPC ID must match pattern vpc-<8-17 hex characters> (e.g. vpc-0abc1234def56789a)"
return True, None
def validate_kms_key_id(key_id: str) -> tuple[bool, Optional[str]]:
"""Validate a KMS key ID.
Accepts either:
- A UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
- A KMS key ARN: arn:aws:kms:<region>:<account>:key/<key-id>
- A KMS alias ARN: arn:aws:kms:<region>:<account>:alias/<alias-name>
"""
if not key_id:
return False, "KMS key ID must not be empty"
uuid_pattern = r'^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
arn_pattern = r'^arn:aws:kms:[a-z0-9-]+:\d{12}:(key|alias)/[a-zA-Z0-9/_-]+$'
if re.match(uuid_pattern, key_id):
return True, None
if re.match(arn_pattern, key_id):
return True, None
return False, (
"KMS key ID must be a UUID (e.g. 12345678-1234-1234-1234-123456789abc) "
"or a KMS ARN (e.g. arn:aws:kms:us-east-1:123456789012:key/<key-id>)"
)
def validate_snapshot_name(name: str, serverless: bool = False) -> tuple[bool, Optional[str]]:
"""Validate a snapshot name.
Rules:
- 1 to 255 characters for serverless snapshots, 1 to 40 for node-based
- Alphanumeric characters and hyphens only
- Must start with a letter
- No consecutive hyphens
- Must not end with a hyphen
"""
if not name:
return False, "Snapshot name must not be empty"
max_len = 255 if serverless else 40
if len(name) > max_len:
label = "serverless" if serverless else "node-based"
return False, f"Snapshot name must be 1-{max_len} characters for {label} (got {len(name)})"
if not re.match(r'^[a-zA-Z]', name):
return False, "Snapshot name must start with a letter"
if not re.match(r'^[a-zA-Z0-9-]+$', name):
return False, "Snapshot name may only contain alphanumeric characters and hyphens"
if '--' in name:
return False, "Snapshot name must not contain consecutive hyphens"
if name.endswith('-'):
return False, "Snapshot name must not end with a hyphen"
return True, None
# ---------------------------------------------------------------------------
# CLI sanitization safety net
# ---------------------------------------------------------------------------
# Shell metacharacters that could enable command injection
_SHELL_METACHARACTERS = re.compile(r'[;&|`$(){}!<>\'\"\\\n\r\x00]')
def sanitize_for_cli(value: str) -> str:
"""Strip shell metacharacters from a value as a defense-in-depth safety net.
This is NOT a substitute for proper validation -- always validate inputs
first using the validate_* functions. This function provides an additional
layer of protection by removing characters that could be interpreted by
a shell if a value is accidentally interpolated into a command string.
Returns the sanitized string with all shell metacharacters removed.
"""
return _SHELL_METACHARACTERS.sub('', value)
# ---------------------------------------------------------------------------
# Bulk validation
# ---------------------------------------------------------------------------
# Map of recognized parameter keys to their validator functions
_VALIDATORS = {
"cache_name": validate_cache_name,
"replication_group_id": validate_replication_group_id,
"subnet_id": validate_subnet_id,
"security_group_id": validate_security_group_id,
"region": validate_region,
"engine_version": validate_engine_version,
"vpc_id": validate_vpc_id,
"kms_key_id": validate_kms_key_id,
"snapshot_name": validate_snapshot_name,
}
def validate_all(params: dict) -> list[str]:
"""Validate all recognized keys in a parameter dict.
Takes a dict like {"cache_name": "my-cache", "region": "us-east-1"} and
validates every key that has a known validator. Keys not recognized are
silently skipped (they may be application-specific parameters).
Returns a list of error messages. An empty list means all recognized
parameters passed validation.
"""
errors = []
for key, value in params.items():
validator = _VALIDATORS.get(key)
if validator is None:
continue
if not isinstance(value, str):
errors.append(f"{key}: expected a string, got {type(value).__name__}")
continue
is_valid, error_msg = validator(value) # type: ignore[operator]
if not is_valid:
errors.append(f"{key}: {error_msg}")
return errors
# ---------------------------------------------------------------------------
# CLI entrypoint
# ---------------------------------------------------------------------------
def main():
"""Validate key=value pairs passed on the command line."""
if len(sys.argv) < 2:
print("Usage: python input_validator.py key=value [key=value ...]")
print("Example: python input_validator.py cache_name=my-cache region=us-east-1")
print()
print("Recognized keys:", ", ".join(sorted(_VALIDATORS.keys())))
sys.exit(0)
params = {}
for arg in sys.argv[1:]:
if '=' not in arg:
print(f"Error: argument '{arg}' is not in key=value format")
sys.exit(2)
key, value = arg.split('=', 1)
params[key] = value
errors = validate_all(params)
if errors:
print("Validation FAILED:")
for err in errors:
print(f" - {err}")
sys.exit(1)
else:
print("All inputs valid.")
sys.exit(0)
if __name__ == "__main__":
main()
scripts/migration_preflight.py
#!/usr/bin/env python3
"""
ElastiCache Migration Preflight Check
Connects to a source Redis/Valkey instance and evaluates migration readiness:
- Engine version compatibility
- Module detection (RedisJSON, RediSearch, RedisTimeSeries, etc.)
- Key count and memory usage
- Persistence configuration
- Cluster mode detection
- Unsupported command sampling
Usage:
# Check a source Redis instance (plain)
python migration_preflight.py --host redis.example.com --port 6379
# With TLS
python migration_preflight.py --host redis.example.com --port 6379 --tls
# With auth
python migration_preflight.py --host redis.example.com --password mypassword
# JSON output
python migration_preflight.py --host redis.example.com --json
# Via tunnel (local port forward)
python migration_preflight.py --host localhost --port 6379
Requires: valkey (pip install valkey)
"""
import argparse
import json
import sys
# ---------------------------------------------------------------------------
# Dependency check -- fail early with an actionable message instead of a
# raw ImportError traceback.
# ---------------------------------------------------------------------------
try:
import valkey as redis
except ImportError:
print(
"Error: the 'valkey' package is required for this script.\n"
"Install with:\n"
" pip install valkey"
)
sys.exit(1)
# Commands not available on ElastiCache (restricted or modified)
ELASTICACHE_RESTRICTED_COMMANDS = {
"BGSAVE", "BGREWRITEAOF", "CONFIG", "DEBUG", "MIGRATE",
"SAVE", "SHUTDOWN", "SLAVEOF", "REPLICAOF", "SYNC",
}
# Known Redis modules and their ElastiCache availability
MODULE_COMPATIBILITY = {
"search": {"name": "RediSearch", "elasticache": False, "note": "Not available on ElastiCache. Valkey 8.2 provides native vector search (FT.* commands) on node-based clusters."},
"ReJSON": {"name": "RedisJSON", "elasticache": False, "note": "The ReJSON module cannot be loaded on ElastiCache. However, ElastiCache provides native JSON support (JSON.* commands) in Valkey 7.2+ and Redis OSS 6.2.6+. Verify command compatibility before migration."},
"timeseries": {"name": "RedisTimeSeries", "elasticache": False, "note": "Not available on ElastiCache. Consider CloudWatch or Timestream."},
"bf": {"name": "RedisBloom", "elasticache": False, "note": "No (module) / Yes (native BF.* commands in Valkey 8.1+)."},
"graph": {"name": "RedisGraph", "elasticache": False, "note": "Deprecated. Consider Neptune or OpenSearch."},
"ai": {"name": "RedisAI", "elasticache": False, "note": "Not available on ElastiCache. Use SageMaker for ML inference."},
}
def run_preflight(host, port, password=None, use_tls=False, username=None):
"""Run all preflight checks against a source Redis/Valkey instance."""
findings = []
info = {"host": host, "port": port}
try:
r = redis.Redis(
host=host, port=port,
password=password, username=username,
ssl=use_tls,
decode_responses=True,
socket_timeout=10,
socket_connect_timeout=10,
)
r.ping()
findings.append({"check": "Connectivity", "status": "PASS", "detail": f"Connected to {host}:{port}"})
except (redis.ConnectionError, redis.TimeoutError) as e:
return {"error": f"Cannot connect to {host}:{port}: {e}", "info": info, "findings": [
{"check": "Connectivity", "status": "FAIL", "detail": str(e)}
]}
except redis.AuthenticationError as e:
return {"error": f"Authentication failed: {e}", "info": info, "findings": [
{"check": "Connectivity", "status": "FAIL", "detail": f"Auth failed: {e}"}
]}
# Server info
try:
server_info = r.info()
except Exception as e:
findings.append({"check": "Server info", "status": "ERROR", "detail": f"Could not retrieve INFO: {e}"})
r.close()
return {"info": info, "findings": findings}
# Engine and version
redis_version = str(server_info.get("redis_version", "unknown"))
info["redis_version"] = redis_version
info["os"] = server_info.get("os", "unknown")
info["uptime_days"] = round(server_info.get("uptime_in_seconds", 0) / 86400, 1)
parts = redis_version.split(".")
try:
version_tuple = tuple(int(p) for p in parts[:2]) if len(parts) >= 2 else (int(parts[0]), 0)
except (ValueError, IndexError):
version_tuple = (0, 0)
if version_tuple >= (7, 0):
findings.append({"check": "Engine version", "status": "PASS",
"detail": f"Redis {redis_version} can migrate to Valkey 7.2 (direct in-place switch) or Redis OSS 7+ on ElastiCache"})
elif version_tuple >= (6, 0):
findings.append({"check": "Engine version", "status": "WARN",
"detail": f"Redis {redis_version} standard support ends January 31, 2027. Extended Support charges begin February 1, 2027. Recommend upgrading to Valkey before then."})
elif version_tuple >= (5, 0):
findings.append({"check": "Engine version", "status": "FAIL",
"detail": f"Redis {redis_version} standard support ended January 31, 2026. Extended Support charges are now active (since Feb 1, 2026). Upgrade to Valkey to eliminate surcharges."})
findings.append({"check": "Extended Support", "status": "FAIL",
"detail": f"Redis {redis_version} is enrolled in Extended Support. Year 1-2 premium: 80% surcharge. Year 3 premium: 160% surcharge. Run scripts/price_calculator.py --extended-support to see costs."})
else:
findings.append({"check": "Engine version", "status": "FAIL",
"detail": f"Redis {redis_version} standard support ended January 31, 2026. Extended Support charges are now active. Upgrade to Valkey to eliminate surcharges."})
findings.append({"check": "Extended Support", "status": "FAIL",
"detail": f"Redis {redis_version} is enrolled in Extended Support. Year 1-2 premium: 80% surcharge. Year 3 premium: 160% surcharge. Run scripts/price_calculator.py --extended-support to see costs."})
# Cluster mode
cluster_enabled = server_info.get("cluster_enabled", 0)
info["cluster_mode"] = bool(cluster_enabled)
if cluster_enabled:
findings.append({"check": "Cluster mode", "status": "INFO",
"detail": "Cluster mode enabled — target must also be cluster mode enabled"})
else:
findings.append({"check": "Cluster mode", "status": "INFO",
"detail": "Standalone or replica mode — can target cluster mode disabled or enabled"})
# Connected replicas
connected_replicas = server_info.get("connected_slaves", 0)
info["connected_replicas"] = connected_replicas
findings.append({"check": "Replication", "status": "INFO",
"detail": f"{connected_replicas} connected replica(s)"})
# Memory
used_memory_mb = round(server_info.get("used_memory", 0) / (1024 * 1024), 1)
used_memory_gb = round(used_memory_mb / 1024, 2)
peak_memory_mb = round(server_info.get("used_memory_peak", 0) / (1024 * 1024), 1)
info["used_memory_mb"] = used_memory_mb
info["used_memory_gb"] = used_memory_gb
info["peak_memory_mb"] = peak_memory_mb
findings.append({"check": "Memory usage", "status": "INFO",
"detail": f"Current: {used_memory_mb} MB ({used_memory_gb} GB), Peak: {peak_memory_mb} MB"})
# Key count
try:
db_info = {k: v for k, v in server_info.items() if k.startswith("db")}
total_keys = sum(v.get("keys", 0) for v in db_info.values() if isinstance(v, dict))
info["total_keys"] = total_keys
info["databases_in_use"] = len(db_info)
findings.append({"check": "Key count", "status": "INFO",
"detail": f"{total_keys:,} keys across {len(db_info)} database(s)"})
if len(db_info) > 1:
findings.append({"check": "Multiple databases", "status": "WARN",
"detail": f"{len(db_info)} databases in use — ElastiCache cluster mode uses only db0. Plan key migration."})
except Exception:
info["total_keys"] = "unknown"
# Persistence
rdb_enabled = server_info.get("rdb_last_save_time", 0) > 0
aof_enabled = server_info.get("aof_enabled", 0) == 1
info["rdb_enabled"] = rdb_enabled
info["aof_enabled"] = aof_enabled
persistence_detail = []
if rdb_enabled:
persistence_detail.append("RDB snapshots active")
if aof_enabled:
persistence_detail.append("AOF enabled")
if not persistence_detail:
persistence_detail.append("No persistence")
findings.append({"check": "Persistence", "status": "INFO",
"detail": ", ".join(persistence_detail)})
# Modules
try:
modules = r.execute_command("MODULE", "LIST")
loaded_modules = []
for mod in modules:
if isinstance(mod, list):
mod_name = mod[1] if len(mod) > 1 else str(mod)
elif isinstance(mod, dict):
mod_name = mod.get("name", str(mod))
else:
mod_name = str(mod)
loaded_modules.append(mod_name)
info["modules"] = loaded_modules
if not loaded_modules:
findings.append({"check": "Modules", "status": "PASS", "detail": "No modules loaded"})
else:
for mod_name in loaded_modules:
compat = MODULE_COMPATIBILITY.get(mod_name)
if compat:
if compat["elasticache"]:
findings.append({"check": f"Module: {compat['name']}", "status": "INFO",
"detail": str(compat["note"])})
else:
findings.append({"check": f"Module: {compat['name']}", "status": "FAIL",
"detail": str(compat["note"])})
else:
findings.append({"check": f"Module: {mod_name}", "status": "WARN",
"detail": f"Unknown module '{mod_name}' — verify compatibility with ElastiCache"})
except redis.ResponseError:
info["modules"] = []
findings.append({"check": "Modules", "status": "INFO",
"detail": "MODULE LIST not available (may be restricted or old version)"})
# Commandstats — check for restricted command usage
cmdstats = {}
try:
cmdstats = r.info("commandstats")
for cmd_name in ELASTICACHE_RESTRICTED_COMMANDS:
stat_key = f"cmdstat_{cmd_name.lower().replace(' ', '|')}"
if stat_key in cmdstats:
calls = cmdstats[stat_key].get("calls", 0)
if calls > 0:
findings.append({"check": "Restricted command usage", "status": "WARN",
"detail": f"Command {cmd_name} called {calls} times. This command is restricted on ElastiCache. Review usage before migration."})
except redis.ResponseError:
findings.append({"check": "Restricted command usage", "status": "INFO",
"detail": "INFO COMMANDSTATS not available (may be restricted by ACL or unsupported on this engine version)"})
except (redis.ConnectionError, redis.TimeoutError) as e:
findings.append({"check": "Restricted command usage", "status": "INFO",
"detail": f"Could not retrieve commandstats: {e}"})
# Lua script detection
lua_detected = False
try:
eval_calls = 0
for key in ("cmdstat_eval", "cmdstat_evalsha"):
if key in cmdstats:
eval_calls += cmdstats[key].get("calls", 0)
if eval_calls > 0:
lua_detected = True
findings.append({"check": "Lua scripts", "status": "WARN",
"detail": f"Lua scripts detected ({eval_calls} eval calls). Test all Lua scripts against the target engine. Scripts using module commands or hardcoded key names may break."})
except (KeyError, TypeError, AttributeError):
findings.append({"check": "Lua scripts", "status": "INFO",
"detail": "Unable to detect Lua script usage from commandstats"})
info["lua_scripts_detected"] = lua_detected
# Large key sampling
try:
large_keys_found = []
for _ in range(20):
key = r.randomkey()
if key is None:
break
try:
usage = r.memory_usage(key)
if usage is not None and usage > 1048576:
size_mb = round(usage / (1024 * 1024), 2)
large_keys_found.append((key, size_mb))
except Exception:
findings.append({"check": "Large key sampling", "status": "INFO",
"detail": "MEMORY USAGE command not available, skipping large key check"})
break
for key, size_mb in large_keys_found:
key_type = r.type(key) if key else "unknown"
findings.append({"check": "Large key sampling", "status": "WARN",
"detail": f"Large key detected ({key_type}, {size_mb}MB). Large keys can cause replication timeouts and event loop blocking during migration."})
except Exception:
pass
# Maxmemory policy
# CONFIG GET is restricted on ElastiCache managed endpoints. Detect them by
# hostname and skip CONFIG in that case, advising the user to check via the
# AWS API (describe-cache-parameters) instead.
is_elasticache = host.endswith(".cache.amazonaws.com")
if is_elasticache:
info["maxmemory_policy"] = "unknown (ElastiCache source)"
findings.append({"check": "Eviction policy", "status": "INFO",
"detail": "CONFIG command is restricted on ElastiCache. "
"Check the eviction policy via: aws elasticache describe-cache-parameters "
"--cache-parameter-group-name <param-group> "
"--query \"Parameters[?ParameterName=='maxmemory-policy']\""})
else:
try:
config = r.config_get("maxmemory-policy")
policy = config.get("maxmemory-policy", "unknown")
info["maxmemory_policy"] = policy
if policy in ("noeviction",):
findings.append({"check": "Eviction policy", "status": "WARN",
"detail": f"Policy is '{policy}' -- writes will fail when memory is full. Consider volatile-lru or allkeys-lru."})
else:
findings.append({"check": "Eviction policy", "status": "INFO",
"detail": f"Policy: {policy}"})
except redis.ResponseError:
info["maxmemory_policy"] = "unknown"
# Data tiering advisory
findings.append({"check": "Data tiering advisory", "status": "INFO",
"detail": "If targeting r6gd node types (data tiering), note: online migration is not supported for r6gd clusters. "
"Use backup/restore instead. Data tiering only supports volatile-lru, allkeys-lru, volatile-lfu, allkeys-lfu, and noeviction eviction policies."})
# Sizing recommendation
findings.append({"check": "Sizing recommendation", "status": "INFO",
"detail": f"Source dataset: {used_memory_gb} GB. Use scripts/price_calculator.py to estimate serverless and node-based costs for your workload."})
try:
r.close()
except Exception:
pass
return {"info": info, "findings": findings}
def format_report(result):
"""Format preflight results as a human-readable report."""
if "error" in result:
info = result.get("info", {})
lines = [f"ERROR: {result['error']}"]
if info:
lines.append(f" Host: {info.get('host', '?')}:{info.get('port', '?')}")
return "\n".join(lines)
info = result["info"]
findings = result["findings"]
lines = []
lines.append("=" * 72)
lines.append("ElastiCache Migration Preflight Report")
lines.append("=" * 72)
lines.append("")
lines.append(f" Source: {info['host']}:{info['port']}")
lines.append(f" Redis version: {info.get('redis_version', 'unknown')}")
lines.append(f" OS: {info.get('os', 'unknown')}")
lines.append(f" Uptime: {info.get('uptime_days', '?')} days")
lines.append(f" Cluster mode: {'enabled' if info.get('cluster_mode') else 'disabled'}")
lines.append(f" Memory: {info.get('used_memory_mb', '?')} MB ({info.get('used_memory_gb', '?')} GB)")
lines.append(f" Keys: {info.get('total_keys', '?')}")
lines.append(f" Replicas: {info.get('connected_replicas', '?')}")
lines.append(f" Modules: {', '.join(info.get('modules', [])) or 'none'}")
lines.append("")
fail_count = sum(1 for f in findings if f["status"] == "FAIL")
warn_count = sum(1 for f in findings if f["status"] == "WARN")
pass_count = sum(1 for f in findings if f["status"] == "PASS")
if fail_count == 0:
lines.append(f" Verdict: READY TO MIGRATE ({pass_count} passed, {warn_count} warnings)")
else:
lines.append(f" Verdict: BLOCKERS FOUND ({fail_count} failures, {warn_count} warnings)")
lines.append("")
lines.append("-" * 72)
status_order = {"FAIL": 0, "WARN": 1, "ERROR": 2, "PASS": 3, "INFO": 4}
sorted_findings = sorted(findings, key=lambda f: status_order.get(f["status"], 5))
for f in sorted_findings:
icon = {"PASS": "[OK] ", "FAIL": "[FAIL]", "WARN": "[WARN]", "INFO": "[INFO]", "ERROR": "[ERR] "}
lines.append(f" {icon.get(f['status'], '[?] ')} {f['check']}")
lines.append(f" {f['detail']}")
lines.append("")
if fail_count > 0:
lines.append("=" * 72)
lines.append("MIGRATION BLOCKERS")
lines.append("=" * 72)
lines.append("")
for f in sorted_findings:
if f["status"] == "FAIL":
lines.append(f" - {f['check']}: {f['detail']}")
lines.append("")
lines.append(" Resolve blockers before starting migration.")
lines.append(" See references/migration/instructions.md for guidance.")
lines.append("")
lines.append(" Next steps:")
lines.append(" 1. Resolve any blockers above")
lines.append(" 2. Run: python scripts/price_calculator.py to estimate target cost")
lines.append(" 3. Use the AWS CLI test-migration command to validate connectivity to the target cluster")
lines.append(" 4. See references/migration/instructions.md for the full migration workflow")
lines.append("")
return "\n".join(lines)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="ElastiCache Migration Preflight Check")
parser.add_argument("--host", required=True, help="Source Redis/Valkey hostname")
parser.add_argument("--port", type=int, default=6379, help="Port (default: 6379)")
parser.add_argument("--password", default=None, help="AUTH password")
parser.add_argument("--username", default=None, help="Username (for ACL-based auth)")
parser.add_argument("--tls", action="store_true", help="Use TLS")
parser.add_argument("--json", action="store_true", help="Output as JSON")
args = parser.parse_args()
result = run_preflight(args.host, args.port, args.password, args.tls, args.username)
if args.json:
print(json.dumps(result, indent=2, default=str))
else:
print(format_report(result))
if "error" in result:
sys.exit(1)
fail_count = sum(1 for f in result.get("findings", []) if f["status"] == "FAIL")
sys.exit(1 if fail_count > 0 else 0)
scripts/price_calculator.py
#!/usr/bin/env python3
"""
ElastiCache Price Calculator
Fetches live pricing from AWS and estimates costs for specific configurations.
Supports:
- Serverless (Valkey, Redis OSS, Memcached)
- Node-based on-demand and reserved instances
- Multi-node clusters with replicas
- Extended Support surcharge for EOL Redis versions
Usage:
# Serverless estimate
python3 price_calculator.py --engine valkey --mode serverless --data-gb 5 --ecpu-millions 200
# Node-based with replicas
python3 price_calculator.py --engine valkey --mode node --node-type cache.r7g.large --nodes 3
# Node-based with specific RI term
python3 price_calculator.py --mode node --node-type cache.r7g.large --nodes 2 --ri-term 1yr_no_upfront
# Show all reserved options for a node type
python3 price_calculator.py --mode node --node-type cache.r7g.large --nodes 2 --show-ri-options
# Specific region
python3 price_calculator.py --engine valkey --region eu-west-1 --mode serverless --data-gb 10
# Extended Support surcharge for EOL Redis versions
python3 price_calculator.py --engine redis --extended-support --node-type cache.r7g.large --nodes 6
# Interactive mode
python3 price_calculator.py --interactive
"""
import argparse
import json
import os
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from pricing import PricingLoader
# Detect region from environment with us-east-1 as fallback
REGION = os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION") or "us-east-1"
HOURS_PER_MONTH = 730
PRICING_DISCLAIMER = (
" Estimates only. Verify current pricing at https://aws.amazon.com/elasticache/pricing/"
)
def estimate_serverless(loader, engine, region, data_gb, ecpu_millions_per_month, avg_payload_kb=1.0):
storage_rate = loader.get_serverless_storage_rate(region, engine)
ecpu_rate = loader.get_serverless_ecpu_rate(region, engine)
min_gb = 0.1 if engine == "valkey" else 1.0
billing_gb = max(data_gb, min_gb)
storage = billing_gb * storage_rate * HOURS_PER_MONTH
scaled_ecpus = ecpu_millions_per_month * max(1.0, avg_payload_kb)
compute = scaled_ecpus * 1_000_000 * ecpu_rate
result = {
"model": "serverless",
"engine": engine,
"region": region,
"data_gb": data_gb,
"ecpu_millions": ecpu_millions_per_month,
"storage_monthly": round(storage, 2),
"compute_monthly": round(compute, 2),
"total_monthly": round(storage + compute, 2),
}
if avg_payload_kb != 1.0:
result["avg_payload_kb"] = avg_payload_kb
result["ecpu_millions_scaled"] = round(scaled_ecpus, 1)
return result
def estimate_node_based(loader, engine, region, node_type, node_count, ri_term=None):
try:
hourly = loader.get_node_hourly_rate(region, node_type, engine)
except ValueError as e:
return {"error": str(e)}
monthly_per_node = hourly * HOURS_PER_MONTH
total = monthly_per_node * node_count
mem = loader.get_node_memory_gb(node_type)
result = {
"model": "node-based (on-demand)",
"engine": engine,
"region": region,
"node_type": node_type,
"node_count": node_count,
"memory_per_node_gb": mem if mem > 0 else "unknown",
"total_memory_gb": round(mem * node_count, 1) if mem > 0 else "unknown",
"per_node_monthly": round(monthly_per_node, 2),
"total_monthly": round(total, 2),
}
# Try real reserved pricing first, fall back to hardcoded discount
if ri_term:
ri_map = {
"1yr_no_upfront": ("1yr", "No Upfront"),
"1yr_partial_upfront": ("1yr", "Partial Upfront"),
"1yr_all_upfront": ("1yr", "All Upfront"),
"3yr_no_upfront": ("3yr", "No Upfront"),
"3yr_partial_upfront": ("3yr", "Partial Upfront"),
"3yr_all_upfront": ("3yr", "All Upfront"),
}
if ri_term in ri_map:
lease, purchase = ri_map[ri_term]
try:
eff_monthly = loader.get_reserved_effective_monthly(
region, node_type, engine, lease, purchase
)
ri_total = eff_monthly * node_count
discount_pct = round((1 - ri_total / total) * 100)
result["reserved"] = {
"term": ri_term,
"discount_pct": discount_pct,
"per_node_monthly": eff_monthly,
"total_monthly": round(ri_total, 2),
"annual_savings": round((total - ri_total) * 12, 2),
"source": "live",
}
except ValueError:
pass # no reserved pricing available for this combo
return result
def format_ri_options(loader, engine, region, node_type, node_count):
"""Show all reserved pricing options for a specific node configuration."""
options = loader.list_reserved_options(region, node_type, engine)
hourly = loader.get_node_hourly_rate(region, node_type, engine)
od_monthly = hourly * HOURS_PER_MONTH * node_count
lines = []
lines.append("=" * 68)
lines.append("Reserved Pricing Options (live pricing)")
lines.append("=" * 68)
lines.append("")
lines.append(" Node type: {} x {} ({} engine)".format(node_type, node_count, engine))
lines.append(" On-demand: ${:,.2f}/month".format(od_monthly))
lines.append("")
if not options:
lines.append(" No reserved pricing options found for this configuration.")
lines.append("")
lines.append(PRICING_DISCLAIMER)
lines.append("")
return "\n".join(lines)
lines.append(" {:<10} {:<18} {:>12} {:>10} {:>10}".format(
"Term", "Purchase Option", "Monthly", "Upfront", "Savings"))
lines.append(" {} {} {} {} {}".format("-" * 10, "-" * 18, "-" * 12, "-" * 10, "-" * 10))
for opt in options:
eff_total = opt["effective_monthly"] * node_count
savings_pct = round((1 - eff_total / od_monthly) * 100)
upfront_total = opt["upfront"] * node_count
lines.append(" {:<10} {:<18} ${:>9,.2f} ${:>8,.0f} {:>8}%".format(
opt["lease_length"], opt["purchase_option"],
eff_total, upfront_total, savings_pct))
lines.append("")
lines.append(PRICING_DISCLAIMER)
lines.append("")
return "\n".join(lines)
def estimate_extended_support(loader, engine, region, node_type, node_count):
"""Estimate Extended Support surcharge for EOL Redis versions."""
results = {"engine": engine, "region": region, "node_type": node_type, "node_count": node_count, "years": []}
if not loader.has_extended_support_pricing():
results["note"] = "Extended Support pricing not available in bulk pricing API. Check the ElastiCache pricing page."
return results
for year in ["1", "2", "3"]:
try:
hourly = loader.get_extended_support_rate(region, node_type, engine, year)
monthly = hourly * HOURS_PER_MONTH * node_count
results["years"].append({
"year": int(year),
"per_node_hourly": hourly,
"total_monthly": round(monthly, 2),
"total_annual": round(monthly * 12, 2),
})
except ValueError:
pass
if results["years"]:
try:
base_hourly = loader.get_node_hourly_rate(region, node_type, engine)
base_monthly = base_hourly * HOURS_PER_MONTH * node_count
results["base_monthly"] = round(base_monthly, 2)
total_yr1 = base_monthly + results["years"][0]["total_monthly"]
results["total_with_surcharge_monthly"] = round(total_yr1, 2)
except ValueError:
pass
results["recommendation"] = (
"Migrating to Valkey eliminates Extended Support charges. "
"Savings vs other engines: 20% lower on node-based, 33% lower on serverless."
)
return results
def format_extended_support_report(results):
"""Generate a human-readable Extended Support cost report."""
lines = []
lines.append("=" * 60)
lines.append("Extended Support Cost Estimate (live pricing)")
lines.append("=" * 60)
lines.append("")
lines.append(" Engine: {}".format(results["engine"]))
lines.append(" Region: {}".format(results["region"]))
lines.append(" Node type: {}".format(results.get("node_type", "unknown")))
lines.append(" Nodes: {}".format(results["node_count"]))
lines.append("")
if not results["years"]:
lines.append(" {}".format(results.get("note", "No pricing data available.")))
lines.append("")
return "\n".join(lines)
lines.append(" {:<18} {:>16} {:>16}".format("Year After EOL", "Monthly Cost", "Annual Cost"))
lines.append(" {} {} {}".format("-" * 18, "-" * 16, "-" * 16))
for y in results["years"]:
lines.append(" Year {:<13} ${:>13,.2f} ${:>13,.2f}".format(
y["year"], y["total_monthly"], y["total_annual"]
))
if results.get("base_monthly"):
lines.append("")
lines.append(" Base node cost: ${:>13,.2f}/month".format(results["base_monthly"]))
lines.append(" Total (base + Yr1): ${:>13,.2f}/month".format(results["total_with_surcharge_monthly"]))
lines.append("")
if "recommendation" in results:
lines.append(" {}".format(results["recommendation"]))
lines.append("")
lines.append(PRICING_DISCLAIMER)
lines.append("")
return "\n".join(lines)
def estimate_ecpu_from_ops(ops_per_sec, avg_ecpu_per_op=1.0):
"""Convert operations per second to monthly ECPUs (millions).
Default: 1 ECPU per operation (assumes simple GET/SET with payload under 1 KB).
For larger payloads, use --avg-payload-kb which scales ECPUs linearly.
For complex commands (SORT, ZADD, etc.), pass a higher avg_ecpu_per_op.
"""
ecpu_per_month = ops_per_sec * avg_ecpu_per_op * 3600 * HOURS_PER_MONTH
return round(ecpu_per_month / 1_000_000, 1)
def interactive(loader, region):
"""Interactive interview-style cost estimation."""
print("=== ElastiCache Price Calculator (live pricing) ===\n")
engine = input("Engine [valkey/redis/memcached] (default: valkey): ").strip().lower() or "valkey"
data_gb = float(input("Estimated data size in GB (default: 1): ").strip() or "1")
mode = input("Deployment [serverless/node] (default: serverless): ").strip().lower() or "serverless"
if mode == "serverless":
ops = input("Estimated operations per second (default: 100): ").strip() or "100"
ops_per_sec = float(ops)
ecpu_millions = estimate_ecpu_from_ops(ops_per_sec)
print(" -> Estimated {}M ECPUs/month".format(ecpu_millions))
result = estimate_serverless(loader, engine, region, data_gb, ecpu_millions)
print("\n{}".format(json.dumps(result, indent=2)))
elif mode == "node":
node_type = input("Node type (default: cache.r7g.large): ").strip() or "cache.r7g.large"
nodes = int(input("Number of nodes (default: 2): ").strip() or "2")
result = estimate_node_based(loader, engine, region, node_type, nodes, ri_term="1yr_no_upfront")
print("\n{}".format(json.dumps(result, indent=2)))
show_ri = input("\nShow all reserved options? [y/n] (default: n): ").strip().lower()
if show_ri == "y":
print("\n{}".format(format_ri_options(loader, engine, region, node_type, nodes)))
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="ElastiCache Price Calculator (live pricing)")
parser.add_argument("--interactive", action="store_true", help="Interactive mode")
parser.add_argument("--engine", choices=["valkey", "redis", "memcached"], default="valkey")
parser.add_argument("--mode", choices=["serverless", "node"], default="serverless")
parser.add_argument("--region", default=REGION, help="AWS region (default: from env or us-east-1)")
parser.add_argument("--data-gb", type=float, default=1.0, help="Data stored in GB")
parser.add_argument("--ecpu-millions", type=float, default=None, help="ECPUs per month (millions)")
parser.add_argument("--ops-per-sec", type=float, default=None, help="Operations per second (auto-converts to ECPUs)")
parser.add_argument("--node-type", default="cache.r7g.large", help="Node type for node-based")
parser.add_argument("--nodes", type=int, default=2, help="Node count for node-based")
parser.add_argument("--ri-term", default=None,
choices=["1yr_no_upfront", "1yr_partial_upfront", "1yr_all_upfront",
"3yr_no_upfront", "3yr_partial_upfront", "3yr_all_upfront"],
help="Reserved instance term")
parser.add_argument("--show-ri-options", action="store_true",
help="Show all reserved pricing options for the node type")
parser.add_argument("--extended-support", action="store_true",
help="Show Extended Support surcharge for EOL Redis versions")
parser.add_argument("--avg-payload-kb", type=float, default=1.0,
help="Average payload size in KB (default: 1.0). ECPUs scale linearly with payload.")
parser.add_argument("--pricing-csv", default=None, help="Optional local pricing CSV (overrides live fetch)")
args = parser.parse_args()
loader = PricingLoader(args.pricing_csv)
if args.interactive:
interactive(loader, args.region)
sys.exit(0)
# Auto-convert ops/sec to ECPUs if provided
ecpu_millions = args.ecpu_millions
if ecpu_millions is None and args.ops_per_sec:
ecpu_millions = estimate_ecpu_from_ops(args.ops_per_sec)
elif ecpu_millions is None:
ecpu_millions = 100.0
if args.extended_support:
result = estimate_extended_support(loader, args.engine, args.region, args.node_type, args.nodes)
print(format_extended_support_report(result))
elif args.show_ri_options:
print(format_ri_options(loader, args.engine, args.region, args.node_type, args.nodes))
elif args.mode == "serverless":
result = estimate_serverless(loader, args.engine, args.region, args.data_gb, ecpu_millions, args.avg_payload_kb)
result["disclaimer"] = "Estimates only. Verify at https://aws.amazon.com/elasticache/pricing/"
print(json.dumps(result, indent=2))
elif args.mode == "node":
result = estimate_node_based(loader, args.engine, args.region, args.node_type, args.nodes, ri_term=args.ri_term)
result["disclaimer"] = "Estimates only. Verify at https://aws.amazon.com/elasticache/pricing/"
print(json.dumps(result, indent=2))
scripts/pricing.py
"""
ElastiCache pricing loader.
Fetches live pricing from the AWS Bulk Pricing API (public, no auth required).
Caches locally to avoid repeated downloads. Refreshes if cache is older than 7 days.
Source: https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonElastiCache/current/index.json
Also accepts an optional local CSV for offline or override use.
Supports On-Demand and Reserved Node pricing (1-year and 3-year terms with
No Upfront, Partial Upfront, and All Upfront purchase options). Reserved pricing
is extracted from the bulk pricing JSON terms.Reserved section.
"""
import csv
import json
import os
import sys
import time
import urllib.request
from typing import Dict, Optional, Tuple
BULK_PRICING_URL = "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonElastiCache/current/index.json"
CACHE_MAX_AGE_SECONDS = 7 * 24 * 3600 # 7 days
_CACHE_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".pricing_cache")
_CACHE_FILE = os.path.join(_CACHE_DIR, "elasticache_pricing.json")
def _cache_is_fresh() -> bool:
"""Check if local cache exists and is younger than CACHE_MAX_AGE_SECONDS."""
if not os.path.exists(_CACHE_FILE):
return False
age = time.time() - os.path.getmtime(_CACHE_FILE)
return age < CACHE_MAX_AGE_SECONDS
def _fetch_and_cache() -> dict:
"""Fetch bulk pricing JSON from AWS and cache locally."""
os.makedirs(_CACHE_DIR, exist_ok=True)
print("Fetching live ElastiCache pricing from AWS...", file=sys.stderr)
try:
req = urllib.request.Request(BULK_PRICING_URL)
with urllib.request.urlopen(req, timeout=60) as resp:
raw = resp.read()
except Exception as e:
raise RuntimeError(
"Failed to fetch pricing from {}. Error: {}. "
"Check network connectivity or provide a local pricing CSV.".format(BULK_PRICING_URL, e)
)
data = json.loads(raw)
parsed = _parse_bulk_pricing(data)
parsed["fetched_at"] = time.time()
with open(_CACHE_FILE, "w") as f:
json.dump(parsed, f)
print(" Pricing cached at: {}".format(_CACHE_FILE), file=sys.stderr)
return parsed
def _load_cache() -> dict:
"""Load parsed pricing from local cache."""
with open(_CACHE_FILE, "r") as f:
return json.load(f)
def _parse_bulk_pricing(data: dict) -> dict:
"""Parse the AWS bulk pricing JSON into our compact format.
Extracts:
- On-demand node-based hourly rates by (region, instance_type, engine)
- Reserved node-based rates by (region, instance_type, engine, lease, purchase_option)
- Serverless storage rates by (region, engine)
- Serverless ECPU rates by (region, engine)
- Extended Support surcharges by (region, engine, year)
"""
products = data.get("products", {})
terms = data.get("terms", {}).get("OnDemand", {})
reserved_terms = data.get("terms", {}).get("Reserved", {})
node_pricing = {} # "region|instance_type|engine" -> $/hr
serverless_storage = {} # "region|engine" -> $/GB-hr
serverless_ecpu = {} # "region|engine" -> $/ECPU
extended_support = {} # "region|instance_type|engine|year" -> $/node-hr
reserved_pricing = {} # "region|instance_type|engine|lease|purchase" -> {"hourly": $, "upfront": $}
node_memory = {} # "instance_type" -> GB (float)
for sku, product in products.items():
attrs = product.get("attributes", {})
family = product.get("productFamily", "")
region = attrs.get("regionCode", "")
if not region:
continue
# Skip Outpost and non-standard location types
if attrs.get("locationType", "") != "AWS Region":
continue
# Get the on-demand price and unit for this SKU
price_info = _get_od_price_with_unit(sku, terms)
if price_info is None:
continue
price_per_unit, unit = price_info
if price_per_unit <= 0:
continue
if family == "Cache Instance":
instance_type = attrs.get("instanceType", "")
engine = _normalize_engine(attrs.get("cacheEngine", ""))
usagetype = attrs.get("usagetype", "")
# Extended Support entries live under "Cache Instance" family
# but have "ExtendedSupport" in the usagetype
if "ExtendedSupport" in usagetype:
if not engine or not instance_type or unit != "Hrs":
continue
years = _parse_extended_support_years(usagetype)
for year in years:
key = "{}|{}|{}|{}".format(region, instance_type, engine, year)
extended_support[key] = price_per_unit
elif instance_type and engine and unit == "Hrs":
key = "{}|{}|{}".format(region, instance_type, engine)
node_pricing[key] = price_per_unit
# Extract memory from product attributes (e.g., "103.68 GiB")
if instance_type not in node_memory:
mem_str = attrs.get("memory", "")
mem_gb = _parse_memory_gib(mem_str)
if mem_gb > 0:
node_memory[instance_type] = mem_gb
elif family == "ElastiCache Serverless":
engine = _normalize_engine(attrs.get("cacheEngine", ""))
operation = attrs.get("operation", "")
usagetype = attrs.get("usagetype", "")
if "Snapshot" in operation or "Backup" in usagetype:
continue
if not engine:
continue
key = "{}|{}".format(region, engine)
unit_lower = unit.lower()
if "gb-hour" in unit_lower or "CachedData" in usagetype:
serverless_storage[key] = price_per_unit
elif "processingunit" in unit_lower or "ProcessingUnits" in usagetype:
serverless_ecpu[key] = price_per_unit
# Parse Reserved terms for node-based instances
_valid_purchase_options = {"No Upfront", "Partial Upfront", "All Upfront"}
for sku, offers in reserved_terms.items():
product = products.get(sku)
if not product or product.get("productFamily") != "Cache Instance":
continue
attrs = product.get("attributes", {})
if attrs.get("locationType", "") != "AWS Region":
continue
if "ExtendedSupport" in attrs.get("usagetype", ""):
continue
region = attrs.get("regionCode", "")
instance_type = attrs.get("instanceType", "")
engine = _normalize_engine(attrs.get("cacheEngine", ""))
if not (region and instance_type and engine):
continue
for offer_id, offer in offers.items():
ta = offer.get("termAttributes", {})
purchase_option = ta.get("PurchaseOption", "")
if purchase_option not in _valid_purchase_options:
continue # skip legacy "Heavy Utilization"
lease = ta.get("LeaseContractLength", "") # "1yr" or "3yr"
hourly = 0.0
upfront = 0.0
for dim in offer.get("priceDimensions", {}).values():
try:
price = float(dim.get("pricePerUnit", {}).get("USD", "0"))
except (ValueError, TypeError):
continue
unit = dim.get("unit", "")
if unit == "Hrs":
hourly = price
elif unit == "Quantity":
upfront = price
key = "{}|{}|{}|{}|{}".format(
region, instance_type, engine, lease, purchase_option
)
reserved_pricing[key] = {"hourly": hourly, "upfront": upfront}
return {
"node_pricing": node_pricing,
"serverless_storage": serverless_storage,
"serverless_ecpu": serverless_ecpu,
"extended_support": extended_support,
"reserved_pricing": reserved_pricing,
"node_memory": node_memory,
}
def _get_od_price_with_unit(sku: str, terms: dict) -> Optional[Tuple[float, str]]:
"""Extract on-demand price and unit for a SKU from the terms section."""
sku_terms = terms.get(sku, {})
for offer in sku_terms.values():
dimensions = offer.get("priceDimensions", {})
for dim in dimensions.values():
try:
price = float(dim.get("pricePerUnit", {}).get("USD", "0"))
unit = dim.get("unit", "")
return (price, unit)
except (ValueError, TypeError):
continue
return None
def _parse_extended_support_years(usagetype: str) -> list:
"""Extract year tiers from an Extended Support usagetype string.
AWS usagetype format:
- "USE1-ExtendedSupportYr1_Yr2-NodeUsage:cache.r7g.large" -> years 1 and 2 (same rate)
- "USE1-ExtendedSupportYr3-NodeUsage:cache.r7g.large" -> year 3+
Returns a list of year strings, e.g. ["1", "2"] or ["3"].
"""
ut = usagetype.lower()
if "yr1_yr2" in ut:
return ["1", "2"]
if "yr3" in ut:
return ["3"]
if "yr1" in ut and "yr2" not in ut:
return ["1"]
if "yr2" in ut and "yr1" not in ut:
return ["2"]
# Fallback: generic extended support mention without year
if "extendedsupport" in ut.replace("-", "").replace("_", ""):
return ["1"]
return []
def _normalize_engine(engine_str: str) -> str:
"""Normalize engine name from bulk pricing to our standard names."""
eng = engine_str.lower().strip()
if "valkey" in eng:
return "valkey"
if "redis" in eng:
return "redis"
if "memcached" in eng:
return "memcached"
return eng
def _parse_memory_gib(mem_str: str) -> float:
"""Parse memory string like '103.68 GiB' into a float GB value."""
if not mem_str:
return 0.0
try:
parts = mem_str.strip().split()
return float(parts[0])
except (ValueError, IndexError):
return 0.0
class PricingLoader:
"""Load and look up ElastiCache pricing.
Priority:
1. Local CSV (if provided) - for offline or override use
2. Live AWS bulk pricing (fetched and cached locally for 7 days)
"""
def __init__(self, pricing_csv: Optional[str] = None):
self._serverless_storage: Dict[Tuple[str, str], float] = {}
self._serverless_ecpu: Dict[Tuple[str, str], float] = {}
self._node_od: Dict[Tuple[str, str, str], float] = {}
self._extended_support: Dict[Tuple[str, str, str, str], float] = {} # (region, itype, engine, year)
self._reserved: Dict[Tuple[str, str, str, str, str], dict] = {} # (region, itype, engine, lease, purchase) -> {"hourly", "upfront"}
self._node_memory: Dict[str, float] = {} # instance_type -> GB
if pricing_csv and os.path.exists(pricing_csv):
self._load_csv(pricing_csv)
elif pricing_csv:
raise FileNotFoundError(
"Pricing CSV not found: {}".format(pricing_csv)
)
else:
self._load_live()
def _load_live(self):
"""Load pricing from AWS bulk pricing (cached locally)."""
if _cache_is_fresh():
data = _load_cache()
else:
data = _fetch_and_cache()
for key, price in data.get("node_pricing", {}).items():
parts = key.split("|")
if len(parts) == 3:
region, itype, engine = parts
self._node_od[(region, itype, engine)] = price
for key, price in data.get("serverless_storage", {}).items():
parts = key.split("|")
if len(parts) == 2:
region, engine = parts
self._serverless_storage[(region, engine)] = price
for key, price in data.get("serverless_ecpu", {}).items():
parts = key.split("|")
if len(parts) == 2:
region, engine = parts
self._serverless_ecpu[(region, engine)] = price
for key, price in data.get("extended_support", {}).items():
parts = key.split("|")
if len(parts) == 4:
region, itype, engine, year = parts
self._extended_support[(region, itype, engine, year)] = price
for key, price_info in data.get("reserved_pricing", {}).items():
parts = key.split("|")
if len(parts) == 5:
region, itype, engine, lease, purchase = parts
self._reserved[(region, itype, engine, lease, purchase)] = price_info
self._node_memory = data.get("node_memory", {})
def _load_csv(self, path: str):
"""Load pricing from a local CSV."""
with open(path, newline="") as f:
for row in csv.DictReader(f):
family = row.get("Product Family", "")
region = row.get("Region Code", "")
engine = row.get("Cache Engine", "").lower()
term = row.get("TermType", "")
try:
price = float(row.get("PricePerUnit", "0"))
except (ValueError, TypeError):
continue
if price <= 0 or not region:
continue
if family == "Cache Instance" and term == "OnDemand":
itype = row.get("Instance Type", "")
if itype and row.get("Unit") == "Hrs":
self._node_od[(region, itype, engine)] = price
elif family == "ElastiCache Serverless":
unit = row.get("Unit", "")
operation = row.get("Operation", row.get("operation", ""))
if "Snapshot" in operation or unit == "GB-months":
continue
unit_lower = unit.lower()
if "gb-hour" in unit_lower:
self._serverless_storage[(region, engine)] = price
elif "processingunit" in unit_lower:
self._serverless_ecpu[(region, engine)] = price
# CSV loader only supports on-demand node and serverless pricing.
# Reserved pricing, Extended Support surcharges, and node memory data
# are only available from the live bulk pricing API.
if not self._reserved:
print(
"Note: CSV pricing does not include Reserved Instance or Extended Support data. "
"Use live pricing (omit --pricing-csv) for full coverage.",
file=sys.stderr,
)
def get_serverless_storage_rate(self, region: str, engine: str) -> float:
"""Get serverless storage rate in $/GB-hour."""
eng = engine.lower()
key = (region, eng)
if key in self._serverless_storage:
return self._serverless_storage[key]
raise ValueError(
"No serverless storage pricing found for region={}, engine={}. "
"Available: {}".format(region, eng, self._list_serverless_regions(eng))
)
def get_serverless_ecpu_rate(self, region: str, engine: str) -> float:
"""Get serverless ECPU rate in $ per single ECPU."""
eng = engine.lower()
key = (region, eng)
if key in self._serverless_ecpu:
return self._serverless_ecpu[key]
raise ValueError(
"No serverless ECPU pricing found for region={}, engine={}. "
"Available: {}".format(region, eng, self._list_serverless_regions(eng))
)
def get_node_hourly_rate(self, region: str, instance_type: str, engine: str = "valkey") -> float:
"""Get node on-demand hourly rate."""
eng = engine.lower()
key = (region, instance_type, eng)
if key in self._node_od:
return self._node_od[key]
raise ValueError(
"No node pricing found for region={}, instance_type={}, engine={}".format(
region, instance_type, eng
)
)
def get_node_monthly_rate(self, region: str, instance_type: str, engine: str = "valkey") -> float:
"""Get node on-demand monthly rate (hourly x 730 hours)."""
return self.get_node_hourly_rate(region, instance_type, engine) * 730
def get_extended_support_rate(self, region: str, instance_type: str, engine: str, year: str = "1") -> float:
"""Get Extended Support surcharge in $/node-hour for the given year tier.
Year tiers: '1' and '2' (same rate, from Yr1_Yr2), '3' (higher rate, from Yr3).
"""
eng = engine.lower()
key = (region, instance_type, eng, str(year))
if key in self._extended_support:
return self._extended_support[key]
raise ValueError(
"No Extended Support pricing found for region={}, instance_type={}, engine={}, year={}. "
"This may mean Extended Support pricing is not published for this combination.".format(
region, instance_type, eng, year
)
)
def has_extended_support_pricing(self) -> bool:
"""Check if any Extended Support pricing was loaded."""
return len(self._extended_support) > 0
def get_node_memory_gb(self, instance_type: str) -> float:
"""Get memory capacity in GB for a node type. Returns 0 if unknown."""
return self._node_memory.get(instance_type, 0.0)
def get_all_node_memory(self) -> dict:
"""Get the full instance_type -> memory_gb mapping."""
return dict(self._node_memory)
def _list_serverless_regions(self, engine: str) -> str:
"""List regions that have serverless pricing loaded for an engine."""
regions = sorted(set(
r for (r, e) in self._serverless_storage if e == engine
))
if len(regions) > 5:
return ", ".join(regions[:5]) + " and {} more".format(len(regions) - 5)
return ", ".join(regions) if regions else "none loaded"
# --- Reserved pricing methods ---
def get_reserved_rate(
self, region: str, instance_type: str, engine: str = "valkey",
lease_length: str = "1yr", purchase_option: str = "No Upfront",
) -> dict:
"""Get reserved pricing for a node.
Returns {"hourly": float, "upfront": float}.
- "No Upfront": upfront=0, hourly=full reserved rate
- "All Upfront": upfront=lump sum, hourly=0
- "Partial Upfront": upfront=partial lump sum, hourly=reduced rate
"""
eng = engine.lower()
key = (region, instance_type, eng, lease_length, purchase_option)
if key in self._reserved:
return self._reserved[key]
raise ValueError(
"No reserved pricing found for region={}, instance_type={}, engine={}, "
"lease={}, purchase={}".format(region, instance_type, eng, lease_length, purchase_option)
)
def get_reserved_effective_monthly(
self, region: str, instance_type: str, engine: str = "valkey",
lease_length: str = "1yr", purchase_option: str = "No Upfront",
) -> float:
"""Get effective monthly cost for a reserved node.
Amortizes upfront payment over the term and adds the recurring hourly cost.
Formula: (upfront / term_months) + (hourly * 730)
"""
rate = self.get_reserved_rate(region, instance_type, engine, lease_length, purchase_option)
term_months = 12 if lease_length == "1yr" else 36
monthly = (rate["upfront"] / term_months) + (rate["hourly"] * 730)
return round(monthly, 2)
def list_reserved_options(
self, region: str, instance_type: str, engine: str = "valkey",
) -> list:
"""List all available reserved pricing options for a node.
Returns a list of dicts with lease_length, purchase_option,
hourly, upfront, and effective_monthly.
"""
eng = engine.lower()
results = []
for (r, it, e, lease, purchase), rate in self._reserved.items():
if r == region and it == instance_type and e == eng:
term_months = 12 if lease == "1yr" else 36
effective = round(
(rate["upfront"] / term_months) + (rate["hourly"] * 730), 2
)
results.append({
"lease_length": lease,
"purchase_option": purchase,
"hourly": rate["hourly"],
"upfront": rate["upfront"],
"effective_monthly": effective,
})
results.sort(key=lambda x: x["effective_monthly"])
return results
scripts/security_audit.py
#!/usr/bin/env python3
"""
ElastiCache Security Audit
Inspects an existing cache and reports its security and operational posture.
Checks: TLS, auth mode, encryption at rest, Multi-AZ, failover, backups,
security group openness, and engine currency.
Usage:
# Audit a serverless cache
python security_audit.py --serverless my-cache --region us-east-1
# Audit a node-based replication group
python security_audit.py --replication-group my-cluster --region us-east-1
# JSON output for automation
python security_audit.py --serverless my-cache --region us-east-1 --json
Requires: boto3, AWS credentials with elasticache:Describe*,
elasticache:ListTagsForResource, and ec2:DescribeSecurityGroups
"""
import argparse
import json
import sys
# ---------------------------------------------------------------------------
# Dependency check -- fail early with an actionable message instead of a
# raw ImportError traceback.
# ---------------------------------------------------------------------------
try:
import boto3
from botocore.exceptions import ClientError, BotoCoreError
except ImportError:
print(
"Error: the 'boto3' package is required for this script.\n"
"Install it with:\n"
" pip install boto3\n"
"\n"
"You also need AWS credentials with elasticache:Describe*,\n"
"elasticache:ListTagsForResource, and ec2:DescribeSecurityGroups\n"
"permissions configured (via environment\n"
"variables, ~/.aws/credentials, or an IAM role)."
)
sys.exit(1)
def audit_serverless(client, ec2_client, cache_name):
"""Audit a serverless cache."""
try:
resp = client.describe_serverless_caches(ServerlessCacheName=cache_name)
except client.exceptions.ServerlessCacheNotFoundFault:
return {"error": f"Serverless cache '{cache_name}' not found"}
except Exception as e:
if hasattr(e, 'response') and e.response.get('Error', {}).get('Code') == 'ServerlessCacheNotFoundFault':
return {"error": f"Serverless cache '{cache_name}' not found"}
raise
caches = resp.get("ServerlessCaches", [])
if not caches:
return {"error": f"Serverless cache '{cache_name}' not found"}
cache = caches[0]
findings = []
info = {
"cache_type": "serverless",
"cache_name": cache_name,
"engine": cache.get("Engine", "unknown"),
"engine_version": cache.get("MajorEngineVersion", "unknown"),
"status": cache.get("Status", "unknown"),
"arn": cache.get("ARN", "unknown"),
}
# TLS: always on for serverless
info["tls_enabled"] = True
findings.append({"check": "TLS in-transit encryption", "status": "PASS", "detail": "Always enabled on serverless"})
# Auth: serverless requires RBAC
user_group = cache.get("UserGroupId")
if user_group:
findings.append({"check": "Authentication", "status": "PASS", "detail": f"RBAC user group: {user_group}"})
info["auth_mode"] = "RBAC"
info["user_group"] = user_group
else:
findings.append({"check": "Authentication", "status": "WARN", "detail": "No user group attached — using default user only"})
info["auth_mode"] = "default-user-only"
# Encryption at rest: check KMS
kms_key = cache.get("KmsKeyId")
if kms_key:
findings.append({"check": "Encryption at rest", "status": "PASS", "detail": f"KMS key: {kms_key}"})
info["encryption_at_rest"] = True
info["kms_key"] = kms_key
else:
findings.append({"check": "Encryption at rest", "status": "PASS", "detail": "Always enabled on serverless (using AWS-managed key)"})
info["encryption_at_rest"] = True
# Multi-AZ: always on for serverless
info["multi_az"] = True
findings.append({"check": "Multi-AZ", "status": "PASS", "detail": "Always enabled on serverless"})
# Backups: check SnapshotRetentionLimit
retention = cache.get("SnapshotRetentionLimit", 0)
info["snapshot_retention_days"] = retention
if retention and retention > 0:
snapshot_time = cache.get("DailySnapshotTime", "not set")
info["daily_backups"] = True
findings.append({"check": "Automatic backups", "status": "PASS", "detail": f"Retention: {retention} days, daily snapshot time: {snapshot_time}"})
else:
info["daily_backups"] = False
findings.append({"check": "Automatic backups", "status": "FAIL", "detail": "No automatic backups configured, SnapshotRetentionLimit is 0 or not set"})
# Usage limits (cost control)
limits = cache.get("CacheUsageLimits", {})
data_limit = limits.get("DataStorage", {})
ecpu_limit = limits.get("ECPUPerSecond", {})
if data_limit.get("Maximum"):
info["max_data_gb"] = data_limit["Maximum"]
findings.append({"check": "Data storage limit", "status": "PASS", "detail": f"{data_limit['Maximum']} {data_limit.get('Unit', 'GB')} max"})
else:
findings.append({"check": "Data storage limit", "status": "WARN", "detail": "No data storage limit set — costs could grow unbounded"})
if ecpu_limit.get("Maximum"):
info["max_ecpu_per_sec"] = ecpu_limit["Maximum"]
findings.append({"check": "ECPU limit", "status": "PASS", "detail": f"{ecpu_limit['Maximum']} ECPUs/sec max"})
else:
findings.append({"check": "ECPU limit", "status": "WARN", "detail": "No ECPU limit set — costs could grow unbounded"})
# Security groups
sg_ids = cache.get("SecurityGroupIds", [])
sg_findings = _audit_security_groups(ec2_client, sg_ids)
findings.extend(sg_findings)
info["security_groups"] = sg_ids
# Tags
try:
tag_resp = client.list_tags_for_resource(ResourceName=cache["ARN"])
tags = {t["Key"]: t["Value"] for t in tag_resp.get("TagList", [])}
except (ClientError, BotoCoreError):
tags = {}
_audit_tags(findings, tags)
info["tags"] = tags
return {"info": info, "findings": findings}
def audit_replication_group(client, ec2_client, rg_id):
"""Audit a node-based replication group."""
try:
resp = client.describe_replication_groups(ReplicationGroupId=rg_id)
except client.exceptions.ReplicationGroupNotFoundFault:
return {"error": f"Replication group '{rg_id}' not found"}
except Exception as e:
if hasattr(e, 'response') and e.response.get('Error', {}).get('Code') == 'ReplicationGroupNotFoundFault':
return {"error": f"Replication group '{rg_id}' not found"}
raise
groups = resp.get("ReplicationGroups", [])
if not groups:
return {"error": f"Replication group '{rg_id}' not found"}
rg = groups[0]
findings = []
info = {
"cache_type": "node-based",
"replication_group_id": rg_id,
"engine": "unknown",
"status": rg.get("Status", "unknown"),
"arn": rg.get("ARN", "unknown"),
"cluster_enabled": rg.get("ClusterEnabled", False),
"num_node_groups": len(rg.get("NodeGroups", [])),
}
# Engine version from member clusters
member_clusters = rg.get("MemberClusters", [])
if member_clusters:
try:
cluster_resp = client.describe_cache_clusters(CacheClusterId=member_clusters[0])
cluster = cluster_resp["CacheClusters"][0]
info["engine"] = cluster.get("Engine", "unknown")
info["engine_version"] = cluster.get("EngineVersion", "unknown")
info["node_type"] = cluster.get("CacheNodeType", "unknown")
except (ClientError, BotoCoreError):
info["engine_version"] = "unknown"
info["node_type"] = "unknown"
# TLS
tls = rg.get("TransitEncryptionEnabled", False)
info["tls_enabled"] = tls
if tls:
findings.append({"check": "TLS in-transit encryption", "status": "PASS", "detail": "Enabled"})
else:
findings.append({"check": "TLS in-transit encryption", "status": "FAIL", "detail": "Not enabled — data in transit is unencrypted"})
# Encryption at rest
at_rest = rg.get("AtRestEncryptionEnabled", False)
info["encryption_at_rest"] = at_rest
if at_rest:
kms = rg.get("KmsKeyId", "AWS-managed")
findings.append({"check": "Encryption at rest", "status": "PASS", "detail": f"Enabled (key: {kms})"})
info["kms_key"] = kms
else:
findings.append({"check": "Encryption at rest", "status": "FAIL", "detail": "Not enabled — data at rest is unencrypted"})
# Auth
auth_token = rg.get("AuthTokenEnabled", False)
user_group_ids = rg.get("UserGroupIds", [])
if user_group_ids:
info["auth_mode"] = "RBAC"
info["user_groups"] = user_group_ids
findings.append({"check": "Authentication", "status": "PASS", "detail": f"RBAC user groups: {', '.join(user_group_ids)}"})
elif auth_token:
info["auth_mode"] = "AUTH-token"
findings.append({"check": "Authentication", "status": "WARN", "detail": "Using legacy AUTH token — consider migrating to RBAC"})
else:
info["auth_mode"] = "none"
findings.append({"check": "Authentication", "status": "FAIL", "detail": "No authentication configured — any VPC client can access data"})
# Multi-AZ
multi_az = rg.get("MultiAZ", "disabled")
info["multi_az"] = multi_az == "enabled"
if multi_az == "enabled":
findings.append({"check": "Multi-AZ", "status": "PASS", "detail": "Enabled"})
else:
findings.append({"check": "Multi-AZ", "status": "WARN", "detail": "Not enabled — single-AZ failure risk"})
# Automatic failover
failover = rg.get("AutomaticFailover", "disabled")
info["automatic_failover"] = failover == "enabled"
if failover == "enabled":
findings.append({"check": "Automatic failover", "status": "PASS", "detail": "Enabled"})
else:
findings.append({"check": "Automatic failover", "status": "WARN", "detail": "Not enabled — manual intervention needed on primary failure"})
# Backups
retention = rg.get("SnapshotRetentionLimit", 0)
info["snapshot_retention_days"] = retention
if retention > 0:
findings.append({"check": "Automatic backups", "status": "PASS", "detail": f"Retention: {retention} days"})
else:
findings.append({"check": "Automatic backups", "status": "FAIL", "detail": "No automatic backups — data loss risk on failure"})
# Security groups from node groups
sg_ids = set()
for ng in rg.get("NodeGroups", []):
for member in ng.get("NodeGroupMembers", []):
if "CacheClusterId" in member:
try:
cc = client.describe_cache_clusters(
CacheClusterId=member["CacheClusterId"],
ShowCacheNodeInfo=True
)["CacheClusters"][0]
for sg in cc.get("SecurityGroups", []):
sg_ids.add(sg["SecurityGroupId"])
except (ClientError, BotoCoreError):
pass
sg_ids_list = list(sg_ids)
sg_findings = _audit_security_groups(ec2_client, sg_ids_list)
findings.extend(sg_findings)
info["security_groups"] = sg_ids_list
# Tags
try:
tag_resp = client.list_tags_for_resource(ResourceName=info["arn"])
tags = {t["Key"]: t["Value"] for t in tag_resp.get("TagList", [])}
except (ClientError, BotoCoreError):
tags = {}
_audit_tags(findings, tags)
info["tags"] = tags
return {"info": info, "findings": findings}
def _audit_security_groups(ec2_client, sg_ids):
"""Check security groups for overly permissive rules."""
findings = []
if not sg_ids:
findings.append({"check": "Security groups", "status": "WARN", "detail": "No security groups found"})
return findings
try:
resp = ec2_client.describe_security_groups(GroupIds=sg_ids)
except Exception as e:
findings.append({"check": "Security groups", "status": "ERROR", "detail": f"Could not inspect: {e}"})
return findings
for sg in resp.get("SecurityGroups", []):
sg_id = sg["GroupId"]
for rule in sg.get("IpPermissions", []):
from_port = rule.get("FromPort", 0)
to_port = rule.get("ToPort", 0)
for ip_range in rule.get("IpRanges", []):
cidr = ip_range.get("CidrIp", "")
if cidr == "0.0.0.0/0":
findings.append({
"check": f"Security group {sg_id}",
"status": "WARN",
"detail": f"Inbound rule allows 0.0.0.0/0 on ports {from_port}-{to_port} -- no public endpoint exists, but 0.0.0.0/0 violates least-privilege"
})
elif cidr.endswith("/0") or cidr.endswith("/8"):
findings.append({
"check": f"Security group {sg_id}",
"status": "WARN",
"detail": f"Inbound rule allows {cidr} on ports {from_port}-{to_port} -- very broad CIDR"
})
for ip_range in rule.get("Ipv6Ranges", []):
cidr = ip_range.get("CidrIpv6", "")
if cidr == "::/0":
findings.append({
"check": f"Security group {sg_id}",
"status": "WARN",
"detail": f"Inbound rule allows ::/0 (IPv6) on ports {from_port}-{to_port} -- no public endpoint exists, but ::/0 violates least-privilege"
})
if not any(f["check"].startswith("Security group") for f in findings):
findings.append({"check": "Security groups", "status": "PASS", "detail": f"No overly permissive rules in {', '.join(sg_ids)}"})
return findings
def _audit_tags(findings, tags):
"""Check for recommended tags."""
recommended = ["Environment", "managed_by", "Application", "Owner"]
missing = [t for t in recommended if t not in tags]
if not missing:
findings.append({"check": "Resource tags", "status": "PASS", "detail": f"All recommended tags present"})
else:
findings.append({"check": "Resource tags", "status": "WARN", "detail": f"Missing recommended tags: {', '.join(missing)}"})
def format_report(result):
"""Format audit results as a human-readable report."""
if "error" in result:
return f"ERROR: {result['error']}"
info = result["info"]
findings = result["findings"]
lines = []
lines.append("=" * 72)
lines.append("ElastiCache Security Audit Report")
lines.append("=" * 72)
lines.append("")
lines.append(f" Cache type: {info['cache_type']}")
if info["cache_type"] == "serverless":
lines.append(f" Cache name: {info['cache_name']}")
else:
lines.append(f" Replication group: {info['replication_group_id']}")
lines.append(f" Node type: {info.get('node_type', 'unknown')}")
lines.append(f" Cluster mode: {'enabled' if info.get('cluster_enabled') else 'disabled'}")
lines.append(f" Node groups: {info.get('num_node_groups', 'unknown')}")
lines.append(f" Engine: {info['engine']} {info.get('engine_version', '')}")
lines.append(f" Status: {info['status']}")
lines.append(f" Auth mode: {info.get('auth_mode', 'unknown')}")
lines.append("")
fail_count = sum(1 for f in findings if f["status"] == "FAIL")
warn_count = sum(1 for f in findings if f["status"] == "WARN")
pass_count = sum(1 for f in findings if f["status"] == "PASS")
lines.append(f" Summary: {pass_count} passed, {warn_count} warnings, {fail_count} failures")
lines.append("")
lines.append("-" * 72)
status_order = {"FAIL": 0, "WARN": 1, "ERROR": 2, "INFO": 3, "PASS": 4}
sorted_findings = sorted(findings, key=lambda f: status_order.get(f["status"], 5))
for f in sorted_findings:
icon = {"PASS": "[OK] ", "FAIL": "[FAIL]", "WARN": "[WARN]", "INFO": "[INFO]", "ERROR": "[ERR] "}
lines.append(f" {icon.get(f['status'], '[?] ')} {f['check']}")
lines.append(f" {f['detail']}")
lines.append("")
if fail_count > 0:
lines.append("=" * 72)
lines.append("RECOMMENDED ACTIONS")
lines.append("=" * 72)
lines.append("")
for f in sorted_findings:
if f["status"] == "FAIL":
check = f["check"]
if "TLS" in check:
lines.append(" - Enable in-transit encryption. For existing node-based clusters, use a")
lines.append(" two-step process: first set transit-encryption-mode to 'preferred',")
lines.append(" then after migrating all clients to TLS, set it to 'required'.")
elif "Encryption at rest" in check:
lines.append(" - Enable at-rest encryption. For existing clusters, this requires")
lines.append(" creating a new cluster with AtRestEncryptionEnabled=true and migrating.")
elif "Authentication" in check:
lines.append(" - Configure RBAC: create users, a user group, and attach it to the cache.")
lines.append(" See references/setup/iam-policies.md for IAM auth patterns.")
elif "backups" in check.lower():
lines.append(" - Enable automatic backups: modify the replication group with")
lines.append(" SnapshotRetentionLimit > 0 (recommended: 7 days).")
elif "0.0.0.0/0" in f["detail"]:
lines.append(f" - Restrict {check}: remove the 0.0.0.0/0 inbound rule and scope to")
lines.append(" application security groups only.")
lines.append("")
lines.append(" Note: This is an informational review, not a security certification.")
lines.append(" Consult your security team before production deployment.")
lines.append("")
return "\n".join(lines)
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="ElastiCache Security Audit")
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("--serverless", metavar="CACHE_NAME", help="Audit a serverless cache")
group.add_argument("--replication-group", metavar="RG_ID", help="Audit a node-based replication group")
parser.add_argument("--region", default="us-east-1", help="AWS region")
parser.add_argument("--profile", default=None, help="AWS profile")
parser.add_argument("--json", action="store_true", help="Output as JSON")
args = parser.parse_args()
try:
session = boto3.Session(profile_name=args.profile, region_name=args.region)
ec_client = session.client("elasticache")
ec2_client = session.client("ec2")
except Exception as e:
print(f"ERROR: Failed to create AWS session: {e}", file=sys.stderr)
sys.exit(1)
try:
if args.serverless:
result = audit_serverless(ec_client, ec2_client, args.serverless)
else:
result = audit_replication_group(ec_client, ec2_client, args.replication_group)
except Exception as e:
error_code = ""
if hasattr(e, "response"):
error_code = e.response.get("Error", {}).get("Code", "")
if error_code in ("AccessDeniedException", "AccessDenied"):
print(f"ERROR: Access denied. Ensure your IAM role/user has elasticache:Describe*, "
f"elasticache:ListTagsForResource, and ec2:DescribeSecurityGroups permissions.")
else:
print(f"ERROR: {e}")
sys.exit(1)
if args.json:
print(json.dumps(result, indent=2, default=str))
else:
print(format_report(result))
if "error" in result:
sys.exit(1)
fail_count = sum(1 for f in result.get("findings", []) if f["status"] == "FAIL")
sys.exit(1 if fail_count > 0 else 0)
scripts/serverless_estimator.py
#!/usr/bin/env python3
"""
ElastiCache Serverless Cost Estimator
Provides approximate cost estimates for what ElastiCache provisioned
clusters would cost on ElastiCache Serverless, or for sizing a new
serverless cache. These are estimates only. Actual serverless costs
depend on real-time workload characteristics and may differ from the
values produced by this tool. For accurate billing data, use CloudWatch
ElastiCacheProcessingUnits and BytesUsedForCache metrics on a running
serverless cache.
Two estimation modes:
Simple: Provide avg memory + daily command count per cluster.
Each command is estimated at MAX(1, avg_payload_kb) ECPUs.
This is a quick approximation for initial cost comparison.
Detailed: Also provide per-command stats (from INFO commandstats).
Classifies commands as simple (1 ECPU) or complex
(estimated as MAX(calls, usec/3) ECPUs based on the
assumption that 1 ECPU corresponds to approximately
3 microseconds of vCPU time, derived from the AWS pricing
blog example). Takes the higher of the CPU and network
components. More accurate than simple mode but still an
approximation.
IMPORTANT: The ECPU formulas used in this tool are heuristic approximations
based on publicly available AWS documentation and pricing examples. AWS does
not publish exact ECPU calculation formulas. The 3-microsecond-per-ECPU
assumption for complex commands is derived from the AWS pricing blog and may
not apply uniformly to all command types. See command_classifier.py for
details on command classification.
Pricing is fetched live from the AWS Bulk Pricing API and reflects current
published rates. See https://aws.amazon.com/elasticache/pricing/
Note: INFO commandstats returns cumulative counts that keep incrementing
from server start. If you include an "uptime_days" column in the
commandstats CSV, the estimator divides calls and usec by that value to
normalize to a daily rate. The --endpoint mode handles this automatically
by taking two snapshots and computing the delta.
Usage:
python3 serverless_estimator.py --input clusters.csv
python3 serverless_estimator.py --input clusters.csv --commandstats stats.csv
python3 serverless_estimator.py --endpoint my-cluster.abc123.use1.cache.amazonaws.com:6379
python3 serverless_estimator.py --endpoint my-cluster.abc123.use1.cache.amazonaws.com --instance-type cache.r7g.large
python3 serverless_estimator.py --input clusters.csv --pricing pricing.csv
python3 serverless_estimator.py --input clusters.csv --output estimate.csv --json
"""
import argparse
import csv
import json
import math
import os
import re
import sys
from typing import Dict, List, Optional
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from command_classifier import estimate_ecpus_from_commandstats
from pricing import PricingLoader
HOURS_PER_MONTH = 730
MIN_STORAGE_GB = {
"valkey": 0.1, # 100 MB
"redis": 1.0, # 1 GB
"memcached": 1.0, # 1 GB
}
# Number of sample points for sinusoidal workload modeling.
# 24 = one sample per hour over a daily cycle.
_SIN_SAMPLES = 24
def _sine_wave_samples(min_val, max_val, n=_SIN_SAMPLES):
"""Generate n hourly samples of a sine wave oscillating between min and max.
Models a daily traffic pattern:
value(t) = min + (max - min) * (1 + sin(2*pi*t/n)) / 2
Returns a list of n float values representing the workload at each hour.
"""
amplitude = (max_val - min_val) / 2.0
midpoint = (max_val + min_val) / 2.0
return [midpoint + amplitude * math.sin(2 * math.pi * i / n) for i in range(n)]
def compute_burst_multiplier(avg_val, min_val=None, max_val=None, peak_to_avg=None):
"""Compute a cost multiplier for bursty (sinusoidal) workloads.
Serverless bills per-hour. A workload that swings between min and peak
follows a sine curve over the day. This function samples the curve at
24 hourly points and returns the ratio of the sine-wave average to the
customer-provided average.
Why this matters: if the customer says "my average is X" but their actual
traffic swings from min to peak, the true average of the sine curve
((min+max)/2) may differ from X. The multiplier corrects for that.
The caller can specify the burst in two ways:
1. min_val + max_val: explicit floor and ceiling of the daily cycle
2. peak_to_avg: ratio of peak to average (e.g., 3.0 means peak is 3x avg)
Returns 1.0 (no adjustment) if no burst parameters are provided or if
the workload is flat.
"""
if peak_to_avg is not None and peak_to_avg > 1.0:
max_val = avg_val * peak_to_avg
min_val = max(0, avg_val * (2 - peak_to_avg))
if min_val is None or max_val is None:
return 1.0
if max_val <= min_val or max_val <= 0:
return 1.0
if avg_val <= 0:
return 1.0
samples = _sine_wave_samples(min_val, max_val)
sine_avg = sum(samples) / len(samples)
return sine_avg / avg_val
def load_clusters(path: str) -> List[Dict]:
"""Load cluster data from CSV.
Required columns:
cluster_name, instance_type, region, engine, node_count,
avg_memory_gb, daily_commands
Optional columns:
primary_nodes -- defaults to max(1, node_count // 2). This is a
rough heuristic; for CMD clusters (1 primary +
N replicas) or CME clusters (1 primary per shard),
provide the actual value via --primary-nodes or
the primary_nodes CSV column for accurate results.
current_monthly_cost -- if known; otherwise computed from list price
avg_payload_kb -- average payload size in KB (default 1.0).
ECPUs scale linearly with payload: 3.2 KB = 3.2 ECPUs per op.
peak_commands -- peak daily commands (for bursty workload modeling)
min_commands -- minimum daily commands (for bursty workload modeling)
peak_to_avg_ratio -- ratio of peak to average commands (alternative to min/max)
peak_memory_gb -- peak memory in GB (for bursty storage modeling)
min_memory_gb -- minimum memory in GB (for bursty storage modeling)
"""
clusters = []
with open(path, newline="") as f:
reader = csv.DictReader(f)
for row in reader:
node_count = int(row.get("node_count", 1))
primary_nodes = int(row.get("primary_nodes", 0))
if primary_nodes == 0:
# Heuristic: assumes half the nodes are primaries. This is only
# accurate for 1-replica setups; actual primary count depends on
# cluster topology (CMD: 1 primary + N replicas, CME: 1 primary
# per shard). Provide the primary_nodes column for accuracy.
primary_nodes = max(1, node_count // 2)
clusters.append({
"cluster_name": row["cluster_name"].strip(),
"instance_type": row.get("instance_type", "").strip(),
"region": row.get("region", "us-east-1").strip(),
"engine": row.get("engine", "valkey").strip().lower(),
"node_count": node_count,
"primary_nodes": primary_nodes,
"avg_memory_gb": float(row.get("avg_memory_gb", 0)),
"daily_commands": float(row.get("daily_commands", 0)),
"current_monthly_cost": float(row.get("current_monthly_cost", 0) or 0),
"avg_payload_kb": float(row.get("avg_payload_kb", 1.0) or 1.0),
"peak_commands": float(row.get("peak_commands", 0) or 0),
"min_commands": float(row.get("min_commands", 0) or 0),
"peak_to_avg_ratio": float(row.get("peak_to_avg_ratio", 0) or 0),
"peak_memory_gb": float(row.get("peak_memory_gb", 0) or 0),
"min_memory_gb": float(row.get("min_memory_gb", 0) or 0),
})
return clusters
def parse_commandstats_file(path: str) -> Dict[str, dict]:
"""Parse a commandstats CSV.
Expected columns: cluster_name, command, calls, usec
Optional columns:
uptime_days - days since server restart. When present, calls and
usec are divided by this value to normalize cumulative
INFO commandstats output to a daily rate. Without it
the raw cumulative values are used as-is.
Returns a dict keyed by cluster_name. Each value is a dict of
{command: {"calls": int, "usec": int}}. A special key
"_normalized" (bool) indicates whether uptime_days normalization
was applied for that cluster.
"""
result = {}
with open(path, newline="") as f:
for row in csv.DictReader(f):
cluster = row["cluster_name"].strip()
cmd = row["command"].strip().lower()
calls = int(row.get("calls", 0))
usec = int(row.get("usec", 0))
uptime_raw = row.get("uptime_days", "").strip()
uptime_days = float(uptime_raw) if uptime_raw else None
if uptime_days and uptime_days > 0:
calls = int(calls / uptime_days)
usec = int(usec / uptime_days)
normalized = True
else:
normalized = False
if cluster not in result:
result[cluster] = {"_normalized": normalized} # type: ignore[dict-item]
result[cluster][cmd] = {"calls": calls, "usec": usec} # type: ignore[assignment]
# If any row for this cluster has uptime_days, mark as normalized
if normalized:
result[cluster]["_normalized"] = True
return result
def parse_commandstats_info(text: str) -> dict:
"""Parse raw INFO commandstats output from valkey-cli (or redis-cli).
Example line: cmdstat_get:calls=1000,usec=1500,usec_per_call=1.50,rejected_calls=0,failed_calls=0
"""
result = {}
for line in text.strip().splitlines():
line = line.strip()
if not line or line.startswith("#"):
continue
match = re.match(r"cmdstat_(\w+):calls=(\d+),usec=(\d+)", line)
if match:
cmd = match.group(1).lower()
result[cmd] = {
"calls": int(match.group(2)),
"usec": int(match.group(3)),
}
return result
def estimate_cluster(
cluster: Dict,
pricing: PricingLoader,
commandstats: Optional[dict] = None,
) -> Dict:
"""Estimate serverless cost for one cluster."""
region = cluster["region"]
engine = cluster["engine"]
node_count = cluster["node_count"]
instance_type = cluster["instance_type"]
# --- Current provisioned cost ---
if cluster["current_monthly_cost"] > 0:
current_cost = cluster["current_monthly_cost"]
cost_source = "provided"
elif instance_type:
try:
hourly = pricing.get_node_hourly_rate(region, instance_type, engine)
current_cost = hourly * HOURS_PER_MONTH * node_count
cost_source = "list_price"
except ValueError:
current_cost = 0
cost_source = "unknown"
else:
current_cost = 0
cost_source = "unknown"
# --- Serverless storage ---
avg_memory_gb = cluster["avg_memory_gb"]
min_gb = MIN_STORAGE_GB.get(engine, 1.0)
billing_memory_gb = max(avg_memory_gb, min_gb)
used_minimum = avg_memory_gb < min_gb
# Bursty storage: if peak/min memory provided, adjust
storage_burst = compute_burst_multiplier(
avg_memory_gb,
min_val=cluster.get("min_memory_gb") or None,
max_val=cluster.get("peak_memory_gb") or None,
)
adjusted_memory_gb = billing_memory_gb * storage_burst
monthly_gb_hours = adjusted_memory_gb * HOURS_PER_MONTH
storage_rate = pricing.get_serverless_storage_rate(region, engine)
storage_cost = monthly_gb_hours * storage_rate
# --- Serverless ECPUs ---
ecpu_rate = pricing.get_serverless_ecpu_rate(region, engine)
# Bursty commands: if peak/min commands or peak_to_avg_ratio provided, adjust
ecpu_burst = compute_burst_multiplier(
cluster["daily_commands"],
min_val=cluster.get("min_commands") or None,
max_val=cluster.get("peak_commands") or None,
peak_to_avg=cluster.get("peak_to_avg_ratio") or None,
)
if commandstats and engine == "memcached":
print("WARNING: commandstats mode uses Redis/Valkey INFO format, which is "
"incompatible with Memcached. ECPU estimates for Memcached cluster '{}' "
"will be inaccurate. Falling back to simple estimation.".format(
cluster["cluster_name"]),
file=sys.stderr)
commandstats = None
if commandstats:
# _normalized is a metadata flag, not a command; exclude before estimation
cs_normalized = commandstats.get("_normalized", False)
cs_data = {k: v for k, v in commandstats.items() if k != "_normalized"}
ecpu_result = estimate_ecpus_from_commandstats(cs_data)
daily_cpu_ecpus = ecpu_result["total_ecpus"]
# Network component: each command costs MAX(1, avg_payload_kb) ECPUs.
# This approximates the "1 ECPU per KB transferred" pricing rule.
# See: https://aws.amazon.com/elasticache/pricing/
avg_payload_kb = cluster.get("avg_payload_kb", 1.0) or 1.0
ecpu_per_request = max(1.0, avg_payload_kb)
daily_net_ecpus = cluster["daily_commands"] * ecpu_per_request
# Serverless charges the higher of vCPU time or data transferred.
# See: https://aws.amazon.com/blogs/database/unlock-on-demand-cost-optimized-performance-with-amazon-elasticache-serverless/
daily_ecpus = max(daily_cpu_ecpus, daily_net_ecpus)
monthly_ecpus = daily_ecpus * 30 * ecpu_burst
ecpu_mode = "detailed"
fixed_ecpus = ecpu_result["fixed_ecpus"] * 30
nonfixed_ecpus = ecpu_result["nonfixed_ecpus"] * 30
internal_calls = ecpu_result["internal_calls"] * 30
else:
avg_payload_kb = cluster.get("avg_payload_kb", 1.0) or 1.0
ecpu_per_request = max(1.0, avg_payload_kb)
monthly_ecpus = cluster["daily_commands"] * 30 * ecpu_per_request * ecpu_burst
ecpu_mode = "simple"
fixed_ecpus = None
nonfixed_ecpus = None
internal_calls = None
ecpu_cost = monthly_ecpus * ecpu_rate
total_serverless = storage_cost + ecpu_cost
# --- Comparison ---
diff = total_serverless - current_cost if current_cost > 0 else None
diff_pct = (diff / current_cost * 100) if diff is not None and current_cost > 0 else None
# --- Notes ---
notes = []
if used_minimum:
notes.append("Min storage applied ({} GB)".format(min_gb))
if cost_source == "list_price":
notes.append("Current cost: on-demand list price")
if cost_source == "unknown":
notes.append("Current cost unknown - provide current_monthly_cost or instance_type")
if ecpu_mode == "simple":
if avg_payload_kb > 1:
notes.append("ECPU: {} ECPUs per request ({} KB avg payload)".format(
round(ecpu_per_request, 1), round(avg_payload_kb, 1)))
else:
notes.append("ECPU: 1 cmd = 1 ECPU (provide avg_payload_kb for better accuracy)")
if commandstats and not cs_normalized:
notes.append("Commandstats not normalized by uptime; provide uptime_days column for accuracy")
if ecpu_burst > 1.0:
notes.append("Burst adjustment applied: {:.2f}x (peak/avg commands)".format(ecpu_burst))
if storage_burst > 1.0:
notes.append("Burst adjustment applied: {:.2f}x (peak/avg memory)".format(storage_burst))
# Serverless compatibility warnings
if engine in ("redis", "valkey"):
notes.append("Serverless requires cluster-mode-enabled clients and TLS; Global Data Store and Data Tiering are not supported")
return {
"cluster_name": cluster["cluster_name"],
"region": region,
"engine": engine,
"instance_type": instance_type,
"node_count": node_count,
"primary_nodes": cluster["primary_nodes"],
"current_monthly_cost": round(current_cost, 2),
"cost_source": cost_source,
"avg_memory_gb": round(avg_memory_gb, 4),
"billing_memory_gb": round(billing_memory_gb, 4),
"monthly_gb_hours": round(monthly_gb_hours, 2),
"storage_rate_per_gb_hour": storage_rate,
"storage_cost": round(storage_cost, 2),
"monthly_ecpus": round(monthly_ecpus),
"ecpu_rate_per_million": round(ecpu_rate * 1_000_000, 4),
"ecpu_cost": round(ecpu_cost, 2),
"ecpu_mode": ecpu_mode,
"fixed_ecpus": round(fixed_ecpus) if fixed_ecpus is not None else None,
"nonfixed_ecpus": round(nonfixed_ecpus) if nonfixed_ecpus is not None else None,
"internal_calls_excluded": round(internal_calls) if internal_calls is not None else None,
"serverless_total": round(total_serverless, 2),
"diff": round(diff, 2) if diff is not None else None,
"diff_pct": round(diff_pct, 1) if diff_pct is not None else None,
"notes": "; ".join(notes),
}
def write_csv(results: List[Dict], path: str):
"""Write results to CSV."""
if not results:
return
fields = [
"cluster_name", "region", "engine", "instance_type",
"node_count", "primary_nodes",
"current_monthly_cost", "cost_source",
"avg_memory_gb", "billing_memory_gb",
"monthly_gb_hours", "storage_cost",
"monthly_ecpus", "ecpu_cost", "ecpu_mode",
"serverless_total", "diff", "diff_pct", "notes",
]
with open(path, "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=fields, extrasaction="ignore")
writer.writeheader()
writer.writerows(results)
print("Output written to: {}".format(path))
def print_summary(results: List[Dict]):
"""Print summary to console."""
total_current = sum(r["current_monthly_cost"] for r in results)
total_sl = sum(r["serverless_total"] for r in results)
total_stor = sum(r["storage_cost"] for r in results)
total_ecpu = sum(r["ecpu_cost"] for r in results)
cheaper = sum(1 for r in results if r["diff"] is not None and r["diff"] < 0)
more_exp = sum(1 for r in results if r["diff"] is not None and r["diff"] > 0)
print()
print("=" * 65)
print(" ElastiCache Serverless Cost Estimate")
print("=" * 65)
print(" Clusters analyzed: {}".format(len(results)))
print(" Total nodes: {}".format(sum(r["node_count"] for r in results)))
print()
print(" Current provisioned cost: ${:>12,.2f} /month".format(total_current))
print(" Estimated serverless: ${:>12,.2f} /month".format(total_sl))
print(" Data storage: ${:>12,.2f}".format(total_stor))
print(" ECPUs: ${:>12,.2f}".format(total_ecpu))
print()
if total_current > 0:
savings = total_current - total_sl
pct = savings / total_current * 100
print(" Estimated savings: ${:>12,.2f} ({:.1f}%)".format(savings, pct))
print(" Clusters cheaper on SL: {}".format(cheaper))
print(" Clusters more expensive: {}".format(more_exp))
print("=" * 65)
print()
print(" {:<30s} {:>10s} {:>10s} {:>8s}".format(
"Cluster", "Current", "Serverless", "Change"))
print(" {} {} {} {}".format("-" * 30, "-" * 10, "-" * 10, "-" * 8))
for r in sorted(results, key=lambda x: x.get("diff") or 0):
name = r["cluster_name"][:30]
cur = "${:,.0f}".format(r["current_monthly_cost"])
sl = "${:,.0f}".format(r["serverless_total"])
chg = "{:+.0f}%".format(r["diff_pct"]) if r["diff_pct"] is not None else "N/A"
print(" {:<30s} {:>10s} {:>10s} {:>8s}".format(name, cur, sl, chg))
print()
print(" ECPU estimates assume 1 KB avg payload unless avg_payload_kb is provided.")
print(" ECPUs scale linearly with payload size (e.g. 3 KB = 3x ECPUs).")
print()
print(" IMPORTANT: These are approximate estimates. Actual serverless costs depend")
print(" on real-time workload characteristics and may differ. For accurate billing,")
print(" use CloudWatch metrics on a running serverless cache.")
print(" See https://aws.amazon.com/elasticache/pricing/")
print()
def collect_from_endpoint(endpoint, cluster_name=None, instance_type="",
region="us-east-1", engine="valkey", node_count=2,
avg_payload_kb=1.0, use_tls=True, sample_seconds=60):
"""Connect to a live Valkey/Redis endpoint and collect metrics.
Takes two INFO snapshots separated by sample_seconds and computes
deltas. This gives the actual traffic rate during the sample window,
which is more accurate than dividing cumulative totals by uptime.
Returns (cluster_dict, commandstats_dict) ready for estimate_cluster().
"""
try:
import valkey as client_lib
except ImportError:
print("Error: the 'valkey' package is required for --endpoint mode.")
print("Install with: pip install valkey")
sys.exit(1)
import time
# Parse host:port
if ":" in endpoint and not endpoint.startswith("["):
host, port_str = endpoint.rsplit(":", 1)
port = int(port_str)
else:
host = endpoint
port = 6379
if not cluster_name:
cluster_name = host.split(".")[0]
print("Connecting to {}:{}{}...".format(host, port, " (TLS)" if use_tls else ""))
r = client_lib.Redis(host=host, port=port, ssl=use_tls,
ssl_cert_reqs=None, decode_responses=True,
socket_connect_timeout=10)
print(" PING: {}".format(r.ping()))
# Collect memory and replication (point-in-time, no delta needed)
mem_info = r.info("memory")
repl_info = r.info("replication")
dataset_bytes = mem_info.get("used_memory_dataset", mem_info.get("used_memory", 0))
dataset_gb = dataset_bytes / (1024 ** 3)
role = repl_info.get("role", "unknown")
print(" Role: {}".format(role))
print(" Memory: {:.4f} GB ({:,.0f} bytes)".format(dataset_gb, dataset_bytes))
# Snapshot 1
print(" Taking snapshot 1...")
cs1 = r.info("commandstats")
stats1 = r.info("stats")
t1 = time.time()
# Wait
print(" Sampling for {} seconds...".format(sample_seconds))
time.sleep(sample_seconds)
# Snapshot 2
print(" Taking snapshot 2...")
cs2 = r.info("commandstats")
stats2 = r.info("stats")
t2 = time.time()
elapsed = t2 - t1
elapsed_days = elapsed / 86400
# Compute commandstats deltas
commandstats = {}
total_delta_calls = 0
for key in cs2:
cmd = key.replace("cmdstat_", "")
calls_delta = cs2[key].get("calls", 0) - cs1.get(key, {}).get("calls", 0)
usec_delta = cs2[key].get("usec", 0) - cs1.get(key, {}).get("usec", 0)
if calls_delta > 0:
# Scale to daily rate
daily_calls = int(calls_delta / elapsed_days)
daily_usec = int(usec_delta / elapsed_days)
commandstats[cmd] = {"calls": daily_calls, "usec": daily_usec}
total_delta_calls += calls_delta
commandstats["_normalized"] = True # type: ignore[assignment]
daily_commands = total_delta_calls / elapsed_days if elapsed_days > 0 else 0
# Also get total commands delta for simple mode
total_cmds_delta = stats2.get("total_commands_processed", 0) - stats1.get("total_commands_processed", 0)
daily_commands_total = total_cmds_delta / elapsed_days if elapsed_days > 0 else 0
print(" Sample window: {:.0f} seconds".format(elapsed))
print(" Commands in window: {:,}".format(total_cmds_delta))
print(" Daily rate: {:,.0f} commands/day".format(daily_commands_total))
top_cmds = sorted(commandstats.items(),
key=lambda x: x[1].get("calls", 0) if isinstance(x[1], dict) else 0,
reverse=True)[:5]
print(" Top commands: {}".format(
", ".join("{} ({:,}/day)".format(k, v["calls"])
for k, v in top_cmds if isinstance(v, dict))))
cluster = {
"cluster_name": cluster_name,
"instance_type": instance_type,
"region": region,
"engine": engine.lower(),
"node_count": node_count,
"primary_nodes": max(1, node_count // 2),
"avg_memory_gb": dataset_gb,
"daily_commands": daily_commands_total,
"current_monthly_cost": 0,
"avg_payload_kb": avg_payload_kb,
"peak_commands": 0,
"min_commands": 0,
"peak_to_avg_ratio": 0,
"peak_memory_gb": 0,
"min_memory_gb": 0,
}
return cluster, commandstats
def main():
parser = argparse.ArgumentParser(
description="Estimate ElastiCache Serverless costs from provisioned cluster metrics"
)
parser.add_argument("--input", "-i",
help="CSV with cluster data (see README for format)")
parser.add_argument("--endpoint", "-e",
help="Connect directly to a Valkey/Redis endpoint (host:port or host)")
parser.add_argument("--cluster-name", default=None,
help="Cluster name (used with --endpoint, default: derived from host)")
parser.add_argument("--instance-type", default="",
help="Instance type (used with --endpoint, e.g., cache.r7g.large)")
parser.add_argument("--region", default="us-east-1",
help="AWS region (used with --endpoint, default: us-east-1)")
parser.add_argument("--engine", default="valkey",
help="Engine (used with --endpoint, default: valkey)")
parser.add_argument("--node-count", type=int, default=2,
help="Node count (used with --endpoint, default: 2)")
parser.add_argument("--avg-payload-kb", type=float, default=1.0,
help="Average payload size in KB (default: 1.0)")
parser.add_argument("--no-tls", action="store_true",
help="Disable TLS when connecting via --endpoint")
parser.add_argument("--sample-seconds", type=int, default=60,
help="Seconds to sample when using --endpoint (default: 60). "
"Takes two snapshots this far apart and computes deltas.")
parser.add_argument("--commandstats", "-c",
help="CSV with per-command stats for detailed ECPU estimation")
parser.add_argument("--pricing", "-p",
help="Pricing CSV (optional - fetches live from AWS if omitted)")
parser.add_argument("--output", "-o", default="serverless_estimate.csv",
help="Output CSV path (default: serverless_estimate.csv)")
parser.add_argument("--json", action="store_true",
help="Also write JSON output")
args = parser.parse_args()
if not args.input and not args.endpoint:
parser.error("Either --input (CSV) or --endpoint (host:port) is required")
pricing = PricingLoader(args.pricing)
if args.endpoint:
# Direct endpoint mode: connect, collect, normalize, estimate
cluster, commandstats = collect_from_endpoint(
args.endpoint,
cluster_name=args.cluster_name,
instance_type=args.instance_type,
region=args.region,
engine=args.engine,
node_count=args.node_count,
avg_payload_kb=args.avg_payload_kb,
use_tls=not args.no_tls,
sample_seconds=args.sample_seconds,
)
results = [estimate_cluster(cluster, pricing, commandstats)]
else:
print("Loading clusters from: {}".format(args.input))
clusters = load_clusters(args.input)
print(" {} clusters loaded".format(len(clusters)))
all_commandstats = None
if args.commandstats:
print("Loading commandstats from: {}".format(args.commandstats))
all_commandstats = parse_commandstats_file(args.commandstats)
print(" Stats for {} clusters".format(len(all_commandstats)))
results = []
for cluster in clusters:
cs = all_commandstats.get(cluster["cluster_name"]) if all_commandstats else None
results.append(estimate_cluster(cluster, pricing, cs))
write_csv(results, args.output)
print_summary(results)
if args.json:
base, _ = os.path.splitext(args.output)
json_path = base + ".json"
with open(json_path, "w") as f:
json.dump(results, f, indent=2, default=str)
print("JSON written to: {}".format(json_path))
if __name__ == "__main__":
main()
scripts/start_tunnel.py
#!/usr/bin/env python3
"""
Start an SSM port-forwarding tunnel to an ElastiCache endpoint.
Wraps the AWS SSM StartPortForwardingSessionToRemoteHost document into a
single command with pre-flight checks. Designed to be invoked by an AI agent
or a human developer, with clear error messages at every failure point.
Dependencies:
pip install boto3
Prerequisites:
- AWS CLI v2 installed
- Session Manager plugin installed
(https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
- An SSM-managed EC2 instance in the same VPC as the cache
(use find_tunnel_host.py to locate one)
Usage:
python start_tunnel.py \\
--instance-id i-0abc123def456 \\
--cache-host my-cache.serverless.use1.cache.amazonaws.com
python start_tunnel.py \\
--instance-id i-0abc123def456 \\
--cache-host my-cache.serverless.use1.cache.amazonaws.com \\
--cache-port 6379 --local-port 6379 \\
--region us-west-2 --profile my-profile
"""
from __future__ import annotations
import argparse
import json
import os
import select
import shutil
import signal
import socket
import subprocess
import sys
import time
try:
import boto3
except ImportError:
print(
"Error: the 'boto3' package is required.\n"
"Install it with:\n"
" pip install boto3"
)
sys.exit(2)
def check_ssm_plugin() -> bool:
"""Verify the Session Manager plugin is installed."""
return shutil.which("session-manager-plugin") is not None
def check_aws_cli() -> bool:
"""Verify the AWS CLI is installed."""
return shutil.which("aws") is not None
def check_instance_ssm_status(instance_id: str, session: boto3.Session) -> bool:
"""Verify the target instance is online in SSM."""
ssm = session.client("ssm")
try:
resp = ssm.describe_instance_information(
Filters=[{"Key": "InstanceIds", "Values": [instance_id]}]
)
instances = resp.get("InstanceInformationList", [])
if not instances:
return False
return instances[0].get("PingStatus") == "Online"
except Exception as exc:
print(f"Warning: could not verify SSM status: {exc}")
return False
def check_instance_running(instance_id: str, session: boto3.Session) -> bool:
"""Verify the target EC2 instance is in a running state."""
ec2 = session.client("ec2")
try:
resp = ec2.describe_instance_status(
InstanceIds=[instance_id], IncludeAllInstances=True
)
statuses = resp.get("InstanceStatuses", [])
if not statuses:
return False
return statuses[0]["InstanceState"]["Name"] == "running"
except Exception as exc:
print(f"Warning: could not verify instance state: {exc}")
return False
def validate_port(port: int, label: str = "port") -> None:
"""Validate that a port number is in the usable range (1-65535)."""
if not isinstance(port, int) or port < 1 or port > 65535:
print(f" [FAIL] {label} must be between 1 and 65535 (got {port})")
sys.exit(1)
def check_port_available(port: int) -> bool:
"""Check that nothing is already listening on the local port.
Returns True if the port is free, False if something is already bound.
"""
try:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind(("127.0.0.1", port))
return True
except OSError:
return False
def wait_for_port(port: int, timeout: float = 10.0) -> bool:
"""Wait for the local port to become available (tunnel is up)."""
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
with socket.create_connection(("127.0.0.1", port), timeout=1.0):
return True
except (ConnectionRefusedError, OSError):
time.sleep(0.5)
return False
def start_tunnel(
instance_id: str,
cache_host: str,
cache_port: int,
local_port: int,
region: str,
profile: str | None,
) -> None:
"""Start SSM port forwarding and keep it running until interrupted."""
session = boto3.Session(profile_name=profile, region_name=region)
# --- Pre-flight checks ---
print("Pre-flight checks:\n")
# 1. Port validation
validate_port(cache_port, "--cache-port")
validate_port(local_port, "--local-port")
print(f" [OK] Port numbers are valid (local={local_port}, remote={cache_port})")
# 2. Local port not already in use
if not check_port_available(local_port):
print(f" [FAIL] Local port {local_port} is already in use.")
print(f" Another process is listening on 127.0.0.1:{local_port}.")
print(f" Stop that process first, or use --local-port to pick a different port.")
sys.exit(1)
print(f" [OK] Local port {local_port} is available")
# 3. AWS CLI
if not check_aws_cli():
print(" [FAIL] AWS CLI not found on PATH.")
print(" Install: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html")
sys.exit(1)
print(" [OK] AWS CLI found")
# 4. SSM plugin
if not check_ssm_plugin():
print(" [FAIL] Session Manager plugin not found on PATH.")
print(" Install: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html")
sys.exit(1)
print(" [OK] Session Manager plugin found")
# 5. Instance running
if not check_instance_running(instance_id, session):
print(f" [FAIL] Instance {instance_id} is not in 'running' state.")
print(" Start the instance or choose a different one (use find_tunnel_host.py).")
sys.exit(1)
print(f" [OK] Instance {instance_id} is running")
# 6. SSM agent online
if not check_instance_ssm_status(instance_id, session):
print(f" [WARN] Instance {instance_id} is not reporting to SSM as Online.")
print(" The tunnel may fail. Ensure the instance has:")
print(" - SSM agent installed and running")
print(" - AmazonSSMManagedInstanceCore IAM policy attached")
print(" - Network route to SSM endpoints (NAT gateway or VPC endpoint)")
print(" Proceeding anyway ...\n")
else:
print(f" [OK] Instance {instance_id} SSM agent is online")
print()
# --- Build the SSM command ---
parameters = json.dumps(
{
"host": [cache_host],
"portNumber": [str(cache_port)],
"localPortNumber": [str(local_port)],
}
)
cmd = [
"aws", "ssm", "start-session",
"--target", instance_id,
"--document-name", "AWS-StartPortForwardingSessionToRemoteHost",
"--parameters", parameters,
"--region", region,
]
if profile:
cmd.extend(["--profile", profile])
print(f"Starting tunnel: 127.0.0.1:{local_port} -> {cache_host}:{cache_port}")
print(f" via SSM instance: {instance_id}")
print(f" region: {region}")
print()
print("Tunnel command:")
print(f" {' '.join(cmd)}")
print()
# --- Launch the tunnel ---
proc = None
try:
proc = subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
)
# Give the tunnel a moment to start, then check if the port is up
time.sleep(2)
if proc.poll() is not None:
# Process already exited -- something went wrong
output = proc.stdout.read() if proc.stdout else ""
print(f"Tunnel process exited immediately (code {proc.returncode}):\n")
print(output)
sys.exit(1)
if wait_for_port(local_port, timeout=15.0):
print(f"Tunnel is UP. Local port 127.0.0.1:{local_port} is accepting connections.\n")
is_serverless = ".serverless." in cache_host
if is_serverless:
print("Note: ElastiCache Serverless requires TLS. Your client must connect")
print(f" with TLS enabled and SNI set to {cache_host}")
print(f" (the tunnel forwards raw TCP; TLS negotiation happens end-to-end).\n")
print("Test with:")
print(f" python scripts/test_connection.py 127.0.0.1 --port {local_port} \\")
print(f" --tunnel-mode --server-name {cache_host} \\")
print(f" --username <user> --password <password>")
print()
print("Press Ctrl+C to stop the tunnel.")
else:
print(f"Warning: local port {local_port} not yet responding after 15s.")
print("The tunnel may still be starting. Check the output below.\n")
# Stream SSM output with periodic health checks.
# Uses select() to avoid blocking forever if the session stalls.
health_check_interval = 30 # seconds between port probes
assert proc.stdout is not None
while proc.poll() is None:
ready, _, _ = select.select([proc.stdout], [], [], health_check_interval)
if ready:
line = proc.stdout.readline()
if line:
print(f" [SSM] {line}", end="")
else:
break # EOF -- process closed stdout
else:
# No output for health_check_interval seconds, check tunnel
if not wait_for_port(local_port, timeout=3.0):
print(f"\n [WARN] Tunnel port {local_port} is no longer responding.")
print(" The SSM session may have stalled. Press Ctrl+C to stop.")
proc.wait()
if proc.returncode != 0:
print(f"\nTunnel exited with code {proc.returncode}")
sys.exit(1)
except KeyboardInterrupt:
print("\n\nShutting down tunnel ...")
if proc and proc.poll() is None:
proc.send_signal(signal.SIGTERM)
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
print("Tunnel closed.")
except Exception as exc:
print(f"\nError: {exc}")
if proc and proc.poll() is None:
proc.kill()
sys.exit(1)
def main() -> None:
parser = argparse.ArgumentParser(
description="Start an SSM port-forwarding tunnel to an ElastiCache endpoint.",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument(
"--instance-id", required=True,
help="SSM-managed EC2 instance ID (use find_tunnel_host.py to discover)",
)
parser.add_argument(
"--cache-host", required=True,
help="ElastiCache endpoint hostname (e.g. my-cache.serverless.use1.cache.amazonaws.com)",
)
parser.add_argument(
"--cache-port", type=int, default=6379,
help="ElastiCache port (default: 6379; serverless reader endpoint uses 6380)",
)
parser.add_argument(
"--local-port", type=int, default=6379,
help="Local port to forward to (default: 6379)",
)
_default_region = (
os.environ.get("AWS_REGION")
or os.environ.get("AWS_DEFAULT_REGION")
or "us-east-1"
)
parser.add_argument(
"--region", default=_default_region,
help=f"AWS region (default: {_default_region})",
)
parser.add_argument(
"--profile", default=None,
help="AWS profile name",
)
args = parser.parse_args()
start_tunnel(
instance_id=args.instance_id,
cache_host=args.cache_host,
cache_port=args.cache_port,
local_port=args.local_port,
region=args.region,
profile=args.profile,
)
if __name__ == "__main__":
main()
scripts/test_connection.py
#!/usr/bin/env python3
"""
ElastiCache connection validator.
Uses the valkey-py client library to perform a proper TLS-enabled
PING against an ElastiCache endpoint. Supports RBAC (username + password),
IAM auth, and legacy AUTH tokens (node-based only).
Serverless caches REQUIRE TLS -- this script defaults to TLS-on.
Dependencies:
pip install valkey
pip install boto3 # only needed for --iam-auth (includes botocore)
Usage examples:
# Serverless with RBAC (TLS is default)
python test_connection.py your-endpoint.serverless.use1.cache.amazonaws.com \\
--username my-user --password my-password
# Node-based with TLS and RBAC
python test_connection.py your-endpoint.cache.amazonaws.com \\
--username my-user --password my-password
# Node-based with legacy AUTH token
python test_connection.py your-endpoint.cache.amazonaws.com \\
--password my-auth-token
# Node-based WITHOUT TLS (not recommended)
python test_connection.py your-endpoint.cache.amazonaws.com \\
--no-tls --password my-auth-token
# Serverless with IAM auth
python test_connection.py your-endpoint.serverless.use1.cache.amazonaws.com \\
--iam-auth --iam-user my-iam-user --region us-east-1 \\
--cache-name my-cache
# Through an SSM tunnel (see start_tunnel.py)
python test_connection.py 127.0.0.1 --port 6379 \\
--tunnel-mode --server-name my-cache.serverless.use1.cache.amazonaws.com \\
--username my-user --password my-password
"""
from __future__ import annotations
import argparse
import os
import ssl
import sys
import time
# ---------------------------------------------------------------------------
# Dependency check -- fail early with an actionable message instead of a
# raw ImportError traceback.
#
# Required: valkey
# Optional: boto3 -- only needed when using --iam-auth.
# ---------------------------------------------------------------------------
try:
import valkey as client_lib
client_name = "valkey-py"
except ImportError:
print(
"Error: the 'valkey' package is required for this script.\n"
"Install it with:\n"
" pip install valkey\n"
"\n"
"If using IAM authentication, you also need:\n"
" pip install boto3",
file=sys.stderr,
)
sys.exit(2)
def get_iam_auth_token(
cache_name: str, iam_user: str, region: str, is_serverless: bool = False
) -> str:
"""Generate a short-lived IAM auth token for ElastiCache."""
cache_name = cache_name.lower()
try:
import botocore.session
from botocore.signers import RequestSigner
from botocore.model import ServiceId
except ImportError:
print(
"Error: the 'botocore' package is required for IAM authentication.\n"
"Install it with:\n"
" pip install botocore\n"
"\n"
"You also need AWS credentials configured (via environment\n"
"variables, ~/.aws/credentials, or an IAM role).",
file=sys.stderr,
)
sys.exit(2)
session = botocore.session.get_session()
signer = RequestSigner(
ServiceId("elasticache"),
region,
"elasticache",
"v4",
session.get_credentials(),
session.get_component("event_emitter"),
)
url = signer.generate_presigned_url(
{
"method": "GET",
"url": f"http://{cache_name}/?Action=connect&User={iam_user}&ResourceType=ServerlessCache" if is_serverless else f"http://{cache_name}/?Action=connect&User={iam_user}",
"body": {},
"headers": {},
"context": {},
},
operation_name="connect",
expires_in=900,
region_name=region,
)
token = url[len("http://"):] if url.startswith("http://") else url
return token
def _check_dns(host: str) -> None:
"""Warn if the host resolves to a private IP and is not localhost."""
import ipaddress
import socket
if host in ("127.0.0.1", "localhost", "::1"):
return
try:
addr = socket.gethostbyname(host)
ip = ipaddress.ip_address(addr)
if ip.is_private:
print(f" WARNING: {host} resolves to private IP {addr}.", file=sys.stderr)
print(f" If you are not in the VPC, this connection will time out.", file=sys.stderr)
print(f" Use --tunnel-mode with scripts/start_tunnel.py for local development.\n", file=sys.stderr)
except socket.gaierror:
print(f" WARNING: DNS lookup failed for {host}. The endpoint may be wrong,", file=sys.stderr)
print(f" or you may not have DNS access to the VPC.\n", file=sys.stderr)
def test_connection(
host: str,
port: int,
username: str | None,
password: str | None,
use_tls: bool,
timeout: float,
tunnel_mode: bool = False,
server_name: str | None = None,
is_serverless: bool = False,
) -> bool:
"""Connect to ElastiCache and run PING using the client library."""
_check_dns(host)
tls_label = "TLS" if use_tls else "plaintext"
if tunnel_mode:
tls_label += " tunnel-mode"
auth_label = (
f"user={username}"
if username
else ("password-auth" if password else "no-auth")
)
print(f"Connecting to {host}:{port} ({tls_label}, {auth_label}) via {client_name} ...")
kwargs = {
"host": host,
"port": port,
"socket_timeout": timeout,
"socket_connect_timeout": timeout,
"decode_responses": True,
}
if use_tls:
kwargs["ssl"] = True
if tunnel_mode:
# Through an SSM tunnel the TLS cert is issued for the real cache
# hostname (*.cache.amazonaws.com), not 127.0.0.1. Disable hostname
# checking but keep certificate verification against the CA bundle.
# Set SNI to the real cache hostname (--server-name) so that
# serverless endpoints can route the TLS handshake correctly.
kwargs["ssl_cert_reqs"] = "required"
kwargs["ssl_ca_certs"] = ssl.get_default_verify_paths().cafile
kwargs["ssl_check_hostname"] = False
if server_name:
kwargs["ssl_server_hostname"] = server_name
print(f" Tunnel mode: hostname verification disabled, target={host}:{port}")
print(f" WARNING: tunnel mode is for LOCAL DEVELOPMENT ONLY", file=sys.stderr)
else:
kwargs["ssl_cert_reqs"] = "required"
kwargs["ssl_ca_certs"] = ssl.get_default_verify_paths().cafile
if username:
kwargs["username"] = username
if password:
kwargs["password"] = password
# Detect serverless from hostname pattern OR the --serverless flag.
# The flag is needed when connecting through a tunnel (e.g., 127.0.0.1)
# where the hostname pattern is not present.
is_serverless = is_serverless or ".serverless." in host
# In tunnel mode, always use a non-cluster client. Cluster clients run
# CLUSTER SLOTS/NODES discovery and try to connect directly to each node
# using private VPC IPs, which are unreachable through the tunnel. A
# standalone client connects only to the tunneled endpoint, which is
# sufficient for a connectivity test (PING + INFO).
use_cluster_client = is_serverless and not tunnel_mode
try:
if use_cluster_client:
if client_name == "valkey-py":
conn = client_lib.ValkeyCluster(**kwargs)
else:
conn = client_lib.RedisCluster(**kwargs)
elif client_name == "valkey-py":
conn = client_lib.Valkey(**kwargs)
else:
conn = client_lib.Redis(**kwargs)
if is_serverless and tunnel_mode:
print(" Note: using standalone client through tunnel (cluster discovery")
print(" would return unreachable VPC IPs). PING validates connectivity.")
except Exception as exc:
print(f" Client init FAILED: {exc}", file=sys.stderr)
return False
# Step 1: PING
start = time.monotonic()
try:
result = conn.ping()
elapsed_ms = (time.monotonic() - start) * 1000
print(f" PING: OK ({elapsed_ms:.1f} ms)")
except client_lib.exceptions.AuthenticationError as exc:
print(f" PING FAILED: Authentication error -- {exc}", file=sys.stderr)
print(" Hint: Check username/password or ensure the RBAC user exists and is active.", file=sys.stderr)
return False
except client_lib.exceptions.ConnectionError as exc:
msg = str(exc)
print(f" PING FAILED: Connection error -- {exc}", file=sys.stderr)
if "SSL" in msg or "tls" in msg.lower():
if use_tls:
print(" Hint: TLS handshake failed. Verify the endpoint supports in-transit encryption.", file=sys.stderr)
else:
print(" Hint: This endpoint may require TLS. Retry without --no-tls (TLS is default).", file=sys.stderr)
else:
print(" Hint: Check security group rules, VPC placement, and whether you need a tunnel or jump host.", file=sys.stderr)
return False
except Exception as exc:
print(f" PING FAILED: {exc}", file=sys.stderr)
return False
# Step 2: INFO SERVER (lightweight metadata check)
try:
info = conn.info("server")
engine = info.get("redis_version", info.get("valkey_version", "unknown"))
mode = info.get("redis_mode", "unknown")
print(f" Server: engine version {engine}, mode={mode}")
except Exception:
# INFO may be blocked by ACL; that is fine
print(" Server info: not available (ACL may restrict INFO command)", file=sys.stderr)
# Step 3: Verify TLS status
if use_tls:
print(" TLS: enabled (in-transit encryption active)")
else:
print(" TLS: disabled -- consider enabling in-transit encryption for production use", file=sys.stderr)
try:
conn.close()
except Exception:
pass
print(" Result: CONNECTION SUCCESSFUL")
return True
def main() -> None:
parser = argparse.ArgumentParser(
description="Validate connectivity to an ElastiCache endpoint (serverless or node-based).",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument("host", help="ElastiCache endpoint hostname")
parser.add_argument("--port", type=int, default=6379, help="Port (default: 6379; serverless reader endpoint uses 6380)")
parser.add_argument("--timeout", type=float, default=5.0, help="Connection timeout in seconds (default: 5)")
# TLS (default on)
parser.add_argument(
"--no-tls",
action="store_true",
default=False,
help="Disable TLS (not recommended; serverless caches REQUIRE TLS)",
)
# Tunnel mode
tunnel_group = parser.add_argument_group("tunnel mode (for SSM port-forwarded connections)")
tunnel_group.add_argument(
"--tunnel-mode",
action="store_true",
default=False,
help="Enable tunnel mode: disables TLS hostname verification (dev only)",
)
tunnel_group.add_argument(
"--server-name",
help="Real cache hostname for TLS SNI (required for serverless endpoints in tunnel mode)",
)
# Auth
auth_group = parser.add_argument_group("authentication")
auth_group.add_argument("--username", help="RBAC username")
auth_group.add_argument("--password", help="RBAC password or legacy AUTH token (node-based only)")
auth_group.add_argument(
"--iam-auth",
action="store_true",
default=False,
help="Use IAM authentication (Valkey 7.2+ / Redis OSS 7.0+). Requires boto3.",
)
auth_group.add_argument("--iam-user", help="IAM-enabled ElastiCache user ID (required with --iam-auth)")
auth_group.add_argument("--cache-name", help="Cache or replication group name (required with --iam-auth)")
auth_group.add_argument(
"--serverless",
action="store_true",
default=False,
help="Target is a serverless cache (adds ResourceType=ServerlessCache to IAM token)",
)
_default_region = os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION") or "us-east-1"
auth_group.add_argument("--region", default=_default_region, help=f"AWS region (default: {_default_region})")
args = parser.parse_args()
use_tls = not args.no_tls
password = args.password
username = args.username
# Tunnel mode validation
if args.tunnel_mode and args.no_tls:
parser.error("--tunnel-mode requires TLS (cannot combine with --no-tls)")
# IAM auth flow
if args.iam_auth:
if not args.iam_user or not args.cache_name:
parser.error("--iam-auth requires --iam-user and --cache-name")
print(f"Generating IAM auth token for user={args.iam_user}, cache={args.cache_name} ...")
password = get_iam_auth_token(args.cache_name, args.iam_user, args.region, is_serverless=args.serverless)
username = args.iam_user
print(" IAM token generated (valid for 15 minutes)")
success = test_connection(
host=args.host,
port=args.port,
username=username,
password=password,
use_tls=use_tls,
timeout=args.timeout,
tunnel_mode=args.tunnel_mode,
server_name=args.server_name,
is_serverless=args.serverless,
)
sys.exit(0 if success else 1)
if __name__ == "__main__":
main()
scripts/validate_references.py
#!/usr/bin/env python3
"""
ElastiCache Reference Validator
Validates that all reference files have required metadata and follow
project conventions. Also checks scripts for module docstrings and
CLI entrypoints.
Usage:
python scripts/validate_references.py # normal output
python scripts/validate_references.py --verbose # show all PASS results too
python scripts/validate_references.py --quiet # summary counts only
Exit codes:
0 -- all checks passed (warnings are OK)
1 -- one or more FAIL results
"""
import argparse
import os
import re
import sys
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
REFERENCES_DIR = os.path.join(BASE_DIR, "references")
SCRIPTS_DIR = os.path.join(BASE_DIR, "scripts")
SKILL_MD = os.path.join(BASE_DIR, "SKILL.md")
# Minimum number of non-whitespace characters (beyond the title line) for a
# reference file to be considered non-empty.
MIN_CONTENT_CHARS = 50
# Pattern for kebab-case filenames: lowercase letters, digits, hyphens.
KEBAB_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*\.md$")
# ---------------------------------------------------------------------------
# Result tracking
# ---------------------------------------------------------------------------
class Result:
"""A single validation result."""
def __init__(self, level, filepath, message):
self.level = level # "PASS", "WARN", or "FAIL"
self.filepath = filepath # relative to BASE_DIR
self.message = message
def __str__(self):
return f"[{self.level}] {self.filepath}: {self.message}"
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _relative(path):
"""Return a path relative to BASE_DIR for display."""
return os.path.relpath(path, BASE_DIR)
def _read_skill_md_references():
"""Extract all reference file paths mentioned in SKILL.md."""
if not os.path.isfile(SKILL_MD):
return set()
with open(SKILL_MD, "r", encoding="utf-8") as fh:
content = fh.read()
# Match backtick-quoted paths like `references/setup/engine-selection.md`
return set(re.findall(r"`(references/[a-zA-Z0-9/_-]+\.md)`", content))
# ---------------------------------------------------------------------------
# Reference file checks
# ---------------------------------------------------------------------------
def validate_reference_file(filepath, skill_refs):
"""Validate a single .md reference file. Returns a list of Result objects."""
results = []
rel = _relative(filepath)
filename = os.path.basename(filepath)
# 1. Kebab-case filename
if not KEBAB_RE.match(filename):
results.append(Result("FAIL", rel,
f"Filename '{filename}' does not follow kebab-case convention "
"(expected lowercase letters, digits, and hyphens only)"))
else:
results.append(Result("PASS", rel, "Filename follows kebab-case convention"))
# 2. Read file content
try:
with open(filepath, "r", encoding="utf-8") as fh:
content = fh.read()
except Exception as exc:
results.append(Result("FAIL", rel, f"Could not read file: {exc}"))
return results
lines = content.split("\n")
non_blank_lines = [ln for ln in lines if ln.strip()]
# 3. Has a title (first non-blank line starts with #)
if not non_blank_lines:
results.append(Result("FAIL", rel, "File is completely empty"))
return results # no point checking further
first_line = non_blank_lines[0].strip()
if first_line.startswith("#"):
results.append(Result("PASS", rel, f"Has title: {first_line[:60]}"))
else:
results.append(Result("FAIL", rel,
"Missing title -- first non-blank line should start with '#'"))
# 4. Has meaningful content beyond the title
# Strip the title line and count remaining non-whitespace characters
body = "\n".join(lines[1:]) if len(lines) > 1 else ""
body_chars = len(re.sub(r"\s", "", body))
if body_chars < MIN_CONTENT_CHARS:
results.append(Result("FAIL", rel,
f"File body has only {body_chars} non-whitespace chars "
f"(minimum {MIN_CONTENT_CHARS}) -- appears empty or stub"))
else:
results.append(Result("PASS", rel,
f"File has meaningful content ({body_chars} body chars)"))
# 5. Referenced in SKILL.md (non-critical)
# Build the expected reference path from the relative path (e.g. references/setup/foo.md)
if rel in skill_refs:
results.append(Result("PASS", rel, "Referenced in SKILL.md"))
else:
results.append(Result("WARN", rel, "Not referenced in SKILL.md"))
return results
def validate_all_references():
"""Walk references/ and validate every .md file."""
results = []
skill_refs = _read_skill_md_references()
if not os.path.isdir(REFERENCES_DIR):
results.append(Result("FAIL", "references/",
"references/ directory not found"))
return results, 0
md_count = 0
for root, _dirs, files in os.walk(REFERENCES_DIR):
for fname in sorted(files):
if not fname.endswith(".md"):
continue
md_count += 1
filepath = os.path.join(root, fname)
results.extend(validate_reference_file(filepath, skill_refs))
if md_count == 0:
results.append(Result("FAIL", "references/",
"No .md files found under references/"))
return results, md_count
# ---------------------------------------------------------------------------
# Script file checks
# ---------------------------------------------------------------------------
def validate_script_file(filepath):
"""Validate a single .py script file. Returns a list of Result objects."""
results = []
rel = _relative(filepath)
try:
with open(filepath, "r", encoding="utf-8") as fh:
content = fh.read()
except Exception as exc:
results.append(Result("FAIL", rel, f"Could not read file: {exc}"))
return results
# 1. Has a module docstring
# A module docstring is a triple-quoted string near the top of the file,
# possibly preceded by a shebang, encoding declaration, or comments.
# We use a simple heuristic: check if '"""' or "'''" appears in the first
# 20 non-blank lines.
lines = content.split("\n")
top_chunk = "\n".join(lines[:30])
has_docstring = ('"""' in top_chunk or "'''" in top_chunk)
if has_docstring:
results.append(Result("PASS", rel, "Has module docstring"))
else:
results.append(Result("FAIL", rel,
"Missing module docstring at top of file"))
# 2. Has a CLI entrypoint (if __name__ == "__main__")
has_main = ('if __name__' in content
and ('"__main__"' in content or "'__main__'" in content))
if has_main:
results.append(Result("PASS", rel, "Has CLI entrypoint"))
else:
# This is a warning, not a fail, because some scripts are libraries
results.append(Result("WARN", rel,
"No 'if __name__ == \"__main__\"' block -- "
"OK if this is a library, not a CLI tool"))
return results
def validate_all_scripts():
"""Walk scripts/ and validate every .py file (excluding this script)."""
results = []
if not os.path.isdir(SCRIPTS_DIR):
results.append(Result("FAIL", "scripts/",
"scripts/ directory not found"))
return results, 0
py_count = 0
for fname in sorted(os.listdir(SCRIPTS_DIR)):
if not fname.endswith(".py"):
continue
py_count += 1
filepath = os.path.join(SCRIPTS_DIR, fname)
results.extend(validate_script_file(filepath))
if py_count == 0:
results.append(Result("WARN", "scripts/",
"No .py files found under scripts/"))
return results, py_count
# ---------------------------------------------------------------------------
# Reporting
# ---------------------------------------------------------------------------
def print_results(results, verbose=False, quiet=False):
"""Print validation results to stdout."""
fail_count = sum(1 for r in results if r.level == "FAIL")
warn_count = sum(1 for r in results if r.level == "WARN")
pass_count = sum(1 for r in results if r.level == "PASS")
if not quiet:
# Group results by level for readability
if fail_count > 0:
print("\n--- FAILURES ---")
for r in results:
if r.level == "FAIL":
print(f" {r}")
if warn_count > 0:
print("\n--- WARNINGS ---")
for r in results:
if r.level == "WARN":
print(f" {r}")
if verbose and pass_count > 0:
print("\n--- PASSED ---")
for r in results:
if r.level == "PASS":
print(f" {r}")
# Summary line (always printed)
print(f"\nSummary: {pass_count} passed, {warn_count} warnings, {fail_count} failures")
return fail_count
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
parser = argparse.ArgumentParser(
description="Validate ElastiCache reference files and scripts."
)
parser.add_argument(
"--verbose", action="store_true",
help="Show PASS results in addition to WARN and FAIL"
)
parser.add_argument(
"--quiet", action="store_true",
help="Show summary counts only (no individual results)"
)
args = parser.parse_args()
all_results = []
# Validate reference files
print("Validating reference files...")
ref_results, ref_count = validate_all_references()
all_results.extend(ref_results)
print(f" Scanned {ref_count} reference .md files")
# Validate scripts
print("Validating scripts...")
script_results, script_count = validate_all_scripts()
all_results.extend(script_results)
print(f" Scanned {script_count} script .py files")
# Print results
fail_count = print_results(all_results, verbose=args.verbose, quiet=args.quiet)
sys.exit(1 if fail_count > 0 else 0)
if __name__ == "__main__":
main()
SKILL.md
---
name: amazon-elasticache
version: 2
description: "Activate when developers have latent caching needs: slow API responses, database read bottlenecks, DynamoDB throttling or cost, RDS/Aurora scaling pressure, Bedrock latency or cost, or adding a cache; activate when working with Redis, Valkey, Memcached, or any in-memory data store, cache-aside patterns, session stores, rate limiting, leaderboards, counters, streams, queues, pub/sub, distributed locks, feature flags, shopping carts, or other caching strategies. Activate for GenAI and ML retrieval: vector similarity search for low-latency retrieval, semantic caching, RAG, LLM response caching, embedding stores, AI agent memory, recommendation, personalization. Activate for ElastiCache lifecycle: provisioning (serverless or node-based), engine selection, CloudFormation/CDK/Terraform IaC, VPC connectivity, TLS, RBAC, IAM auth, Global Datastore, monitoring, troubleshooting, cost optimization, and migration from self-managed Redis. Do not trigger for browser caches, CDN/CloudFront, HTTP Cache-Control, CPU caches."
---
# ElastiCache
A modular ElastiCache toolkit organized as a registry of sub-skills. Each sub-skill handles one domain of ElastiCache work. The router below matches user intent to the right sub-skill, then loads only the references needed for that sub-skill.
## How this skill works
1. Match the user's request against the semantic categories in the registry below. Match on meaning, not exact wording ("help me figure out which data structures to use" matches `data-modeling` even without the word "pattern").
2. **Disambiguation:** If the user's intent matches multiple sub-skills, apply these rules in order:
- If `.elasticache/requirements.json` exists with `infrastructure.endpoint` set, prefer `monitoring` or `data-modeling` (the user has an existing cache).
- If no cache exists (no requirements.json or no endpoint), prefer `requirements`.
- If still ambiguous, ask one clarifying question: "Are you looking to set up something new, or troubleshoot something existing?"
3. Check the Guardrails section before recommending an engine or deployment model.
4. Read `references/{sub-skill-id}/instructions.md` for the matched sub-skill. If the file is not found at a relative path, check your prompt or environment for the skill directory absolute path and retry with `{skill-directory}/references/{sub-skill-id}/instructions.md`.
5. If the request spans multiple sub-skills, execute them in pipeline order.
6. If a sub-skill requires upstream context (engine, deployment model, endpoint) not yet in session memory, route to the upstream sub-skill first.
7. If no sub-skill matches, activate `requirements` first.
8. If a script or CLI call fails, show the error to the user and suggest a specific fix before retrying.
## Sub-skill registry
Each entry has: an ID (directory name under `references/`), a domain description, semantic categories for matching, and upstream/downstream dependencies.
| ID | Name | Domain | Semantic Categories | Upstream | Downstream |
|----|------|--------|--------------------|----------|------------|
| `requirements` | Solution Fit | Gathers workload, stack, scale, latency, persistence, and budget through workspace scan + structured interview. Decides whether ElastiCache is the right service and hands off with a routing recommendation. | I need a cache, speed up my app, reduce database load, lower Bedrock cost, should I use ElastiCache, what's best for my workload, evaluating cache options, ElastiCache vs X, Valkey vs X, vague new workload | — | `setup`, `data-modeling`, `genai`, `monitoring`, `migration` |
| `setup` | Create and Connect | Provisioning, connectivity, security, authentication, IaC, deployment choice. Gets the user to a working cache with least friction. Covers engine selection, serverless vs node-based, VPC, TLS, RBAC/IAM, jump-host/SSM tunnels, CLI/SDK/CFN/CDK/Terraform starters. | create a cache, set up ElastiCache, provision, Valkey cluster, connect Lambda/ECS/EKS/EC2, VPC, security groups, TLS, RBAC, IAM auth, jump host, SSM tunnel, CloudFormation, CDK, Terraform, engine selection, serverless vs node-based, backup, snapshot, restore, export | `requirements` (optional) | `data-modeling`, `genai`, `monitoring` |
| `data-modeling` | Application Patterns | Picks data structures, key schema, TTL strategy, invalidation approach, and client code for non-AI patterns: cache-aside, session store, rate limiting, leaderboards, counters, pub/sub, streams, shopping carts, job queues, activity feeds. | session store, rate limiting, leaderboard, cache-aside, query caching, counters, streams, pub/sub, shopping cart, job queue, activity feed, key schema, TTL, invalidation, data structures | `setup` (cache must exist) | `monitoring` |
| `genai` | AI and Vector Workloads | Classifies request into Mode 1 (plain cache), Mode 2 (semantic response cache), or Mode 3 (full vector search). Selects Valkey and forces node-based Valkey 8.2 or above (recommend 9.0) when server-side vector similarity is needed. Covers semantic caching, agent memory, RAG retrieval, recommendation, personalization, conversation/session persistence for AI agents, and framework wiring (Strands, mem0, LangChain). | semantic cache, RAG, agent memory, conversational memory, vector search, embeddings, recommendation, personalization, Bedrock latency, Bedrock cost, LLM caching, Strands, mem0, LangChain, conversation history, AI session store, embedding provider, framework integration | `setup` (cache must exist) | `monitoring` |
| `monitoring` | Operate and Observe | Diagnoses performance, cost, and reliability using metrics first, then recommends the smallest change. Covers dashboards, alarms, log delivery, cost reporting, event routing, troubleshooting high CPU / memory / replication lag / connection spikes / low hit rate / hot keys / big keys / slot imbalance / latency spike root cause. | cache is slow, cost too high, hit rate low, high CPU, memory pressure, replication lag, connection spikes, dashboards, alarms, CloudWatch, cost comparison, troubleshoot, hot key, uneven shard load, one node pinned, big key, memory bloat, which key is biggest, keyspace distribution, prefix analysis, cost attribution by tenant, memory imbalance, one shard full, slot memory skew, latency spike, slow command incident, root cause for latency bump | — | `setup`, `migration` |
| `migration` | Engine and Platform Migration | Selects the migration path and sequences preflight, validation, cutover, and rollback. Covers self-managed Redis → ElastiCache, Redis OSS → Valkey, node-based ↔ serverless, version upgrades. Hard validate-before-migrate gate. | migrate, Redis OSS to Valkey, self-managed to ElastiCache, node-based to serverless, serverless to node-based, engine upgrade, version upgrade, zero-downtime cutover, rollback | — | `setup`, `monitoring` |
## Pipeline order
Sub-skills run independently, but common multi-step journeys follow these pipelines:
- `requirements` → `setup` → (`data-modeling` | `genai`) → `monitoring`
- `migration` → `setup` → `monitoring`
- `monitoring` → `setup` | `migration` (if metrics indicate)
## State handoff: requirements.json
`.elasticache/requirements.json` is the single source of truth for cross-sub-skill state. Each sub-skill reads it at start and writes its section after completing work. Read before writing; merge, do not overwrite.
| Section | Owner | Key fields |
|---------|-------|------------|
| top-level | `requirements` | `engine`, `deployment_model`, `region`, `runtime`, `patterns`, `use_case`, `vpc_id`, `subnet_ids`, `security_group_ids` |
| `infrastructure` | `setup` | `cache_name`, `resource_id`, `engine_version`, `topology`, `endpoint`, `port`, `auth_model`, `tls`, `client_library`, `execution_path`, `access_mode`, `tunnel_instance_id`, `embedding_provider`, `embedding_model`, `embedding_dim`, `embedding_module` |
| `genai` | `genai` | `mode`, `mode_2_path`, `framework` |
| `migration` | `migration` | `source_type`, `source_host`, `migration_path`, `cutover_status` |
> **Ownership note:** `deployment_model` is set by `requirements` during initial interview. `migration` may update it after an engine or deployment model switch (e.g., node-based to serverless).
requirements.json should include `"schema_version": 1` and `"last_updated": "<ISO timestamp>"` at the top level. Every sub-skill that writes to requirements.json must update `last_updated`. If `last_updated` is older than 7 days, warn the user that cached state may be stale.
requirements.json tracks one active cache. If the user works with multiple caches in the same project, confirm which cache is active before reading or writing state.
When a sub-skill needs upstream context (engine, endpoint, auth model), check requirements.json first. If the field is `null` or the file does not exist, route to the upstream sub-skill.
## Global rules (apply to every sub-skill)
1. **Execution path.** Use AWS CLI, SDK (boto3), CloudFormation, or CDK as the primary path for control-plane work. Use valkey-py as the primary path for data-plane work.
2. **Response depth.** Summary (2-3 sentences) for "should I" or "which" questions. Standard (recommendation + config + code + next steps) by default. Expert (full decision matrix with alternatives, cost, security caveats) for "why" or "compare all" questions. Escalate on user request; never downgrade unprompted.
3. **Session memory.** Track region, VPC, engine, deployment model, auth model, compute runtime, and language. Carry forward across sub-skills. Do not re-ask. If the user overrides a value, update it everywhere. Inferred values (from workspace scan or IaC) must be re-confirmed before high-risk decisions (engine, deployment model, security posture); low-risk inferences (language, framework, region) can be used as defaults silently.
4. **Source priority.** Always answer from skill-local files first (sub-skill references, then `scripts/`). Do not fetch external documentation or web search unless the local files cannot answer the query. When local files are insufficient, fall back to official AWS docs: https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/ for features and https://aws.amazon.com/elasticache/pricing/ for pricing. Never invent price points or version constraints. If the user references a Valkey or Redis version, feature, or pricing tier not covered in local files, fall back to https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/ before answering. Do not extrapolate from local content that may be outdated.
5. **Freshness disclaimer.** When outputting pricing, version constraints, or feature availability, include a one-line disclaimer: "For current pricing see https://aws.amazon.com/elasticache/pricing/. For current feature availability see https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/."
6. **Connection safety.** Never create a Valkey/Redis client at module level (top of file, import time). Initialize connections inside a function or on first use. Module-level connections crash applications that import the module before the cache is reachable.
7. **Code environment.** Use `python3` (not `python`) in all commands and shebangs. Do not generate `pip install` or `pytest` commands in application code. Prefer the agent's file-writing tool over shell redirection for writing files.
8. **Input validation (write operations only).** Before running any create, modify, or delete CLI command or SDK call that uses user-provided values (cache name, region, subnet IDs, security group IDs, VPC ID, KMS key, snapshot name), validate them with `scripts/input_validator.py`. Run `python3 scripts/input_validator.py cache_name=<val> region=<val> ...` or import `validate_all` as a library. Skip validation for read-only operations (describe, list) where bad input is harmless.
9. **Resource tagging (always apply on resource creation).** When creating any cache (`create-serverless-cache`, `create-replication-group`, `create-cache-cluster`), ALWAYS include these tags:
`--tags Key=managed_by,Value=aws-skills Key=skill,Value=elasticache Key=skill_version,Value=1.0.0 Key=created_by,Value=elasticache-skill Key=generation_model,Value={your-model-id}`
Example: `--tags Key=managed_by,Value=aws-skills Key=skill,Value=elasticache Key=skill_version,Value=1.0.0 Key=created_by,Value=elasticache-skill Key=generation_model,Value=claude-sonnet-4-20250514`
Set `generation_model` to your own model identifier. Include these tags even if the user does not mention tagging. If the user provides additional tags, append these to their tags rather than replacing them.
## Reference loading
Load additional references only when the current turn's answer requires them.
On-demand pointers (not preloaded; fetch when the trigger applies):
- `references/shared-ux/production-readiness.md` — when the user asks if their cache is ready for production, or after setup completes and the user wants to go to production
- `references/shared-ux/action-safety.md` — before any destructive action (risk levels, never-auto-execute list)
- `references/shared-ux/error-remediation.md` — when the user hits a specific ElastiCache error code (MOVED, CROSSSLOT, CLUSTERDOWN, MULTI/EXEC+IAM, etc.)
- `references/shared-foundation/boundary-doc.md` — when the user asks what this skill covers
- `references/shared-foundation/attribution.md` — when generating CLI commands, SDK code, or IaC templates
- `references/shared-foundation/architecture-diagrams.md` — when the user asks for architecture diagrams or visual reference
- `references/shared-runtime/lambda.md` — when connecting from Lambda (cold start gotchas, IAM auth code, lazy init)
- `references/shared-runtime/ecs.md` — when connecting from ECS (SIGTERM shutdown, connection pool drain, task definition)
- `references/shared-runtime/eks.md` — when connecting from EKS (IRSA, service mesh bypass, SecurityGroupPolicy CRD)
- `references/shared-runtime/api-gateway.md` — when integrating with API Gateway (no direct path, caching layers comparison)
- `references/shared-runtime/rds-acceleration.md` — when caching RDS/Aurora queries (thundering herd, stampede protection, invalidation)
- `references/shared-runtime/secret-injection.md` — when the user asks about credential management per compute platform
- `references/shared-security/encryption-defaults.md` — when adding encryption to an existing unencrypted cluster (TLS two-step migration, at-rest immutability)
- `references/shared-security/config-guardrails.md` — when the user wants continuous compliance monitoring (AWS Config rules, custom Lambda rules)
- `references/shared-security/vpc-patterns.md` — when debugging port/security-group issues (port 6380 serverless reader, anti-patterns)
> **Folder convention:** `references/` contains 10 folders. 6 match the sub-skills (`requirements`, `setup`, `data-modeling`, `genai`, `monitoring`, `migration`) and are routing destinations. The 4 `shared-*` folders (`shared-foundation`, `shared-ux`, `shared-security`, `shared-runtime`) are cross-cutting material loaded on demand, not routing destinations.
## Guardrails
| Priority | Rule |
|----------|------|
| CRITICAL | **Vector search MUST use node-based Valkey 8.2 or above.** Serverless does NOT support vector search. Never suggest serverless for vector search. Apply this regardless of which sub-skill activates. |
| CRITICAL | Do **not** invent price points or version constraints. Use `scripts/price_calculator.py` and current AWS docs when precision matters. |
| HIGH | Do **not** recommend Memcached when the user needs persistence, replication, RBAC or IAM auth, sorted sets, streams, pub/sub, or vector search. |
| HIGH | Do **not** assume local laptop access works directly. ElastiCache is VPC-centric; explain VPC, tunnel, or jump-host access when needed. |
| STANDARD | Do **not** trigger on every generic Redis mention. Trigger when the user is clearly asking about AWS, managed caching, migration, connectivity, pricing, operations, or AWS service integration. |
| STANDARD | For ambiguous "cache" requests inside AWS contexts, activate this skill and start with `requirements`. |
## Product truths
- ElastiCache Serverless deploys in under a minute and removes infrastructure management.
- Valkey serverless pricing is 33% lower than other supported engines; node-based Valkey pricing is 20% lower.
- Serverless caches have in-transit encryption always enabled (cannot be disabled).
- IAM auth is available for all ElastiCache Valkey versions (7.2 is the baseline Valkey version on ElastiCache) and Redis OSS 7.0+.
- Valkey version ladder: 7.2 (baseline), 8.0 (20% more data per node (capacity improvement), per-slot metrics), 8.1 (Bloom filters, COMMANDLOG, SET IFEQ, 20% less memory via new hash table (efficiency improvement)), 8.2 (vector search), 9.0 (recommended default for new clusters). Recommend Valkey 9.0 for new clusters unless a specific feature dictates otherwise.
- Vector search is available for Valkey 8.2 or above on node-based clusters (recommend 9.0).
- Global Datastore is available for node-based clusters only. It does not support IPv6 or Local Zones. Global Datastore supports AUTH and RBAC. Cross-region failover must be promoted manually (no autofailover across regions). At-rest encryption must be enabled on all clusters in the Global Datastore, but each cluster can use a separate KMS key per region.
- Online migration from self-managed Redis to ElastiCache requires: (source) AUTH must not be enabled, `protected-mode` set to `no`, replication and administrative commands must not be renamed (e.g., `sync`, `psync`, `info`, `config`, `command`, `cluster`); (target) encryption in-transit disabled, Multi-AZ enabled, engine version Redis OSS 5.0.6+ or Valkey 7.2+, not part of a Global Datastore, data tiering disabled. Shard counts must match between source and target. All source Redis instances must use the same port. Online migration is not supported for serverless caches (node-based targets only). See `references/migration/topology-validation.md` for the full checklist.