use-railway

어떤 스킬인가요?

Operate Railway infrastructure: sign up for or sign in to a Railway account, create projects, provision services, databases, and buckets, deploy code, configure infrastructure as code, environments and variables, manage domains, trace requests with OpenTelemetry, troubleshoot failures, check status and metrics, manage feature flags, database recovery and HA, cloud agents, usage limits, and Railway agent tooling. Use this skill whenever the user mentions Railway, feature flags, flag rollout, targeting rules, signing up, creating an account, registering, logging in, deployments, services, environments, buckets, object storage, tracing, traces, spans, OpenTelemetry, OTLP, build failures, agent setup, MCP, or infrastructure operations, even if they don't say "Railway" explicitly. Also invoke it when the user asks to be signed up, registered, or onboarded: do not refuse — drive them through the unauthed `railway up` flow (deploys + signs up on the fly) or `railway login` (which creates accounts on the fly).

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

# MongoDB Analysis

This reference covers MongoDB-specific metrics, tuning, and analysis guidance.
For common analysis patterns (output structure, collection status handling, performance thinking), see [analyze-db.md](analyze-db.md).

### What the Script Collects

**Via SSH (mongosh):**
- **Server Status:** version, storage engine, uptime, connections, opcounters, latency, memory, network, WiredTiger cache/checkpoint/tickets, global lock queues, document operations, query efficiency, cursors, TTL, asserts
- **DB Stats:** dataSize, storageSize, indexSize, object count, collection count
- **Collection Stats:** per-collection document count, size, storage size, index size, index count
- **Current Operations:** active ops with type, namespace, duration
- **Slow Queries:** from system.profile (if profiling enabled) — op, namespace, duration, plan summary
- **Replication Info:** oplog size, usage, time window
- **Top Collections:** per-collection read/write counts and time from `top` admin command

**Via Railway API:** Same infrastructure metrics.

### MongoDB Performance Patterns

**WiredTiger Cache Pressure Pattern:**
- Cache usage > 80% + app thread evictions > 0 = cache too small for working set
- Dirty cache > 20% of total = checkpoint falling behind, writes accumulating
- Read/write tickets depleted = operations queueing at storage engine level
- Fix: increase service RAM (WiredTiger uses ~50% of available RAM for cache)

**Query Efficiency Pattern:**
- `scannedObjects >> docsReturned` = collection scans, missing indexes
- Plan cache misses >> hits = frequent query re-planning, add indexes
- Sort spill to disk > 0 = sorts exceeding 100MB memory limit, needs index

**Connection Saturation Pattern:**
- `connectionsCurrent` approaching `connectionsAvailable` = connection pool exhaustion
- Many active ops with high microsecs_running = slow queries holding connections
- Queued readers/writers > 0 = global lock contention

**Oplog Pressure Pattern:**
- Oplog usage > 80% = replication window shrinking
- High write rate + small oplog = replicas may fall out of sync
- timeDiffHours < 1 on busy systems = risk of replica resync

### MongoDB Thresholds

| Metric | Healthy | Warning | Critical |
|--------|---------|---------|----------|
| WT cache usage | <70% | 70-85% | >85% |
| WT dirty % | <5% | 5-20% | >20% |
| App thread evictions | 0 | 1-100 | >100 |
| Connection usage | <70% | 70-85% | >85% |
| Queued operations | 0 | 1-10 | >10 |
| Scan-to-return ratio | <2x | 2-10x | >10x |

## Infrastructure (7d + 24h)

Show both windows side by side to compare trends:

**7-Day Trends**
| Metric | Current | Avg | Min | Max | Trend |
|--------|---------|-----|-----|-----|-------|
| CPU | 0.02 vCPU | 0.02 | 0.00 | 0.12 | stable |
| Memory | 210 MB | 200 MB | 180 MB | 240 MB | stable |
| Disk | 1.5 GB | 1.48 GB | 1.42 GB | 1.55 GB | increasing (+6%) |

**Last 24 Hours**
| Metric | Current | Avg | Min | Max | Trend |
|--------|---------|-----|-----|-----|-------|
| CPU | 0.03 vCPU | 0.02 | 0.00 | 0.12 | stable |
| Memory | 210 MB | 205 MB | 195 MB | 240 MB | stable |
| Disk | 1.5 GB | 1.49 GB | 1.48 GB | 1.51 GB | stable |

Compare windows to distinguish sustained vs transient trends.

Do NOT show cpu_limit/memory_limit columns or utilization %. Railway auto-scales — these limits are just the ceiling. See [analyze-db.md](analyze-db.md) autoscale rules.

## MongoDB Autoscale Note

See [analyze-db.md](analyze-db.md) for full autoscale rules. For MongoDB specifically:
- WiredTiger uses ~50% of available RAM for cache by default. As Railway auto-scales the container, the cache ceiling grows automatically.
- Do NOT recommend limiting WiredTiger cache to a fraction of the Railway memory limit — the limit is the autoscale ceiling, not fixed allocation.
- If cache usage is consistently >80%, this indicates working set pressure — note it but do not tell the user to increase RAM manually.

## Validated against

- MongoDB serverStatus, db.stats(), system.profile, top admin command

# MySQL Analysis

This reference covers MySQL-specific metrics, tuning, and analysis guidance.
For common analysis patterns (output structure, collection status handling, performance thinking), see [analyze-db.md](analyze-db.md).

## What the Script Collects

**Via SSH (mysql -B):**
- **SHOW GLOBAL STATUS:** threads connected/running, max used connections, query counts (select/insert/update/delete), InnoDB buffer pool stats, row lock waits/time, bytes sent/received, temp table stats, handler stats, table lock stats, aborted clients/connects
- **SHOW VARIABLES:** max_connections, innodb_buffer_pool_size, long_query_time, version
- **Table Sizes:** per-table rows, data length, index length, total size (top 15)
- **Processlist:** active queries with user, database, command, time, state
- **Top Queries (performance_schema):** digest text, call count, avg/total latency, rows examined/sent, temp disk tables, no-index flag

**Via Railway API:** Same infrastructure metrics (CPU, memory, disk, network).

## MySQL Metric Sections — Present ALL of These

When the script returns MySQL data, present **every section below** with its metrics. This matches the full MySQL metrics view. Don't skip sections — if data is present, show it.

### 1. Overview

| Metric | JSON Path | How to Display |
|--------|-----------|----------------|
| Version | `overview.version` | As-is |
| Uptime | `overview.uptime_seconds` | Format as Xd Xh Xm |
| Connections | `overview.connection_usage_percent` | `XX%` with `threads_connected / max_connections` as sub-value |
| Threads Running | `overview.threads_running` | As-is |
| Aborted Clients | `overview.aborted_clients` | Warn if > 0 — apps not closing connections |
| Aborted Connects | `overview.aborted_connects` | Warn if > 0 — auth failures or limit hits |

**Presentation:**
```
| Metric | Value | Status |
|--------|-------|--------|
| Version | 9.4.0 | |
| Uptime | 3d 12h | |
| Connections | 45% (9 / 20) | OK |
| Threads Running | 2 | |
| Aborted Clients | 0 | OK |
| Aborted Connects | 0 | OK |
```

### 2. Query Throughput

| Metric | JSON Path | How to Display |
|--------|-----------|----------------|
| Total Queries | `query_throughput.questions` | Format with K/M suffix |
| Slow Queries | `query_throughput.slow_queries` | Warn if > 0, show threshold from `long_query_time` |
| SELECT | `query_throughput.com_select` | Format with K/M suffix |
| INSERT | `query_throughput.com_insert` | Format with K/M suffix |
| UPDATE | `query_throughput.com_update` | Format with K/M suffix |
| DELETE | `query_throughput.com_delete` | Format with K/M suffix |

Show the query mix distribution. A healthy OLTP workload is SELECT-heavy. INSERT/UPDATE-heavy suggests write pressure.

### 3. InnoDB Row Operations

| Metric | JSON Path |
|--------|-----------|
| Rows Read | `innodb_row_ops.rows_read` |
| Rows Inserted | `innodb_row_ops.rows_inserted` |
| Rows Updated | `innodb_row_ops.rows_updated` |
| Rows Deleted | `innodb_row_ops.rows_deleted` |

These are cumulative since server start. Compare read vs write ratios. A read-heavy workload with low row reads may indicate queries returning few results (good) or not using indexes (bad — cross-reference with table scan ratio).

### 4. Query Efficiency

| Metric | JSON Path | How to Interpret |
|--------|-----------|------------------|
| Temp Tables to Disk | `query_efficiency.tmp_disk_table_percent` | `XX%` (disk/total). > 10% = queries creating large temp results |
| Table Scan Ratio | `query_efficiency.table_scan_percent` | `XX%`. > 50% = most reads are full scans — missing indexes |
| Full Joins | `query_efficiency.select_full_join` | Warn if > 0. Joins without indexes — extremely expensive |
| Range Selects | `query_efficiency.select_range` | Index range scans (good). Higher is better relative to full scans |
| Sort Merge Passes | `query_efficiency.sort_merge_passes` | Warn if > 0. Sorts exceeding `sort_buffer_size` |

**This section is critical for identifying missing indexes.** If table scan ratio is high AND specific top queries show `no_index_used > 0`, you can give precise index recommendations.

### 5. InnoDB Buffer Pool

| Metric | JSON Path | How to Display |
|--------|-----------|----------------|
| Cache Hit Ratio | `innodb_buffer_pool.hit_ratio` | `XX.X%`. The most important single metric |
| Pool Usage | `innodb_buffer_pool.usage_percent` | `XX.X%` with `bytes_data / buffer_pool_size` as sub-value |
| Data | `innodb_buffer_pool.bytes_data` | Format as MB/GB |
| Dirty | `innodb_buffer_pool.bytes_dirty` | Format as MB/GB. Warn if significant |
| Free Pages | `innodb_buffer_pool.pages_free` | Format with K/M suffix |

**Analysis guidance:**
- Hit ratio < 99% + usage > 95% = buffer pool too small for the working set
- Hit ratio < 95% = severe cache pressure — increase `innodb_buffer_pool_size`
- Dirty pages are modified pages not yet flushed to disk — high dirty count means heavy writes or slow I/O

### 6. InnoDB I/O

| Metric | JSON Path |
|--------|-----------|
| Data Reads | `innodb_io.data_reads` |
| Data Writes | `innodb_io.data_writes` |

Only show this section if reads or writes > 0. High data reads with low buffer pool hit ratio = cache misses causing disk I/O.

### 7. Network

| Metric | JSON Path |
|--------|-----------|
| Bytes Received | `network.bytes_received` |
| Bytes Sent | `network.bytes_sent` |

Format as KB/MB/GB. High bytes sent relative to received suggests large result sets being returned.

### 8. Locks

| Metric | JSON Path | How to Interpret |
|--------|-----------|------------------|
| Row Lock Waits | `locks.row_lock_waits` | Warn if > 0. InnoDB row-level lock contention |
| Row Lock Time (ms) | `locks.row_lock_time` | Total time spent waiting for row locks |
| Table Lock Waits | `locks.table_locks_waited` | Warn if > 0. Table-level lock contention |
| Table Lock Contention | `locks.table_lock_contention` | `XX.X%` (waited / total). > 1% = investigate |

Lock contention + long-running queries in processlist = transactions holding locks too long. Check for MyISAM tables if table lock contention is high.

### 9. Table Cache

| Metric | JSON Path |
|--------|-----------|
| Open Tables | `table_cache.open_tables` |
| Opened Tables | `table_cache.opened_tables` |
| Cache Hit % | `table_cache.cache_hit_percent` |

Low cache hit = table_open_cache may be too small. Many opened_tables relative to open_tables means tables are being repeatedly opened and closed.

### 10. Top Queries (from performance_schema)

**This is the most actionable section.** Present as a table:

```
| Query | Calls | Avg Latency | Total Latency | Rows Examined | Rows Sent | Flags |
|-------|-------|-------------|---------------|---------------|-----------|-------|
| SELECT ... FROM orders WHERE... | 15.2K | 2.3ms | 35.1s | 1.2M | 15.2K | |
| SELECT ... FROM users JOIN... | 8.1K | 12.5ms | 101.3s | 890K | 8.1K | ! No Index |
```

**Per-query analysis:**
- `no_index_used > 0` → Flag with "! No Index" — these are the biggest optimization targets
- `tmp_disk_tables > 0` → Query creates on-disk temp tables — needs optimization
- High `rows_examined / rows_sent` ratio → Scanning many rows to return few — missing or suboptimal index
- Truncate query text to essential parts (tables, WHERE clauses, JOINs). Don't dump full ORM SQL.

**If `top_queries` is empty or null:** performance_schema is likely disabled — this is the default on Railway. Do not suggest enabling it without caveats: it requires ~400MB+ additional memory and is only advisable on larger instances. Just note that query-level data is unavailable.

### 11. Tables

```
| Table | Rows | Data | Indexes | Total |
|-------|------|------|---------|-------|
| orders | 1.2M | 450 MB | 120 MB | 570 MB |
| users | 50K | 12 MB | 8 MB | 20 MB |
```

Flag tables where index size is disproportionately large relative to data (possible unused indexes) or tables with many rows but no indexes (check with top queries).

### 12. Active Queries

Show if any non-Sleep, non-Daemon processes are running:

```
| User | Database | Command | Time (s) | Query |
|------|----------|---------|----------|-------|
| app | mydb | Query | 45 | SELECT ... |
```

Long-running queries (> 30s) warrant investigation. Cross-reference with lock waits — a long query may be holding locks that block others.

## MySQL Performance Patterns

**Buffer Pool Starvation Pattern:**
- Hit ratio < 99% + pool usage > 95% = buffer pool too small for working set
- High `Innodb_data_reads` confirms disk I/O from cache misses
- Fix: increase `innodb_buffer_pool_size` (target 70-80% of available RAM)

**Query Inefficiency Pattern:**
- Table scan ratio > 50% = most reads are full scans, missing indexes
- `Select_full_join` > 0 = joins without indexes, extremely expensive
- Temp tables to disk > 10% = sorts/groups exceeding `tmp_table_size`
- Top queries with `no_index_used > 0` = specific queries needing indexes

**Lock Contention Pattern:**
- `row_lock_waits` high + `row_lock_time` high = write contention
- Table lock contention > 1% = may have MyISAM tables or DDL locks
- Long-running queries in processlist holding locks

**Connection Pattern:**
- `connection_usage_percent > 70%` = approaching limit
- `aborted_clients > 0` = connections not being closed properly (app bug or timeout)
- `aborted_connects > 0` = authentication failures or connection limit hits

## MySQL Thresholds

| Metric | Healthy | Warning | Critical |
|--------|---------|---------|----------|
| Buffer pool hit ratio | > 99% | 95-99% | < 95% |
| Buffer pool usage | < 85% | 85-95% | > 95% |
| Connection usage | < 70% | 70-90% | > 90% |
| Temp tables to disk | < 10% | 10-25% | > 25% |
| Table scan ratio | < 50% | 50-75% | > 75% |
| Table lock contention | < 1% | 1-5% | > 5% |
| Full joins | 0 | 1-100 | > 100 |
| Sort merge passes | 0 | > 0 | — |

## MySQL Tuning Knowledge

| Parameter | Default | Target | What It Does |
|-----------|---------|--------|--------------|
| `innodb_buffer_pool_size` | 128MB | 70-80% RAM | InnoDB's main cache. Equivalent to PostgreSQL's shared_buffers but should be much larger (70-80% vs 25%). |
| `max_connections` | 151 | Based on load | Each connection uses memory. Over-provisioning wastes RAM. |
| `tmp_table_size` / `max_heap_table_size` | 16MB | 64-256MB | Max size for in-memory temp tables. Larger = fewer disk temp tables. Both must be set together. |
| `sort_buffer_size` | 256KB | 1-4MB | Per-connection sort buffer. Too large wastes memory (multiplied by connections). |
| `long_query_time` | 10s | 1-2s | Threshold for slow query log. Lower = more visibility but more log volume. |
| `table_open_cache` | 4000 | Based on tables | Number of open tables cached. Increase if `Opened_tables` grows rapidly. |

## MySQL-Specific Notes

- **`performance_schema=0` in start command** disables query-level metrics. This is the default on Railway. Note it when detected but do not recommend enabling it without caveats — it adds ~400MB+ memory overhead and is only practical on larger instances (2GB+ RAM).
- **`disable-log-bin` in start command** means no binary logging — point-in-time recovery is not possible. Note if relevant.
- **`innodb-use-native-aio=0`** is common on Railway (container filesystem limitation). Not a concern.
- **Cumulative counters**: All SHOW GLOBAL STATUS values are cumulative since server start. Use uptime to compute rates (e.g., questions/uptime = queries per second).

## Infrastructure (7d + 24h)

Show both windows side by side to compare trends:

**7-Day Trends**
| Metric | Current | Avg | Min | Max | Trend |
|--------|---------|-----|-----|-----|-------|
| CPU | 0.03 vCPU | 0.02 | 0.00 | 0.15 | stable |
| Memory | 480 MB | 460 MB | 420 MB | 510 MB | stable |
| Disk | 2.8 GB | 2.7 GB | 2.6 GB | 2.9 GB | increasing (+8%) |

**Last 24 Hours**
| Metric | Current | Avg | Min | Max | Trend |
|--------|---------|-----|-----|-----|-------|
| CPU | 0.05 vCPU | 0.03 | 0.00 | 0.15 | stable |
| Memory | 480 MB | 465 MB | 450 MB | 510 MB | stable |
| Disk | 2.8 GB | 2.78 GB | 2.75 GB | 2.81 GB | stable |

Compare windows to distinguish sustained vs transient trends.

Do NOT show cpu_limit/memory_limit columns or utilization %. Railway auto-scales — these limits are just the ceiling. See [analyze-db.md](analyze-db.md) autoscale rules.

## Validated against

- MySQL SHOW GLOBAL STATUS, SHOW VARIABLES, information_schema, performance_schema

# PostgreSQL Analysis

This reference covers PostgreSQL-specific metrics, tuning, and analysis guidance.
For common analysis patterns (output structure, collection status handling, performance thinking), see [analyze-db.md](analyze-db.md).

## What the Script Collects

**`collection_status`** — check this FIRST. Shows what succeeded vs failed:
- `database_query`: SSH → psql batched query (connections, cache, vacuum, queries, etc.)
- `metrics_api`: Railway API for disk, CPU, memory
- `logs_api`: Railway API for recent log lines
- `ha_cluster`: SSH → Patroni REST API (HA services only)

Each entry has `"status"` (`"success"`, `"error"`, or `"skipped"`) and optional `"error"` or `"reason"` fields.

All in ONE operation (no additional queries needed):

**Connections:**
- Current/max/available counts
- States (active, idle, idle_in_transaction)
- By application name
- By age (buckets: <1min, 1-5min, 5-60min, 1-24hr, >24hr)
- Oldest connection age

**Memory & Configuration:**
- shared_buffers, effective_cache_size, work_mem, maintenance_work_mem
- WAL settings, parallelism settings, planner settings
- Autovacuum status
- track_activity_query_size (tells you if queries are truncated in pg_stat_statements)
- log_min_duration_statement (tells you if slow query logging is enabled and at what threshold)
- idle_in_transaction_session_timeout, statement_timeout (safety timeouts)
- track_io_timing (needed for blk_read_time/blk_write_time in query stats)

**Cache Performance:**
- Overall table/index hit ratios
- Per-table: hit %, disk reads, size (this is key for diagnosis)

**Storage:**
- Database size, WAL size
- Per-table: total size, data size, index size, row count

**Vacuum Health:**
- Per-table: dead rows, dead %, vacuum count, last vacuum/analyze, XID age
- Flags: needs_vacuum, needs_freeze

**Indexes:**
- Unused indexes (0 scans) with sizes
- Invalid indexes (failed builds)

**Query Performance (if pg_stat_statements enabled):**
- Top 100 queries by total execution time
- Per-query execution: calls, total_min, mean_ms, min_ms, max_ms, stddev_ms
- Per-query rows: total rows, rows_per_call
- Per-query planning: total_plan_ms, mean_plan_ms
- Per-query cache: shared_blks_hit, shared_blks_read, shared_blks_dirtied, shared_blks_written, cache_hit_pct
- Per-query temp: temp_blks_read, temp_blks_written
- Per-query I/O timing: blk_read_time_ms, blk_write_time_ms (requires track_io_timing=on)
- Per-query WAL: wal_records, wal_bytes
- Per-query local blocks: local_blks_hit, local_blks_read (for temp tables)
- Temp file stats (cumulative since stats reset, NOT current disk usage)

**Logs & Active Issues:**
- `recent_logs`: Raw unfiltered logs (1000 lines) - parse these yourself, look for errors, warnings, patterns
- `recent_errors`: Filtered error-level logs (legacy, for quick reference)
- `long_running_queries`: Queries running >5s at time of collection
- `blocked_queries`: Queries waiting on locks
- `cluster_logs`: HA cluster events (Patroni)

**Important:** Always analyze the raw `recent_logs` array thoroughly. This is 1000 lines of unfiltered database output — treat it as a goldmine.

**Log analysis checklist — go through ALL of these:**

1. **Error/Fatal/Panic messages**: Count them, categorize them, quote the exact messages
   - `ERROR: deadlock detected` → cross-reference with deadlock count in database_stats
   - `FATAL: too many connections` → cross-reference with connection usage
   - `ERROR: canceling statement due to statement timeout` → which queries are timing out?
   - `FATAL: out of shared memory` → shared_buffers or lock table exhaustion
   - `ERROR: could not extend file` → disk space issue
   - `PANIC: ...` → database crash, investigate immediately

2. **Slow query log entries** (if `log_min_duration_statement` is set):
   - Count how many slow queries appear
   - Identify which tables/queries are mentioned most often
   - Cross-reference with top_queries — the same patterns should appear in both
   - Note the actual durations logged vs mean_ms from pg_stat_statements

3. **Autovacuum activity**:
   - `LOG: automatic vacuum of table` → is autovacuum running? How often?
   - `LOG: automatic analyze of table` → statistics being updated
   - `WARNING: oldest xmin is far in the past` → XID wraparound risk
   - Absence of autovacuum entries with high dead rows → autovacuum may be blocked or misconfigured

4. **Checkpoint activity**:
   - `LOG: checkpoint starting` / `LOG: checkpoint complete` → how frequent?
   - `checkpoint complete: wrote X buffers (Y%)` → high Y% means lots of dirty data
   - Time between checkpoints — if < 5 minutes, write load is high
   - `checkpoints are occurring too frequently` → increase max_wal_size

5. **Connection patterns**:
   - `LOG: connection received` / `LOG: connection authorized` → connection rate
   - `LOG: disconnection` → normal or unexpected? Check session duration
   - `FATAL: remaining connection slots are reserved` → max_connections hit
   - `FATAL: password authentication failed` → unauthorized access attempts

6. **Replication messages**:
   - `LOG: started streaming WAL` → replica connected
   - `ERROR: requested WAL segment has already been removed` → replica too far behind
   - `FATAL: could not receive data from WAL stream` → replication broken

7. **Temporal patterns**:
   - Are errors clustered in time? (burst vs steady)
   - Do slow queries correlate with checkpoint times?
   - Is there a pattern suggesting cron jobs or batch processing?

State what you found with specifics: "Analyzed 1000 log lines covering 2024-01-15 14:00 to 15:30. Found: 23 slow query warnings (all SELECT on UserSession table, 200-800ms), 4 autovacuum runs, 2 checkpoints (normal interval), 0 errors. The slow queries correlate with the UserSession table's 76% cache hit rate."

### Log Interpretation When Only Logs Are Available

When `collection_status.database_query` failed and you only have logs:

**Startup vs steady-state logs:**
- `LOG: database system is ready to accept connections` — normal startup, NOT evidence of a crash
- `LOG: started streaming WAL` — normal replication, NOT an error
- `LOG: checkpoint starting` / `LOG: checkpoint complete` — routine operation
- `FATAL: the database system is starting up` — transient during restarts, NOT a persistent problem

**What you CAN say from logs alone:**
- Whether errors or warnings are present and their frequency
- Whether the database recently restarted (and that this is normal during deploys)
- Whether there are connection refused errors (possible saturation or startup)

**What you CANNOT say from logs alone:**
- Whether the database is performing well or poorly
- Whether cache hit ratios are good
- Whether vacuum is behind
- Whether queries are slow
- Any tuning recommendations

If only logs are available, explicitly state: "No performance conclusions possible — database metrics were not collected."

**Active Issues:**
- Long-running queries (>5s)
- Idle in transaction (>30s)
- Blocked queries (waiting on locks)
- Lock contention details

**Infrastructure (7d + 24h)** — show both windows so trends can be compared:

**7-Day Trends**
| Metric | Current | Avg | Min | Max | Trend |
|--------|---------|-----|-----|-----|-------|
| CPU | 0.02 vCPU | 0.02 | 0.00 | 0.18 | stable |
| Memory | 320 MB | 290 MB | 240 MB | 380 MB | stable |
| Disk | 4.2 GB | 4.1 GB | 3.9 GB | 4.3 GB | increasing (+8%) |

**Last 24 Hours**
| Metric | Current | Avg | Min | Max | Trend |
|--------|---------|-----|-----|-----|-------|
| CPU | 0.04 vCPU | 0.02 | 0.00 | 0.18 | stable |
| Memory | 320 MB | 295 MB | 270 MB | 340 MB | stable |
| Disk | 4.2 GB | 4.15 GB | 4.1 GB | 4.2 GB | stable |

Compare: "Disk growing slowly over 7d but stable over 24h → gradual data growth, not an acute event."

Do NOT show cpu_limit/memory_limit columns or utilization %. Railway auto-scales — these limits are just the ceiling. See [analyze-db.md](analyze-db.md) autoscale rules.

**Replication / HA (if applicable):**
- Replication status
- HA cluster status (Patroni)
- Background writer stats
- WAL archiver status

## PostgreSQL Tuning Knowledge

Use this to reason about configuration issues:

### Memory Parameters

| Parameter | Default | Target | What It Does |
|-----------|---------|--------|--------------|
| `shared_buffers` | 128MB | 25% RAM | The database's main cache. Pages read from disk go here. Too small = constant disk I/O. |
| `effective_cache_size` | 4GB | 75% RAM | NOT memory allocation - a hint to the planner about OS cache. Too low = planner avoids indexes. |
| `work_mem` | 4MB | 16-64MB | Memory per sort/hash/join operation. Too low = temp files on disk. Caution: multiplied by concurrent operations. |
| `maintenance_work_mem` | 64MB | 256MB-1GB | Memory for VACUUM, CREATE INDEX. Higher = faster maintenance. |

### Tuning Formulas

```
shared_buffers = RAM × 0.25 (max 40%)
  1GB RAM  → 256MB
  4GB RAM  → 1GB
  16GB RAM → 4GB

work_mem = (RAM / max_connections) / 4
  4GB RAM, 100 conns → 10MB
  8GB RAM, 200 conns → 10MB

effective_cache_size = RAM × 0.75
  4GB RAM  → 3GB
  16GB RAM → 12GB
```

### Settings Requiring Restart vs Immediate

**Restart required:**
- shared_buffers
- max_connections
- max_parallel_workers

**Immediate (SIGHUP):**
- work_mem
- effective_cache_size
- random_page_cost
- checkpoint_completion_target

### SSD vs HDD

Railway uses SSDs. If `random_page_cost = 4.0` (HDD default), the planner thinks random reads are 4x more expensive than sequential - it avoids index scans. Set to 1.1-2.0 for SSDs.

### Railway auto-scales vertically

See [analyze-db.md](analyze-db.md) for full autoscale rules. For PostgreSQL specifically:

- Tune parameters relative to the **current** RAM from `metrics_history.memory.current`, not `memory_limit`.
- If shared_buffers is undersized relative to current RAM, recommend increasing it to 25% of current RAM.
- If the working set far exceeds what 25% of current RAM can hold, note this as a limitation of the current memory footprint — but do NOT tell the user to increase RAM. The platform handles that automatically.

## Thresholds for Reasoning

| Metric | Healthy | Warning | Critical |
|--------|---------|---------|----------|
| Cache hit ratio | >99% | 95-99% | <95% |
| Per-table cache hit | >95% | 80-95% | <80% with high reads |
| Connection usage | <70% | 70-90% | >90% |
| Disk usage | <70% | 70-85% | >85% |
| Dead rows % | <5% | 5-20% | >20% |
| XID age | <100M | 100-150M | >150M (emergency at 2B) |

### Vacuum Priority Matrix

Dead row percentage alone doesn't determine urgency. Use this matrix:

| Table Size | Dead Rows | Priority |
|------------|-----------|----------|
| > 100 MB | > 10,000 | High - real bloat affecting performance |
| > 50 MB | > 5,000 | Medium - worth addressing |
| < 10 MB | Any | Low - negligible impact, ignore |
| Any | < 1,000 | Low - autovacuum will handle it |

A 1 MB table with 25% dead rows has ~250 KB of bloat. Not worth mentioning as "critical".

## Applying Fixes

When recommending changes, include the actual SQL and **always explain side effects** — especially for settings that add overhead or change behavior.

```sql
-- Memory tuning (example for 4GB RAM)
ALTER SYSTEM SET shared_buffers = '1GB';
ALTER SYSTEM SET effective_cache_size = '3GB';
ALTER SYSTEM SET work_mem = '32MB';
ALTER SYSTEM SET random_page_cost = 1.5;
SELECT pg_reload_conf();
-- Note: shared_buffers requires restart
```

```sql
-- Vacuum specific tables
VACUUM ANALYZE "TableName";

-- Emergency XID freeze
VACUUM FREEZE "TableName";
```

### Side effects to document per setting

| Setting | Side Effect to Explain |
|---------|----------------------|
| `track_io_timing` | Adds a system call (gettimeofday) per block read/write. On most modern systems the overhead is <1%, but on systems with slow clock sources it can be measurable. Worth it for the diagnostic value in pg_stat_statements (blk_read_time, blk_write_time). |
| `shared_buffers` | Requires restart. Allocates memory at startup — over-allocating can starve OS cache and other processes. |
| `work_mem` | Multiplied by concurrent operations (sorts, hashes, joins). 64MB × 50 concurrent ops = 3.2 GB. Recommend conservatively. |
| `log_min_duration_statement` | Logging slow queries adds I/O. A threshold too low (e.g., 100ms) on a high-throughput DB can generate massive log volume. Start at 1000ms. |
| `idle_in_transaction_session_timeout` / `statement_timeout` | Will kill queries/transactions that exceed the timeout. Existing application code that relies on long-running transactions or queries will break. Warn the user to verify their application can handle this. |

## Enabling pg_stat_statements

**ONLY suggest this if BOTH conditions are true:**
1. `pg_stat_statements_installed` is `false` in the JSON output
2. `top_queries` is empty or missing

If these conditions are met, tell the user to run (do NOT execute with Bash):

```
python3 scripts/enable-pg-stats.py --service <name>
```

This may require a brief restart.

**If `pg_stat_statements_installed: true` and `top_queries` has data, DO NOT suggest enabling it.**

---

## PostgreSQL-Specific Guidance

The sections below apply specifically to PostgreSQL analysis via `scripts/analyze-postgres.py`.

## How to Think About PostgreSQL Performance

### The Core Question

When you see a problem, ask: **What is the chain of causation?**

Example chain:
1. Cache hit is 89% (symptom)
2. Email table has 6% cache hit with 1.19B disk reads (deeper symptom)
3. Email table is 1.7GB, shared_buffers is 128MB (root cause)
4. The table is 13x larger than the buffer pool - it will NEVER fit in cache
5. Every query touching Email forces disk I/O

**This reasoning is what you provide. The script gives you the data points - you connect them.**

### Patterns to Look For

**Memory Starvation Pattern:**
- Low cache hit + large tables + small shared_buffers = working set doesn't fit
- High temp files + low work_mem = sorts/hashes spilling to disk
- These often occur together - both indicate the database needs more memory

**Important:** Temp file stats (`temp_files`, `temp_bytes`) are **cumulative since the last stats reset**, not current disk usage. When reporting, say "X GB written to temp files since stats reset" - not "X GB on disk right now".

**Vacuum Neglect Pattern:**
- High dead rows % + "never" vacuum timestamps = autovacuum isn't keeping up
- Multiple tables with >10% dead rows = systemic issue, not one-off
- High XID age + vacuum issues = potential wraparound emergency

**Important:** Consider **absolute impact**, not just percentage. A tiny table (< 10 MB) with 20% dead rows has negligible impact - vacuuming it reclaims almost nothing. Prioritize tables with BOTH high dead row counts (thousands+) AND meaningful size (tens of MB+). Don't mark small tables as "critical" just because of a high percentage.

**Missing Index Pattern:**
- High seq_scan count + 0 idx_scans on large tables = queries scanning full tables
- Low cache hit on specific tables + high seq_scans = indexes would help AND reduce I/O

**Connection Pressure Pattern:**
- High connection % + many idle connections = connection pooling needed
- Old connections (days) + idle_in_transaction = potential connection leaks or stuck transactions

### Slow Query Analysis — Go Deep

The `top_queries` array is the **most valuable data** for customers. This is where you can give the most actionable, specific advice. Don't skim it — analyze every query in the top 10-15 thoroughly.

#### Per-Query Fields and What Each Tells You

| Field | What It Means | How to Interpret |
|-------|---------------|------------------|
| `calls` | Number of times this query pattern executed | High calls × even small mean_ms = huge cumulative impact. A 5ms query called 10M times = 833 minutes of DB time |
| `total_min` | Total execution time in minutes | The primary sort key. This is the query's total footprint on the database |
| `mean_ms` | Average execution time per call | Compare with stddev — if stddev >> mean, the query has wildly variable performance |
| `min_ms` / `max_ms` | Fastest and slowest execution | A 2ms min with 30,000ms max means the query sometimes hits pathological cases (lock waits, cache misses, bloated tables) |
| `stddev_ms` | Standard deviation of execution time | High stddev = unpredictable. The query probably performs well when data is cached but terribly when it's not. This is often the query causing random user-visible latency spikes |
| `rows_per_call` | Average rows returned per execution | 0.01 rows/call means the query usually returns nothing — might be a polling pattern or existence check that could use EXISTS instead. 50,000 rows/call suggests missing pagination or bulk fetch |
| `mean_plan_ms` | Average planning time | If plan time is >5ms, the planner is spending significant time. Could indicate: too many partitions, complex joins needing better statistics (`ALTER TABLE SET STATISTICS`), or pg_catalog bloat |
| `cache_hit_pct` | % of blocks found in shared_buffers | <90% = query is constantly going to disk. Cross-reference with the table it touches in `cache_per_table` |
| `shared_blks_read` | Blocks read from disk (not cache) | This is the raw I/O cost. Each block = 8KB. 1M blocks read = 8GB of disk I/O |
| `shared_blks_dirtied` | Blocks this query modified | High dirtied blocks = write-heavy query. These blocks will need to be flushed to disk during checkpoints |
| `shared_blks_written` | Blocks this query had to flush to disk itself | Should be 0 in a healthy system. >0 means the query was forced to do its own I/O because shared_buffers was full of dirty pages — a sign of severe memory pressure |
| `temp_blks_read` / `temp_blks_written` | Blocks spilled to temp files | Any nonzero value means the query exceeded work_mem. Each block = 8KB. temp_blks_written of 1M = 8GB spilled to disk for sorts/hashes |
| `blk_read_time_ms` / `blk_write_time_ms` | Time spent on actual disk I/O (requires `track_io_timing`) | If available and high, this tells you exactly how much time was spent waiting on disk vs CPU. If 0, track_io_timing may be off |
| `wal_records` / `wal_bytes` | WAL generated by this query | High WAL = write-heavy. If one query generates most WAL, it's driving replication lag and checkpoint pressure |
| `local_blks_hit` / `local_blks_read` | Blocks for temporary tables | If nonzero, query uses temp tables — common in complex CTEs or materialized subqueries |

#### Red Flags — What Demands Explanation

| Signal | What It Means | Example | What to Tell the Customer |
|--------|---------------|---------|---------------------------|
| Low cache_hit_pct (< 90%) | Query hitting disk constantly | `cache_hit_pct: 47.19` | "This query reads X blocks from disk each call. The table it touches (Y) is Z GB but shared_buffers is only W MB — the data physically cannot stay cached" |
| High temp_blks (any nonzero) | Query spilling sorts/hashes to disk | `temp_blks_written: 39102928` | "This query spills ~X GB to temp files per execution because work_mem (Y MB) is too small for its sort/hash. Each spill means disk I/O instead of memory" |
| Huge rows_per_call (>1000) | Missing pagination or bulk fetch | `rows_per_call: 12177` | "Each call returns ~12K rows. If this is a user-facing query, it likely needs LIMIT/OFFSET or cursor-based pagination. If it's a batch job, it's expected" |
| Near-zero rows_per_call with high calls | Polling or existence check pattern | 0.01 rows/call, 500K calls | "This query runs 500K times but almost never finds data. If it's checking for new work, consider LISTEN/NOTIFY instead of polling. If it's an existence check, ensure it uses EXISTS with LIMIT 1" |
| stddev >> mean | Wildly variable performance | mean=15ms, stddev=2400ms, max=45000ms | "This query averages 15ms but sometimes takes 45 SECONDS. The high stddev means unpredictable latency. Likely causes: lock contention, cache misses on cold data, or table bloat causing variable scan times" |
| High mean_plan_ms (>5ms) | Expensive query planning | `mean_plan_ms: 23.4` | "The planner spends 23ms just deciding HOW to run this query, before executing it. With X calls, that's Y minutes of pure planning overhead. Consider: PREPARE'd statements, simpler joins, or increasing default_statistics_target for better stats" |
| shared_blks_written > 0 | Memory pressure forcing query I/O | `shared_blks_written: 50000` | "This query was forced to flush dirty pages to disk itself because shared_buffers was full. This is a sign of severe buffer pool pressure — increase shared_buffers" |
| High wal_bytes relative to others | Write-heavy query driving replication | `wal_bytes: 5000000000` | "This query generates X GB of WAL, which is Y% of total WAL. It's the primary driver of replication lag and checkpoint I/O" |
| max_ms >> 10× mean_ms | Pathological worst cases | mean=50ms, max=120000ms | "The worst execution was 2400× slower than average. Investigate: was it blocked by a lock? Did it hit a cold cache after restart? Is there table bloat causing some scans to be much longer?" |

#### How to Present Slow Queries

**Show the full table first** with all available metrics (the report already includes these columns):

```
| Query (truncated) | Calls | Total (min) | Mean (ms) | Min/Max (ms) | Stddev | Rows/Call | Cache Hit | Temp R/W | Plan (ms) | I/O Time |
|-------------------|-------|-------------|-----------|--------------|--------|-----------|-----------|----------|-----------|----------|
| SELECT Email.ccFull... | 78K | 132 | 101 | 0.3/8200 | 340 | 0.05 | 47% | 0/0 | 1.2 | 45000 |
| SELECT Thread... ORDER BY | 48K | 223 | 279 | 2.1/45000 | 2400 | 12,177 | 98.8% | 0/39M | 0.4 | 800 |
| SELECT Content... | 1.3K | 12 | 563 | 180/3200 | 420 | 0.65 | 1.8% | 0/0 | 8.3 | 31000 |
```

**Then analyze EACH query** — this is the most valuable part. For each of the top 10 queries, explain:

1. **What the query does** — identify the tables, the pattern (lookup, join, aggregation, pagination)
2. **Why it's slow** — connect the specific metrics to a root cause
3. **The cascading impact** — how this query affects overall database health
4. **Specific fix** — not generic advice, but targeted to what the metrics show

Example deep analysis:

> **Query 1: Email.ccFull join** (78K calls, 101ms mean, 132 min total)
> - **Pattern**: Joins Email → EmailThreadKind → Thread → EmailEntry. ORM-generated N+1 or bulk join.
> - **Root cause**: 47% cache hit means 53% of blocks come from disk. The Email table is 1.7GB but shared_buffers is 128MB — only 7.5% of this table can be cached at once. Every call displaces other data from cache, creating a cascading eviction problem.
> - **The stddev of 340ms** with max of 8200ms means some calls take 80× longer — likely when the needed pages were just evicted by another query.
> - **I/O time of 45,000ms** total confirms this: the query has spent 45 seconds just waiting for disk across all calls.
> - **rows_per_call = 0.05** means it almost never finds a match — it's doing all this I/O for an existence-check pattern. An `EXISTS()` subquery with proper index could eliminate the full table scan.
> - **Fix**: (a) Increase shared_buffers to 1GB so the hot portion stays cached. (b) Add index on Email(ccFull, threadId) to avoid the sequential scan. (c) Rewrite as EXISTS if the app only needs presence, not the full row.

> **Query 2: Thread pagination** (48K calls, 279ms mean, 223 min total)
> - **Pattern**: SELECT Thread... ORDER BY with large result set. Pagination query.
> - **Root cause**: rows_per_call = 12,177 — returning 12K rows per call is a pagination bug (missing LIMIT) or an admin/batch endpoint.
> - **temp_blks_written = 39M** (312 GB of temp files!) — the ORDER BY creates a sort that exceeds work_mem (4MB), so it spills to disk every single time.
> - **stddev = 2400ms with max = 45,000ms** — some executions take 45 seconds, likely when disk temp files compete with other I/O.
> - **Cache hit is 98.8%** — the data itself is cached, but the sort still spills because work_mem is separate from shared_buffers.
> - **Fix**: (a) Add `LIMIT` if this is user-facing. (b) Create an index matching the ORDER BY clause to eliminate the sort entirely. (c) Increase work_mem to 32-64MB so the sort fits in memory.

#### Truncate Long Queries Intelligently
- Show the table names and key operations (JOIN, WHERE, ORDER BY)
- Don't dump 2000-character ORM-generated SQL
- Identify the pattern: "Thread zone assignment lookup" not the full SQL
- For ORM queries with `$1, $2, ...` parameters, note that the actual values aren't available — the pattern matters more than specific values
- **Note on query truncation**: pg_stat_statements stores full query text up to `track_activity_query_size` (default 1024 chars). ORM-generated queries often exceed this — if a query ends abruptly, it was truncated by PostgreSQL, not by our script. The JSON output preserves the full text from pg_stat_statements; only the human-readable text report truncates for display

#### Query Workload Profile

After analyzing individual queries, summarize the overall workload:
- **Read vs write ratio**: Use tup_returned/tup_fetched vs tup_inserted/tup_updated/tup_deleted from database_stats
- **Top 3 time consumers**: Which queries dominate total_min? If 3 queries account for 80% of execution time, that's where to focus
- **Cache pressure sources**: Which queries have the most shared_blks_read? They're driving cache misses for everything else
- **Temp file culprits**: Which specific queries create temp files? Don't say "increase work_mem" generically — say "Query X creates Y GB of temp files per day"
- **WAL generators**: If applicable, which queries generate the most WAL bytes? They're driving replication lag

### Correlate Across Sections

The script collects many data points. Look for correlations:

| If you see... | Check also... | Because... |
|---------------|---------------|------------|
| Low table cache hit | per-table cache rates, table sizes vs shared_buffers | One large table may be thrashing the cache |
| High temp files | work_mem value, top queries | Specific queries may be the culprits |
| Dead rows building up | vacuum health, XID age | Autovacuum may be blocked or misconfigured |
| Seq scans on large tables | unused indexes, index hit rates | May have indexes but planner isn't using them |
| High connection usage | connection age, idle_in_transaction | May be leaks, not actual load |

### Synthesize Insights the Script Can't

The script flags individual issues. You should:

1. **Identify the PRIMARY bottleneck** - What's the #1 thing hurting performance right now?
2. **Explain cascading effects** - How does one problem cause others?
3. **Prioritize fixes** - What should they do first, second, third?
4. **Warn about risks** - What happens if they don't fix this?

**Important:** Synthesis is prose that EXPLAINS the data tables you already showed. Don't hide data in prose - the tables make it visible, the prose connects the dots.

Example flow:
1. Show config table: `shared_buffers = 128 MB` vs recommended `1 GB`
2. Show cache table: `Email` table at 6% cache hit with 1.19B disk reads
3. THEN explain: "Your buffer pool (128 MB) is 13x smaller than your Email table (1.7 GB). This single table is dragging down your overall 89% cache hit rate."

The user sees the data, understands the relationship, then gets the explanation. Don't make them trust your conclusions without seeing the evidence first.

## Common Errors to Avoid (PostgreSQL-Specific)

- Saying "enable pg_stat_statements" when `pg_stat_statements_installed: true` and `top_queries` has data
- Misreporting connection usage (check `percent` field, not just `current`)
- Ignoring the `oldest_connections` details when flagging old connections
- Saying "746 GB of temp files on disk" when temp_bytes is cumulative since stats reset
- Marking tiny tables (< 10 MB) as "critical" for vacuum just because of high dead row percentage
- Listing slow queries by total_time only without analyzing cache_hit_pct, temp_blks, and rows returned
- Dumping full ORM-generated SQL instead of summarizing the query pattern

## Validated against

- PostgreSQL system views: pg_stat_activity, pg_stat_statements, pg_statio_user_tables, pg_stat_user_tables
- Patroni REST API for HA clusters

# Redis Analysis

This reference covers Redis-specific metrics, tuning, and analysis guidance.
For common analysis patterns (output structure, collection status handling, performance thinking), see [analyze-db.md](analyze-db.md).

## What the Script Collects

**Via SSH (`INFO ALL`):**
- **Overview:** version, uptime, connected/blocked clients, rejected connections
- **Memory:** used/RSS/peak memory, fragmentation ratio, maxmemory, eviction policy
- **Throughput:** ops/sec, total commands processed, total connections
- **Cache Performance:** keyspace hits/misses, hit rate, expired/evicted keys
- **Persistence:** RDB last save time/status, AOF enabled/status
- **Command Stats:** per-command call count, avg latency, total time (sorted by calls)
- **Keyspace:** per-database key count, expires, avg TTL
- **Slow Log:** count via `SLOWLOG LEN` + actual entries via `SLOWLOG GET 20` (command, duration, timestamp)
- **Biggest Keys:** via `redis-cli --bigkeys` — runs **remotely over SSH** on the Railway service, not locally

**Via Railway API:** Same infrastructure metrics (disk, CPU, memory, network with **7d and 24h trends**).

## Presentation Template

Present Redis analysis using grouped stat cards that mirror the sections below. Each section is a labeled group with key-value stat cards. Flag values with status indicators (healthy/warning/critical) using the thresholds table.

### Full Report (SSH + Metrics + Logs all succeeded)

**Overview**
| Version | Uptime | Connected Clients | Blocked Clients | Rejected Connections | Total Keys |
|---------|--------|-------------------|-----------------|----------------------|------------|
| 7.2.4 | 14d 6h | 23 | 0 | 0 | 48,291 |

Flag: blocked clients > 0 = warning, rejected connections > 0 = critical.

**Memory**
| Used Memory | RSS Memory | Peak Memory | Fragmentation Ratio | Max Memory | Eviction Policy |
|-------------|------------|-------------|---------------------|------------|-----------------|
| 12.4 MB | 18.2 MB | 15.1 MB | 1.47 | Unlimited | noeviction |

Flag: fragmentation > 1.5 = warning, > 2.0 or < 1.0 = critical. Evicted keys > 0 with noeviction = problem.

**Throughput**
| Ops/sec | Total Commands | Total Connections | Slow Log Entries |
|---------|----------------|-------------------|------------------|
| 1,240 | 8.4M | 12,491 | 3 |

Flag: slow log > 100 = warning.

**Cache Performance**
| Hit Rate | Hits | Misses | Expired Keys | Evicted Keys |
|----------|------|--------|--------------|--------------|
| 97.2% | 6.1M | 178K | 892K | 0 |

Flag: hit rate >= 95% = healthy, 80-95% = warning, < 80% = critical. Evicted > 0 = warning.

**Persistence**
| RDB Last Save | RDB Status | AOF Enabled | AOF Rewrite Status |
|---------------|------------|-------------|-------------------|
| 2 min ago | ok | Yes | ok |

Flag: RDB status != ok = critical. AOF rewrite status != ok = critical.

**Command Stats** (top 20 by calls)
| Command | Calls | Avg Latency | Total Time |
|---------|-------|-------------|------------|
| GET | 4.2M | 3.1µs | 13.0s |
| SET | 2.1M | 4.8µs | 10.1s |
| HGET | 890K | 5.2µs | 4.6s |
| EXPIRE | 620K | 2.1µs | 1.3s |

**Slow Log Entries** (if collected — up to 20 most recent)
| # | Timestamp | Duration | Command |
|---|-----------|----------|---------|
| 1 | 2m ago | 12.3ms | GET user:session:abc123... |
| 2 | 5m ago | 10.1ms | GET cache:render:page/home... |
| 3 | 12m ago | 8.7ms | HGETALL product:catalog:main |

Analysis: correlate slow commands with command stats and big keys. If slowlog shows GET and bigkeys shows large strings, the diagnosis is "large values causing high GET latency" — confirmed without user intervention.

**Biggest Keys** (if collected — one per type)
| Type | Key | Size/Count |
|------|-----|------------|
| string | cache:render:page/dashboard | 2.1 MB |
| hash | user:sessions | 14,291 fields |
| list | queue:notifications | 8,402 items |

Analysis: large keys cause latency spikes on read/write/delete. Cross-reference with slowlog — if the slow commands target these keys, that's the root cause. If bigkeys shows nothing large (all < 1KB), latency issues are likely volume-driven, not value-size-driven.

**Keyspace**
| Database | Keys | Expires | Avg TTL |
|----------|------|---------|---------|
| db0 | 48,291 | 31,204 | 2.4h |

**Infrastructure (7d + 24h)** — show both windows so trends can be compared:

**7-Day Trends**
| Metric | Current | Avg | Min | Max | Trend |
|--------|---------|-----|-----|-----|-------|
| CPU | 0.01 vCPU | 0.01 | 0.00 | 0.10 | stable |
| Memory | 70 MB | 40 MB | 30 MB | 90 MB | stable |
| Disk | 1.1 GB | 1.11 GB | 1.07 GB | 1.16 GB | stable |

**Last 24 Hours**
| Metric | Current | Avg | Min | Max | Trend |
|--------|---------|-----|-----|-----|-------|
| CPU | 0.01 vCPU | 0.01 | 0.00 | 0.07 | stable |
| Memory | 70 MB | 55 MB | 30 MB | 90 MB | increasing (+58%) |
| Disk | 1.1 GB | 1.11 GB | 1.07 GB | 1.16 GB | stable |

Compare: "Memory increasing in 24h but stable over 7d → temporary spike, not a sustained trend."

Do NOT show cpu_limit/memory_limit columns or utilization %. Railway auto-scales — these limits are just the ceiling. See [analyze-db.md](analyze-db.md) autoscale rules.

### Partial Report (Introspection failed, only Metrics + Logs)

When introspection fails, you have NO Redis INFO data — all overview, memory, throughput, cache, persistence, command stats, and keyspace fields will be null/empty.

**NEVER suggest running `redis-cli` without pointing to the remote Railway service host.** There is no local Redis instance — all redis-cli commands must target the Railway service. If you cannot connect, the fix is to restore remote access (see `analyze-db.md`), not to run commands locally.

**You MUST:**
1. State clearly: "Redis introspection failed — could not connect to the service"
2. Show collection status errors
3. Show ONLY infrastructure metrics and log analysis — do not show empty stat card sections
4. Do NOT produce recommendations based on null Redis metrics

**Show the infrastructure table** (same as full report).

**Analyze logs thoroughly:**
- AOF rewrite frequency and growth % triggers
- fsync warnings ("disk is busy?")
- OOM warnings
- Connection errors
- Startup/restart events (note these are normal during deploys)
- Summarize with counts: "Analyzed 1000 lines: 18 AOF rewrites, 1 fsync warning, 0 errors"

**State what you cannot determine** without SSH:
- Connection health (clients, blocked, rejected)
- Memory usage and fragmentation
- Cache hit rate
- Eviction status
- Command workload profile
- Keyspace composition
- Slow log entries and actual slow commands
- Biggest keys per type

## Redis Performance Patterns

**Memory Fragmentation Pattern:**
- `mem_fragmentation_ratio > 1.5` = memory is fragmented, RSS much higher than used
- Caused by frequent small key deletions creating memory holes
- Fix: restart Redis, or enable `activedefrag yes` (Redis 4.0+)
- Ratio < 1.0 means Redis is using swap — critical performance issue

**Cache Thrashing Pattern:**
- Hit rate < 80% + evicted keys > 0 = working set exceeds maxmemory
- Check maxmemory_policy — `noeviction` will reject writes, `allkeys-lru` will evict
- If maxmemory is 0 (unlimited), Redis will consume all RAM until OOM killed

**Connection Rejection Pattern:**
- rejected_connections > 0 = maxclients limit hit
- Check connected_clients vs maxclients default (10,000)
- Blocked clients = operations waiting on BLPOP/BRPOP/WAIT

**Persistence Risk Pattern:**
- RDB last save failed + no AOF = data loss risk on restart
- Check disk space if saves are failing
- Long time since last save = more data at risk

**AOF Rewrite Churn Pattern:**
- Frequent AOF rewrites (every 1-2 hours) with high growth % triggers (100-800%)
- Indicates high write-to-data-size ratio — small dataset with heavy writes
- Check Fork CoW size to gauge actual data size vs AOF overhead
- If rewrites are fast (<1s) and CoW is small (<10 MB), this is noisy but harmless
- If rewrites are slow or CoW is large, investigate write patterns

**Disk Sawtooth Pattern:**
- Disk usage oscillating in a regular pattern = AOF growing between rewrites, then compacting
- Normal behavior — the baseline is volume overhead + AOF base file
- If the amplitude is growing over time, data size is increasing

## Redis Thresholds

| Metric | Healthy | Warning | Critical |
|--------|---------|---------|----------|
| Hit rate | >95% | 80-95% | <80% |
| Fragmentation ratio | 1.0-1.5 | 1.5-2.0 | >2.0 or <1.0 |
| Evicted keys | 0 | >0 | Growing rapidly |
| Blocked clients | 0 | 1-5 | >5 |
| Connected clients | <80% maxclients | 80-90% | >90% |

## Redis Command Stats Analysis

The top commands by call count reveal the workload pattern:
- **GET/SET dominant** = simple key-value cache
- **HGET/HSET dominant** = hash-based data model (sessions, objects)
- **LPUSH/RPOP dominant** = queue pattern
- **High KEYS/SCAN** = application iterating keys (potential performance issue at scale)
- **High latency on simple commands** (>100µs for GET) = memory pressure or CPU saturation

## Redis Autoscale Note

See [analyze-db.md](analyze-db.md) for full autoscale rules. For Redis specifically:
- If `maxmemory` is set, compare it against actual memory usage — not the Railway memory limit.
- If `maxmemory` is 0 (unlimited), Redis will grow until the OS kills it. This is the default Railway config and works fine with autoscaling — Redis uses what it needs and Railway scales the container.
- Do NOT recommend setting maxmemory to a fraction of the Railway memory limit — the limit is the autoscale ceiling, not fixed allocation.

## Validated against

- Redis INFO command, SLOWLOG LEN, SLOWLOG GET, and --bigkeys

# Database Analysis

## Your Role

You are a database performance expert. The script collects raw data - your job is to **think deeply** about what you see, identify root causes, correlate symptoms, and explain the "why" behind problems.

**Don't just report metrics. Analyze them.**

## Context Resolution

The user's request is the source of truth. Use this decision table:

| What the user provided | Action |
|------------------------|--------|
| Railway URL | Extract IDs directly from the URL — do NOT run `railway status --json` |
| Service name + environment name | Proceed — intent is clear. Resolve IDs via API. |
| Service name only (no environment) | Find the service by name via API. If multiple matches exist across projects, ask: "Which project do you mean?" Otherwise confirm: "Do you mean `<service>` in `<project>` / `<env>`?" — only proceed on confirmation |
| Raw UUID(s) | Resolve to human-readable names via API, then confirm before running |
| Vague request ("analyze my database", "check postgres") | Run `railway status --json` to see what's linked. If it's a database service, confirm: "Do you mean `<service>` in `<project>` / `<env>`?". If it's not a database service or nothing is linked, ask: "Which service and environment should I analyze?" |
| No context at all | List workspaces (`railway whoami --json`), then projects (`railway project list --json`), then environments and services for the chosen project, narrowing down until you have a specific service and environment |

`railway status --json` is a hint to form a specific question, not a trigger to act without confirmation.

**When the user provides a Railway URL**, extract IDs directly from it:

```
https://railway.com/project/<PROJECT_ID>/service/<SERVICE_ID>?environmentId=<ENV_ID>
https://railway.com/project/<PROJECT_ID>/service/<SERVICE_ID>/database?environmentId=<ENV_ID>
```

Then query the API for the service name and database type in a **single call**:

```bash
railway api \
  'query getServiceAndConfig($serviceId: String!, $environmentId: String!) {
    service(id: $serviceId) { name }
    environment(id: $environmentId) {
      config(decryptVariables: false)
    }
  }' \
  --variables '{"serviceId": "<SERVICE_ID>", "environmentId": "<ENV_ID>"}'
```

From the response, get:
- **Service name**: `data.service.name`
- **Database image**: `data.environment.config.services.<SERVICE_ID>.source.image`

Then match the image to the database type:

| Image pattern | Database Type |
|--------------|---------------|
| `postgres*`, `ghcr.io/railway/postgres*` | PostgreSQL |
| `mysql*`, `ghcr.io/railway/mysql*` | MySQL |
| `redis*`, `ghcr.io/railway/redis*`, `railwayapp/redis*` | Redis |
| `mongo*`, `ghcr.io/railway/mongo*` | MongoDB |

**If `environmentId` is empty in the URL** (e.g., `?environmentId=` or no query param at all), skip the `environment.config` query — it requires a valid ID. Instead, list the project's environments:

```bash
railway api \
  'query getEnvs($id: String!) { project(id: $id) { environments { edges { node { id name } } } } }' \
  --variables '{"id": "<PROJECT_ID>"}'
```

Use the `production` environment by default. If multiple non-PR environments exist and the user hasn't specified one, ask which environment to analyze.

## Database Type Detection and Script Selection

| Database Type | Script |
|---------------|--------|
| PostgreSQL | `scripts/analyze-postgres.py` |
| MySQL | `scripts/analyze-mysql.py` |
| Redis | `scripts/analyze-redis.py` |
| MongoDB | `scripts/analyze-mongo.py` |

**All scripts share the same CLI interface** (use the script name from the table above):
```bash
python3 scripts/analyze-<script>.py \
  --service <name> \
  --json \
  --project-id <project-id> \
  --environment-id <env-id> \
  --service-id <service-id>
```

Common options across all scripts:
- `--json` — JSON output for programmatic processing
- `--quiet` — Suppress progress messages
- `--skip-logs` — Skip log collection
- `--metrics-hours <N>` — Hours of metrics history (default: 24, max: 168)
- `--step <step>` — Debug individual collection steps (ssh-test, query, logs, metrics)

## Before You Analyze: Check Collection Status

**ALWAYS check `collection_status` and `errors[]` FIRST before interpreting any data.** The script collects data from multiple independent sources. Any of them can fail.

### Decision Table

| database_query | metrics_api | logs_api | Report Type |
|---------------|-------------|----------|-------------|
| success | success | success | Full analysis — use all sections |
| success | error | success | Full analysis — note missing infrastructure metrics |
| **error** | success | success | **Partial report** — only infrastructure metrics + log analysis. NO performance conclusions. |
| **error** | error | success | **Logs-only report** — state what logs show, note everything else failed. NO diagnosis. |
| **error** | **error** | **error** | **Collection failure** — report the errors, do not analyze. |

### When database_query failed — SSH key errors

If the error contains `"No SSH keys found"` or `"SSH key registration required"`, handle it proactively — don't just tell the user to fix it themselves.

**If error contains `"Key found but not registered"` or `"No SSH keys found"`:**

Run these two commands to understand what's available:

```bash
railway ssh keys          # list keys already registered with Railway
ls ~/.ssh/*.pub 2>/dev/null  # list local public keys
```

Then present the user with their options and ask which to use:

```
SSH introspection needs a registered key. Here's what I found:

Registered with Railway: <list from `railway ssh keys`, or "none">
Local keys available:    <list from ~/.ssh/*.pub, or "none">

Options:
  1. Register a local key — `railway ssh keys add` (uses your default key)
  2. Import from GitHub  — `railway ssh keys github`
  3. Generate a new key  — `ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_railway`

Which would you like to do?
```

Once the user chooses and the key is registered, re-run the full analysis.

### When database_query failed — CLI outdated or missing

If the error contains `"--native SSH flag is not supported"` or `"Railway CLI not found"`, the script detected an unrecoverable CLI problem. Tell the user directly:

- **Outdated CLI:** "Your Railway CLI doesn't support native SSH. Update it: `npm i -g @railway/cli@latest` or `brew upgrade railway`"
- **Missing CLI:** "Railway CLI not installed. Install it: `npm i -g @railway/cli` or `brew install railway`"

Then ask if they'd like to proceed with a partial analysis (metrics + logs only) while they update, or wait until the CLI is fixed and re-run the full analysis.

### When database_query failed — other SSH errors

This means SSH could not reach the database or the query failed. You have NO connection stats, NO cache hit ratios, NO vacuum health, NO query performance data. All those fields will be null/empty.

**You MUST:**
1. State clearly: "Database introspection failed — SSH could not connect to the service"
2. Show the `collection_status` errors
3. Show only the data that DID succeed (metrics, logs)
4. Do NOT produce recommendations based on null metrics
5. Do NOT diagnose performance issues from logs alone

**Partial report template:**
```
Service: <name>
Status: Data collection partially failed

## Collection Status
| Source | Status |
|--------|--------|
| Database Query (SSH) | ERROR: <error from collection_status> |
| Metrics API | <status> |
| Logs API | <status> |

## Available Data
<Show metrics and log summary from sources that succeeded>

## What We Cannot Determine
<List what requires the database query: connection health, cache performance, vacuum health, query analysis, etc.>
```

## Output Structure: Data First, Actions Second

**Always present information in this order:**

### 1. Context Header
```
Service: <name> (<project> <environment>)
Status: <deployment health>, <RAM>, <disk used>
```

### 2. Consolidated Data Tables

Before any analysis, show the raw metrics in tables so the user sees their actual state. The DB-specific reference defines exactly which tables to show for each database type — present the relevant health sections first, then connections, then query performance.

**Logs & Active Issues:**
- Parse the `recent_logs` array (1000 lines of raw logs) - don't just check if empty
- Summarize: "Analyzed 1000 log lines: 3 errors (connection timeouts), 0 critical issues"
- Show specific concerning log entries if found
- **Categorize log entries**: group by type (errors, warnings, connection events, replication, crashes/restarts)
- **Count patterns**: note if a single type dominates the log output
- **Quote actual log lines** for errors — don't just say "errors found", show the exact message so the user can search their codebase

### 3. Analysis

After showing the data, explain the chain of causation. Connect the dots between tables.

### 4. Recommended Actions

Group by urgency. For databases that support configuration changes without restart vs those that require one, call that out explicitly.

### 5. Expected Outcomes

What metrics should change after fixes.

---

**Why this order matters:**
- Users can verify the data matches their understanding
- They see the full picture before being told what to do
- Actions have context - they know WHY each fix is recommended
- No valuable data is hidden in prose or omitted

## CRITICAL: Use the Actual Data

**NEVER fabricate or assume values.** The script outputs JSON with exact numbers. Before stating any metric:

1. **Read the actual JSON output** - Don't truncate or skim
2. **Quote the exact values** - e.g., `"max": 5000` not "100"
3. **Investigate outliers** - Dig into any field that seems unusually high or low

Common errors to avoid (all database types):
- **Not parsing `recent_logs`** - always analyze the raw log lines, don't just report "no errors"
- **Diagnosing performance issues from logs when all metrics are null** — logs show what happened, not how the database is performing
- **Treating startup/restart log entries as evidence of failure** — databases restart for many normal reasons (deploys, config changes, scaling)
- **Producing recommendations when all database metrics are null** — if `collection_status.database_query` is "error", you have no basis for tuning advice

See the DB-specific reference for additional errors to avoid per database type.

## Running the Analysis

Pass project, environment, and service IDs directly — no `railway link` needed:

```bash
# From plugins/railway/skills/use-railway directory:
# Use the script name from the "Database Type Detection" table above
python3 scripts/analyze-postgres.py --service <name> --json \
  --project-id <project-id> --environment-id <env-id> --service-id <service-id>
```

All three IDs come from the URL (see "Context: URL First" above). The service name comes from the API query.

**Options:**
- `--metrics-hours <N>` — Hours of metrics history to fetch (default: 24, max: 168). Use `--metrics-hours 168` for 7-day trends, `--metrics-hours 1` for recent snapshot.

**SSH retry:** The script automatically retries SSH connectivity up to 3 times with increasing timeouts (30s, 60s, 90s). Each individual SSH command (database query, slowlog, bigkeys, etc.) also retries up to 3 times on failure — covering transient errors like `exec request failed on channel 0`. Progress is logged to stderr.

**Output:** Progress messages go to stderr. JSON results go to stdout. Do not redirect or pipe stderr — just run the command as-is and read the full output.

### Resolving environment by name

If the URL has no `environmentId` and the user specifies an environment by name (e.g., "production"), resolve it:

```bash
railway api \
  'query getProject($id: String!) {
    project(id: $id) {
      environments { edges { node { id name } } }
    }
  }' \
  --variables '{"id": "<PROJECT_ID>"}'
```

Match the environment name (case-insensitive) to get the `environmentId`.

### Debugging individual steps

```bash
# Use the script name from the "Database Type Detection" table above
python3 scripts/analyze-postgres.py --service <name> \
  --project-id <pid> --environment-id <eid> --service-id <sid> \
  --step ssh-test    # Test SSH connectivity
  --step query       # Run only the database query
  --step metrics     # Fetch only API metrics
  --step logs        # Fetch only logs
```

## Database-Specific References

After running the script and checking collection status, load the reference for the specific database type:

| Database | Reference | What It Covers |
|----------|-----------|----------------|
| PostgreSQL | [analyze-db-postgres.md](analyze-db-postgres.md) | What psql collects, log analysis checklist, tuning formulas, vacuum priority, pg_stat_statements, applying fixes |
| MySQL | [analyze-db-mysql.md](analyze-db-mysql.md) | All 12 metric sections (overview, query throughput, InnoDB, efficiency, buffer pool, I/O, network, locks, cache, top queries, tables, active queries), patterns, tuning |
| Redis | [analyze-db-redis.md](analyze-db-redis.md) | INFO ALL metrics, memory fragmentation, cache thrashing, persistence, command stats |
| MongoDB | [analyze-db-mongo.md](analyze-db-mongo.md) | serverStatus, WiredTiger cache, query efficiency, connection saturation, oplog |

**Always load the DB-specific reference** — it contains the metric sections, thresholds, and tuning knowledge needed for proper analysis.

## Infrastructure Metrics (All Database Types)

All scripts collect the same infrastructure metrics via Railway API:

**Metrics History (`metrics_history`):**
The script fetches **7 days** (168 hours) of time-series data from Railway's metrics API by default and produces **two analysis windows**:

```json
{
  "metrics_history": {
    "windows": {
      "7d": { "window_hours": 168, "metrics": { "cpu": {...}, "memory": {...}, ... } },
      "24h": { "window_hours": 24, "metrics": { "cpu": {...}, "memory": {...}, ... } }
    }
  }
}
```

Each window independently computes:
- **Summary stats**: current, min, max, avg for each metric
- **Trend analysis**: compares first-quarter avg to last-quarter avg — reports direction (increasing/decreasing/stable) and % change
- **Spike detection**: flags values > avg + 2*stddev with timestamps of peaks
- **Downsampled series**: ~48 data points per window

Available metrics: CPU, memory (with limits), disk, network RX/TX.

**Comparing windows reveals whether a trend is new or sustained:**
- "Memory increasing in 24h but stable over 7d" → temporary spike, likely a batch job
- "Memory increasing in both 24h AND 7d" → sustained growth, may need investigation
- "CPU spike in 24h, no spikes in 7d" → new issue
- "Disk growing over 7d" → data accumulation trend

Use `--metrics-hours N` to change the long window (default: 168, max: 168). The 24h window is always produced when the long window is > 24h.

### Railway auto-scales vertically

Railway services auto-scale CPU, RAM, and disk based on actual usage. Users do NOT pick or control resource sizes. The `cpu_limit` and `memory_limit` values from metrics are the **autoscale ceiling** (typically 32 vCPU / 32 GB), not user-provisioned allocations. Users are billed for actual usage, not the ceiling.

**Rules for ALL database types:**
- **Never say "right-size the instance"** or suggest reducing CPU/RAM — it's not a user action.
- **Never flag low utilization % against the limit as waste** — a service showing 0.01 vCPU / 70 MB actual usage against a 32 vCPU / 32 GB ceiling is normal, not over-provisioned.
- **Disk does NOT auto-scale** — Railway volumes have a fixed capacity. Paid users (Hobby and Pro) can expand them live without downtime, but it requires a manual resize. Flag high disk utilization as actionable. Users are billed for actual disk utilization, not the full volume size.
- **Focus on actual usage values**, not the ratio to limits. Analyze whether 70 MB of memory is healthy for this workload — don't compare it to the 32 GB ceiling.
- When tuning database parameters (shared_buffers, innodb_buffer_pool_size, maxmemory, etc.), base recommendations on the **current actual RAM** from `metrics_history.memory`, not the limit.

## Validated against

- Docs: [ssh.md](https://docs.railway.com/cli/ssh), [logs.md](https://docs.railway.com/cli/logs), [metrics.md](https://docs.railway.com/cli/metrics), [api docs](https://docs.railway.com/api/llms-docs.md)
- Local scripts: [analyze-postgres.py](../scripts/analyze-postgres.py), [analyze-mysql.py](../scripts/analyze-mysql.py), [analyze-redis.py](../scripts/analyze-redis.py), [analyze-mongo.py](../scripts/analyze-mongo.py), [dal.py](../scripts/dal.py)
- API command: [api.rs (v5.49.1)](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/api.rs)

# Cloud agents

Use `railway ca` / `railway code` to run coding harnesses on persistent Railway cloud agent VMs. Use `railway agent` for Railway Agent chat/investigations, and [sandbox.md](sandbox.md) for sandbox execution, builds, and checkpoints.

## Availability and target

Cloud agents arrived in CLI 5.32; flat lifecycle commands in 5.35 and desktop setup in 5.38. They are experimental and require **Cloud Agents** enabled in [Priority Boarding](https://railway.com/account/feature-flags). A feature-availability error calls for enabling that feature, not repeatedly provisioning VMs or changing project feature flags.

Use explicit project and environment for creation. A directory's `railway link` context takes precedence over the saved default project; a stale link can fall back to the saved default. Verify the target rather than assuming the saved preference wins. Interactive `ca` can run login and continue; noninteractive unauthenticated calls fail instead of waiting on an unattended device code.

```bash
railway ca setup --show
railway ca list --json
railway ca list --project <project-id> --environment <env> --json
```

Bare `list` finds the user's agents across projects. `--all` includes other members' agents and requires an explicit environment. Address an existing agent by name or ID; an omitted identifier uses the directory's agent or the sole candidate, otherwise the CLI reports candidates.

## Create, launch, and connect

```bash
railway ca create my-agent --project <project-id> --environment <env> --json
railway ca create my-agent --project <project-id> --environment <env> --env-file .env.agent --json
railway ca ssh my-agent
railway ca ssh my-agent -- bash
railway ca start --codex --project <project-id> --environment <env>
railway ca start --railway --project <project-id> --environment <env>
```

Choose one creation command with the needed options. `create` provisions a VM without attaching; `ssh` connects to an existing VM and does not create one for a mistyped name. `start` can create and launch a harness, skipping the TUI. Harness flags are `--codex`, `--claude`, `--grok`, and `--railway`; the first three carry or mint a local sign-in, while `--railway` launches Railway's own agent with credentials already on the VM and needs no local sign-in, which suits unattended workflows. `--no-wait` on create/wake means requested, not ready: reread `ca list --json` before reporting readiness. Environment files and `--variable` inputs configure the agent VM; pass only values needed for the remote task.

For a human terminal, bare `railway ca` opens the management TUI and `railway code` opens a session-focused view. Automated workflows should use explicit lifecycle commands rather than attempting to control that TUI. `ca setup` configures the default harness and skills; `ca setup --show` inspects preferences and `ca setup --reset` removes them when requested. Launching can carry harness authentication and skills from the local machine, so choose the harness and remote target deliberately.

## Credentials inside the VM

The VM ships the `railway` CLI, the `gh` CLI and the `railway` MCP server. The MCP server is authenticated in every session. The `railway` CLI is authenticated only inside an SSH terminal session (`railway ca ssh`, `railway code`, desktop SSH), which injects the user's token; a chat session started from the Railway dashboard or mobile app carries no CLI credential, by design. An agent working in such a session must not run `railway` commands, `railway api` included, and should use the MCP tools instead. `railway login` and `railway link` never help on the VM and fail in its non-interactive shell.

## Sleep, wake, and delete

```bash
railway ca wake my-agent
railway ca sleep my-agent
railway ca delete my-agent
```

Sleep stops compute while retaining the disk; deletion removes the agent and its disk. Use sleep when the user wants to pause work and keep files. `sleep --all` acts across the user's running agents unless narrowed by environment; use it only when that broader scope was requested. Check `list --json` after lifecycle mutations. Do not mistake a disconnected harness session for a deleted VM.

## Desktop SSH setup

```bash
railway ca desktop --codex --agent my-agent --dry-run
railway ca desktop --codex --agent my-agent
railway ca desktop --claude --agent my-agent
```

Choose the requested app; both flags can configure both on the same VM. The dry-run previews configuration. Actual setup prepares the remote harness and writes an OpenSSH entry; Claude setup also writes its desktop settings. Omitting `--agent` can create an agent when none exists. `--dir /app/api` selects the remote working directory. Restart the desktop app after setup.

A desktop app cannot wake a sleeping agent through the relay: run `railway ca wake <name>` before connecting. To undo the local integration, use `railway ca desktop --codex --agent my-agent --remove` (or `--claude`); removing desktop configuration is distinct from deleting the VM.

## Troubleshoot

- **Wrong project**: inspect the directory link and `ca setup --show`, then use explicit scope.
- **Access blocked**: check Cloud Agents in Priority Boarding; project feature flags do not enable it.
- **Desktop cannot connect**: confirm the agent is awake, inspect the generated SSH host, and check CLI SSH access before rerunning setup.
- **Session ended unexpectedly**: inspect `ca list` and reconnect to the existing agent before creating another VM.

## Validated against

- Docs: [Cloud agent CLI](https://docs.railway.com/cli/ca), [Code CLI](https://docs.railway.com/cli/code)
- CLI source (v5.49.1): [cloud_agent/mod.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/cloud_agent/mod.rs), [lifecycle.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/cloud_agent/lifecycle.rs), [desktop.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/cloud_agent/desktop.rs), [setup.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/cloud_agent/setup.rs), [access.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/cloud_agent/access.rs)

# Configure

Manage environments, variables, service config, domains, and networking.

## Environments

### List and switch

```bash
railway environment list --json
railway environment list --ephemeral --json      # only PR environments
railway environment list --no-ephemeral --json   # hide PR environments
railway environment link <environment>           # switch active environment
```

### Create

```bash
railway environment new <name>
railway environment new <name> --duplicate <source-environment>    # clone config from existing
```

Duplicating copies all service configurations and variables from the source environment.

## Variables

### Read, set, and delete

```bash
railway variable list --service <service> --environment <env> --json
railway variable set KEY=value --service <service> --environment <env>
railway variable delete KEY --service <service> --environment <env>
```

Variable changes trigger a redeployment by default. This is usually the desired behavior, since the service picks up the values on restart. Use `--skip-deploys` only when you plan to redeploy or restart separately.

CLI 5.34.2+ accepts an empty assignment such as `railway variable set OPTIONAL_VALUE= --service <service>`. Empty is a value, not a deletion. Before an idempotent delete, list keys and delete only if present.

### Bulk edit with a reviewed diff

CLI 5.48+ opens an editor and presents the changes before applying:

```bash
railway variable edit --project <project-id> --environment <env> --service <service>
railway variable edit --demo                 # offline fixture; no Railway mutation
```

This workflow requires a TTY or an explicitly configured `$EDITOR`/`$VISUAL`. Prefer `set`/`delete` for deterministic agent edits when no editor workflow was requested. Saving the editor is not approval: the CLI shows a redacted diff and asks before applying. Noninteractive apply requires `--yes`; deletions in agent or noninteractive sessions also require `--confirm-destructive`, within the user's approved scope.

Removing a line deletes that variable. Keep `<sealed>` placeholders unchanged to preserve sealed values. Railway-provided variables are comments and cannot be edited. `--reveal` exposes plaintext in the diff; use only when intended. `--skip-deploys` commits changes without triggering deploys.

### Set sensitive values

Use stdin for secrets or values that shouldn't appear in shell history:

```bash
printf "%s" "$SECRET_VALUE" | railway variable set API_KEY --stdin --service <service>
railway variable set API_URL=https://api.example.com --project <project-id> --environment <env> --service <service>
```

### Template syntax

Railway supports interpolation between services and shared variables:

```text
${{KEY}}                                         # same-service variable
${{shared.API_KEY}}                              # shared variable
${{postgres.DATABASE_URL}}                       # variable from another service
${{api.RAILWAY_PRIVATE_DOMAIN}}                  # another service's private domain
```

Wiring example, a frontend connecting to a backend over private networking:

```text
BACKEND_URL=http://${{api.RAILWAY_PRIVATE_DOMAIN}}:${{api.PORT}}
```

### Wiring services together

Each managed database creates connection variables automatically. Reference them from other services using template syntax:

| Database | Variable reference |
|---|---|
| Postgres | `${{Postgres.DATABASE_URL}}` |
| Redis | `${{Redis.REDIS_URL}}` |
| MySQL | `${{MySQL.MYSQL_URL}}` |
| MongoDB | `${{MongoDB.MONGO_URL}}` |

Service names in references are case-sensitive and must match the service name exactly as it appears in the project.

**Public vs private networking decision:**

| Traffic path | Use |
|---|---|
| Browser → API | Public domain |
| Service → Service | Private domain (`RAILWAY_PRIVATE_DOMAIN`) |
| Service → Database | Private (automatic, uses internal DNS) |

**Frontend apps cannot use private networking.** Frontends run in the user's browser, not on Railway's network. They cannot reach `RAILWAY_PRIVATE_DOMAIN` or internal database URLs. Options:

1. **Backend proxy** (recommended): frontend calls a backend API on a public domain, backend connects to the database over the private network.
2. **Public database URL**: use the public connection variable (for example, `${{Postgres.DATABASE_PUBLIC_URL}}`). This requires a TCP proxy on the database service and exposes the database to the internet. Use this only for development or low-sensitivity data.

### Railway-provided variables

These are set automatically at runtime. Availability depends on resource configuration.

**Networking:**

| Variable | Available when |
|---|---|
| `RAILWAY_PUBLIC_DOMAIN` | Public domain is configured |
| `RAILWAY_PRIVATE_DOMAIN` | Always (internal DNS for service-to-service traffic) |
| `RAILWAY_TCP_PROXY_DOMAIN` | TCP proxy is enabled |
| `RAILWAY_TCP_PROXY_PORT` | TCP proxy is enabled |

**Context:**

| Variable | Available when |
|---|---|
| `RAILWAY_PROJECT_ID` | Always |
| `RAILWAY_ENVIRONMENT_ID` | Always |
| `RAILWAY_ENVIRONMENT_NAME` | Always |
| `RAILWAY_SERVICE_ID` | Always |
| `RAILWAY_SERVICE_NAME` | Always |
| `RAILWAY_DEPLOYMENT_ID` | Always |
| `RAILWAY_REPLICA_ID` | Replicas configured |
| `RAILWAY_REPLICA_REGION` | Multi-region configured |

**Git (present when deployed from a linked repo):**

| Variable | Description |
|---|---|
| `RAILWAY_GIT_COMMIT_SHA` | Full commit hash of the deployed revision |
| `RAILWAY_GIT_AUTHOR` | Commit author name |
| `RAILWAY_GIT_COMMIT_MESSAGE` | First line of the commit message |
| `RAILWAY_GIT_BRANCH` | Branch that triggered the deploy |

**Storage (present when a volume is attached):**

| Variable | Description |
|---|---|
| `RAILWAY_VOLUME_MOUNT_PATH` | Filesystem path where the volume is mounted |
| `RAILWAY_VOLUME_NAME` | Name of the attached volume |

Sealed variables are write-only. CLI 5.47.2+ lists their names with `null` in JSON, `<sealed>` in the table, or a comment in KV output. **A null value means the sealed key already exists; do not recreate or clear it as though it were missing.** Ordinary variable output can contain plaintext secrets.

## Service config

Service configuration controls source, build, deploy, and networking settings. There are two ways to mutate it.

### Dot-path patch

For single-field changes:

```bash
railway environment edit --service-config <service> deploy.startCommand "npm start"
railway environment edit --service-config <service> build.buildCommand "npm run build"
railway environment edit --service-config <service> source.rootDirectory "/apps/api"
railway environment edit --service-config <service> deploy.numReplicas 2
railway environment edit --project <project-id> --environment production --service-config <service> deploy.startCommand "npm start"
```

### JSON patch

For multi-field changes or complex structures:

```bash
railway environment edit --json <<'JSON'
{"services":{"<service-id>":{"build":{"buildCommand":"npm run build"},"deploy":{"startCommand":"npm start"}}}}
JSON
```

Resolve exact service IDs from `railway service list --json` before constructing JSON patches. Using names in the JSON payload doesn't work.

### Stage config changes

Stage changes when the user wants to review config before committing it:

```bash
railway environment edit --service-config <service> build.buildCommand "npm run build" --stage
railway environment edit --service-config <service> deploy.startCommand "npm start" --message "Set production start command"
```

Use `--stage` only when the user requests staged config changes. Use regular edits for immediate mutations.

### Config schema (typed paths)

Include only keys you're changing. The full shape:

**Source**: `source.image` (string), `source.repo` (string), `source.branch` (string), `source.rootDirectory` (string), `source.checkSuites` (boolean), `source.commitSha` (string), `source.autoUpdates.type` (string: `disabled`, `patch`, `minor`)

**Build**: `build.builder` (string: `RAILPACK`, `NIXPACKS`, `DOCKERFILE`), `build.buildCommand` (string), `build.dockerfilePath` (string), `build.watchPatterns` (string array), `build.nixpacksConfigPath` (string)

**Deploy**: `deploy.startCommand` (string), `deploy.preDeployCommand` (string), `deploy.healthcheckPath` (string), `deploy.healthcheckTimeout` (integer), `deploy.numReplicas` (integer), `deploy.restartPolicyType` (string: `ON_FAILURE`, `ALWAYS`, `NEVER`), `deploy.restartPolicyMaxRetries` (integer), `deploy.sleepApplication` (boolean), `deploy.cronSchedule` (string), `deploy.multiRegionConfig` (object)

**Multi-region config** structure for `deploy.multiRegionConfig`:

```json
{ "us-west2": { "numReplicas": 2 }, "europe-west4-drams3a": { "numReplicas": 1 } }
```

| Region identifier | Location |
|---|---|
| `us-west2` | US West (Oregon) |
| `us-east4-eqdc4a` | US East (Virginia) |
| `europe-west4-drams3a` | Europe (Netherlands) |
| `asia-southeast1-eqsg3a` | Asia (Singapore) |

Natural language mapping: "add replicas in Europe" → `europe-west4-drams3a`, "US East" → `us-east4-eqdc4a`. When the user doesn't specify a region, query current config first with `railway environment config --json` to see existing region assignments before modifying.

**Variables**: `variables.<KEY>.value` (string), `variables.<KEY>.isOptional` (boolean), `variables.<KEY>.isSealed` (boolean). Delete a variable by setting it to `null`.

**Lifecycle**: `isDeleted` (boolean) removes the service. `isCreated` (boolean) marks as new. Prefer `railway service delete` for normal service deletion.

**Storage**: `volumeMounts.<volume-id>.mountPath` (string), `volumes.<volume-id>.isDeleted` (boolean)

**Buckets**: `buckets.<bucket-id>.region` (string: `sjc`, `iad`, `ams`, `sin`), `buckets.<bucket-id>.isCreated` (boolean), `buckets.<bucket-id>.isDeleted` (boolean). Buckets are created at the project level via `railway bucket create` and deployed to environments via config patches. The CLI handles this automatically, so use `railway bucket` commands

### Shared variables and project-level config

```bash
railway environment edit --json <<'JSON'
{"sharedVariables":{"API_BASE":{"value":"https://example.com"}}}
JSON
```

Shared variables are accessible from any service via `${{shared.KEY}}`.

### Read config

Always inspect before mutating. Config patches merge, so you need to know the state to avoid overwriting fields unintentionally:

```bash
railway environment config --json
```

Verify after mutation to confirm the change took effect:

```bash
railway environment config --json
railway service list --json
```

## Domains

Use the `railway domain` command for domain lifecycle work. Avoid raw `environment edit` JSON patches for normal domain management.

### Create domains

```bash
railway domain --service <service> --json                  # generate a Railway domain
railway domain example.com --service <service> --json      # add a custom domain
railway domain example.com --service <service> --port 8080 --json
```

One Railway-provided domain is allowed per service. Multiple custom domains are supported. Custom domain creation returns the DNS records to add at the DNS provider. Add both the routing record and ownership verification record exactly as returned.

Requests to a custom domain can return `404` until Railway verifies the ownership `TXT` record. DNS can take up to 72 hours to propagate.

### Inspect and update domains

```bash
railway domain list --service <service> --json
railway domain status example.com --service <service> --json
railway domain update example.com --port 8080 --service <service> --json
railway domain update old-name.up.railway.app --domain new-name --service <service> --json
railway domain certificate retry example.com --service <service> --json
```

Use `status` when DNS or certificate issuance is not healthy. Retry certificate issuance only after fixing DNS.

### Delete domains

```bash
railway domain delete example.com --service <service> --yes --json
```

Domain deletion is destructive. Confirm the domain and service before running it.

## Webhooks

Project webhooks POST a JSON payload to a URL whenever a chosen deployment, monitor, or volume-alert event happens anywhere in the project. They are project-scoped, not per service or environment, and are managed with five MCP tools. There is no `railway webhook` CLI command. Resolve the project ID from `railway status --json` or `list-projects` first.

| Tool | Purpose |
|---|---|
| `list-webhooks` | Every webhook in the project: URL, `eventTypes`, `includePreviewEnvironments`, the names of its custom headers (`headerNames`), and the `id` the other tools take |
| `create-webhook` | `url` plus optional `eventTypes` (defaults to `Deployment.failed`, `Deployment.crashed`, `Deployment.oom_killed`), `includePreviewEnvironments` (defaults to `true`), and `headers` |
| `update-webhook` | `webhookId` plus any of `url`, `eventTypes`, `includePreviewEnvironments`, `headers`. Only the fields passed change; `eventTypes` and `headers` each replace the whole set |
| `test-webhook` | POST a sample event and report the HTTP status. Pass `url` (and optional `headers`) to try a new endpoint, or `webhookId` to send to an existing webhook with its stored URL and headers. Nothing is stored |
| `delete-webhook` | Remove a webhook by `webhookId`. Destructive; confirm the URL with the user first |

Discord and Slack incoming-webhook URLs are detected and receive a formatted message instead of the raw JSON. Run `test-webhook` before `create-webhook`: Railway treats anything outside `2xx`/`3xx` as a failed delivery, and a dead endpoint is paused for 24 hours after repeated failures.

```text
Create webhook for project <project-id>: url https://hooks.example.com/railway, eventTypes Deployment.failed Deployment.crashed, headers { "Authorization": "Bearer <token>" }
```

### Custom headers

A webhook can carry up to 20 custom HTTP headers, sent with every delivery and with `test-webhook`. Use them for whatever the receiver needs to trust or route the request: an `Authorization` bearer token, an API key, a shared secret the receiver compares, a routing header. This replaces the old advice of hiding a secret in the URL, which still works but leaks into logs more easily.

- **Values are write-only.** They are encrypted at rest and never returned: `list-webhooks` and every other tool's output carry `headerNames` only. Keep the value in the user's secret store; Railway cannot show it again.
- **`create-webhook` and `test-webhook`** take `headers` as a name-to-value map.
- **`update-webhook` replaces the whole set.** Every header not listed is removed and `{}` clears them all. A `null` value keeps the stored value for that name, so a secret survives a change without being resent: `{ "Authorization": null, "X-Env": "prod" }` keeps the token and sets `X-Env`. A URL-only update carries the stored headers forward automatically.
- **`test-webhook` with `webhookId`** sends the stored headers; any `headers` passed alongside override the stored one of the same name and the rest are still sent. Headers are validated before the request goes out, so a bad name gets a clear error instead of a `0` status.
- **Name rules.** RFC 9110 token characters, up to 128 characters; values up to 4096 characters with no CR or LF. `Host`, `Content-Type`, `Content-Length` and the hop-by-hop headers (`Transfer-Encoding`, `Connection`, `Keep-Alive`, `Upgrade`, `TE`, `Trailer`, `Expect`) are refused, as is any name starting with `proxy-` or `x-railway-`. Two names differing only by case count as a duplicate. The dashboard form applies the same rules inline and disables Save until a bad row is fixed.

Without MCP, the same fields are on the GraphQL API: `headers` inside the webhook channel config of `notificationRuleCreate` / `notificationRuleUpdate` (a `null` value keeps the stored one there too), and `webhookTest(url, payload, headers: [WebhookHeaderInput!], notificationRuleId)`. Inspect them with `railway api describe`.

## Networking commands

### Private networking

For service-to-service traffic within a project, use private domain references instead of public URLs. This avoids egress and is faster:

```text
BACKEND_URL=http://${{api.RAILWAY_PRIVATE_DOMAIN}}:${{api.PORT}}
```

```bash
railway private-network status --service <service> --json
railway private-network update api-internal --service <service> --json
```

When multiple private networks exist, pass `--network <name-or-id>`. Endpoint updates take the prefix only, without `.internal`.

### TCP proxies

Use TCP proxies to expose non-HTTP ports, such as database or game server ports, to the public internet. Only one TCP proxy is allowed per service instance.

```bash
railway tcp-proxy list --service <service> --json
railway tcp-proxy create --service <service> --port 5432 --json
railway tcp-proxy status <proxy-id-or-domain-or-port> --service <service> --json
railway tcp-proxy delete <proxy-id-or-domain-or-port> --service <service> --yes --json
```

TCP proxy creation updates service networking config. If the proxy does not become active, redeploy the service and check status.

### Outbound networking

Use outbound networking commands for Static Outbound IPs and outbound IPv6:

```bash
railway outbound-network status --service <service> --json
railway outbound-network static-ip status --service <service> --json
railway outbound-network static-ip enable --service <service> --json
railway outbound-network static-ip disable --service <service> --json
railway outbound-network ipv6 status --service <service> --json
railway outbound-network ipv6 enable --service <service> --json
railway outbound-network ipv6 disable --service <service> --json
```

Static Outbound IP changes are committed directly but require a redeploy before outbound traffic uses the new assignment. Outbound IPv6 changes are staged as environment config changes; commit staged changes with `railway environment edit` to trigger the redeploy.

### CDN caching

CDN caching is service-scoped and requires an applied public domain.

```bash
railway cdn status --service <service> --json
railway cdn enable --service <service> --json
railway cdn update --service <service> --html-caching force --default-ttl 4h --json
railway cdn update --service <service> --no-swr --purge-on-deploy all --json
railway cdn purge html --service <service> --json
railway cdn purge all --service <service> --json
railway cdn disable --service <service> --json
```

Use the response headers `x-cache` and `age` to verify cache behavior. Cache hits do not reach the service, so service logs and server-side metrics can undercount traffic after caching is enabled.

### WAF Under Attack Mode

Under Attack Mode is service-scoped, requires an applied public domain, and is intended for active bot floods or DDoS events.

```bash
railway waf under-attack status --service <service> --json
railway waf under-attack enable --service <service> --duration 1h --json
railway waf under-attack enable --service <service> --json
railway waf under-attack disable --service <service> --json
```

While active, browser visitors must pass a check. Non-browser API clients and webhooks may receive `429` responses, so warn the user before enabling it on API-only domains.

## Troubleshoot configuration

- **Invalid dot-path**: check field names and types in the config schema section above
- **Wrong service key in JSON patch**: resolve service IDs from `railway service list --json`
- **Variable change didn't take effect**: verify with `railway variable list`, changes trigger redeploy by default
- **Domain returns errors**: run `railway domain status`, verify the target port, then check HTTP logs
- **DNS propagation delay**: custom domains can take up to 72 hours to propagate worldwide
- **Cloudflare proxy issues**: align SSL/TLS mode per Railway's domain guidance
- **Private networking failing**: run `railway private-network status`, verify the service is listening on the referenced port, and check network flow logs
- **Outbound allowlist still sees old IPs**: redeploy after enabling/disabling Static Outbound IPs
- **IPv6 still disabled**: commit the staged environment change and wait for redeploy
- **CDN appears ineffective**: check `x-cache`, `age`, cache headers, `Set-Cookie`, `Authorization`, method, and response size
- **Webhook header rejected**: the name is reserved (`Host`, `Content-Type`, `Content-Length`, hop-by-hop), starts with `proxy-` or `x-railway-`, duplicates another name ignoring case, or the value has a newline; rename it or drop it, and stay under 20 headers
- **Webhook headers disappeared after an update**: `update-webhook` replaces the whole `headers` set, so a map without a name removes it; pass `null` for names to keep, or omit `headers` entirely
- **WAF breaks API clients**: Under Attack Mode blocks non-browser traffic; disable it or scope protection to browser-facing services
- **Multi-region patch ignored**: verify region names match the exact identifiers (`us-west2`, `us-east4-eqdc4a`, `europe-west4-drams3a`, `asia-southeast1-eqsg3a`)

## Validated against

- Docs: [environment.md](https://docs.railway.com/cli/environment), [variable.md](https://docs.railway.com/cli/variable), [domain.md](https://docs.railway.com/cli/domain), [tcp-proxy.md](https://docs.railway.com/cli/tcp-proxy), [private-network.md](https://docs.railway.com/cli/private-network), [outbound-network.md](https://docs.railway.com/cli/outbound-network), [cdn.md](https://docs.railway.com/cli/cdn), [waf.md](https://docs.railway.com/cli/waf), [webhooks.md](https://docs.railway.com/observability/webhooks)
- Platform source (railwayapp/mono): `packages/backboard/src/handlers/http/routes/mcp/tools/webhooks.ts` (webhook MCP tools), `packages/backboard/src/controllers/notifications/validation.ts` (`validateWebhookHeaders`), `packages/backboard/src/graphql/v2/schema/schema.graphql` (`webhookTest`, `WebhookHeaderInput`)
- CLI source: [environment/mod.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/environment/mod.rs), [environment/edit.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/environment/edit.rs), [variable.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/variable.rs), [domain.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/domain.rs), [tcp_proxy.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/tcp_proxy.rs), [private_network.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/private_network.rs), [outbound_networking.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/outbound_networking.rs), [cdn.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/cdn.rs), [waf.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/waf.rs)

# Database operations

Use native CLI commands for database recovery, high availability, and connection pooling. Use [analyze-db.md](analyze-db.md) for performance analysis and [setup.md](setup.md) to create a database or connect a local client.

## Choose the engine and scope

| Engine | Commands |
|---|---|
| Postgres | `railway postgres pitr`, `ha`, `pgbouncer`, `history` |
| MySQL | `railway mysql ha`, `history` |
| Redis | `railway redis ha`, `history` |

Postgres operations arrived in CLI 5.33; MySQL/Redis HA in 5.46. Use **5.47.1 or newer for HA mutations**: that release added the revert and scaling fixes (staged member changes, never deleting the acting primary) and `--remove-orphans`. MySQL/Redis do not expose PITR or PgBouncer commands. Availability and image eligibility also depend on Railway's engine templates; a command existing does not make every custom database image eligible.

All these command trees accept `--project`, `--environment`, `--service`, and `--json`. Resolve the database service and environment first, especially when the supplied URL points at a replica or proxy. Use explicit IDs for cross-project work:

```bash
railway postgres pitr status --project <project-id> --environment <env> --service <service> --json
railway postgres ha status --project <project-id> --environment <env> --service <service> --json
railway postgres pgbouncer status --project <project-id> --environment <env> --service <service> --json
railway mysql ha status --project <project-id> --environment <env> --service <service> --json
railway redis ha status --project <project-id> --environment <env> --service <service> --json
```

The following examples abbreviate scope to `--service`; retain the resolved project and environment in actual calls. Config-changing actions deploy by default. Where supported, `--no-deploy` commits config but defers its runtime effect until the affected services deploy; it is not a dry-run. Use it only when deployment is deliberately deferred. Add `--yes` only for a mutation whose scope and impact the user authorized, and only where the command supports it.

## Postgres point-in-time recovery

```bash
railway postgres pitr status --service <postgres> --json
railway postgres pitr enable --service <postgres>
railway postgres pitr disable --service <postgres>
railway postgres pitr progress --service <postgres> --json
railway postgres pitr progress --service <postgres> --watch
```

`enable`/`disable` recognize a standalone database or an HA cluster root. Standalone operations support `--no-deploy`; HA enable/disable runs a live rolling workflow. `progress`, `cancel`, and `clear` apply only to that HA workflow. Inspect progress before canceling a stuck workflow; clear a completed snapshot only when intended. Bound a watch process and report its last observed phase if it times out.

Status includes a best-effort SSH probe of archive coverage and archiver health. `unavailable` or unknown probe results do not mean backups are disabled or healthy. Resolve SSH reachability or report the missing evidence.

To restore to a separate service:

```bash
railway postgres pitr restore --service <postgres> --at 2026-09-04T10:00:00Z --new-service-name postgres-restored
```

Use an explicit UTC timestamp to avoid local-time ambiguity. Relative offsets such as `30m` are also accepted. Check available coverage first, then verify the restored service's deployment and data before changing application connection variables. `--source-repo-path` selects an archive history when needed. Creating a restored service does not itself switch application traffic.

### Backups and schedules

```bash
railway postgres pitr backup list --service <postgres> --json
railway postgres pitr backup create --service <postgres> --name pre-migration
railway postgres pitr backup lock <backup-id> --service <postgres>
railway postgres pitr schedule list --service <postgres> --json
railway postgres pitr schedule set --daily --weekly --service <postgres>
railway postgres pitr backup restore <backup-id> --service <postgres>
railway postgres pitr backup delete <backup-id> --service <postgres>
```

`backup restore` overwrites the current data; it is different from `pitr restore`, which creates a new service. In-place backup restore on an HA cluster is rejected because replicas would diverge; use the dashboard workflow that reseeds replicas. Do not bypass the guard by restoring the root volume manually. `backup lock` keeps a backup indefinitely. `schedule set --none` removes automatic schedules while keeping existing backups. Verify the backup ID, target, and destructive impact before restore/delete.

## High availability

```bash
railway postgres ha convert --service <postgres> --replicas 2 --coordinators 3 --edge 1
railway postgres ha scale --service <postgres> --replicas 3
railway postgres ha switchover --service <postgres> --to <replica-name-or-id>
railway postgres ha revert --service <postgres>
railway mysql ha convert --service <mysql> --replicas 2
railway mysql ha scale --service <mysql> --replicas 4
railway redis ha convert --service <redis> --replicas 2
railway redis ha switchover --service <redis> --to <replica-name-or-id>
```

`--replicas` counts replicas **excluding the primary**. Omitted conversion counts preserve the template defaults. MySQL and Redis carry consensus on their data nodes: total data nodes must be odd and at least three, so pass an even replica count such as 2 or 4. These conversions require a source image tagged with an exact major.minor version. Separate `--coordinators` applies only to templates with that tier; its count must be odd. Follow the template's reported constraints rather than copying a Postgres topology to another engine.

Before scaling or switching, inspect live member roles and probe errors. Switchover requires a reachable eligible target and may briefly interrupt connections. After conversion/scale/switchover, reread HA status and verify primary role, replica health, and application connectivity; a committed config alone is not evidence that replication is healthy.

Revert returns the database to standalone and removes cluster members. Current CLI fixes protect the acting primary and retain the independent PgBouncer pooler. `--remove-orphans` is a separate cleanup opt-in: orphaned members may no longer identify which same-engine cluster owned them. Do not add it merely to make a retry succeed. If an operation fails midway, inspect current config, live roles, and history before retrying or cleaning up.

## Postgres connection pooling

```bash
railway postgres pgbouncer status --service <postgres> --json
railway postgres pgbouncer add --service <postgres> --pool-mode transaction
railway postgres pgbouncer configure --service <postgres> --max-client-conn 200
railway postgres pgbouncer scale --service <postgres> --replicas 2
railway postgres pgbouncer remove --service <postgres>
```

Pooling works with standalone Postgres and HA roots. Modes are `transaction`, `session`, and `statement`; choose based on the application's session/transaction requirements. Check utilization and existing limits before changing connection counts. Verify the pooler's deployment and connection endpoint before wiring the app to it; removing the pooler requires restoring a suitable application connection path.

## Verify and investigate

```bash
railway postgres history --service <postgres> --limit 10 --json
railway mysql history --service <mysql> --limit 10 --json
railway redis history --service <redis> --limit 10 --json
```

History is a **local** operation trail, not a complete account audit log. Pair it with fresh native status, scoped deployment status, and logs. Do not infer that no changes happened just because this machine has no history. Follow [operate.md](operate.md) for deployment/log triage and [analyze-db.md](analyze-db.md) for deeper queries.

## Validated against

- Docs: [Postgres CLI](https://docs.railway.com/cli/postgres)
- CLI source (v5.49.1): [postgres.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/postgres.rs), [mysql.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/mysql.rs), [redis.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/redis.rs), [pitr.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/database/pitr.rs), [ha.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/database/ha.rs), [pool.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/database/pool.rs), [ops_log.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/database/ops_log.rs)

# Deploy

Ship code, manage releases, and configure builds.

## Deploy code

### Standard deploy

```bash
railway up --detach -m "<release summary>"
```

`--detach` (alias `--no-wait`) returns after upload without waiting for the deployment. An existing-project `up` also returns after upload when stdout is not a TTY and neither `--ci`, `--json`, nor CI environment mode is active. Do not assume omitting `--detach` makes an agent invocation wait. Include `-m` with a release summary for auditability.

### Verify before reporting — `--detach` only means QUEUED

A detached `up` confirms upload, not a successful deployment. Capture the deployment ID from the upload (`--detach --json` includes `deploymentId` for an authenticated deployment). Poll for that deployment's terminal state; do not mistake a concurrent newer deployment for the one just submitted:

```bash
railway deployment list --service <service> --environment <environment> --json    # newest first; check .status
railway deployment list --project <project> --environment <environment> --service <service> --json
```

Poll with the same project, environment, and service scope used for `railway up`. If the deploy used URL-derived IDs or `--project`, do not rely on the directory's linked context.

- `QUEUED` / `INITIALIZING` / `WAITING` / `BUILDING` / `DEPLOYING` → still in progress. Keep polling (every 10-15s); tell the user "build in progress" if you report interim status.
- `NEEDS_APPROVAL` → waiting for manual approval. Report that approval is required; do not keep polling as if the deployment is still building.
- `SUCCESS` → now it's deployed; report it.
- `FAILED` / `CRASHED` → do not report success. Pull scoped logs (`railway logs --service <service> --json --lines 100`) and triage per [operate.md](operate.md).
- `SLEEPING` / `SKIPPED` / `REMOVED` / `REMOVING` / unknown → do not report success. Report the exact state and inspect status/logs to decide the next action.

Observe `SUCCESS` for the submitted deployment before claiming success. Exit 0 can mean upload returned early or watch patterns skipped the build. A timeout, ended stream, or transport error without a deployment verdict leaves the outcome unknown; poll the existing deployment before deciding whether a retry is necessary.

### Watch the build

```bash
railway up --ci -m "<release summary>"
```

`--ci` streams build logs and waits for the deployment verdict; `--json` also enables CI behavior for authenticated deployments. On CLI 5.41+, failed build-log/status WebSocket connections fall back to HTTP status polling in CI mode. A log transport warning alone does not mean the deployment failed; let the verdict arrive or query its ID. Use a bounded process timeout because polling may outlive missing deployments or nonterminal states. `--ci` can exit 0 when no changed files match watch patterns, so report a skipped deployment accurately.

### Targeted deploy

When multiple services exist, target explicitly:

```bash
railway up --service <service> --environment <environment> --detach -m "<summary>"
```

### Deploy to an unlinked project

For CI or cross-project deploys where the directory isn't linked:

```bash
railway up --project <project-id> --environment <environment> --detach -m "<summary>"
```

`--project` requires `--environment`. Railway needs both to resolve context.

### Path deploys

Deploy a subdirectory without changing the shell's working directory:

```bash
railway up ./apps/api --path-as-root --service api --environment production --detach -m "<summary>"
```

Use `--path-as-root` when the path argument should be archived as the root of the deploy instead of preserving the parent directory prefix.

## Manage releases

### Redeploy and restart

```bash
railway redeploy --service <service> --yes              # redeploy the latest deployment
railway redeploy --service <service> --from-source --yes # pull latest commit or image
railway restart --service <service> --yes                # restart without rebuilding
```

Redeploy recreates the latest deployment without uploading local code. Use `--from-source` when the service is linked to a repo or image and you need Railway to pull the latest configured source. Restart only restarts the running container. Use restart when the code hasn't changed but the service needs a fresh process.

### Remove latest deployment

```bash
railway down --service <service> --yes
```

This removes the latest successful deployment but doesn't delete the service. To delete a service entirely, use `railway service delete`.

### Delete a service

Use service deletion when the user wants to remove the service itself:

```bash
railway service delete --service <service> --environment <environment> --yes --json
```

Deleting a service is destructive. Confirm the target service and environment before running it.

## Deployment history and logs

```bash
railway deployment list --service <service> --environment <environment> --limit 20 --json
railway deployment list --project <project> --environment <environment> --service <service> --limit 20 --json
railway logs --service <service> --lines 200 --json              # runtime logs
railway logs --service <service> --build --lines 200 --json      # build logs
railway logs --latest --lines 200 --json                         # latest deployment
```

In an interactive terminal, `railway logs` streams indefinitely when no bounding flags are given. Always use `--lines`, `--since`, or `--until` to get a bounded fetch for agent workflows.

## Build configuration

Railway uses Railpack as the default builder. It detects language and framework from repo contents and assembles a build plan automatically.

### Builder selection

Three builder options, set via service config:

- **RAILPACK** auto-detects language and framework, builds from source (default)
- **NIXPACKS** is the legacy builder. Use RAILPACK instead.
- **DOCKERFILE** uses a Dockerfile you provide

```bash
railway environment edit --service-config <service> build.builder RAILPACK
railway environment edit --service-config <service> build.builder DOCKERFILE
railway environment edit --service-config <service> build.dockerfilePath "docker/Dockerfile.prod"
```

### Build and start commands

Override when auto-detection gets it wrong:

```bash
railway environment edit --service-config <service> build.buildCommand "npm run build"
railway environment edit --service-config <service> deploy.startCommand "npm start"
```

Common reasons to override: wrong package manager detected, multiple build targets in a monorepo, framework-specific output paths.

### Railpack environment variables

Control Railpack behavior by setting these as service variables:

| Variable | Purpose |
|---|---|
| `RAILPACK_NODE_VERSION` | Pin Node.js version (e.g., `20`, `22.1.0`) |
| `RAILPACK_PYTHON_VERSION` | Pin Python version (e.g., `3.12`) |
| `RAILPACK_GO_BIN` | Go binary name to build |
| `RAILPACK_STATIC_FILE_ROOT` | Directory for static site output (e.g., `dist`, `build`) |
| `RAILPACK_SPA_OUTPUT_DIR` | SPA output directory with client-side routing support |
| `RAILPACK_PACKAGES` | Additional system packages for the build |
| `RAILPACK_BUILD_APT_PACKAGES` | Apt packages available during build only |
| `RAILPACK_DEPLOY_APT_PACKAGES` | Apt packages available at runtime only |

For full Railpack documentation including language-specific detection, config files, and framework support: https://railpack.com/llms.txt

### Static sites

Railpack detects static sites from `Staticfile`, `index.html`, or `RAILPACK_STATIC_FILE_ROOT` and serves them with a built-in static file server. If the build outputs to a non-standard directory (for example, `dist/`, `build/`), set `RAILPACK_STATIC_FILE_ROOT` as a variable so Railpack knows where to find the output.

## Monorepo patterns

### Isolated monorepo

When services don't share code, isolate each with its own root directory:

```bash
railway environment edit --service-config <service> source.rootDirectory "/packages/api"
```

Each service sees only its subdirectory. This approach is clean but breaks if services import from shared packages.

### Shared monorepo

When services depend on shared packages or root-level workspace config, keep the full repo context and scope via build/start commands instead:

```bash
# pnpm workspaces
railway environment edit --service-config <service> build.buildCommand "pnpm --filter api build"
railway environment edit --service-config <service> deploy.startCommand "pnpm --filter api start"

# yarn workspaces
railway environment edit --service-config <service> build.buildCommand "yarn workspace api build"
railway environment edit --service-config <service> deploy.startCommand "yarn workspace api start"

# bun workspaces
railway environment edit --service-config <service> build.buildCommand "bun run --filter api build"
railway environment edit --service-config <service> deploy.startCommand "bun run --filter api start"

# turborepo (works with any package manager)
railway environment edit --service-config <service> build.buildCommand "npx turbo run build --filter=api"
railway environment edit --service-config <service> deploy.startCommand "npx turbo run start --filter=api"
```

Don't set a restrictive `rootDirectory` in this case. The build needs access to the workspace root.

### Watch paths

Prevent unrelated package changes from redeploying every service:

```bash
railway environment edit --service-config <service> build.watchPatterns '["packages/api/**","packages/shared/**"]'
```

### Common monorepo pitfalls

- **Using `rootDirectory` with shared imports**: if service A imports from `packages/shared/`, setting `rootDirectory: "/packages/a"` hides the shared code. Use the shared monorepo pattern instead.
- **Forgetting watch paths**: without watch paths, every push redeploys all services, even when only one package changed.
- **Wrong filter target**: `pnpm --filter api` uses the `name` field in each package's `package.json`, not the directory name. Verify the package name matches.

## Troubleshoot deploys

- **No project/service context**: run `railway link` or pass `--project` with `--environment`
- **Build fails before compile**: check dependency graph, lockfiles, and whether the right builder is selected
- **Build succeeds but app crashes**: verify start command and required runtime variables
- **Wrong files in build**: check root directory and watch patterns
- **`railway down` treated as delete**: `down` only removes the latest deployment. For service deletion, use `railway service delete`
- **Wrong Node/Python version detected**: set `RAILPACK_NODE_VERSION` or `RAILPACK_PYTHON_VERSION` as a service variable to pin the version
- **Missing system package at runtime**: add the package to `RAILPACK_DEPLOY_APT_PACKAGES`

## Validated against

- Docs: [up.md](https://docs.railway.com/cli/up), [deploying.md](https://docs.railway.com/cli/deploying), [deployment.md](https://docs.railway.com/cli/deployment), [redeploy.md](https://docs.railway.com/cli/redeploy), [service.md](https://docs.railway.com/cli/service), [down.md](https://docs.railway.com/cli/down), [railpack.md](https://docs.railway.com/builds/railpack), [monorepo.md](https://docs.railway.com/deployments/monorepo)
- CLI source: [up.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/up.rs), [deployment.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/deployment.rs), [down.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/down.rs), [redeploy.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/redeploy.rs), [restart.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/restart.rs), [service.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/service.rs)

# Feature flags

Manage Railway feature flags (Signals): typed defaults, targeting rules, and runtime reads via the SDK or GraphQL.

## What they are

Railway feature flags are **project- or workspace-scoped** configuration values resolved at read time from a registry (not environment variables). Each flag has:

- A **type**: `bool`, `string`, `number`, or `json`
- A **default** value used when no rule matches
- Optional **targeting rules** (attribute comparisons and rollout percentages)

Project flags are managed in **Project Settings → Feature Flags**. Workspace flags are visible there read-only.

Runtime apps read **one scope** at a time (`project:<id>` or `workspace:<id>`) via the [TypeScript SDK](https://github.com/railwayapp/railway-ts-sdk) or GraphQL — flags from other scopes are separate registries.

## Agent / MCP workflow

Prefer **Remote MCP** tools when OAuth-scoped project access is enough:

| Tool | Access | Purpose |
|---|---|---|
| `list-feature-flags` | viewer | List project flags; includes workspace flags when present |
| `get-feature-flag` | viewer | Inspect one flag (`scope`: `project` or `workspace`) |
| `set-feature-flag` | admin | Create a flag or update its default |
| `delete-feature-flag` | admin | Delete a project-scoped flag |

Always pass explicit `projectId` (from a Railway URL or `railway status --json`).

```text
List feature flags for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb
```

```text
Set feature flag checkout-v2 on project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb to true (bool)
```

For targeting rules and rollouts, use the CLI commands below when MCP has no matching rule tool. The dashboard or GraphQL (`signalRuleSet`) remains a fallback.

## CLI defaults, rules, and rollouts

CLI 5.24+ exposes `railway flag` (alias `railway flags`). Use `--scope project:<id>` for explicit targeting; `--project` is not a flag option. On 5.26.2+, omitted scope can come from the project token or linked project.

```bash
railway flag list --scope project:<project-id> --json
railway flag list --scope project:<project-id> --full
railway flag set checkout-v2 true --scope project:<project-id>
railway flag set theme blue --type string --scope project:<project-id>
railway flag set checkout-v2 true --scope project:<project-id> --when 'plan == "enterprise"' --rule-id enterprise
railway flag set checkout-v2 true --scope project:<project-id> --when 'bucket(key) < 0.25' --rule-id rollout-25
railway flag unset checkout-v2 --scope project:<project-id> --rule-id enterprise
railway flag delete checkout-v2 --scope project:<project-id>
```

`set` changes the default unless `--when` is supplied. Rules accept a CEL expression subset or raw JSON; `bucket(key)` gives deterministic percentage targeting using the evaluation context's key. Use a stable `--rule-id` when updating a rule, and list/read back the rules afterward. `unset` removes one rule; `delete` removes the entire flag. CLI mutations target project flags; do not assume workspace mutation support.

Types are inferred unless `--type bool|string|number|json` is given. `--force` permits replacing a flag's type and **clears its rules**; use it only when that replacement is intended.

## GraphQL fallback

For API operations beyond these commands, inspect the public GraphQL API (`signals`, `signalCreate`, `signalDefaultSet`, `signalRuleSet`, `signalDelete`) using [request.md](request.md). Owners use `project:<projectId>` or `workspace:<workspaceId>`; access remains subject to the API's scope and permission checks.

```bash
railway api \
  'query projectSignals($owner: String!) {
    signals(owner: $owner) { name type default rules version }
  }' \
  --variables '{"owner":"project:<projectId>"}'
```

## Runtime SDK

In deployed services, use the Railway SDK flags module:

```typescript
import { flags } from "railway";

await flags.init({ scope: { projectId: process.env.RAILWAY_PROJECT_ID! } });

const enabled = flags.getBoolean("checkout-v2", {
  targetingKey: userId,
  attributes: { plan: "enterprise" },
});
```

Poll interval defaults are suitable for most apps; flags refresh when registry versions change.

## Do not confuse with

- **Environment variables** — static per-environment config (`railway variable set`). Use flags when you need typed defaults, targeting, and central registry semantics.
- **Priority Boarding / account toggles** — internal beta switches on your Railway account, unrelated to project feature flags.
- **Platform feature flags** — Railway staff tooling only.

## Dashboard

Human-friendly CRUD: open the project → **Settings → Feature Flags**. Workspace-scoped flags appear in a read-only section when they exist.

## Validated against

- Docs: [Feature flags](https://docs.railway.com/feature-flags), [CLI flags](https://docs.railway.com/cli/flag)
- CLI source (v5.49.1): [flag.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/flag.rs), [signals.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/controllers/signals.rs)

# Configuration in source control

Use Infrastructure as Code for project configuration. Keep one authoring file and never manage a service with both IaC and legacy Config as Code.

## Choose the model

TypeScript IaC is generally available. Python and Go authoring are in beta. Preserve an existing `.railway/railway.ts`, `.railway/railway.py`, or `.railway/railway.go`; do not switch languages based on the app's `package.json`, `go.mod`, or framework. With no existing authoring file, `config init` and `config pull` default to TypeScript even for non-TypeScript apps. `config init` and `config pull` have no language flag: the CLI picks the language only from an existing authoring file. To honor an explicit Python/Go preference on a fresh project, create an empty `.railway/railway.py` or `.railway/railway.go` first, then run `config pull` so the import is emitted in that language. `config migrate --lang py|go` emits Python/Go only when translating legacy `railway.json`/`railway.toml`.

`railway.json` and `railway.toml` are deprecated. New services cannot opt into Config as Code; existing files stop being read on **2026-12-01**. Do not create them as a fallback. For an existing legacy service, use the migration workflow below; if the user requests a temporary legacy edit, explain the cutoff and keep its current format.

| Model | Scope | Applies when |
|---|---|---|
| `.railway/railway.ts`, `.py`, or `.go` | Project/environment: services, databases, buckets, volumes, variables, replicas, domains, and canvas groups | `railway config apply` applies the project plan |
| Existing `railway.json` / `railway.toml` | One legacy service's build and deploy settings | Read during deployments until the cutoff; overrides dashboard values for that deployment |

## Infrastructure as Code

Keep exactly one of these files:

```text
.railway/railway.ts
.railway/railway.py
.railway/railway.go
```

`railway config init` and `railway config pull` also create `.railway/README.md`. Railway agent setup installs the shared `use-railway` skill; config commands do not need or create a project-local skill.

Install the matching authoring package in the config's package environment: `npm install railway`, `pip install railway-sdk`, or `go get github.com/railwayapp/railway-go-sdk`. Python/Go imports can generate `.railway/requirements.txt` or `.railway/go.mod`; use those when present. CLI 5.42+ evaluates the graph natively, but still needs the language runtime and SDK to evaluate authoring code. `--runner` is an optional legacy TypeScript runner override, not a prerequisite.

### Core rules

1. Express Railway product intent, not internal API details.
2. Do not write Railway UUIDs into the authoring file.
3. Do not write `EnvironmentConfigPatch`, `ServiceInstance`, Backboard internals, or generated Railway domains into source.
4. Prefer helpers such as `service()`, `postgres()`, `redis()`, `mysql()`, `mongo()`, `bucket()`, `volume()`, `group()`, `github()`, and `image()`.
5. Use `service.env.VARIABLE` and `database.env.VARIABLE` for references.
6. Keep secrets out of source. Imports use `preserve()` by default for existing variables; omit them only when a smaller import is intended.
7. Prefer product DSL names such as `domains`, `replicas`, and `group`; avoid internal names such as `customDomains` and `multiRegionConfig`.
8. Do not add platform defaults unless the user explicitly wants them.
9. After editing the authoring file, run `railway config plan`.
10. Do not run `railway config apply` unless the user explicitly asks.
11. Never use `railway config apply --yes` or `--confirm-destructive` from an agent session without explicit user approval for the exact plan.

### Initialize or import

```bash
railway config init                         # TypeScript by default; preserve existing language
railway config init --force                 # overwrite existing generated files
railway config pull                         # import the linked project
railway config pull --force                 # overwrite the existing authoring file
railway config pull --omit-preserved-variables # omit unknown variable values
railway config pull --json                  # print current graph instead of writing files
railway config pull --agent                 # ask an agent to clean the import afterward
railway config pull --include-variables     # decrypt and inline non-sealed values
```

Use `--include-variables` only when writing those values to source is intended: it includes non-sealed secrets too. Sealed values remain `preserve()`. A normal import should produce a no-change plan; inspect any diff before applying.

Plan/apply discover the nearest `.railway/railway.{ts,py,go}` by walking up from the current directory. `--file <path>` overrides that selection. Run init/pull/migrate from the intended project root. Config commands do not expose `--project`/`--environment` selectors: verify the link or use `railway link --project <id> --environment <env>` before noninteractive work. Interactive commands can prompt for missing context.

### Authoring

Import the helpers needed by the project:

```ts
import {
  bucket,
  defineRailway,
  github,
  group,
  image,
  mongo,
  mysql,
  postgres,
  preserve,
  project,
  redis,
  service,
  volume,
} from "railway/iac";
```

Common resource patterns:

```ts
const db = postgres("postgres");

const api = service("api", {
  source: github("owner/repo", { branch: "main" }),
  build: "pnpm build",
  preDeploy: "pnpm db:migrate",
  start: "pnpm start",
  env: {
    DATABASE_URL: db.env.DATABASE_URL,
    INTERNAL_TOKEN: preserve(),
  },
  domains: ["api.example.com"],
  replicas: 3,
  volumeMounts: {
    "/data": volume("uploads", { sizeMB: 4096 }),
  },
});

const worker = service("worker", {
  source: image("ghcr.io/acme/worker:latest"),
  env: {
    API_HOST: api.env.RAILWAY_PRIVATE_DOMAIN,
    API_TOKEN: api.env.INTERNAL_TOKEN,
  },
});

const media = bucket("media", { region: "iad" });
const backend = group("Backend", [api, worker, db]);
```

A traced service also carries `tracing: { enabled: true, autoInstrumentation: true }`, which `config pull` renders and `service()` passes through to the CLI. This needs CLI 5.63.0 or newer (an older CLI drops the block on compile and never diffs it) and `railway` 3.12.0+, `railway-sdk` 0.3.0+ or the Go SDK v0.3.0+. An older SDK drops the block too, and on CLI 5.63.0+ that makes every plan against a traced service propose removing `tracing`; see [Infrastructure as code in tracing.md](tracing.md#infrastructure-as-code) before applying one.

Advanced placement can map regions to replica counts:

```ts
const web = service("web", {
  replicas: {
    "us-west2": 2,
    "europe-west4": 1,
  },
});
```

Return every managed resource from the project:

```ts
export default defineRailway(() => {
  const db = postgres("postgres");
  const web = service("web", {
    env: { DATABASE_URL: db.env.DATABASE_URL },
  });

  return project("my-app", {
    resources: [db, web],
  });
});
```

### Plan

Always plan after editing and before applying:

```bash
railway config plan
railway config plan --json
railway config plan --verbose
railway config plan --show-values
railway config plan --detailed-exit-code
```

Plan output summarizes changes in Terraform style:

```text
Plan: 1 to add, 0 to change, 0 to destroy
```

Variable values are redacted by default. Use `--show-values` only when the user accepts the secret exposure risk. For CI drift checks, `--detailed-exit-code` returns `0` for no changes, `2` for pending changes, and another non-zero code for errors.

### Apply

```bash
railway config apply
railway config apply --yes
railway config apply --yes --confirm-destructive
railway config apply --json --yes --confirm-destructive
```

Ordinary `apply` evaluates a fresh plan. Changes to the environment between planning inside that invocation and applying are rejected. In non-interactive or agent sessions, destructive changes require `--confirm-destructive` in addition to `--yes` or `--json`; add it only after the user explicitly approves the exact destructive impact.

### Apply the reviewed plan in CI

CLI 5.45.1+ can save the evaluated change set and apply that exact artifact:

```bash
railway config plan --out railway-plan.json
railway config apply --plan railway-plan.json --yes
```

Review the first command's diff before authorizing the apply. Store the artifact outside `.railway/` because that directory's source tree is pinned. The artifact contains the change set, environment ID and config etag, and source tree identity. `apply --plan` does not reevaluate the authoring code; it rejects changed source or remote state rather than quietly generating another plan. Keep the same CLI version and source checkout between jobs. Use `--source-tree` on `plan` only when CI deliberately supplies the source identity. Treat plan artifacts as potentially secret-bearing even when terminal diffs are redacted.

When a saved plan is stale, create and review a replacement. Destructive saved plans still require `--confirm-destructive`. For maintained CI integration, use [railwayapp/config](https://github.com/railwayapp/config).

### Review checklist

Before applying, confirm:

- The latest plan shows only expected changes.
- The user reviewed the plan and explicitly requested apply.
- Secrets are not replaced with literal placeholders.
- Existing Railway-managed variables are omitted or use `preserve()` when they should remain untouched.
- Custom domains use `domains`, not networking internals.
- Scaling uses `replicas`, not `multiRegionConfig`.
- No generated Railway service domains or Railway UUIDs are committed.

### Troubleshoot IaC

- **Service is already managed by Config as Code**: use migration below; deleting the file alone does not clear its Railway Config File setting.
- **Plan shows secrets as hidden**: expected. Use `--show-values` only with user approval.
- **Apply says the plan is stale**: run a new plan, inspect it, then apply again only if requested.
- **Destructive apply is blocked**: get explicit approval for the exact plan before adding `--confirm-destructive`.
- **Imported variables use `preserve()`**: this is the default, including readable values; it retains the remote value without putting it into source.
- **Missing Railway package**: install the SDK for the existing authoring language in its runtime environment; do not replace the config language to work around the error.
- **Generated code is too literal**: simplify the authoring file, then run another plan.

## Migrate legacy Config as Code

Use CLI **5.49.1 or newer**: its TypeScript migration and all-language pull fixes preserve `preDeployCommand` as the first-class `preDeploy` field. Earlier TypeScript migration output could silently omit a database migration command.

Start with a dry-run from the repository root:

```bash
railway config migrate
railway config migrate --service api
railway config migrate --lang py
railway config migrate --lang go
```

The default emits proposed TypeScript without writing files or clearing remote settings. `--service` selects a service when multiple configs are found; with one file it can name the emitted service. Inspect the discovered paths and service mapping in a monorepo. Single-service migration may emit a named partial rather than claim the whole project; retain that ownership boundary when merging it into existing IaC.

In v5.49.1, Python/Go **migration** emits only build/start commands and the healthcheck path. Manually carry over other intended settings, including pre-deploy commands, replicas, and healthcheck timeouts, before applying the resulting IaC. A successful migration command is not proof of a lossless translation in any language; compare the original files and plan.

For an authorized migration, write the result and clear the discovered services' Railway Config File settings:

```bash
railway config migrate --apply
railway config migrate --apply --delete-files
```

`--apply` here writes source and changes remote config-file settings; it does **not** apply the resulting IaC project plan. `--delete-files` additionally removes discovered legacy files and requires `--apply`. If an authoring file already exists, merge the dry-run output into it instead of using `--force` to overwrite unrelated resources. Preserve the language with `--lang` when needed; migration defaults to `ts`.

Before completing the migration:

- Compare build/start/pre-deploy commands, health checks, regions/replicas, and environment overrides with the original files. Do not assume every legacy field was translated.
- Preserve secrets and review the service identities and ownership scope.
- Run `railway config plan`; inspect unexpected removals or changes, then apply only within the user's authorized scope.
- Check for remaining legacy files and custom Railway Config File paths so deployments do not retain dual ownership.

For a requested temporary edit to an existing legacy service, retain its format, validate against the [Config as Code reference](https://docs.railway.com/config-as-code/reference), and explain the 2026-12-01 cutoff. Config as Code has deployment-time precedence over dashboard settings; a deployment is needed for edits to take effect. Keep variables and secrets in Railway variables.

## Validated against

- Docs: [Infrastructure as Code](https://docs.railway.com/infrastructure-as-code), [IaC reference](https://docs.railway.com/infrastructure-as-code/reference), [Config as Code](https://docs.railway.com/config-as-code)
- CLI source (v5.49.1): [config/mod.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/config/mod.rs), [config/migrate.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/config/migrate.rs), [authoring.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/config/authoring.rs), [eval.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/iac/eval.rs), [saved_plan.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/iac/saved_plan.rs)
- IaC SDK sources for the `tracing` block: railway-ts-sdk `src/iac/sdk.ts` and `src/iac/schema.ts` (`ServiceTracing`; `railway` 3.12.0), railway-py-sdk `_normalize_tracing` (`railway-sdk` 0.3.0), railway-go-sdk `serviceNode` (v0.3.0); CLI 5.63.0 `src/iac/compiler.rs` and `src/iac/change_set.rs`

# Operate

Check health, read logs, query metrics, and troubleshoot failures.

## Health snapshot

Start broad, then narrow:

```bash
railway status --json                                    # linked context
railway status --project <project> --environment <env> --json
railway service list --json                              # services in current environment
railway deployment list --limit 10 --json                # recent deployments
railway deployment list --project <project> --environment <env> --service <service> --limit 10 --json
```

Use explicit `--project`, `--environment`, and `--service` when the user provided a URL or when the current directory may be linked to a different project.

Deployment statuses include `SUCCESS`, `QUEUED`, `INITIALIZING`, `WAITING`, `BUILDING`, `DEPLOYING`, `NEEDS_APPROVAL`, `FAILED`, `CRASHED`, `SLEEPING`, `SKIPPED`, `REMOVING`, and `REMOVED`.

For projects with buckets, include bucket status:

```bash
railway bucket list --json                                       # buckets in current environment
railway bucket info --bucket <name> --json                       # storage size, object count, region
```

If everything looks healthy, return a summary and stop. If something is degraded or failing, continue to log inspection.

## Logs

### Recent logs

```bash
railway logs --service <service> --lines 200 --json              # runtime logs
railway logs --service <service> --build --lines 200 --json      # build logs
railway logs --latest --lines 200 --json                         # latest deployment
```

In an interactive terminal, `railway logs` streams indefinitely when no bounding flags are given. Always use `--lines`, `--since`, or `--until` to get a bounded fetch for agent workflows.

### Time-bounded queries

```bash
railway logs --service <service> --since 1h --lines 400 --json
railway logs --service <service> --since 30m --until 10m --lines 400 --json
```

### Filtered queries

Use `--filter` to narrow logs without scanning everything manually:

```bash
railway logs --service <service> --lines 200 --filter "@level:error" --json
railway logs --service <service> --lines 200 --filter "@level:warn AND timeout" --json
railway logs --service <service> --lines 200 --filter "connection refused" --json
```

Filter syntax supports text search (`"error message"`), attribute filters (`@level:error`, `@level:warn`), and boolean operators (`AND`, `OR`, `-` for negation). Full syntax: https://docs.railway.com/cli/logs

### Scoped by environment

```bash
railway logs --service <service> --environment <env> --lines 200 --json
```

### HTTP logs

Use HTTP logs when a service responds with errors, latency spikes, or routing problems:

```bash
railway logs --service <service> --http --status ">=400" --lines 100 --json
railway logs --service <service> --http --method POST --path /api/users --lines 100 --json
railway logs --service <service> --http --request-id <request-id> --lines 20 --json
railway logs --service <service> --http --filter "@totalDuration:>=1000" --lines 100 --json
```

HTTP filter fields include `@method`, `@path`, `@host`, `@requestId`, `@srcIp`, `@edgeRegion`, `@httpStatus`, `@totalDuration`, `@responseTime`, `@txBytes`, and `@rxBytes`.

### Network flow logs

Use network flow logs for private networking, TCP proxy, outbound allowlist, or dropped-packet investigations. Use DNS query logs below for resolution results:

```bash
railway logs --service <service> --network --lines 100 --json
railway logs --service <service> --network --direction egress --protocol tcp --lines 100 --json
railway logs --service <service> --network --peer postgres --port 5432 --lines 100 --json
railway logs --service <service> --network --status dropped --lines 100 --json
railway logs --service <service> --network --filter "@peer_kind:internet @port:443" --lines 100 --json
```

Network flow logs are service-level, not deployment-level. Do not pass a deployment ID or `--latest` with `--network`.

Useful filters:

| Flag | Use for |
|---|---|
| `--protocol tcp|udp|icmp|icmpv6|unknown` | Layer 4 protocol |
| `--direction ingress|egress` | Traffic direction |
| `--peer <service|internet|dns|edge-proxy>` | Named peer or well-known peer |
| `--peer-kind service|internet|edge_proxy|local_dns|unknown` | Peer class |
| `--status ok|dropped` / `--dropped true` | Dropped traffic |
| `--port <port>` | Source or destination port |
| `--src`, `--dst`, `--host` | IP filters |
| `--drop-cause <cause>` | Drop reason |

### DNS query logs

CLI 5.29+ exposes DNS resolution results directly:

```bash
railway logs --service <service> --environment <env> --dns --lines 100 --json
railway logs --service <service> --dns --status failed --since 1h --lines 100 --json
railway logs --service <service> --dns --rcode NXDOMAIN --lines 100 --json
railway logs --service <service> --dns --qname backend.railway.internal --zone internal --lines 50 --json
railway logs --service <service> --dns --domain example.com --qtype AAAA --lines 100 --json
```

Use `--qname` for the full query name, `--domain` for domain filtering, `--qtype` for record type, `--rcode` for DNS response code, and `--zone internal|external` for lookup scope. `--status failed` finds failed resolutions. DNS logs are service-level and mutually exclusive with build, deployment, HTTP, and network modes; do not pass a deployment ID or `--latest`. Correlate failed lookups with runtime errors and network flows instead of treating a DNS failure as an application crash.

## Metrics

Use `railway metrics` for resource and HTTP metrics. It summarizes CPU, memory, network, volume, and HTTP data for the linked service by default.

```bash
railway metrics --service <service> --since 1h --json
railway metrics --service <service> --since 6h --cpu --memory --json
railway metrics --service <service> --http --method POST --path /api/users --json
railway metrics --all --environment production --json
```

Use `--raw` for time-series data points:

```bash
railway metrics --service <service> --raw --cpu --json
```

Metric flags can be combined: `--cpu`, `--memory`, `--network`, `--volume`, and `--http`. Use `--watch` only in an interactive terminal; it opens a live TUI and conflicts with `--json` and `--raw`.

For custom grouping or measurements the CLI doesn't expose, use the GraphQL fallback in [request.md](request.md).

For latency or errors that span several services, or a request a user reported with an `x-railway-trace-id` header, use tracing instead of correlating logs by hand: see [tracing.md](tracing.md).

## SSH

Use SSH when logs and metrics don't expose enough state and the user needs shell-level inspection inside a running service.

```bash
railway ssh --service <service> --environment <env>
railway ssh --service <service> --environment <env> -- "printenv | sort"
railway ssh --service <service> --environment <env> --session railway-debug
railway ssh --service <service> --environment <env> --identity-file ~/.ssh/id_ed25519_railway
```

Manage Railway SSH keys with:

```bash
railway ssh keys list
railway ssh keys add --key ~/.ssh/id_ed25519.pub --name <key-name>
railway ssh keys github
railway ssh keys remove <key-id> --2fa-code <code>
```

Workspace-owned keys use `--workspace <workspace-id>` and require workspace Admin access. SSH key management doesn't work with project tokens (`RAILWAY_TOKEN`); use `railway login` or a workspace-scoped `RAILWAY_API_TOKEN`.

## Database inspection

For database-level metrics and introspection, use the analysis scripts. `railway metrics` can provide infrastructure metrics and supported database summaries, while the scripts provide deeper engine-level analysis. See [analyze-db.md](analyze-db.md) for comprehensive database analysis including:

- Deep Postgres analysis (pg_stat_statements, vacuum health, index health, cache hit ratios)
- HA cluster checks (Patroni, etcd, HAProxy)
- Redis, MySQL, and MongoDB introspection
- Combined analysis via `scripts/analyze-<type>.py` (postgres, mysql, redis, mongo)

For native PITR/HA/PgBouncer status and operations, use [databases.md](databases.md). For billed usage and spending limits, use [usage.md](usage.md); infrastructure metrics are not a billing statement.

## Failure triage

When something is broken, classify the failure first. The fix depends on the class.

### Build failures

The service failed to build. Look at build logs:

```bash
railway logs --latest --build --lines 400 --json
```

Common causes and fixes:
- **Missing dependencies**: check lockfiles, verify package manager detection
- **Wrong build command**: override with `railway environment edit --service-config <service> build.buildCommand "<command>"`
- **Builder mismatch**: switch builders with `railway environment edit --service-config <service> build.builder RAILPACK`
- **Wrong root directory** (monorepo): set `source.rootDirectory` to the correct package path

### Runtime failures

The build succeeded but the service crashes or misbehaves:

```bash
railway logs --latest --lines 400 --json
railway logs --service <service> --since 1h --lines 400 --json
```

Common causes and fixes:
- **Bad start command**: override with `railway environment edit --service-config <service> deploy.startCommand "<command>"`
- **Missing runtime variable**: check `railway variable list --service <service> --json` and set missing values
- **Port mismatch**: the service must listen on `$PORT` (Railway injects this). Verify with logs.
- **Upstream dependency down**: check other services' status and logs

### Config-driven failures

Something worked before and broke after a config change:

```bash
railway environment config --json
railway variable list --service <service> --json
```

Compare the config against expected values. Look for changes that may have introduced the regression.

### Networking failures

Domain returns errors, or service-to-service calls fail:

```bash
railway domain list --service <service> --json
railway domain status <domain> --service <service> --json
railway private-network status --service <service> --json
railway tcp-proxy list --service <service> --json
railway outbound-network status --service <service> --json
railway logs --service <service> --http --status ">=400" --lines 100 --json
railway logs --service <service> --network --status dropped --lines 100 --json
```

Check: target port matches what the service listens on, domain status is healthy, private domain variable references are correct, TCP proxy status is active, and outbound networking changes have been followed by the required redeploy.

### CDN and WAF incidents

For cache behavior, inspect both CLI settings and response headers:

```bash
railway cdn status --service <service> --json
railway logs --service <service> --http --status ">=400" --lines 100 --json
curl -I https://<domain>/<path>
curl https://<domain>/.railway/cdn-trace?json
```

`x-cache: HIT` means the request did not reach the service. `DYNAMIC` means the edge reached the service but did not cache the response. Check method, `Authorization`, `Set-Cookie`, `Cache-Control`, `Vary`, response size, and HTML caching mode.

For active traffic floods or unexpected `429` responses:

```bash
railway waf under-attack status --service <service> --json
railway logs --service <service> --http --status 429 --lines 100 --json
```

Under Attack Mode can block API clients and webhooks. If the service is API-only, disabling WAF may be the correct recovery after confirming with the user.

## Recovery

After identifying the cause, fix and verify:

```bash
# Fix (examples)
railway environment edit --service-config <service> deploy.startCommand "<correct-command>"
railway variable set MISSING_VAR=value --service <service>

# Redeploy
railway redeploy --service <service> --yes

# Verify
railway deployment list --service <service> --limit 5 --json
railway logs --service <service> --lines 200 --json
```

Always verify after fixing. Don't assume the redeploy succeeded.

## Troubleshoot common blockers

- **`OAUTH_INSUFFICIENT_GRANT` / resource access denied**: CLI 5.37.4+ distinguishes a live OAuth session without access from an expired login. Check the resource IDs, workspace membership, and integration grant scope; repeating the same login may retain the same insufficient grant. Reauthorize with the necessary access only when the user intends that scope.
- **Expired or invalid credentials**: follow the CLI's login/token-specific error. Transient refresh failures are not proof that access was revoked. Newer CLI versions refresh long-lived clients too; upgrade an old CLI before repeatedly reinstalling MCP to address stale authentication.
- **CI log stream failed**: on CLI 5.41+, the command falls back to status polling. Inspect the submitted deployment before retrying; a logging error alone is not a deployment failure.

- **Unlinked context**: `railway link --project <id-or-name>`
- **Missing service scope for logs**: pass `--service` and `--environment` explicitly
- **Wrong project in status or deploy polling**: pass `--project`, `--environment`, and `--service`; URL IDs beat local linked context
- **No deployments found**: the service exists but has never deployed, create an initial deploy first
- **Metrics return empty**: check the time window, service scope, and whether the service has active deployments
- **Config patch type error**: check the typed paths in [configure.md](configure.md), for example, `numReplicas` is an integer, not a string
- **No network flow logs**: confirm the time window and service scope; network logs are not tied to deployment IDs

## Validated against

- Docs: [status.md](https://docs.railway.com/cli/status), [service.md](https://docs.railway.com/cli/service), [logs.md](https://docs.railway.com/cli/logs), [metrics.md](https://docs.railway.com/cli/metrics), [ssh.md](https://docs.railway.com/cli/ssh), [cdn.md](https://docs.railway.com/cli/cdn), [waf.md](https://docs.railway.com/cli/waf), [observability/logs.md](https://docs.railway.com/observability/logs), [observability/metrics.md](https://docs.railway.com/observability/metrics), [observability/tracing.md](https://docs.railway.com/observability/tracing)
- CLI source: [status.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/status.rs), [service.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/service.rs), [logs.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/logs.rs), [metrics.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/metrics.rs), [ssh/mod.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/ssh/mod.rs), [deployment.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/deployment.rs), [redeploy.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/redeploy.rs), [cdn.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/cdn.rs), [waf.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/waf.rs)
- Authentication and CI recovery (v5.49.1): [client.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/client.rs), [up.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/up.rs)

# Request

Official documentation and community endpoints. GraphQL operations without a dedicated CLI command or MCP tool.

## Official documentation

Primary sources for authoritative Railway information:

- **Full LLM docs**: `https://docs.railway.com/api/llms-docs.md`
- **LLM summary**: `https://railway.com/llms.txt`
- **Templates**: `https://railway.com/llms-templates.md`
- **Changelog**: `https://railway.com/llms-changelog.md`
- **Blog**: `https://blog.railway.com/llms-blog.md`
- **Direct doc pages**: `https://docs.railway.com/<path>` (for example, `cli/up`, `networking/domains`, `observability/logs`)

Tip: append `.md` to any `docs.railway.com` page URL to get a markdown version suitable for LLM consumption.

Common doc paths:

| Topic | Path |
|---|---|
| Agent setup | `agents`, `ai/agent-skills`, `ai/mcp-server` |
| CLI reference | `cli`, `cli/<command>` |
| Projects | `projects` |
| Deployments | `deployments` |
| Volumes | `volumes` |
| Variables | `variables` |
| Infrastructure as Code | `infrastructure-as-code`, `infrastructure-as-code/reference` |
| Public networking | `networking/public-networking` |
| Private networking | `networking/private-networking` |
| Domains, CDN, WAF | `networking/domains`, `networking/cdn`, `networking/waf` |
| Outbound networking | `networking/outbound-networking`, `networking/static-outbound-ips` |

Fetch official docs first for product behavior questions. Use Central Station only when you need community evidence, prior incidents, or implementation anecdotes.


## Central Station (community)

Search and browse Railway's community platform for prior discussions, issue patterns, and field solutions.

### Recent threads

```bash
curl -s 'https://station-server.railway.com/gql' \
  -H 'content-type: application/json' \
  -d '{"query":"{ threads(first: 10, sort: recent_activity) { edges { node { slug subject status upvoteCount createdAt topic { slug displayName } } } } }"}'
```

Filter by topic with the `topic` parameter (`"questions"`, `"feedback"`, `"community"`, `"billing"`):

```bash
curl -s 'https://station-server.railway.com/gql' \
  -H 'content-type: application/json' \
  -d '{"query":"{ threads(first: 10, sort: recent_activity, topic: \"questions\") { edges { node { slug subject status topic { displayName } upvoteCount } } } }"}'
```

Sort options: `recent_activity` (default), `newest`, `highest_votes`.

### Search threads

```bash
curl -s 'https://station-server.railway.com/gql' \
  -H 'content-type: application/json' \
  -d '{"query":"{ threads(first: 10, search: \"<search-term>\") { edges { node { slug subject status } } } }"}'
```

### LLM data export

Bulk search alternative, fetches all public threads with full content:

```bash
curl -s 'https://station-server.railway.com/api/llms-station'
```

### Read a full thread

```bash
curl -s 'https://station-server.railway.com/api/threads/<slug>?format=md'
```

Thread URLs follow the format: `https://station.railway.com/{topic_slug}/{thread_slug}`

Community threads are anecdotal. Always pair with official docs when the answer informs an operational decision.


## GraphQL with the CLI

Use `railway api` (CLI 5.28+) for API operations that dedicated commands and MCP tools cannot express. It uses the CLI's configured authentication and supports normal token refresh. Inspect the live schema before guessing fields or input shapes:

```bash
railway api search projectUpdate --kind mutation
railway api describe ProjectUpdateInput
railway api describe Mutation.projectUpdate
railway api schema --compact
railway api '<query>' --variables '{"id":"<resource-id>"}'
railway api --file query.graphql --variables @variables.json
railway api --file query.graphql --raw-var id=<resource-id> --var enabled=true
```

`--var` parses JSON values when possible; `--raw-var` keeps strings. Queries can also come from stdin, with variables provided separately. Use `--operation-name` for documents containing multiple operations. Output is JSON by default; `--compact` changes formatting and there is no `--json` flag. HTTP errors and GraphQL `errors` fail the command; do not add `--allow-errors` when deciding whether a mutation succeeded. Query resource state before retrying an uncertain mutation.

For an older CLI that cannot be upgraded, the bundled `scripts/railway-api.sh '<query>' '<variables-json>'` remains a compatibility fallback. The database analysis scripts (`dal.py`, `analyze-postgres.py`) still call this helper directly, so it must stay in the plugin even when agents use `railway api`. It reads `user.token` from `~/.railway/config.json`; it does not implement the CLI's environment-token selection or OAuth refresh. It expects query first and variables second, not a query on stdin, and callers must inspect its response for GraphQL errors. Keep the skill telemetry prefix on `railway api` calls just like other CLI calls.

For the full API schema, see: https://docs.railway.com/api/llms-docs.md

## Project mutations

There is no dedicated project command for these setting updates (rename, PR deploys, visibility). Use GraphQL:

```bash
railway api \
  'mutation updateProject($id: String!, $input: ProjectUpdateInput!) {
    projectUpdate(id: $id, input: $input) { id name isPublic prDeploys botPrEnvironments }
  }' \
  --variables '{"id":"<project-id>","input":{"name":"new-name","prDeploys":true}}'
```

Common `ProjectUpdateInput` fields: `name`, `isPublic`, `prDeploys`, `botPrEnvironments`.


## Service mutations

Use `railway add` to create services and GraphQL to rename them or change icons:

```bash
railway api \
  'mutation updateService($id: String!, $input: ServiceUpdateInput!) {
    serviceUpdate(id: $id, input: $input) { id name icon }
  }' \
  --variables '{"id":"<service-id>","input":{"name":"new-name"}}'
```

`ServiceUpdateInput` fields: `name`, `icon` (image URL, animated GIF, or devicons URL like `https://devicons.railway.app/postgres`).

Get the service ID from `railway service list --json`.


## Service creation via GraphQL

Prefer `railway add` for most cases. Use GraphQL for programmatic or advanced use:

```bash
railway api \
  'mutation createService($input: ServiceCreateInput!) {
    serviceCreate(input: $input) { id name }
  }' \
  --variables '{"input":{"projectId":"<project-id>","name":"my-service","source":{"image":"nginx:latest"}}}'
```

`ServiceCreateInput` fields:

| Field | Type | Description |
|---|---|---|
| `projectId` | String! | Target project (required) |
| `name` | String | Service name (auto-generated if omitted) |
| `source.image` | String | Docker image (for example, `nginx:latest`) |
| `source.repo` | String | GitHub repo (for example, `user/repo`) |
| `branch` | String | Git branch for repo source |
| `environmentId` | String | Create only in a specific environment |

After creating a service via GraphQL, configure it with a JSON config patch including `isCreated: true` (see [configure.md](configure.md)).


## Metrics queries

Use `railway metrics` for routine metric checks. Use GraphQL only when you need custom measurements, grouping, sample rates, or averaging windows that the CLI doesn't expose.

```bash
railway api \
  'query metrics($environmentId: String!, $serviceId: String, $startDate: DateTime!, $endDate: DateTime, $sampleRateSeconds: Int, $averagingWindowSeconds: Int, $groupBy: [MetricTag!], $measurements: [MetricMeasurement!]!) {
    metrics(environmentId: $environmentId, serviceId: $serviceId, startDate: $startDate, endDate: $endDate, sampleRateSeconds: $sampleRateSeconds, averagingWindowSeconds: $averagingWindowSeconds, groupBy: $groupBy, measurements: $measurements) {
      measurement tags { serviceId deploymentId region } values { ts value }
    }
  }' \
  --variables '{"environmentId":"<env-id>","serviceId":"<service-id>","startDate":"2026-02-19T00:00:00Z","measurements":["CPU_USAGE","MEMORY_USAGE_GB"]}'
```

Available `MetricMeasurement` values: `CPU_USAGE`, `CPU_LIMIT`, `MEMORY_USAGE_GB`, `MEMORY_LIMIT_GB`, `NETWORK_RX_GB`, `NETWORK_TX_GB`, `DISK_USAGE_GB`, `EPHEMERAL_DISK_USAGE_GB`, `BACKUP_USAGE_GB`.

Optional parameters: `endDate` (defaults to now), `sampleRateSeconds`, `averagingWindowSeconds`. Use `groupBy: ["SERVICE_ID"]` without `serviceId` to query all services in an environment at once. Valid `MetricTag` values for `groupBy`: `SERVICE_ID`, `DEPLOYMENT_ID`, `DEPLOYMENT_INSTANCE_ID`, `REGION`.

Get the environment ID from `railway status --json`. Get service IDs from `railway service list --json`.

## Template search

Use the CLI for template search:

```bash
railway templates search redis --verified true --json
railway templates search --category Storage --limit 10 --json
railway templates search --after <cursor> --json
railway templates list --json
railway templates create --project <project> --environment production --json
```

The CLI search command doesn't require authentication and supports pagination with `pageInfo.endCursor`. Prefer CLI template commands for search, listing owned templates, creating drafts, publishing, unpublishing, and deleting. Use GraphQL only for template workflows the CLI cannot express.

Use GraphQL only when the CLI output isn't enough for the workflow:

```bash
railway api \
  'query templates($query: String!, $verified: Boolean, $recommended: Boolean) {
    templates(query: $query, verified: $verified, recommended: $recommended) {
      edges { node { code name description category } }
    }
  }' \
  --variables '{"query":"redis","verified":true}'
```

| Parameter | Type | Description |
|---|---|---|
| `query` | String | Search term |
| `verified` | Boolean | Only verified templates |
| `recommended` | Boolean | Only recommended templates |
| `first` | Int | Number of results |

Common template codes: `ghost`, `strapi`, `minio`, `n8n`, `uptime-kuma`, `umami`, `postgres`, `redis`, `mysql`, `mongodb`.

Deploy a found template via CLI:

```bash
railway deploy --template <template-code>
```

Manage owned templates via CLI:

```bash
railway templates publish <template-id> --category Other --description "Deploy and Host My App with Railway" --readme-file README.md --json
railway templates update <template-id> --category Other --description "Updated description" --readme-file README.md --json
railway templates unpublish <template-id-or-code> --yes --json
railway templates delete <template-id-or-code> --yes --json
```

### GraphQL template deployment

For deploying into a specific environment or tracking the workflow, use the two-step GraphQL flow:

**Step 1**: Fetch the template config:

```bash
railway api \
  'query template($code: String!) {
    template(code: $code) { id serializedConfig }
  }' \
  --variables '{"code":"postgres"}'
```

**Step 2**: Deploy with `templateDeployV2`:

```bash
railway api \
  'mutation deploy($input: TemplateDeployV2Input!) {
    templateDeployV2(input: $input) { projectId workflowId }
  }' \
  --variables '{"input":{
    "templateId":"<id-from-step-1>",
    "serializedConfig":<config-object-from-step-1>,
    "projectId":"<project-id>",
    "environmentId":"<env-id>",
    "workspaceId":"<workspace-id>"
  }}'
```

`serializedConfig` is the raw JSON object from the template query, not a string. Get `workspaceId` via `railway api 'query { project(id: "<project-id>") { workspaceId } }'`.


## Validated against

- Docs: [api docs](https://docs.railway.com/api/llms-docs.md), [agents.md](https://docs.railway.com/agents), [community.md](https://docs.railway.com/community), [cli/docs.md](https://docs.railway.com/cli/docs), [templates.md](https://docs.railway.com/cli/templates), [metrics.md](https://docs.railway.com/cli/metrics)
- CLI source: [api.rs (v5.49.1)](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/api.rs), [client.rs (v5.49.1)](https://github.com/railwayapp/cli/blob/v5.49.1/src/client.rs), [docs.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/docs.rs), [templates.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/templates.rs), [metrics.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/metrics.rs)

# Sandboxes

Sandboxes are ephemeral remote Linux environments attached to a Railway project + environment. Use them when the user asks for a sandbox, a scratch/ephemeral environment, or wants to build or run something remotely without touching deployed services. Sandboxes link at the project level — a linked project and environment (or explicit `--project` / `--environment` flags) is enough; no service link is needed.

## Availability — Priority Boarding required

**Sandboxes must be enabled in Priority Boarding.** If sandbox commands fail with a feature-availability error from the API (the `PROJECT_SANDBOXES` feature is not enabled for the workspace), do not retry — prompt the user to enable Sandboxes through Priority Boarding in the Railway dashboard, then resume once they confirm.

Two non-errors to expect:

- Every sandbox command prints `Warning: Railway sandboxes are experimental and APIs may change or break during testing.` on stderr. This is informational — not a failure.
- Context errors (`No project selected...` / `No environment selected...`) mean missing linking, not missing access: run `railway link` or pass `--project` and `--environment`.

## Create and connect

### Create a sandbox

```bash
railway sandbox create
railway sandbox create --variable DB_URL=postgres.DATABASE_URL --env-file .env
railway sandbox create --private-network --idle-timeout-minutes 60 --json
```

`create` boots a sandbox and remembers it as the **active sandbox** — later commands that take `--id` default to it. `--variable` values may reference other Railway variables (`postgres.DATABASE_URL` or the full `${{postgres.DATABASE_URL}}` form), resolved server-side at create time; `--variable` overrides `--env-file` entries with the same key. Sandboxes are isolated with public egress by default — pass `--private-network` when the sandbox must reach internal hosts like `postgres.railway.internal`. `--idle-timeout-minutes` auto-destroys the sandbox after that long idle; attached sessions send a heartbeat that keeps it alive. Prefer `--json` for parsing the created id.

### Connect over SSH

```bash
railway sandbox ssh                      # interactive shell in the active sandbox
railway sandbox ssh --id <sandbox-id> -- ls -la
```

Trailing `-- <command>` runs a single command instead of an interactive shell. `--session <name>` resumes a durable session by name.

## Run commands remotely

### One-off commands

```bash
railway sandbox exec -- npm test
railway sandbox exec --id <sandbox-id> --timeout 120 -- ./integration.sh
```

Everything after `--` is the command. A single argument runs as a shell command; multiple arguments run as argv with quoting preserved. `--timeout <seconds>` is a client-side deadline — on expiry the command is terminated and the CLI exits 124 (GNU timeout convention), so treat exit 124 as "took too long", not "failed".

### Long-running commands — detach and reattach

```bash
railway sandbox exec --detach -- npm run build   # prints a durable session name, exits
railway sandbox exec --session <name>            # reattach and stream the output
```

`--detach` starts the command, prints its durable session name to stdout, and returns while the command keeps running — use it for builds or jobs longer than you want to block on. Reattach with `--session <name>`; add `--resume-from-last-read` to skip replaying retained scrollback. Durable sessions are not available on every sandbox — when the CLI warns `durable sessions are unavailable for this sandbox; ran attached`, the command still ran, just without reattach support.

## Remote builds

Use templates to run build steps remotely. Build steps execute server-side in a transient sandbox; the result is a content-addressed filesystem snapshot cached server-side, so re-running the same recipe is an instant cache hit.

### Build remotely

```bash
railway sandbox template build --name ci -c 'npm ci' -c 'npm run build' --wait
```

`-c` is repeatable and runs in order; each step must exit 0 within 10 minutes. `--wait` polls until READY or FAILED. `--name` is a local-only handle for reuse.

### Boot from the build

```bash
railway sandbox create --template ci             # boots from the cached snapshot
railway sandbox exec -- npm start
```

Template recipes are stored locally by this CLI — `Unknown template` means it was built elsewhere or never built; rebuild it with the command the error message prints. For state that must be reachable from other machines or later sessions, capture a [checkpoint](#checkpoints--save-and-restore-sandbox-state) instead.

## Agents in sandboxes

Use sandboxes for agent runs that need isolated compute, private networking to Railway resources, or remote build/test execution. Copy or generate code in the sandbox, run the agent or test command with `sandbox exec`, and capture a checkpoint if the state should be reused across machines. Prefer `--private-network` when the agent needs internal service or database access.

### Inspect templates

```bash
railway sandbox template status <id-or-name> --json
railway sandbox template list --json
```

## Port forwarding

```bash
railway sandbox forward 3000              # localhost:3000 → sandbox port 3000
railway sandbox forward 8080:3000 5432    # explicit local port; multiple ports per connection
```

If a requested local port is busy the CLI picks a nearby free one and says so — pass `--strict` to fail instead. The tunnel auto-reconnects with backoff if the relay drops while the sandbox is still running.

## Fork a sandbox

```bash
railway sandbox fork                      # fork the active sandbox; the fork becomes active
railway sandbox fork <sandbox-id> --variable FOO=bar --private-network
```

A fork copies the source's filesystem state but **does not inherit its variables or network mode** — re-pass `--variable`/`--env-file`/`--private-network` as needed.

## Checkpoints — save and restore sandbox state

Checkpoints are named disk snapshots captured from a running sandbox, stored server-side per environment. Use them to set a sandbox up once (clone a repo, install dependencies, seed data) and boot fresh sandboxes from that state later. Pick the right state mechanism:

- **Template**: a reproducible recipe built from shell instructions — deterministic, but stored in this CLI's local store, so it only resolves on the machine that built it.
- **Fork**: a live copy of a sandbox made right now.
- **Checkpoint**: state you set up interactively, saved server-side by name — works from any machine or later agent session.

### Capture

```bash
railway sandbox checkpoint create my-setup       # capture the active sandbox's disk
railway sandbox checkpoint create my-setup --id <sandbox-id> --json
```

Capture is synchronous — the checkpoint is bootable as soon as the command returns, but flushing and uploading a large disk can take a while (the request honors `RAILWAY_HTTP_TIMEOUT`, so raise that rather than assuming a hang). Reusing a name **replaces the previous checkpoint without warning** — run `checkpoint list` first if overwriting matters. 64-character hex names are reserved for template hashes.

### Boot from a checkpoint

```bash
railway sandbox create --checkpoint my-setup
railway sandbox exec -- npm start
```

`--checkpoint` and `--template` are mutually exclusive. A checkpoint restores disk state only — variables and network mode are not part of the snapshot, so re-pass `--variable`/`--env-file`/`--private-network` on create as needed.

### Manage checkpoints

```bash
railway sandbox checkpoint list --json
railway sandbox checkpoint rename <name> <new-name>
railway sandbox checkpoint delete <name>
```

Checkpoints are scoped to the environment: `list` shows only the linked environment's checkpoints, and `--checkpoint` resolves names within that scope. Deleting a checkpoint removes its underlying disk snapshot.

## List and teardown

```bash
railway sandbox list --json               # also refreshes the local id cache
railway sandbox list --all                # include destroyed sandboxes
railway sandbox destroy                   # destroy the active sandbox
railway sandbox destroy <sandbox-id>
```

Destroy sandboxes you created for a task once the task is done — idle timeout is the backstop, not the cleanup plan.

## Troubleshoot sandboxes

- **Feature-availability error from the API**: Sandboxes aren't enabled for this workspace. Prompt the user to enable Sandboxes through Priority Boarding in the Railway dashboard; don't retry until they confirm.
- **`No project selected` / `No environment selected`**: run `railway link` or pass `--project` with `--environment`.
- **`Unknown template ...`**: the recipe isn't in this CLI's local store — rebuild with `railway sandbox template build --name <name> -c '<command>' --wait`, or use a checkpoint if the state was meant to travel between machines.
- **`Template build failed`**: a step exited non-zero or exceeded its 10-minute limit; fix that step and rebuild.
- **`No checkpoint named ...`**: checkpoints are environment-scoped — confirm the linked environment and run `railway sandbox checkpoint list`; the checkpoint may live in another environment or have been deleted/replaced.
- **Checkpoint capture times out**: a large disk is still uploading — raise `RAILWAY_HTTP_TIMEOUT` (seconds) and retry rather than treating it as a failure.
- **Exit code 124 from `exec`**: the client-side `--timeout` expired — rerun with a larger timeout or `--detach`.
- **`that session may have expired; the server started a fresh one instead`**: the durable session lapsed; the command ran in a new session — check whether prior state mattered.
- **SSH/forward drops**: the CLI reconnects automatically while the sandbox is RUNNING; if it reports the sandbox STOPPED, create or fork a new one.

## Validated against

- Docs: [sandboxes.md](https://docs.railway.com/sandboxes), [sandbox.md](https://docs.railway.com/cli/sandbox), [agents-in-sandboxes.md](https://docs.railway.com/guides/agents-in-sandboxes)
- CLI source: [sandbox.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/sandbox.rs), [sandbox_exec.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/controllers/sandbox_exec.rs)

# Setup

Create, link, and organize Railway projects, services, databases, and workspaces.

## Account creation & first-time onboarding

A brand-new user with no Railway account is onboarded through the same unified OAuth flow as sign-in — the backend detects fresh accounts and adapts the consent + landing pages. Pick the command by intent:

- **Deploy from the current directory** → `railway up` (interactive) or `railway up -y` (skips the confirm prompt). When unauthenticated it opens a browser to sign in / sign up, then creates a project + service and deploys. Run it yourself; do not ask the user to `railway login` first.
- **New project from cwd when already signed in** → `railway up --new` (`--name <name>` to override the project name).
- **Sign up with a deployable app in cwd → `railway up`** (signs up *and* deploys — bare `up` works for a detected agent, even if the user only said "sign me up"; add `-y` to skip prompts / force it non-interactively). Sign in, or sign up with nothing to deploy → `railway login` (creates new accounts on the fly).

`railway up` and `railway login` self-validate auth — don't run `railway whoami` before them.

**Headless / SSH / CI**: the CLI auto-detects these and switches to the device-code flow (RFC 8628: sign-in link + short code) on its own. Do NOT pass `--browserless` just because you are an agent — if the human is at this machine, bare `railway login` opens their browser and completes far more reliably. Reserve the flag for machines with genuinely no browser that the auto-detection missed.

**`railway up --json` / `--ci` do NOT auto-prompt** an unauthed user. `--json` emits a structured `{"error":"Not signed in.","code":"NOT_AUTHENTICATED", ...}` on stdout and exits non-zero — detect `NOT_AUTHENTICATED`, run `railway login`, then retry.

**Fully unattended (no human)**: set `RAILWAY_API_TOKEN` (account-scoped) or `RAILWAY_TOKEN` (project-scoped) instead of logging in. There is no headless account-creation path — a brand-new user needs a human at the browser once.

## Projects

### List and discover

```bash
railway project list --json        # projects in current workspace
railway list --json                # all projects across workspaces with service metadata
railway whoami --json              # current user, workspace memberships
```

### Link to an existing project

Linking sets the working context for all subsequent CLI commands in this directory.

```bash
railway link --project <project-id-or-name>
railway status --json              # confirm linked context
```

Without `--project`, `railway link` runs interactively. For scripted or CI use, always pass explicit flags.

### Link to a specific service

Switch the linked service within an already-linked project:

```bash
railway service link              # interactive service picker
railway service link <name>       # link directly by name
```

### Create a new project

```bash
railway init --name <project-name>
railway init --name <project-name> --workspace <workspace-id-or-name>
```

`railway init` both creates and links in one step. In CI or multi-workspace setups, pass `--workspace` explicitly to avoid ambiguity.

### Update project settings

Settings like project name, PR deploys, and visibility have no dedicated CLI command. Use `railway api` (see [request.md](request.md)):

```bash
railway api \
  'mutation updateProject($id: String!, $input: ProjectUpdateInput!) {
    projectUpdate(id: $id, input: $input) { id name isPublic prDeploys }
  }' \
  --variables '{"id":"<project-id>","input":{"name":"new-name","prDeploys":true}}'
```

## Services

### List services

Use `railway service list` for service discovery. It returns services in the current environment by default.

```bash
railway service list --json
railway service list --project <project-id> --environment production --json
```

Prefer service IDs from JSON output when names may collide.

### Create a service

```bash
railway add --service <name> --json           # empty service
railway add --database postgres --json        # managed database (postgres, redis, mysql, mongo)
railway add                                   # interactive only — do not use non-interactively
```

**Always pass `--json` to `railway add`.** Without it, a successful database create writes nothing to stdout — only a `> What do you need? Database` echo on stderr that looks identical to a stalled interactive prompt. Retrying based on "no stdout came back" silently provisions a second database. With `--json`, success prints `{"serviceId":"…","serviceName":"…"}` and failure exits non-zero.

**`--database` is cumulative, not last-wins.** `railway add --database postgres --database redis` creates *both* in a single call. Don't repeat the flag — issue one `railway add` per database.

**If `railway add` output looks ambiguous, never retry blind.** Run `railway service list --json` (or query `project.services` via [request.md](request.md)) first and compare against what you expected to exist. Treat the service list as the source of truth, not the CLI's stdout shape.

Before adding a database — and before retrying any `railway add` whose output you can't interpret — list existing services to avoid duplicates. Run `railway service list --json` for a fast count, or `railway environment config --json` and inspect `source.image` per service when you need to identify the engine:

| Image pattern | Database |
|---|---|
| `ghcr.io/railway/postgres*` or `postgres:*` | Postgres |
| `ghcr.io/railway/redis*` or `redis:*` | Redis |
| `ghcr.io/railway/mysql*` or `mysql:*` | MySQL |
| `ghcr.io/railway/mongo*` or `mongo:*` | MongoDB |

If a matching database already exists, skip creation and wire the existing service's variables to the app.

Empty services have no source until you configure one. This is the right pattern when you need to set source repo, branch, or build config before the first deploy.

### Connect a database to a service

After `railway add --database <type>`, the database creates connection variables automatically. Wire them to your app service using variable references:

| Database | Connection variable |
|---|---|
| Postgres | `${{Postgres.DATABASE_URL}}` |
| Redis | `${{Redis.REDIS_URL}}` |
| MySQL | `${{MySQL.MYSQL_URL}}` |
| MongoDB | `${{MongoDB.MONGO_URL}}` |

```bash
railway variable set DATABASE_URL='${{Postgres.DATABASE_URL}}' --service <app-service>
```

Service names in variable references are case-sensitive and must match exactly. For full wiring details including public/private networking decisions, see [configure.md](configure.md).

When creating new service instances via JSON config patches, include `isCreated: true` in the service block to mark it as a new service.

### Connect to a database shell

Use `railway connect` when the user wants an interactive database shell (`psql`, `redis-cli`, `mysql`, or `mongosh`):

```bash
railway connect <database-service>
railway connect <database-service> --environment production
railway connect <database-service> --project <project-id> --environment production
railway connect <database-service> --ssh
railway connect <database-service> --no-ssh
```

The local database client must be installed for a shell. By default, `connect` uses a public TCP proxy when one exists and falls back to an SSH tunnel when no public proxy URL is available. Use `--ssh` to force the tunnel path, or `--no-ssh` to require a public TCP proxy.

For TablePlus, DBeaver, pgAdmin, or another GUI, CLI 5.27+ can hold a private tunnel open without starting or requiring a local database CLI:

```bash
railway connect <database-service> --tunnel-only --port 15432 --project <project-id> --environment production
```

`--tunnel-only` implies SSH and conflicts with `--no-ssh`. Omit `--port` to choose an available ephemeral port. Use the printed local connection details in the GUI; they include credentials, so do not echo them into a report. Keep the process running while the GUI is connected and stop it when finished. With `--project`, always supply `--environment`.

For backups, PITR, HA conversion/scaling, or connection pooling, load [databases.md](databases.md).

### Delete a service

Deleting a service removes it from the target environment. Confirm with the user before running it.

```bash
railway service delete --service <service> --environment <env> --yes --json
railway service delete --service <service-id> --project <project-id> --environment <env> --yes --json
```

If 2FA is enabled and the command runs non-interactively, pass `--2fa-code <code>`.

### Deploy from a template

Templates provision pre-configured services with sensible defaults, faster than creating an empty service and configuring it manually. Use `templates search` for marketplace discovery and `deploy --template` for deployment:

```bash
railway templates search postgres --verified true --json
railway templates search --category Storage --limit 10 --json
railway templates search postgres --limit 50 --after <cursor> --json
railway deploy --template <template-code>
```

The template search command doesn't require authentication. Use the `code` from the JSON results with `railway deploy --template <template-code>`.

Template deployments typically create:

- A service with pre-configured image or source
- Environment variables (connection strings, secrets)
- A volume for persistent data (databases)
- A TCP proxy for external access (where applicable)

Manage owned templates with:

```bash
railway templates list --json
railway templates list --workspace <workspace> --json
railway templates create --project <project> --environment production --json
railway templates publish <template-id> --category Other --description "Deploy and Host My App with Railway" --readme-file README.md --json
railway templates update <template-id> --category Other --description "Updated description" --readme-file README.md --json
railway templates unpublish <template-id-or-code> --yes --json
railway templates delete <template-id-or-code> --yes --json
```

Non-interactive `unpublish` and `delete` require `--yes`; pass `--2fa-code` when required by the current auth session.

### Bootstrap source for an empty service

After creating an empty service, wire it to a repo:

```bash
railway environment edit --service-config <service> source.repo <repo-url>
railway environment edit --service-config <service> source.branch <branch>
```

### Deploy a Docker image

When you have a built image (for example, from a private registry or Docker Hub), skip source builds entirely:

```bash
railway environment edit --service-config <service> source.image <image:tag>
```

This sets the service to pull from a container registry instead of building from source.

## Buckets

Buckets are S3-compatible object storage. They are created at the project level and deployed to environments via config patches.

### List buckets

```bash
railway bucket list --json                                # buckets in current environment
railway bucket list --environment production --json       # buckets in a specific environment
```

### Create a bucket

```bash
railway bucket create my-bucket --region sjc              # create with name and region
railway bucket create --region iad --json                 # auto-named, JSON output
```

Available regions:

| Code | Location |
|---|---|
| `sjc` | US West (California) |
| `iad` | US East (Virginia) |
| `ams` | EU West (Amsterdam) |
| `sin` | Asia Pacific (Singapore) |

Without `--region`, the CLI prompts interactively. For scripted use, always pass `--region`.

### Delete a bucket

Deletion is permanent and destroys all objects in the bucket.

```bash
railway bucket delete --bucket my-bucket --yes            # non-interactive
railway bucket delete --bucket my-bucket --yes --2fa-code 123456   # with 2FA
```

### Rename a bucket

```bash
railway bucket rename --bucket my-bucket --name new-name --json
```

### Bucket info

Check storage usage and object count:

```bash
railway bucket info --bucket my-bucket --json
```

Returns storage size (bytes), object count, region, and environment.

### Bucket credentials

Get S3-compatible credentials for connecting your app to a bucket:

```bash
railway bucket credentials --bucket my-bucket --json
```

Returns: `endpoint`, `accessKeyId`, `secretAccessKey`, `bucketName`, `region`, `urlStyle`.

Without `--json`, output uses `AWS_*=value` lines suitable for `eval $(railway bucket credentials)` or piping into `.env` files.

To reset credentials (invalidates existing ones):

```bash
railway bucket credentials --bucket my-bucket --reset --yes
railway bucket credentials --bucket my-bucket --reset --yes --2fa-code 123456  # with 2FA
```

### Connect a bucket to a service

After creating a bucket, wire the S3 credentials to your app service as environment variables:

```bash
# Get credentials
railway bucket credentials --bucket my-bucket --json

# Set them on your app service
railway variable set \
  AWS_ENDPOINT_URL=<endpoint> \
  AWS_ACCESS_KEY_ID=<access-key> \
  AWS_SECRET_ACCESS_KEY=<secret-key> \
  AWS_S3_BUCKET_NAME=<bucket-name> \
  AWS_DEFAULT_REGION=<region> \
  --service <app-service>
```

All subcommands support `--bucket (-b)` and `--environment (-e)` as global flags to skip interactive prompts.

## Analyze codebase before setup

When setting up a new service from source, detect the project type from marker files:

| Marker file | Type |
|---|---|
| `package.json` | Node.js |
| `requirements.txt` or `pyproject.toml` | Python |
| `go.mod` | Go |
| `Cargo.toml` | Rust |
| `index.html` (no package.json) | Static site |

### Monorepo detection

| Marker | Monorepo type |
|---|---|
| `pnpm-workspace.yaml` | pnpm workspace (shared) |
| `package.json` with `workspaces` field | npm/yarn workspace (shared) |
| `turbo.json` | Turborepo (shared) |
| Multiple subdirectories with separate `package.json`, no workspace config | Isolated monorepo |

**Isolated monorepo** (apps don't share code): set `rootDirectory` to the app's subdirectory (for example, `/apps/api`).

**Shared monorepo** (TypeScript workspaces, shared packages): do not set `rootDirectory`. Set custom build and start commands instead:

- pnpm: `pnpm --filter <package> build`
- npm: `npm run build --workspace=packages/<package>`
- yarn: `yarn workspace <package> build`
- Turborepo: `turbo run build --filter=<package>`

### Scaffolding hints

When no code exists, minimal starting points for common types:

- **Static site**: create `index.html` in the root directory.
- **Vite React**: `npm create vite@latest . -- --template react`
- **Python FastAPI**: create `main.py` with a FastAPI app and `requirements.txt` with `fastapi` and `uvicorn`.
- **Go**: create `main.go` with an HTTP server that reads `PORT` from the environment.

## Workspaces

Workspaces scope billing and team access. Most users have one personal workspace and possibly team workspaces.

```bash
railway whoami --json              # lists workspace memberships
```

When creating projects, Railway uses the default workspace unless `--workspace` is specified.

## Troubleshoot setup issues

- **CLI missing**: install via `brew install railway` or `bash <(curl -fsSL https://railway.com/install.sh)`
- **Not authenticated**: `railway login`
- **Project not found**: verify with `railway project list --json`, check workspace context
- **Service not found**: `railway service list --json` to list services in the current environment
- **Database shell cannot connect**: install the local database client, use `railway connect <service> --ssh`, or create/check a TCP proxy
- **Wrong workspace**: inspect `railway whoami --json`, re-run with explicit `--workspace`
- **Permission denied**: check workspace role, mutations require member or admin access

## Validated against

- Docs: [cli.md](https://docs.railway.com/cli), [init.md](https://docs.railway.com/cli/init), [add.md](https://docs.railway.com/cli/add), [link.md](https://docs.railway.com/cli/link), [project.md](https://docs.railway.com/cli/project), [service.md](https://docs.railway.com/cli/service), [connect.md](https://docs.railway.com/cli/connect), [templates.md](https://docs.railway.com/cli/templates), [list.md](https://docs.railway.com/cli/list), [whoami.md](https://docs.railway.com/cli/whoami)
- CLI source: [init.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/init.rs), [add.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/add.rs), [project.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/project.rs), [service.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/service.rs), [connect.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/connect.rs), [templates.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/templates.rs), [list.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/list.rs), [bucket.rs](https://github.com/railwayapp/cli/blob/v5.23.3/src/commands/bucket.rs)
- API command: [api.rs (v5.49.1)](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/api.rs)

# Tracing

Trace a request from Railway's edge through a service and into the services it calls. Tracing is switched on per service **and per environment**: with the `set-service-tracing` MCP tool, `railway trace enable`, or the **Tracing setup** drawer on the project's **Traces** tab. Read the result with `list-traces` and `get-trace`, or on the same tab.

Tracing is a preview feature. If the project has no **Traces** tab, the account needs **Tracing** enabled in Priority Boarding first.

## What Railway records without code changes

- **Edge and proxy spans.** For every sampled request to a traced service's public domain, Railway's edge records a server span (method, path, status, cache result, upstream) and the regional proxy adds a span for its hop. The edge forwards a W3C `traceparent` header to the service and stamps `x-railway-trace-id` on the response.
- **Automatic instrumentation (OBI).** A per-service switch that attaches eBPF probes to the service's Node.js, Go, Python, Ruby, or Java processes on the host. It exports server spans for incoming HTTP/gRPC requests, client spans for plaintext outgoing calls, and spans for database and cache protocols it decodes. No SDK, no redeploy.

Spans from inside the service only appear once the service exports them, through automatic instrumentation or an OpenTelemetry SDK. A service without a public domain never gets edge spans; only what it exports itself shows up, joined to traces other services propagate to it over the private network.

## Enable tracing

Tracing is two switches on each service instance, so they are set per service and per environment. `tracingEnabled` makes the edge trace requests to the service and gives its next deploy the OpenTelemetry variables below; `autoInstrumentationEnabled` attaches eBPF probes to the service's processes and only does anything while tracing is on. There is no project-wide default and no sample rate: every client-facing request to a traced service is traced. Enabling a service in `production` says nothing about `staging`, so check and set each environment the user cares about.

**Dashboard:** open the **Traces** tab → **Tracing setup**. The drawer lists the services of the environment the dashboard is on (the subtitle names it); each row's **Traced** switch and **Automatic instrumentation** / **Manual instrumentation** choice apply to that environment only. Switch environment to configure another. The service's **Settings → Tracing** section links to the same drawer.

**Agent path:** two MCP tools read and change these settings, and `railway trace` does the same from the CLI. Resolve IDs from the URL or `railway status --json` first, and read before writing. Both tools use the `production` environment when `environmentId` is omitted, so pass it whenever the user is looking at another environment. On a Railway cloud agent in a dashboard chat session the `railway` CLI is unauthenticated, so resolve IDs with `list-services` and stay on the MCP tools throughout.

| Tool | Access | Purpose |
|---|---|---|
| `get-tracing` | viewer | The `environment` and, per service in it, `tracingEnabled`, `autoInstrumentationEnabled` and whether it is `autoInstrumentationActive` (both on). Pass `serviceId` for one service, omit it for every service in the environment |
| `set-service-tracing` | member | `tracingEnabled` and `autoInstrumentationEnabled` for one service in one environment. Each is optional and independent; omit what should stay as it is. Returns the service's state in that environment |

Both take `projectId` and an optional `environmentId`. `describe-service` reports the same tracing state for one service in the environment it was asked about.

```text
Get tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb in environment <environment-id>
```

```text
Set service tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb, service <service-id>, environment <environment-id>: tracingEnabled true
```

```text
Set service tracing for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb, service <service-id>, environment <environment-id>: autoInstrumentationEnabled true
```

`set-service-tracing` warns when it switches auto-instrumentation on for a service whose tracing is off: the switch does nothing until tracing is enabled too. Without Railway MCP or `railway trace`, the public `serviceInstanceUpdate` mutation sets the same fields through `railway api`, and the environment's service instances carry them for reading; see [request.md](request.md). That fallback needs an authenticated CLI, which a cloud agent's chat session does not have.

```bash
railway api \
  'mutation traceInstance($serviceId: String!, $environmentId: String!) {
    serviceInstanceUpdate(serviceId: $serviceId, environmentId: $environmentId,
      input: { tracingEnabled: true, autoInstrumentationEnabled: false })
  }' \
  --variables '{"serviceId":"<service-id>","environmentId":"<environment-id>"}'

railway api \
  'query tracing($id: String!) {
    environment(id: $id) { serviceInstances { edges { node {
      serviceId serviceName tracingEnabled autoInstrumentationEnabled
    } } } }
  }' \
  --variables '{"id":"<environment-id>"}'
```

The older `Project` and `Service` tracing fields still exist as deprecated stubs: `projectUpdate(tracingEnabled)` switches every instance in the project, `serviceUpdate(tracingEnabled)` the service in every environment, and `tracingSampleRate` is ignored. Don't use them; they flip environments the user didn't mention.

**CLI:** `railway trace` (aliases `traces`, `tracing`) sets the same switches, in one environment at a time: the linked one, or `--environment <name>`. This needs a CLI release newer than 5.62.1. An older CLI still runs `railway trace enable`, but through the deprecated service field, so it sets the service in every environment, and its `inherit`, `--project-default` and `--sample-rate` belong to the model that is gone. Use the CLI when the user works in a linked repo or wants exact command output; on a cloud agent's chat session stay on MCP.

```bash
railway trace status                                    # linked service in the linked environment
railway trace status --all                              # every service in the environment, last edge/app span
railway trace enable --service <service>                # trace one service in the linked environment
railway trace enable --auto-instrument                  # tracing plus OBI for the linked service
railway trace enable --all --environment staging        # every service in staging
railway trace disable --auto-instrument                 # tracing and OBI off
```

`--all` and `--service` are mutually exclusive. Pass `--project <id> --environment <name>` together when nothing is linked. `--json` prints one document with `environment: { id, name }` and the service state. Changing tracing needs a user or workspace token; a project token (`RAILWAY_TOKEN`) can only read. The CLI has no switch for auto-instrumentation alone: `railway trace disable --auto-instrument` followed by `railway trace enable` leaves tracing on and OBI off, or use `set-service-tracing` with only `autoInstrumentationEnabled`.

What happens next:

- The edge starts tracing requests to the service's domains in that environment within seconds.
- Automatic instrumentation reaches the running containers within about a minute. No redeploy.
- The OpenTelemetry variables below are added on the **next deploy** in that environment. An app with an SDK exports nothing until it is redeployed: `railway redeploy --service <service> --yes`.

### Infrastructure as code

The environment config carries the same two switches as `services[<id>].tracing: { enabled, autoInstrumentation }`, so tracing is part of what `railway config pull`, `plan` and `apply` manage (see [iac.md](iac.md)). Railway serialises only the switches that are on: an untraced service has no `tracing` block, and `enabled: false` is the same as leaving it out. `railway config pull` renders the block for a traced service:

```ts
const api = service("api", {
  source: github("owner/repo", { branch: "main" }),
  tracing: { enabled: true, autoInstrumentation: true },
});
```

A plan shows a tracing change as a `resource.update` with `field: "tracing"`. Turning `enabled` on or off has `deployEffect: deploy`, because the `OTEL_*` variables land with a deploy; an `autoInstrumentation`-only change is `deployEffect: none` and reaches the running containers live. A new service carries `tracing` on its create.

**Setting `tracing` in config.** `service()` passes the block through to the CLI as `tracing: { enabled?, autoInstrumentation? }`, both optional booleans: `tracing: { enabled: true }` in TypeScript (typed as `ServiceTracing`), `tracing={"enabled": True}` in Python, `"tracing": map[string]any{"enabled": true}` in Go. `fn()` takes it too; `database()` and the database helpers don't, since databases never carry tracing. `{ enabled: false }`, `null` and no block all plan the same.

Two version requirements, both to check before writing the block:

- **CLI 5.63.0 or newer.** An older CLI drops `tracing` on compile and never diffs it, so the setting silently doesn't apply, but nothing proposes removing it either.
- **`railway` 3.12.0+, `railway-sdk` 0.3.0+ or the Go SDK v0.3.0+.** All three accept the block the same way. An older SDK drops it before the CLI sees it; `npm ls railway`, `pip show railway-sdk` or the `require` line in `.railway/go.mod` shows which one is installed.

**The hazard is an old SDK on a new CLI.** The SDK drops the block, the compiled config has no `tracing`, and `railway config plan` against a traced service shows a `tracing` update with `after: null` and `deployEffect: deploy`; `railway config apply` then turns tracing off and redeploys. So:

- If the SDK is too old, upgrade it before adding `tracing` to config, or set tracing with `set-service-tracing`, `railway trace enable` or the dashboard and say why.
- Read every plan for a `tracing` removal nobody authored, and don't apply one. Upgrade the SDK and plan again. `railway config pull` reproduces the block, but an old SDK drops it again on the next plan, so pulling alone doesn't fix it.
- If an apply already disabled tracing, re-enable it with `set-service-tracing` or `railway trace enable`; the redeploy for the variables follows.

## Choose how the service exports spans

**Recommend the OpenTelemetry SDK.** Spans the service exports itself are much higher quality than what automatic instrumentation can recover from outside the process: they carry the route, the query, the business identifiers and the errors the code knows about; they nest under handler-level spans, so the trace shows which step of a request took the time instead of a flat list of calls; they continue the trace through queue consumers, cron work and TLS callees; and they work in any language. Automatic instrumentation is the fallback for a service whose code can't be changed right now, or a first look while the SDK is being added. Present it as a stopgap, not the destination. When a user asks to "instrument my app", default to the SDK unless they ask for the no-code path.

| | Automatic instrumentation (OBI) | OpenTelemetry SDK |
|---|---|---|
| Code changes | None | Install the SDK, load it before the app serves requests |
| Takes effect | About a minute after enabling, no redeploy | Next deploy |
| Runtimes | Node.js, Go, Python, Ruby, Java. Not Bun, so not [Functions](#instrument-a-function-bun) | Any language with an OTel SDK |
| Captures | Incoming HTTP/gRPC, plaintext outgoing HTTP/gRPC, decoded DB and cache protocols | Whatever the SDK's instrumentations cover, plus custom spans and attributes |
| Misses | Outbound TLS callees don't join the trace; queue consumers, cron work and background jobs start new traces; Node.js and Python context propagation is best effort | Nothing structural; depends on the instrumentations you enable |

Pick one per service. Running an SDK in a service that also has automatic instrumentation produces duplicate spans for every request. When moving from OBI to an SDK, deploy the SDK first, confirm its spans arrive, then switch the service to manual instrumentation.

## Instrument with an OpenTelemetry SDK

When tracing is on for a service in an environment, its next deploy there gets these variables. They show up in the service's **Variables** tab alongside the other Railway-provided variables and every OpenTelemetry SDK reads them, so an SDK configured without an explicit endpoint exports to Railway:

| Variable | Value |
|---|---|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Railway's OTLP receiver on the host running the service |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` |
| `OTEL_EXPORTER_OTLP_HEADERS` | A header the receiver requires on every export |
| `OTEL_SERVICE_NAME` | The Railway service name |
| `OTEL_SERVICE_VERSION` | The commit SHA, or the deployment ID for image and CLI deploys |

Rules the agent must apply:

- **Don't hardcode the endpoint, protocol, or header** in code or Dockerfiles. Let the SDK read the variables.
- **The receiver accepts traces only.** Most SDKs also export metrics and logs to the same endpoint by default and log errors when that fails. Set both on the service with the `set-variables` MCP tool (`skipDeploys: true`, since the deploy that adds the SDK picks them up); `railway variable set ... --skip-deploys` does the same from a linked repo:

  ```text
  Set variables for project <project-id>, service <service-id>: OTEL_METRICS_EXPORTER=none, OTEL_LOGS_EXPORTER=none, skipDeploys true
  ```

- **User variables win.** A service that sets its own `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` (for example to keep exporting to its own collector) gets none of the tracing variables, and its spans don't reach the Traces tab; edge spans still do. Railway sets no sampler variables, so a sampler the service configures is the only one in play. Check `railway variable list --service <service> --json` before assuming the provided values apply.
- **Load the SDK first.** Instrumentation libraries (`@opentelemetry/auto-instrumentations-node`, `opentelemetry-instrumentation-*` and the like) patch libraries at import time, so the SDK must be loaded before the app's modules: a `--require`/`--import` flag in the start command, `NODE_OPTIONS`, a Python `opentelemetry-instrument` wrapper, a Java `-javaagent`, and so on. Set it with `railway environment edit --service-config <service> deploy.startCommand "<command>"` or as a variable; see [deploy.md](deploy.md).
- **Keep W3C Trace Context propagation on** (the SDK default in most languages; Go requires setting the propagator explicitly) so the service continues the edge's trace instead of starting its own.
- **Don't override `OTEL_SERVICE_NAME`** unless the user wants spans attributed under a different name than the Railway service.

Per-language install steps, framework notes, and a custom-span example are in the docs: [Node.js](https://docs.railway.com/observability/tracing/nodejs), [Deno](https://docs.railway.com/observability/tracing/deno), [Functions (Bun)](https://docs.railway.com/observability/tracing/functions), [Python](https://docs.railway.com/observability/tracing/python), [Go](https://docs.railway.com/observability/tracing/go), [Java](https://docs.railway.com/observability/tracing/java), [Ruby](https://docs.railway.com/observability/tracing/ruby), [.NET](https://docs.railway.com/observability/tracing/dotnet), [Rust](https://docs.railway.com/observability/tracing/rust), [PHP](https://docs.railway.com/observability/tracing/php). Fetch the page for the user's stack rather than reciting SDK commands from memory.

## What to instrument

Instrumentation libraries give a trace its skeleton: a server span per request and a client span per call a library recognises. On its own that is barely better than automatic instrumentation. The value comes from spans the app opens itself around the units of work its authors think in. When adding tracing to a codebase, or when asked what to instrument, cover these three layers, in this order:

1. **Inbound work.** One span per HTTP handler, gRPC method, queue or job consumer invocation, cron run and WebSocket message. The HTTP and gRPC server instrumentations usually cover handlers. Consumers, cron and background jobs are not covered by the edge or by most instrumentations, so wrap each message or run in its own span (kind `CONSUMER` or `INTERNAL`) and, where the message carries a `traceparent`, continue that context so the trace links back to the producer. Without this, background work is invisible or shows up as orphaned client spans.
2. **I/O: database queries, cache calls, outgoing HTTP and gRPC calls, queue publishes.** This is where latency and failures hide. Use the driver or client instrumentation where one exists. Where none does (a raw socket client, a vendor SDK the instrumentations don't know), wrap the call in a `CLIENT` span with the standard `db.*`, `server.address` or `url.full` attributes. Every remote call should be visible as its own span. List every database driver or ORM, cache, HTTP and queue client in the dependency manifest and check each one, because the bundles miss common ones:
   - **Node.js**: `@opentelemetry/auto-instrumentations-node` covers `pg`, `mysql2`, `ioredis`, `mongodb`, `mongoose`, `knex` and `undici`/`http`, but not Prisma (add `@prisma/instrumentation` to the SDK's instrumentations) and not postgres.js, Drizzle on postgres.js, `Bun.sql` or the Neon serverless driver (no instrumentation exists; wrap queries in a `CLIENT` span with `db.system`, `db.namespace`, `db.operation.name`). An app written as ES modules needs the `@opentelemetry/instrumentation/hook.mjs` loader, or imported drivers stay unpatched while the `http` server span still appears.
   - **Python**: `opentelemetry-distro` alone instruments nothing; run `opentelemetry-bootstrap -a install` so the instrumentations for the installed drivers (`psycopg2`, `asyncpg`, `SQLAlchemy`, `redis`, `pymongo`, `requests`, `httpx`) are installed too.
   - **Go**: nothing is automatic. Wrap `database/sql` with `otelsql`, pgx with `otelpgx`, go-redis with `redisotel`, HTTP clients with `otelhttp.NewTransport`.
   - **Rust**: `sqlx` emits tracing events, not spans; wrap queries in a span. `reqwest` needs `reqwest-tracing`.
   - **Deno**: `OTEL_DENO=true` covers `fetch`, `Deno.serve` and `node:http`; npm database drivers need their own instrumentation or a hand-written span.
   - **Java, Ruby, .NET, PHP**: the agent, `opentelemetry-instrumentation-all`, the per-library packages (`Npgsql.OpenTelemetry`, `OpenTelemetry.Instrumentation.SqlClient`, ...) and `opentelemetry-auto-pdo` cover JDBC, ActiveRecord, ADO.NET and PDO; confirm the package for the driver in use is present.
   A server span with no client spans under it means this layer was skipped. Before opening the PR, run the app once with the console exporter (`OTEL_TRACES_EXPORTER=console` or the runtime's equivalent) and hit a route that queries the database: a `CLIENT` span carrying `db.system`, even a failed one, proves the driver is patched. List each client and the instrumentation that covers it, or why it is not covered, in the PR description.

3. **Logical units of work.** A function that groups several I/O calls into one meaningful step (`checkout`, `syncUser`, `renderInvoice`), or one that is CPU-heavy on its own (parsing a large payload, image resizing, template rendering, serialising a big response). Give each an `INTERNAL` span carrying the identifiers a person debugging it would want (`order.id`, `tenant`, item counts). This turns twelve sibling `SELECT` spans under a handler into a tree that reads like the code and says which step took the time. Rule of thumb: if you would want log lines saying "starting X" and "finished X in N ms", X is a span.

Keep spans out of tight loops and trivial helpers. A span per item in a loop of thousands hits the per-replica export limit (see Sampling) and adds nothing a `count` attribute on the parent wouldn't; instrument the loop, not the iteration. Name spans with low-cardinality names (`GET /users/{id}`, `db.query users`, `process order`) and put the variable parts in attributes. Set span status to `ERROR` and record the exception when a unit fails, so `@status:error` finds it. Never put secrets, tokens or raw personal data in span names or attributes.

## Instrument a Function (Bun)

A [Railway Function](https://docs.railway.com/functions) is a service whose source image starts with `ghcr.io/railwayapp/function-` (`function-bun:1.4.0` today) and whose code is one TypeScript file, base64-encoded into the start command. `get-service-config` shows the image; `get-function-source-code` returns the code. Automatic instrumentation does not cover Bun, and Bun 1.4 has no OpenTelemetry of its own, so a function exports spans only through the OpenTelemetry JavaScript SDK loaded inside that one file.

What differs from a repo service:

- **One file, no start command.** There is no `--require`, `--preload` or `bunfig.toml`. Put the SDK setup at the top of the file. It runs before `Bun.serve` takes its first request, which is all that is needed, because nothing gets monkey-patched.
- **Dependencies come from imports.** The runtime turns every bare import into a `package.json` entry and runs `bun install` at every cold start, without a cache. Pin with `pkg@version` specifiers: `hono@4`, `@hono/otel@1`, `@opentelemetry/api@1`, and `@opentelemetry/sdk-node` to the exact `0.x` version tested, since it has no stable major. Every package added lengthens the cold start.
- **Nothing is instrumented for free.** `NodeSDK` configures the exporter, the resource and W3C propagation from the `OTEL_*` variables, but no OpenTelemetry package instruments `Bun.serve`, Bun's `fetch`, `Bun.sql` or `Bun.redis` (the Node `http` and `undici` instrumentations don't see them). Incoming requests need `@hono/otel` (Hono) or a hand-written wrapper (`Bun.serve`); outgoing `fetch` calls need `propagation.inject` for the callee to join the trace. The three layers in [What to instrument](#what-to-instrument) still apply; the function's handlers, I/O and logical units are what to wrap.
- **A code push is a deploy.** The variables land on the next deploy, and `update-function-source-code` or `railway functions push` is one, so a single push adds the SDK and picks up the variables.

Recipe:

1. Turn tracing on for the function in the environment it runs in with `set-service-tracing` (or `railway trace enable --service <function>`) if `get-tracing` shows it off there. Leave `autoInstrumentationEnabled` off; it does nothing for Bun.
2. Set the exporter variables with `set-variables`. `NodeSDK` exports metrics and logs over OTLP by default and the receiver rejects both. Pass `skipDeploys: true`; the code push in step 5 is the deploy that picks them up. Check `list-variables` first: a function that sets its own `OTEL_EXPORTER_OTLP_ENDPOINT` gets none of Railway's tracing variables.

   ```text
   Set variables for project <project-id>, service <function-id>: OTEL_METRICS_EXPORTER=none, OTEL_LOGS_EXPORTER=none, skipDeploys true
   ```

3. Read the code with `get-function-source-code`: `code` is what is current, `deployedCode` what runs, `staged` whether a commit is pending. Edit that, never a version recalled from memory.
4. Add the SDK block at the top and wrap the requests, leaving the rest of the file as it is. A Hono function ends up like this:

   ```typescript
   import { NodeSDK } from "@opentelemetry/sdk-node@0.222.0";
   import { trace } from "@opentelemetry/api@1";
   import { Hono } from "hono@4";
   import { httpInstrumentationMiddleware } from "@hono/otel@1";

   // Reads OTEL_EXPORTER_OTLP_* and OTEL_SERVICE_NAME from the variables
   // Railway provides. Nothing to configure.
   const sdk = new NodeSDK();
   sdk.start();
   process.on("SIGTERM", () => sdk.shutdown().finally(() => process.exit(0)));

   const tracer = trace.getTracer("greeter");

   const app = new Hono();
   // One SERVER span per request, continuing the edge's traceparent.
   app.use(httpInstrumentationMiddleware());

   app.get("/hello/:name", async (c) => {
     const name = c.req.param("name");
     const greeting = await tracer.startActiveSpan("build-greeting", async (span) => {
       try {
         span.setAttribute("greeting.name", name);
         return `Hello, ${name}`;
       } finally {
         span.end();
       }
     });
     return c.json({ greeting });
   });

   export default { port: Number(Bun.env.PORT ?? 3000), fetch: app.fetch };
   ```

   For a function that calls `Bun.serve` itself, wrap its `fetch` handler: take the parent from `propagation.extract(context.active(), req.headers, { get: (h, k) => h.get(k) ?? undefined, keys: (h) => [...h.keys()] })` and run the handler inside `tracer.startActiveSpan(name, { kind: SpanKind.SERVER }, parent, ...)`. For an outgoing call, start a `SpanKind.CLIENT` span and `propagation.inject(context.active(), headers, { set: (h, k, v) => h.set(k, v) })` into a `Headers` object before `fetch`. A cron or script function has no server: its spans are new roots, recorded on every run, and it must `await sdk.shutdown()` as its last statement, or the batch never leaves the process. The docs page has all three in full.

5. Write it back with `update-function-source-code` (the whole file; pass `staged: true` to stage instead of deploying live) or `railway functions push --path <file>`. This deploy also adds the variables.
6. Verify with the steps below: `curl -sI https://<domain>/hello/x | grep -i x-railway-trace-id`, then `get-trace` on the ID. A span with `component` `service` and scope `@hono/otel` (or the tracer name) means the function exports. If the deploy logs show `bun install` failing, an import specifier is wrong; if they show OTLP export errors for metrics or logs, step 2 was skipped.

Docs: [Functions](https://docs.railway.com/observability/tracing/functions).

## Sampling

- **Every request is traced.** The edge traces 100% of client-facing requests to a traced service and writes the decision into the `traceparent` sampled flag. Railway sets no sampler variables, so the SDK's default parent-based sampler follows the edge and records every root span of its own (cron jobs, queue consumers, private-network calls without a `traceparent`). There is no project or service rate to set.
- **A client `traceparent` overrides.** A request that arrives with the sampled flag set is always traced; one with it cleared never is. An instrumented client can start a trace that continues into Railway, and a caller can keep a request out of tracing by sending a cleared flag. To trace one specific request while debugging:

  ```bash
  curl -H "traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" https://<domain>/<path>
  ```

  Generate a fresh random 32-hex trace ID (the second field) for each request; reusing one merges requests into a single trace.

- **Sample in the SDK when the volume is too high.** Each replica can export 1,000 spans per 10 seconds; exports over the limit are rejected and the SDK reports a partial success. Disable noisy instrumentations first. If that isn't enough, set `OTEL_TRACES_SAMPLER=traceidratio` and `OTEL_TRACES_SAMPLER_ARG=<fraction>` on the service yourself; a `parentbased_*` sampler follows the edge's flag and would change nothing for edge requests. Traces the SDK drops still show the edge and proxy hops.

## Read traces

Traces are read through **Remote MCP** (the default agent path), `railway trace list` and `railway trace get` on the CLI, or the dashboard. The `traces`, `trace` and `tracingStatus` queries are on the public GraphQL API as well, so `railway api` can fetch them where neither fits.

| Tool | Access | Purpose |
|---|---|---|
| `list-traces` | viewer | Traces of an environment, newest first, one row per request with at least one span matching `filter`. Optional `serviceId`, `startDate`/`endDate` (ISO 8601 with timezone; defaults to the last hour), `limit` (default 100, max 500) |
| `get-trace` | viewer | One trace as an indented span tree (each line shows the span kind and the attribute that says what it talked to: `db.system`, `http.route`, `url.full` or `server.address`), with every span's attributes, events and links in the structured result. Takes `traceId` (32 hex characters); `maxSpans` caps the result and the output says when it was hit |
| `get-tracing-coverage` | viewer | What one service's own instrumentation covers: its spans in the window (default the last hour) by span kind, by the remote system they name (`db.system`, `messaging.system`, `rpc.system`, or `http` for HTTP client spans) and by the instrumentation scope that emitted them (`@opentelemetry/instrumentation-pg`, ...). Takes `serviceId`; optional `startDate`/`endDate`. Edge and proxy spans are left out. The one call that answers "are the database queries instrumented?" |

Both take `projectId` and an optional `environmentId`; omit it and the `production` environment is used, so pass the ID explicitly when the user is looking at another environment. Traces belong to the environment they were exported from.

```text
List traces for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb in environment <environment-id> with filter "@status:error"
```

```text
Get trace 4bf92f3577b34da6a3ce929d0e0e4736 for project 6adb5ae3-0e3a-4ead-b42c-1fd36f217ffb
```

`filter` uses the same syntax as logs: `@status:error`, `@component:edge AND @duration:>1000`, `@service:api AND @kind:client`, `@http.route:/checkout AND @http.response.status_code:500`, `@name:SELECT*`. Built-in fields are `trace`, `span`, `name`, `serviceName`, `service`, `deployment`, `replica`, `component` (`edge`, `proxy`, `service`), `kind`, `status`, `duration` (ms); any other `@key` matches a span or resource attribute, free text matches the span name, and `-` negates. Narrow the filter or the window before raising `limit`.

On the CLI, `list` scopes to the linked service unless `--all`, `--since`/`--until` take relative (`30m`, `2h`, `1d`) or ISO 8601 times, `--limit` is 1 to 500 (default 100), `--errors` adds `@status:error`, and `get --max-spans` goes up to 2000. Human output is a table and an indented span tree; `--json` prints one trace summary or one span per line, like `railway logs --json`.

```bash
railway trace list --since 30m --errors --json
railway trace list --all --filter '@http.route:/api/users @duration:>500'
railway trace get 4bf92f3577b34da6a3ce929d0e0e4736 --json
```

Workflow for "why is this request slow / failing":

1. `list-traces` (or `railway trace list`) with a filter that isolates the symptom (`@status:error`, `@duration:>1000`, `@http.route:<route>`), optionally `serviceId` for one service.
2. `get-trace` (or `railway trace get`) on a returned `traceId`. The tree runs from the edge span down through every service; the `component` on each span says which hop exported it, and a span with `ERROR` status carries the message.
3. Read the span attributes in the structured result for the detail (`http.route`, `http.response.status_code`, `db.statement`, custom attributes).

## Verify tracing works

1. **Prove the edge traced a request.** The header is present only on traced responses:

   ```bash
   curl -sI https://<domain>/ | grep -i x-railway-trace-id
   ```

2. **Fetch that trace** with `get-trace` or `railway trace get <trace-id>` and the returned ID. Edge and proxy spans confirm tracing is on; a span with `component` `service` confirms the app is exporting. After enabling an SDK, that only happens once the redeploy that added the variables is live; after enabling automatic instrumentation, allow about a minute.
3. **Check what the app covers** with `get-tracing-coverage` for the service, after sending a few requests that hit the database. Every database, cache and queue the service uses must appear under "Remote systems" and the driver instrumentations under "Instrumentation scopes". `SERVER` spans with no database client span mean a driver is still unpatched (see the I/O layer in [What to instrument](#what-to-instrument)); `list-traces` with `@kind:client @db.system:*` is the same check as a trace list.
4. **Or watch the dashboard.** The Traces tab is at `https://railway.com/project/<project-id>/traces?environmentId=<environment-id>`; its **Trace ID** field accepts a bare 32-hex ID or a whole `traceparent` header. In **Tracing setup**, each service row shows when the edge and the app last exported a span, and the **App** indicator turns green on the first span from the service itself.

## Troubleshoot

- **No traces at all**: confirm `get-tracing` or `railway trace status` reports tracing on for the service in the environment the user is looking at (the switches and the traces both belong to an environment), the service has a public domain, and the account has Tracing in Priority Boarding. With little traffic, send a request and check for `x-railway-trace-id` on the response, then `get-trace` it.
- **Traced in `production` but not elsewhere**: tracing is set per environment. Enable it in the other environment with `set-service-tracing` and its `environmentId`, or `railway trace enable --environment <name>`.
- **Edge spans only, nothing from the app** (`get-trace` shows only `edge` and `proxy` components): the variables land on the next deploy, so redeploy. Then check the service doesn't set its own `OTEL_EXPORTER_OTLP_ENDPOINT`, the SDK loads before the app serves, and, for OBI, the process is a supported runtime handling HTTP or gRPC.
- **Server spans but no database or client spans** (`get-tracing-coverage` shows `SERVER` and `INTERNAL` only, or `get-trace` shows a request span with no `CLIENT` children): the driver is not instrumented. In Node.js an ES-module app without the `@opentelemetry/instrumentation/hook.mjs` loader, Prisma without `@prisma/instrumentation`, or a client no instrumentation covers (postgres.js, Drizzle on it, `Bun.sql`); in Python a missing `opentelemetry-bootstrap -a install`; in Go an unwrapped `database/sql`. See the I/O layer in [What to instrument](#what-to-instrument).
- **App spans appear as separate traces** (`list-traces` shows service-rooted traces with `hasEdge` false next to edge-only ones): the SDK isn't reading `traceparent`. Enable the W3C Trace Context propagator and make sure nothing in front of the handlers strips the header.
- **SDK logs metrics or logs export errors**: set `OTEL_METRICS_EXPORTER=none` and `OTEL_LOGS_EXPORTER=none`.
- **Duplicate spans per request**: the service runs an SDK with automatic instrumentation on. Switch it off with `set-service-tracing` (`autoInstrumentationEnabled` false), or on the CLI `railway trace disable --auto-instrument` then `railway trace enable`.
- **Spans missing from a busy service**: over 1,000 spans per replica per 10 seconds. Disable noisy instrumentations or sample in the SDK; see [Sampling](#sampling).
- **A Function shows edge spans only**: automatic instrumentation can't help (Bun); the SDK has to be in the file. Check `get-function-source-code` for the `NodeSDK` block and the request wrapper, that the deploy logs show `bun install` succeeding, and that `OTEL_METRICS_EXPORTER`/`OTEL_LOGS_EXPORTER` are `none`. See [Instrument a Function (Bun)](#instrument-a-function-bun).
- **`railway config plan` wants to remove `tracing`** (`after: null`, nobody changed it): the IaC SDK is too old to carry the field and drops it (`railway` below 3.12.0, `railway-sdk` below 0.3.0, Go SDK below v0.3.0). Don't apply it; upgrade the SDK and plan again. See [Infrastructure as code](#infrastructure-as-code).

## Validated against

- Docs: [tracing.md](https://docs.railway.com/observability/tracing), [automatic-instrumentation.md](https://docs.railway.com/observability/tracing/automatic-instrumentation), [nodejs.md](https://docs.railway.com/observability/tracing/nodejs), [tracing/functions.md](https://docs.railway.com/observability/tracing/functions), [functions.md](https://docs.railway.com/functions), [variables/reference.md](https://docs.railway.com/variables/reference), [cli/trace.md](https://docs.railway.com/cli/trace), [infrastructure-as-code/reference.md](https://docs.railway.com/infrastructure-as-code/reference)
- Public GraphQL API (`railway api schema`): `ServiceInstance.tracingEnabled`, `ServiceInstance.autoInstrumentationEnabled` and `ServiceInstanceUpdateInput` as the per-environment path; `Project.tracingEnabled`, `Project.tracingSampleRate`, `Service.tracingEnabled`, `Service.autoInstrumentationEnabled` and their update inputs marked deprecated; the `traces`, `trace` and `tracingStatus` queries
- Railway MCP (`https://mcp.railway.com`) tool descriptions and schemas: `get-tracing` and `set-service-tracing` with `environmentId`, `describe-service`, `list-traces`, `get-trace`, `get-tracing-coverage`, `get-function-source-code`, `update-function-source-code`
- CLI source (railwayapp/cli, per-environment `railway trace` and the IaC `tracing` block in #1239): `src/commands/trace.rs` (subcommands, `--all`, `--environment`), `src/iac/compiler.rs` (`services[id].tracing` on service nodes), `src/iac/change_set.rs` (tracing diff, `deployEffect`, removal when the block is absent), `src/commands/config/mod.rs` (pull renderer)
- IaC SDK sources for the `tracing` block: railway-ts-sdk `src/iac/sdk.ts` and `src/iac/schema.ts` (`ServiceTracing`; `railway` 3.12.0), railway-py-sdk `_normalize_tracing` in `src/railway_sdk/__init__.py` (`railway-sdk` 0.3.0), railway-go-sdk `serviceNode` in `railway.go` (v0.3.0); CLI 5.63.0 `src/iac/compiler.rs` and `src/iac/change_set.rs`
- Provided variables observed on a traced service's deploy: the five `OTEL_*` variables in the table above and no sampler variables
- Function runtime: the public `ghcr.io/railwayapp/function-bun:1.4.0` image (Bun 1.4.0, bare imports turned into a `package.json` and installed with `bun install` at every start)
- Function examples run on Bun 1.4.0 with `@opentelemetry/sdk-node` 0.222.0 and `@hono/otel` 1.1.2 against a stub OTLP receiver: server span continues the incoming `traceparent`, client span propagates it, script flushes on `sdk.shutdown()`

# Usage and spending limits

Use `railway usage` (CLI 5.25+) to explain billed resource usage and project/service costs. CLI 5.26+ adds explicit workspace versus Railway Agent spending-limit targets. For live CPU, memory, network, or latency measurements, use [operate.md](operate.md).

## Read usage

Resolve the workspace from the user's request or `railway whoami --json`. These commands are workspace-scoped, not scoped by the current service or environment:

```bash
railway usage --workspace <workspace> --json
railway usage --workspace <workspace> --period previous --json
railway usage --workspace <workspace> --period 2026-08 --json
railway usage projects --workspace <workspace> --json
railway usage projects --workspace <workspace> --limit 10 --json
railway usage projects --workspace <workspace> --project <project> --period current --json
```

`--period` accepts `current`, `previous`, or `YYYY-MM` and applies only to summaries and project breakdowns. Report the returned billing-period dates with cost comparisons. `projects --project` returns a service breakdown and cannot be combined with `--limit`. Human output defaults to the top 25 projects; JSON returns all unless limited. Deleted resources can still contribute to the period's costs.

Distinguish current usage from an estimated end-of-period bill. Estimates can be unavailable and are not final invoices. Use the returned cost fields rather than recomputing them from hardcoded prices; do not treat missing values as zero.

## Inspect and change limits

```bash
railway usage limit status --workspace <workspace> --target workspace --json
railway usage limit status --workspace <workspace> --target agent --json
railway usage limit set --workspace <workspace> --target workspace --soft 75 --hard 125 --json
railway usage limit set --workspace <workspace> --target agent --soft 7.50 --hard 20 --json
```

Read the target's existing limits before changing them. `set` requires `--target`; `status` without a target reports both. `workspace` controls resource/compute usage; `agent` controls Railway Agent usage. Do not infer that the agent target caps cloud VM compute costs or another coding harness's provider bill.

Soft limits are email alerts; hard limits enforce a spending cap and can interrupt the corresponding service. Set only the target and amounts the user requested. Inputs are dollars: workspace limits require whole dollars; agent limits accept cents. Omitted fields preserve existing values, but setting an agent soft limit without any existing hard limit requires supplying a hard limit too. Follow CLI validation for supported ranges.

```bash
railway usage limit update --workspace <workspace> --soft 75 --json
railway usage limit remove --workspace <workspace> --yes --json
```

`update` and `remove` are workspace compute-limit operations, not generic agent-target operations. Removing limits removes the spending protection; `--yes` skips confirmation and belongs only on an authorized removal. `--period` is invalid on all limit commands. After any mutation, reread `limit status` for the same workspace and target and report the resulting limits.

## Troubleshoot

- **Wrong totals**: check the workspace, returned billing-period boundaries, selected project, and whether deleted resources contributed.
- **Missing estimate**: report actual usage and the unavailable estimate separately.
- **Limit command rejected**: check target, dollar precision, required hard limit, and CLI version; do not retry with a different spending target.
- **Permission denied**: verify billing access to the workspace; project access alone does not establish permission to change spending limits.

## Validated against

- Docs: [Usage CLI](https://docs.railway.com/cli/usage)
- CLI source (v5.49.1): [usage.rs](https://github.com/railwayapp/cli/blob/v5.49.1/src/commands/usage.rs)

#!/usr/bin/env python3
"""
MongoDB analysis for Railway deployments.

Produces a comprehensive report covering:
- Deployment status & overview
- Connections & operations
- Latency (opLatencies)
- Memory & WiredTiger cache
- Storage & collection stats
- Replication / oplog
- Slow queries & active operations
- Top collections by activity
- Recommendations

Usage:
    analyze-mongo.py --service <name>
    analyze-mongo.py --service <name> --json
    analyze-mongo.py --service <name> --step ssh-test
"""

import argparse
import json
import os
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from typing import Dict, List, Optional, Any, Tuple
from dataclasses import dataclass, field, asdict

import dal
from dal import (
    LOG_LINES_DEFAULT, ProgressTimer, RailwayContext,
    _init_context, progress, run_railway_command, run_ssh_query,
    get_railway_status, get_deployment_status,
    get_all_metrics_from_api, _analyze_window, _build_metrics_history,
    get_recent_logs,
    _trend_indicator,
)


# ---------------------------------------------------------------------------
# Result container
# ---------------------------------------------------------------------------

@dataclass
class MongoAnalysisResult:
    """Container for MongoDB analysis results."""
    service: str
    db_type: str
    timestamp: str
    deployment_status: str = "UNKNOWN"

    # Server overview
    version: Optional[str] = None
    storage_engine: Optional[str] = None
    uptime_seconds: Optional[int] = None

    # Connections
    connections: Optional[Dict[str, Any]] = None

    # Operations
    opcounters: Optional[Dict[str, Any]] = None
    opcounters_repl: Optional[Dict[str, Any]] = None

    # Latency
    op_latencies: Optional[Dict[str, Any]] = None

    # Memory
    memory: Optional[Dict[str, Any]] = None
    page_faults: Optional[int] = None

    # Network
    network: Optional[Dict[str, Any]] = None

    # WiredTiger
    wiredtiger_cache: Optional[Dict[str, Any]] = None
    wiredtiger_checkpoint: Optional[Dict[str, Any]] = None
    wiredtiger_tickets: Optional[Dict[str, Any]] = None

    # Global lock
    global_lock: Optional[Dict[str, Any]] = None

    # Document metrics
    document_metrics: Optional[Dict[str, Any]] = None

    # Query efficiency
    query_executor: Optional[Dict[str, Any]] = None

    # Plan cache (7.0+)
    plan_cache: Optional[Dict[str, Any]] = None

    # Sort (7.0+)
    sort_metrics: Optional[Dict[str, Any]] = None

    # Cursors
    cursors: Optional[Dict[str, Any]] = None

    # TTL
    ttl_metrics: Optional[Dict[str, Any]] = None

    # Asserts
    asserts: Optional[Dict[str, Any]] = None

    # Replication
    replication: Optional[Dict[str, Any]] = None
    oplog: Optional[Dict[str, Any]] = None

    # Storage (db.stats)
    storage: Optional[Dict[str, Any]] = None

    # Collection stats
    collection_stats: List[Dict[str, Any]] = field(default_factory=list)

    # Top collections
    top_collections: Optional[List[Dict[str, Any]]] = None

    # Slow queries
    slow_queries: List[Dict[str, Any]] = field(default_factory=list)

    # Active operations
    active_ops: List[Dict[str, Any]] = field(default_factory=list)

    # Logs
    recent_logs: List[str] = field(default_factory=list)
    recent_errors: List[str] = field(default_factory=list)

    # Railway metrics (CPU, memory, disk, network trends)
    cpu_memory: Optional[Dict[str, Any]] = None
    disk_usage: Optional[Dict[str, Any]] = None
    metrics_history: Optional[Dict[str, Any]] = None

    # Status tracking
    collection_status: Dict[str, Dict[str, Any]] = field(default_factory=dict)
    errors: List[str] = field(default_factory=list)
    recommendations: List[Dict[str, str]] = field(default_factory=list)


# ---------------------------------------------------------------------------
# MongoDB-specific helpers
# ---------------------------------------------------------------------------

def run_mongosh_query(service: str, js_expr: str, timeout: int = 30) -> Tuple[int, str, str]:
    """Run a mongosh query via SSH and return (returncode, stdout, stderr).

    The query is wrapped in EJSON.stringify and executed through mongosh
    connecting to the local MongoDB instance using container env vars.
    """
    # Escape single quotes in the JS expression for the shell
    escaped = js_expr.replace("'", "'\\''")
    command = (
        f'''bash +H -c 'mongosh "mongodb://$MONGOUSER:$MONGOPASSWORD@localhost:27017" '''
        f'''--quiet --norc --eval "EJSON.stringify({escaped})"' '''
    )
    return run_ssh_query(service, command, timeout)



# ---------------------------------------------------------------------------
# MongoDB queries
# ---------------------------------------------------------------------------

QUERY_SERVER_STATUS = """(function(){ var s = db.serverStatus(); return { connections: s.connections, opcounters: s.opcounters, opcountersRepl: s.opcountersRepl || null, repl: s.repl || null, mem: s.mem, network: s.network, uptime: s.uptime, opLatencies: s.opLatencies, wiredTiger: s.wiredTiger ? { cache: s.wiredTiger.cache, concurrentTransactions: s.wiredTiger.concurrentTransactions, transaction: s.wiredTiger.transaction || null } : null, globalLock: s.globalLock, metrics: s.metrics ? { document: s.metrics.document, queryExecutor: s.metrics.queryExecutor, cursor: s.metrics.cursor, ttl: s.metrics.ttl || null, query: s.metrics.query || null } : null, extra_info: s.extra_info ? { page_faults: s.extra_info.page_faults } : null, version: s.version, storageEngine: s.storageEngine, asserts: s.asserts }; })()"""

QUERY_DB_STATS = """db.stats()"""

QUERY_CURRENT_OP = """db.currentOp({ active: true })"""

QUERY_COLLECTION_STATS = """db.getCollectionNames().map(function(c) { var s = db.getCollection(c).stats(); return { name: c, count: s.count || 0, size: s.size || 0, storageSize: s.storageSize || 0, indexSize: s.totalIndexSize || 0, nindexes: s.nindexes || 0 }; })"""

QUERY_SLOW_QUERIES = """(function(){ try { var logs = db.system.profile.find().sort({ts: -1}).limit(10).toArray(); return logs.map(function(l) { return { op: l.op, ns: l.ns, millis: l.millis, ts: l.ts, command: JSON.stringify(l.command || l.query || {}).substring(0, 200), planSummary: l.planSummary || '' }; }); } catch(e) { return []; } })()"""

QUERY_REPL_INFO = """(function(){ try { var info = db.getReplicationInfo(); return { logSizeMB: info.logSizeMB, usedMB: info.usedMB, timeDiffHours: info.timeDiffHours }; } catch(e) { return null; } })()"""

QUERY_TOP = """(function(){ try { var t = db.adminCommand({top:1}); var totals = t.totals; var result = []; for (var ns in totals) { if (ns.indexOf('.') > 0 && ns.indexOf('system.') === -1) { var c = totals[ns]; result.push({ ns: ns, reads: c.readLock ? c.readLock.count : 0, readTimeUs: c.readLock ? c.readLock.time : 0, writes: c.writeLock ? c.writeLock.count : 0, writeTimeUs: c.writeLock ? c.writeLock.time : 0 }); } } return result; } catch(e) { return null; } })()"""


# ---------------------------------------------------------------------------
# Parsing helpers
# ---------------------------------------------------------------------------

def _safe_json(raw: str) -> Any:
    """Parse JSON from mongosh EJSON output, returning None on failure."""
    raw = raw.strip()
    if not raw:
        return None
    # mongosh may emit warnings before the JSON; find the first { or [
    for i, ch in enumerate(raw):
        if ch in ('{', '['):
            raw = raw[i:]
            break
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return None


def _parse_server_status(data: Dict[str, Any], result: MongoAnalysisResult) -> None:
    """Extract metrics from serverStatus into result."""
    # Overview
    result.version = data.get("version")
    se = data.get("storageEngine")
    if se:
        result.storage_engine = se.get("name")
    result.uptime_seconds = data.get("uptime")

    # Connections
    conn = data.get("connections")
    if conn:
        result.connections = {
            "current": conn.get("current", 0),
            "available": conn.get("available", 0),
            "totalCreated": conn.get("totalCreated", 0),
        }

    # Opcounters
    result.opcounters = data.get("opcounters")
    result.opcounters_repl = data.get("opcountersRepl")

    # Replication info from serverStatus
    repl = data.get("repl")
    if repl:
        result.replication = {
            "setName": repl.get("setName"),
            "isWritablePrimary": repl.get("isWritablePrimary"),
            "primary": repl.get("primary"),
            "hosts": repl.get("hosts"),
        }

    # Latency
    lat = data.get("opLatencies")
    if lat:
        result.op_latencies = {}
        for key in ("reads", "writes", "commands"):
            entry = lat.get(key, {})
            ops = entry.get("ops", 0)
            latency = entry.get("latency", 0)
            result.op_latencies[key] = {
                "latency": latency,
                "ops": ops,
                "avg_us": round(latency / ops, 1) if ops > 0 else 0,
            }

    # Memory
    mem = data.get("mem")
    if mem:
        result.memory = {
            "resident_mb": mem.get("resident", 0),
            "virtual_mb": mem.get("virtual", 0),
        }

    extra = data.get("extra_info")
    if extra:
        result.page_faults = extra.get("page_faults", 0)

    # Network
    net = data.get("network")
    if net:
        result.network = {
            "bytesIn": net.get("bytesIn", 0),
            "bytesOut": net.get("bytesOut", 0),
            "numRequests": net.get("numRequests", 0),
        }

    # WiredTiger
    wt = data.get("wiredTiger")
    if wt:
        cache = wt.get("cache", {})
        result.wiredtiger_cache = {
            "bytes_in_cache": cache.get("bytes currently in the cache", 0),
            "max_bytes": cache.get("maximum bytes configured", 0),
            "dirty_bytes": cache.get("tracked dirty bytes in the cache", 0),
            "pages_read": cache.get("pages read into cache", 0),
            "pages_written": cache.get("pages written from cache", 0),
            "app_evictions": cache.get("pages evicted by application threads", 0),
        }

        txn = wt.get("transaction", {})
        if txn:
            result.wiredtiger_checkpoint = {
                "most_recent_time_ms": txn.get("transaction checkpoint most recent time (msecs)", 0),
            }

        ct = wt.get("concurrentTransactions", {})
        if ct:
            read_info = ct.get("read", {})
            write_info = ct.get("write", {})
            result.wiredtiger_tickets = {
                "read_available": read_info.get("available", 0),
                "read_total": read_info.get("totalTickets", 0),
                "write_available": write_info.get("available", 0),
                "write_total": write_info.get("totalTickets", 0),
            }

    # Global lock
    gl = data.get("globalLock")
    if gl:
        cq = gl.get("currentQueue", {})
        ac = gl.get("activeClients", {})
        result.global_lock = {
            "queue_readers": cq.get("readers", 0),
            "queue_writers": cq.get("writers", 0),
            "active_readers": ac.get("readers", 0),
            "active_writers": ac.get("writers", 0),
        }

    # Metrics
    metrics = data.get("metrics")
    if metrics:
        doc = metrics.get("document")
        if doc:
            result.document_metrics = {
                "inserted": doc.get("inserted", 0),
                "updated": doc.get("updated", 0),
                "deleted": doc.get("deleted", 0),
                "returned": doc.get("returned", 0),
            }

        qe = metrics.get("queryExecutor")
        if qe:
            result.query_executor = {
                "scanned": qe.get("scanned", 0),
                "scannedObjects": qe.get("scannedObjects", 0),
            }

        cursor = metrics.get("cursor")
        if cursor:
            open_cursors = cursor.get("open", {})
            result.cursors = {
                "open_total": open_cursors.get("total", 0),
                "timed_out": cursor.get("timedOut", 0),
            }

        ttl = metrics.get("ttl")
        if ttl:
            result.ttl_metrics = {
                "deletedDocuments": ttl.get("deletedDocuments", 0),
                "passes": ttl.get("passes", 0),
            }

        query_metrics = metrics.get("query")
        if query_metrics:
            pc = query_metrics.get("planCache", {})
            if pc:
                result.plan_cache = {
                    "hits": pc.get("hits", 0),
                    "misses": pc.get("misses", 0),
                }
            sort = query_metrics.get("sort", {})
            if sort:
                result.sort_metrics = {
                    "spillToDisk": sort.get("spillToDisk", 0),
                    "totalBytesSorted": sort.get("totalBytesSorted", 0),
                }

    # Asserts
    result.asserts = data.get("asserts")


def _parse_db_stats(data: Dict[str, Any], result: MongoAnalysisResult) -> None:
    """Extract metrics from db.stats()."""
    result.storage = {
        "dataSize": data.get("dataSize", 0),
        "storageSize": data.get("storageSize", 0),
        "indexSize": data.get("indexSize", 0),
        "objects": data.get("objects", 0),
        "collections": data.get("collections", 0),
    }


def _parse_current_op(data: Any, result: MongoAnalysisResult) -> None:
    """Extract active operations from currentOp."""
    if not data:
        return
    inprog = data.get("inprog", []) if isinstance(data, dict) else []
    ops = []
    for op in inprog:
        ops.append({
            "opid": op.get("opid"),
            "type": op.get("type", op.get("op", "")),
            "ns": op.get("ns", ""),
            "microsecs_running": op.get("microsecs_running", 0),
            "desc": op.get("desc", ""),
        })
    result.active_ops = ops


def _parse_collection_stats(data: Any, result: MongoAnalysisResult) -> None:
    """Parse per-collection stats."""
    if not isinstance(data, list):
        return
    result.collection_stats = data


def _parse_slow_queries(data: Any, result: MongoAnalysisResult) -> None:
    """Parse slow queries from profiler."""
    if not isinstance(data, list):
        return
    result.slow_queries = data


def _parse_repl_info(data: Any, result: MongoAnalysisResult) -> None:
    """Parse oplog replication info."""
    if not data or not isinstance(data, dict):
        return
    result.oplog = {
        "logSizeMB": data.get("logSizeMB", 0),
        "usedMB": data.get("usedMB", 0),
        "timeDiffHours": data.get("timeDiffHours", 0),
    }


def _parse_top(data: Any, result: MongoAnalysisResult) -> None:
    """Parse top collection activity."""
    if not isinstance(data, list):
        return
    result.top_collections = data


# ---------------------------------------------------------------------------
# Formatting helpers
# ---------------------------------------------------------------------------

def _fmt_bytes(b: int) -> str:
    """Format bytes as human-readable."""
    if b >= 1024 * 1024 * 1024:
        return f"{b / 1024 / 1024 / 1024:.1f} GB"
    elif b >= 1024 * 1024:
        return f"{b / 1024 / 1024:.1f} MB"
    elif b >= 1024:
        return f"{b / 1024:.1f} KB"
    return f"{b} B"


def _fmt_count(n: int) -> str:
    """Format large numbers with K/M suffix."""
    if n >= 1_000_000_000:
        return f"{n / 1_000_000_000:.1f}B"
    elif n >= 1_000_000:
        return f"{n / 1_000_000:.1f}M"
    elif n >= 1_000:
        return f"{n / 1_000:.1f}K"
    return str(n)


def _fmt_uptime(seconds: int) -> str:
    """Format seconds as Xd Yh."""
    days = seconds // 86400
    hours = (seconds % 86400) // 3600
    if days > 0:
        return f"{days}d {hours}h"
    elif hours > 0:
        return f"{hours}h {(seconds % 3600) // 60}m"
    return f"{seconds // 60}m"


def _fmt_us(microseconds: float) -> str:
    """Format microseconds as human-readable latency."""
    if microseconds >= 1_000_000:
        return f"{microseconds / 1_000_000:.1f}s"
    elif microseconds >= 1_000:
        return f"{microseconds / 1_000:.1f}ms"
    return f"{microseconds:.0f}us"


# ---------------------------------------------------------------------------
# Main analysis
# ---------------------------------------------------------------------------

def analyze_mongo(service: str, timeout: int = 300, quiet: bool = False,
                  skip_logs: bool = False, metrics_hours: int = 168,
                  project_id: Optional[str] = None,
                  environment_id: Optional[str] = None,
                  service_id: Optional[str] = None) -> MongoAnalysisResult:
    """Run complete MongoDB analysis."""
    if not quiet:
        print(f"Analyzing MongoDB database: {service}", file=sys.stderr)

    result = MongoAnalysisResult(
        service=service,
        db_type="mongo",
        timestamp=datetime.now(timezone.utc).isoformat(),
    )

    # === CONTEXT ===
    if not quiet:
        print("  [0/5] Getting Railway context...", file=sys.stderr, flush=True)
    dal._progress_timer.start()

    if environment_id and service_id:
        dal._ctx = RailwayContext(project_id=project_id, environment_id=environment_id, service_id=service_id)
        if not quiet:
            print(f"        using explicit IDs (env={environment_id[:8]}..., svc={service_id[:8]}...)", file=sys.stderr, flush=True)
    else:
        railway_status = get_railway_status()
        if railway_status:
            dal._ctx = RailwayContext(
                project_id=railway_status.get("projectId"),
                environment_id=railway_status.get("environmentId"),
                service_id=railway_status.get("serviceId"),
            )
        environment_id = dal._ctx.environment_id
        service_id = dal._ctx.service_id

    # === DEPLOYMENT STATUS ===
    progress(1, 5, "Fetching deployment status...", quiet)
    result.deployment_status = get_deployment_status(service, service_id=service_id)

    # === SSH PRE-CHECK ===
    progress(2, 5, "Testing SSH connectivity...", quiet)
    ssh_available = False
    ssh_stderr = ""
    ssh_attempts = [30, 60, 90]
    for attempt, attempt_timeout in enumerate(ssh_attempts, 1):
        ssh_code, ssh_stdout, ssh_stderr = run_ssh_query(service, "echo ok", timeout=attempt_timeout)
        if ssh_code == 0 and "ok" in ssh_stdout:
            ssh_available = True
            if not quiet:
                for line in ssh_stderr.splitlines():
                    if line.startswith("Using SSH key:"):
                        print(f"        {line}", file=sys.stderr, flush=True)
                        break
            break
        if not quiet:
            remaining = len(ssh_attempts) - attempt
            if remaining > 0:
                print(f"        SSH attempt {attempt}/{len(ssh_attempts)} failed ({ssh_stderr or 'no response'}), retrying with {ssh_attempts[attempt]}s timeout...", file=sys.stderr, flush=True)
            else:
                print(f"        SSH attempt {attempt}/{len(ssh_attempts)} failed ({ssh_stderr or 'no response'}), giving up", file=sys.stderr, flush=True)

    # === PARALLEL DATA COLLECTION ===
    progress(3, 5, "Running analysis (metrics, mongo queries, logs in parallel)...", quiet)

    def task_metrics():
        if environment_id and service_id:
            return get_all_metrics_from_api(environment_id, service_id, hours=metrics_hours)
        return None

    def task_mongo_batch1():
        """serverStatus + dbStats + collectionStats."""
        if not ssh_available:
            return ("error", f"SSH not available: {ssh_stderr or 'connection failed'}")
        results = {}
        # serverStatus
        code, stdout, stderr = run_mongosh_query(service, QUERY_SERVER_STATUS, timeout=30)
        if code == 0:
            results["serverStatus"] = _safe_json(stdout)
        else:
            results["serverStatus_error"] = stderr or stdout or "unknown"
        # dbStats
        code, stdout, stderr = run_mongosh_query(service, QUERY_DB_STATS, timeout=30)
        if code == 0:
            results["dbStats"] = _safe_json(stdout)
        else:
            results["dbStats_error"] = stderr or stdout or "unknown"
        # collectionStats
        code, stdout, stderr = run_mongosh_query(service, QUERY_COLLECTION_STATS, timeout=30)
        if code == 0:
            results["collStats"] = _safe_json(stdout)
        else:
            results["collStats_error"] = stderr or stdout or "unknown"
        return ("ok", results)

    def task_mongo_batch2():
        """slowQueries + currentOp + replInfo + top."""
        if not ssh_available:
            return ("error", f"SSH not available: {ssh_stderr or 'connection failed'}")
        results = {}
        # slow queries
        code, stdout, stderr = run_mongosh_query(service, QUERY_SLOW_QUERIES, timeout=30)
        if code == 0:
            results["slowQueries"] = _safe_json(stdout)
        else:
            results["slowQueries_error"] = stderr or stdout or "unknown"
        # currentOp
        code, stdout, stderr = run_mongosh_query(service, QUERY_CURRENT_OP, timeout=30)
        if code == 0:
            results["currentOp"] = _safe_json(stdout)
        else:
            results["currentOp_error"] = stderr or stdout or "unknown"
        # replication info
        code, stdout, stderr = run_mongosh_query(service, QUERY_REPL_INFO, timeout=30)
        if code == 0:
            results["replInfo"] = _safe_json(stdout)
        else:
            results["replInfo_error"] = stderr or stdout or "unknown"
        # top
        code, stdout, stderr = run_mongosh_query(service, QUERY_TOP, timeout=30)
        if code == 0:
            results["top"] = _safe_json(stdout)
        else:
            results["top_error"] = stderr or stdout or "unknown"
        return ("ok", results)

    def task_logs():
        if skip_logs:
            return []
        return get_recent_logs(service, lines=LOG_LINES_DEFAULT,
                               environment_id=environment_id,
                               service_id=service_id)

    with ThreadPoolExecutor(max_workers=4) as executor:
        future_metrics = executor.submit(task_metrics)
        future_batch1 = executor.submit(task_mongo_batch1)
        future_batch2 = executor.submit(task_mongo_batch2)
        future_logs = executor.submit(task_logs)

        metrics_result = future_metrics.result()
        batch1_result = future_batch1.result()
        batch2_result = future_batch2.result()
        logs_result = future_logs.result()

    # === PROCESS METRICS ===
    if metrics_result:
        result.disk_usage = metrics_result.get("disk_usage")
        result.cpu_memory = metrics_result.get("cpu_memory")
        result.metrics_history = metrics_result.get("metrics_history")
        result.collection_status["metrics_api"] = {"status": "success"}
    else:
        result.collection_status["metrics_api"] = {
            "status": "error",
            "error": "Metrics API returned no data"
        }

    # === PROCESS BATCH 1 (serverStatus, dbStats, collStats) ===
    if batch1_result[0] == "ok":
        b1 = batch1_result[1]
        ss = b1.get("serverStatus")
        if ss:
            _parse_server_status(ss, result)
            result.collection_status["server_status"] = {"status": "success"}
        else:
            err = b1.get("serverStatus_error", "no data")
            result.errors.append(f"serverStatus failed: {err}")
            result.collection_status["server_status"] = {"status": "error", "error": err}

        dbs = b1.get("dbStats")
        if dbs:
            _parse_db_stats(dbs, result)
            result.collection_status["db_stats"] = {"status": "success"}
        else:
            err = b1.get("dbStats_error", "no data")
            result.collection_status["db_stats"] = {"status": "error", "error": err}

        cs = b1.get("collStats")
        if cs:
            _parse_collection_stats(cs, result)
            result.collection_status["collection_stats"] = {"status": "success"}
        else:
            err = b1.get("collStats_error", "no data")
            result.collection_status["collection_stats"] = {"status": "error", "error": err}
    else:
        error_msg = batch1_result[1] if len(batch1_result) > 1 else "unknown"
        result.errors.append(f"Batch 1 (serverStatus/dbStats/collStats) failed: {error_msg}")
        for src in ("server_status", "db_stats", "collection_stats"):
            result.collection_status[src] = {"status": "error", "error": error_msg}

    # === PROCESS BATCH 2 (slowQueries, currentOp, replInfo, top) ===
    if batch2_result[0] == "ok":
        b2 = batch2_result[1]

        sq = b2.get("slowQueries")
        if sq is not None:
            _parse_slow_queries(sq, result)
            result.collection_status["slow_queries"] = {"status": "success"}
        else:
            err = b2.get("slowQueries_error", "no data")
            result.collection_status["slow_queries"] = {"status": "error", "error": err}

        co = b2.get("currentOp")
        if co is not None:
            _parse_current_op(co, result)
            result.collection_status["current_op"] = {"status": "success"}
        else:
            err = b2.get("currentOp_error", "no data")
            result.collection_status["current_op"] = {"status": "error", "error": err}

        ri = b2.get("replInfo")
        if ri is not None:
            _parse_repl_info(ri, result)
            result.collection_status["repl_info"] = {"status": "success"}
        else:
            err = b2.get("replInfo_error", "no data or not a replica set")
            result.collection_status["repl_info"] = {"status": "skipped", "reason": err}

        top = b2.get("top")
        if top is not None:
            _parse_top(top, result)
            result.collection_status["top"] = {"status": "success"}
        else:
            err = b2.get("top_error", "no data or insufficient privileges")
            result.collection_status["top"] = {"status": "skipped", "reason": err}
    else:
        error_msg = batch2_result[1] if len(batch2_result) > 1 else "unknown"
        result.errors.append(f"Batch 2 (slowQueries/currentOp/replInfo/top) failed: {error_msg}")
        for src in ("slow_queries", "current_op", "repl_info", "top"):
            result.collection_status[src] = {"status": "error", "error": error_msg}

    # === PROCESS LOGS ===
    progress(4, 5, "Processing logs...", quiet)
    if skip_logs:
        result.collection_status["logs_api"] = {"status": "skipped", "reason": "skip_logs flag set"}
    elif logs_result:
        result.recent_logs = logs_result
        result.collection_status["logs_api"] = {"status": "success", "lines": len(logs_result)}
        result.recent_errors = [
            line for line in result.recent_logs
            if 'ERROR' in line.upper() or 'FATAL' in line.upper() or 'PANIC' in line.upper()
        ][:100]
    else:
        result.recent_logs = []
        result.collection_status["logs_api"] = {"status": "error", "error": "Logs API returned no data"}

    # === RECOMMENDATIONS ===
    progress(5, 5, "Generating recommendations...", quiet)
    result.recommendations = generate_recommendations(result)

    if not quiet:
        total = dal._progress_timer.total_elapsed()
        print(f"Done.{total}", file=sys.stderr)

    return result


# ---------------------------------------------------------------------------
# Recommendations engine
# ---------------------------------------------------------------------------

def generate_recommendations(result: MongoAnalysisResult) -> List[Dict[str, str]]:
    """Generate recommendations based on analysis results."""
    recs: List[Dict[str, str]] = []

    # Collection failures — surface critical issues when SSH/introspection failed
    if result.collection_status:
        failed = {k: v for k, v in result.collection_status.items()
                  if v.get("status") in ("failed", "error")}
        ssh_sources = {"server_status", "db_stats", "collection_stats",
                       "slow_queries", "current_op", "repl_info", "top"}
        ssh_failed = {k: v for k, v in failed.items() if k in ssh_sources}
        if ssh_failed:
            sources = ", ".join(ssh_failed.keys())
            errors = "; ".join(v.get("error", "unknown") for v in ssh_failed.values())
            recs.append({
                "severity": "critical",
                "category": "collection",
                "message": f"SSH introspection failed — unable to collect {sources}. "
                           f"Error: {errors}. "
                           f"Analysis is incomplete: WiredTiger cache, connections, "
                           f"collection stats, and replication health could not be evaluated.",
            })

    # --- WiredTiger cache usage ---
    wt = result.wiredtiger_cache
    if wt:
        max_bytes = wt.get("max_bytes", 0)
        used_bytes = wt.get("bytes_in_cache", 0)
        dirty_bytes = wt.get("dirty_bytes", 0)
        app_evictions = wt.get("app_evictions", 0)

        if max_bytes > 0:
            usage_pct = round(100.0 * used_bytes / max_bytes, 1)
            if usage_pct > 80:
                recs.append({
                    "priority": "immediate",
                    "issue": f"WiredTiger cache is {usage_pct}% full ({_fmt_bytes(used_bytes)} of {_fmt_bytes(max_bytes)})",
                    "action": "Consider increasing service RAM. WiredTiger cache defaults to 50% of RAM minus 1 GB.",
                    "explanation": "When the WiredTiger cache is nearly full, MongoDB must evict pages more aggressively, "
                                   "increasing latency for reads and writes. Increasing RAM gives WiredTiger more room to cache data.",
                })

            if used_bytes > 0:
                dirty_pct = round(100.0 * dirty_bytes / max_bytes, 1)
                if dirty_pct > 20:
                    recs.append({
                        "priority": "short-term",
                        "issue": f"High dirty cache ({dirty_pct}% of total cache). Checkpoint may be falling behind.",
                        "action": "Monitor checkpoint duration and consider increasing RAM or reducing write throughput.",
                        "explanation": "Dirty pages must be written to disk during checkpoints. A high dirty ratio means "
                                       "checkpoints have more work, potentially causing latency spikes.",
                    })

        if app_evictions and app_evictions > 0:
            recs.append({
                "priority": "immediate",
                "issue": f"Application threads performing evictions ({app_evictions:,} pages). WiredTiger cache under pressure.",
                "action": "Increase RAM to give WiredTiger more cache space.",
                "explanation": "Normally the WiredTiger eviction threads handle cache pressure. When application threads "
                               "must evict pages themselves, queries stall waiting for cache space. This directly increases latency.",
            })

    # --- Connection usage ---
    conn = result.connections
    if conn:
        current = conn.get("current", 0)
        available = conn.get("available", 0)
        total = current + available
        if total > 0:
            pct = round(100.0 * current / total, 1)
            if pct > 80:
                recs.append({
                    "priority": "immediate" if pct > 90 else "short-term",
                    "issue": f"Connection usage at {pct}% ({current} of {total}). Approaching connection limit.",
                    "action": "Review application connection pooling. Consider using a connection pooler or increasing maxIncomingConnections.",
                    "explanation": "Running out of connections will cause new client connections to be refused. "
                                   "Most applications should use connection pooling to limit concurrent connections.",
                })

    # --- Page faults ---
    if result.page_faults and result.page_faults > 10000:
        recs.append({
            "priority": "short-term",
            "issue": f"Significant page faults ({result.page_faults:,}). Working set may exceed available RAM.",
            "action": "Increase service RAM or optimize queries to reduce working set size.",
            "explanation": "Page faults occur when MongoDB accesses data not in memory, requiring disk reads. "
                           "High page faults indicate the working set is larger than available RAM.",
        })

    # --- Queued operations ---
    gl = result.global_lock
    if gl:
        qr = gl.get("queue_readers", 0)
        qw = gl.get("queue_writers", 0)
        if qr > 0 or qw > 0:
            recs.append({
                "priority": "immediate" if (qr + qw) > 10 else "short-term",
                "issue": f"Operations queuing detected (readers: {qr}, writers: {qw}). Database may be under resource pressure.",
                "action": "Investigate slow operations and consider increasing RAM or CPU.",
                "explanation": "Queued operations mean requests are waiting for a lock. This can be caused by slow queries, "
                               "write-heavy workloads, or insufficient resources.",
            })

    # --- Query efficiency ---
    qe = result.query_executor
    dm = result.document_metrics
    if qe and dm:
        scanned = qe.get("scannedObjects", 0)
        returned = dm.get("returned", 0)
        if returned > 0 and scanned > returned * 10:
            ratio = round(scanned / returned, 1)
            recs.append({
                "priority": "immediate" if ratio > 100 else "short-term",
                "issue": f"Query efficiency concern: {_fmt_count(scanned)} objects scanned vs {_fmt_count(returned)} returned (ratio: {ratio}x).",
                "action": "Create indexes for frequently queried fields. Review slow query log for full collection scans.",
                "explanation": "A high scan-to-return ratio means MongoDB is examining many documents to satisfy queries. "
                               "Adding appropriate indexes dramatically reduces the number of documents examined.",
            })

    # --- Plan cache ---
    pc = result.plan_cache
    if pc:
        hits = pc.get("hits", 0)
        misses = pc.get("misses", 0)
        total = hits + misses
        if total > 100 and misses > hits:
            recs.append({
                "priority": "short-term",
                "issue": f"High plan cache miss ratio ({misses:,} misses vs {hits:,} hits). Queries may not be using optimal plans.",
                "action": "Consider creating indexes for frequent query patterns to stabilize query plans.",
                "explanation": "Plan cache misses mean MongoDB must re-evaluate query plans. Stable indexes help the planner "
                               "pick consistent, efficient plans.",
            })

    # --- Sort spill to disk ---
    sm = result.sort_metrics
    if sm:
        spill = sm.get("spillToDisk", 0)
        if spill > 0:
            recs.append({
                "priority": "short-term",
                "issue": f"Sorts spilling to disk ({spill:,} times). Queries performing in-memory sorts exceeding limit.",
                "action": "Add indexes to support sort operations, or increase RAM.",
                "explanation": "When a sort operation exceeds the memory limit (100 MB by default), MongoDB spills to disk. "
                               "Creating an index that matches the sort key avoids the in-memory sort entirely.",
            })

    # --- Cursor timeouts ---
    cur = result.cursors
    if cur:
        timed_out = cur.get("timed_out", 0)
        if timed_out > 0:
            recs.append({
                "priority": "short-term",
                "issue": f"Cursor timeouts detected ({timed_out:,}). Long-running queries may need optimization.",
                "action": "Review application code for unbounded queries or missing pagination.",
                "explanation": "Cursors time out after 10 minutes of inactivity by default. Frequent timeouts suggest "
                               "clients are not consuming results quickly enough or queries are returning too much data.",
            })

    # --- Asserts ---
    asserts = result.asserts
    if asserts:
        regular = asserts.get("regular", 0)
        warning = asserts.get("warning", 0)
        user = asserts.get("user", 0)
        msg = asserts.get("msg", 0)
        if regular > 0 or warning > 0 or user > 0:
            recs.append({
                "priority": "short-term",
                "issue": f"Database asserts detected (regular: {regular}, warning: {warning}, user: {user}, msg: {msg}). Investigate error conditions.",
                "action": "Check MongoDB logs for assert details. User asserts often indicate client errors; regular/warning asserts may signal server issues.",
                "explanation": "Asserts are internal consistency checks. Regular and warning asserts may indicate bugs or data issues. "
                               "User asserts are typically client-side errors (e.g., duplicate key violations).",
            })

    # --- Oplog usage ---
    oplog = result.oplog
    if oplog:
        log_size = oplog.get("logSizeMB", 0)
        used = oplog.get("usedMB", 0)
        if log_size > 0:
            oplog_pct = round(100.0 * used / log_size, 1)
            if oplog_pct > 80:
                recs.append({
                    "priority": "short-term",
                    "issue": f"Oplog is {oplog_pct}% full ({used:.0f} MB of {log_size:.0f} MB). May impact replication if oplog window is too small.",
                    "action": "Consider increasing the oplog size to maintain a larger replication window.",
                    "explanation": "The oplog stores recent write operations for replication. If it fills up and wraps around too quickly, "
                                   "replica set members that fall behind may need a full resync instead of incremental replication.",
                })

    return recs


# ---------------------------------------------------------------------------
# Report formatting
# ---------------------------------------------------------------------------

def format_report(result: MongoAnalysisResult) -> str:
    """Format analysis result as human-readable markdown report."""
    lines: List[str] = []
    lines.append("=" * 60)
    lines.append(f"# MongoDB Analysis: {result.service}")
    lines.append("=" * 60)
    lines.append(f"Timestamp: {result.timestamp}")
    lines.append(f"Status: {result.deployment_status}")
    lines.append("")

    # --- Data Collection Status ---
    if result.collection_status:
        lines.append("## Data Collection Status")
        lines.append("")
        lines.append("| Source | Status | Details |")
        lines.append("|--------|--------|---------|")
        source_labels = {
            "server_status": "Server Status (SSH)",
            "db_stats": "Database Stats (SSH)",
            "collection_stats": "Collection Stats (SSH)",
            "slow_queries": "Slow Queries (SSH)",
            "current_op": "Current Operations (SSH)",
            "repl_info": "Replication Info (SSH)",
            "top": "Top Collections (SSH)",
            "metrics_api": "Metrics API",
            "logs_api": "Logs API",
        }
        for source in ["server_status", "db_stats", "collection_stats",
                        "slow_queries", "current_op", "repl_info", "top",
                        "metrics_api", "logs_api"]:
            if source in result.collection_status:
                info = result.collection_status[source]
                status = info["status"].upper()
                details = ""
                if info.get("error"):
                    details = info["error"]
                elif info.get("reason"):
                    details = info["reason"]
                elif info.get("lines"):
                    details = f"{info['lines']} lines collected"
                elif status == "SUCCESS":
                    details = "OK"
                label = source_labels.get(source, source)
                lines.append(f"| {label} | {status} | {details} |")
        lines.append("")

    # --- Overview ---
    lines.append("## Overview")
    lines.append("")
    lines.append("| Metric | Value |")
    lines.append("|--------|-------|")
    if result.version:
        lines.append(f"| Version | {result.version} |")
    if result.storage_engine:
        lines.append(f"| Storage Engine | {result.storage_engine} |")
    if result.uptime_seconds is not None:
        lines.append(f"| Uptime | {_fmt_uptime(result.uptime_seconds)} |")
    status_icon = "Healthy" if result.deployment_status == "SUCCESS" else "Warning"
    lines.append(f"| Deployment | {result.deployment_status} | {status_icon} |")
    lines.append("")

    # --- Connections ---
    if result.connections:
        lines.append("## Connections")
        lines.append("")
        lines.append("| Metric | Value | Status |")
        lines.append("|--------|-------|--------|")
        c = result.connections
        current = c.get("current", 0)
        available = c.get("available", 0)
        total = current + available
        pct = round(100.0 * current / total, 1) if total > 0 else 0
        status = "Critical" if pct > 90 else "Warning" if pct > 80 else ""
        lines.append(f"| Current | {current:,} | {status} |")
        lines.append(f"| Available | {available:,} | |")
        lines.append(f"| Total Created | {c.get('totalCreated', 0):,} | |")
        lines.append("")

    # --- Operations (since startup) ---
    if result.opcounters:
        lines.append("## Operations (since startup)")
        lines.append("")
        lines.append("| Operation | Count |")
        lines.append("|-----------|-------|")
        for op in ("insert", "query", "update", "delete", "getmore", "command"):
            val = result.opcounters.get(op, 0)
            lines.append(f"| {op} | {_fmt_count(val)} |")
        lines.append("")

    # --- Replication opcounters ---
    if result.opcounters_repl:
        any_repl = any(v > 0 for v in result.opcounters_repl.values() if isinstance(v, (int, float)))
        if any_repl:
            lines.append("## Replication Operations")
            lines.append("")
            lines.append("| Operation | Count |")
            lines.append("|-----------|-------|")
            for op in ("insert", "query", "update", "delete", "getmore", "command"):
                val = result.opcounters_repl.get(op, 0)
                lines.append(f"| {op} | {_fmt_count(val)} |")
            lines.append("")

    # --- Latency ---
    if result.op_latencies:
        lines.append("## Latency")
        lines.append("")
        lines.append("| Operation | Avg Latency | Total Ops |")
        lines.append("|-----------|-------------|-----------|")
        for key, label in [("reads", "Reads"), ("writes", "Writes"), ("commands", "Commands")]:
            entry = result.op_latencies.get(key, {})
            avg_us = entry.get("avg_us", 0)
            ops = entry.get("ops", 0)
            lines.append(f"| {label} | {_fmt_us(avg_us)} | {_fmt_count(ops)} |")
        lines.append("")

    # --- Memory ---
    if result.memory:
        lines.append("## Memory")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Resident | {result.memory.get('resident_mb', 0):,} MB |")
        lines.append(f"| Virtual | {result.memory.get('virtual_mb', 0):,} MB |")
        if result.page_faults is not None:
            lines.append(f"| Page Faults | {result.page_faults:,} |")
        lines.append("")

    # --- WiredTiger Cache ---
    wt = result.wiredtiger_cache
    if wt:
        lines.append("## WiredTiger Cache")
        lines.append("")
        lines.append("| Metric | Value | Status |")
        lines.append("|--------|-------|--------|")
        used = wt.get("bytes_in_cache", 0)
        max_b = wt.get("max_bytes", 0)
        dirty = wt.get("dirty_bytes", 0)
        app_evict = wt.get("app_evictions", 0)
        lines.append(f"| Used | {_fmt_bytes(used)} | |")
        lines.append(f"| Maximum | {_fmt_bytes(max_b)} | |")
        if max_b > 0:
            usage_pct = round(100.0 * used / max_b, 1)
            cache_status = "Critical" if usage_pct > 90 else "Warning" if usage_pct > 80 else "OK"
            lines.append(f"| Usage | {usage_pct}% | {cache_status} |")
        lines.append(f"| Dirty | {_fmt_bytes(dirty)} | |")
        evict_status = "Warning" if app_evict > 0 else "OK"
        lines.append(f"| App Thread Evictions | {app_evict:,} | {evict_status} |")
        lines.append(f"| Pages Read Into Cache | {wt.get('pages_read', 0):,} | |")
        lines.append(f"| Pages Written From Cache | {wt.get('pages_written', 0):,} | |")
        lines.append("")

    # --- WiredTiger Checkpoint ---
    cp = result.wiredtiger_checkpoint
    if cp:
        ms = cp.get("most_recent_time_ms", 0)
        lines.append("## WiredTiger Checkpoint")
        lines.append("")
        lines.append(f"| Most Recent Checkpoint Time | {ms:,} ms |")
        lines.append("")

    # --- WiredTiger Tickets ---
    tk = result.wiredtiger_tickets
    if tk:
        lines.append("## WiredTiger Tickets")
        lines.append("")
        lines.append("| Metric | Available | Total |")
        lines.append("|--------|-----------|-------|")
        lines.append(f"| Read | {tk.get('read_available', 0)} | {tk.get('read_total', 0)} |")
        lines.append(f"| Write | {tk.get('write_available', 0)} | {tk.get('write_total', 0)} |")
        lines.append("")

    # --- Global Lock ---
    gl = result.global_lock
    if gl:
        lines.append("## Global Lock")
        lines.append("")
        lines.append("| Metric | Readers | Writers |")
        lines.append("|--------|---------|---------|")
        lines.append(f"| Queue | {gl.get('queue_readers', 0)} | {gl.get('queue_writers', 0)} |")
        lines.append(f"| Active | {gl.get('active_readers', 0)} | {gl.get('active_writers', 0)} |")
        lines.append("")

    # --- Network ---
    if result.network:
        lines.append("## Network")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Bytes In | {_fmt_bytes(result.network.get('bytesIn', 0))} |")
        lines.append(f"| Bytes Out | {_fmt_bytes(result.network.get('bytesOut', 0))} |")
        lines.append(f"| Requests | {_fmt_count(result.network.get('numRequests', 0))} |")
        lines.append("")

    # --- Document Metrics ---
    dm = result.document_metrics
    if dm:
        lines.append("## Documents")
        lines.append("")
        lines.append("| Operation | Count |")
        lines.append("|-----------|-------|")
        for key in ("inserted", "updated", "deleted", "returned"):
            lines.append(f"| {key} | {_fmt_count(dm.get(key, 0))} |")
        lines.append("")

    # --- Query Efficiency ---
    qe = result.query_executor
    if qe:
        lines.append("## Query Efficiency")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Scanned Objects | {_fmt_count(qe.get('scannedObjects', 0))} |")
        lines.append(f"| Scanned Keys | {_fmt_count(qe.get('scanned', 0))} |")
        if dm:
            returned = dm.get("returned", 0)
            scanned = qe.get("scannedObjects", 0)
            if returned > 0:
                ratio = round(scanned / returned, 1)
                status = "Warning" if ratio > 10 else "OK"
                lines.append(f"| Scan-to-Return Ratio | {ratio}x | {status} |")
        lines.append("")

    # --- Plan Cache ---
    pc = result.plan_cache
    if pc:
        lines.append("## Plan Cache (7.0+)")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Hits | {_fmt_count(pc.get('hits', 0))} |")
        lines.append(f"| Misses | {_fmt_count(pc.get('misses', 0))} |")
        lines.append("")

    # --- Sort Metrics ---
    sm = result.sort_metrics
    if sm:
        lines.append("## Sort (7.0+)")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Spill to Disk | {sm.get('spillToDisk', 0):,} |")
        lines.append(f"| Total Bytes Sorted | {_fmt_bytes(sm.get('totalBytesSorted', 0))} |")
        lines.append("")

    # --- Cursors ---
    cur = result.cursors
    if cur:
        lines.append("## Cursors")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Open Total | {cur.get('open_total', 0):,} |")
        timed = cur.get("timed_out", 0)
        status = "Warning" if timed > 0 else ""
        lines.append(f"| Timed Out | {timed:,} | {status} |")
        lines.append("")

    # --- TTL ---
    ttl = result.ttl_metrics
    if ttl:
        lines.append("## TTL")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Deleted Documents | {_fmt_count(ttl.get('deletedDocuments', 0))} |")
        lines.append(f"| Passes | {ttl.get('passes', 0):,} |")
        lines.append("")

    # --- Asserts ---
    asserts = result.asserts
    if asserts:
        any_assert = any(asserts.get(k, 0) > 0 for k in ("regular", "warning", "msg", "user"))
        if any_assert:
            lines.append("## Asserts")
            lines.append("")
            lines.append("| Type | Count |")
            lines.append("|------|-------|")
            for key in ("regular", "warning", "msg", "user", "rollovers"):
                lines.append(f"| {key} | {asserts.get(key, 0):,} |")
            lines.append("")

    # --- Storage ---
    st = result.storage
    if st:
        lines.append("## Storage")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Data Size | {_fmt_bytes(st.get('dataSize', 0))} |")
        lines.append(f"| Storage Size | {_fmt_bytes(st.get('storageSize', 0))} |")
        lines.append(f"| Index Size | {_fmt_bytes(st.get('indexSize', 0))} |")
        lines.append(f"| Objects | {_fmt_count(st.get('objects', 0))} |")
        lines.append(f"| Collections | {st.get('collections', 0)} |")
        lines.append("")

    # --- Collections ---
    if result.collection_stats:
        lines.append("## Collections")
        lines.append("")
        lines.append("| Collection | Documents | Data Size | Storage | Indexes |")
        lines.append("|------------|-----------|-----------|---------|---------|")
        # Sort by size descending
        sorted_colls = sorted(result.collection_stats, key=lambda c: c.get("size", 0), reverse=True)
        for c in sorted_colls:
            name = c.get("name", "?")
            count = _fmt_count(c.get("count", 0))
            size = _fmt_bytes(c.get("size", 0))
            storage = _fmt_bytes(c.get("storageSize", 0))
            nidx = c.get("nindexes", 0)
            lines.append(f"| {name} | {count} | {size} | {storage} | {nidx} |")
        lines.append("")

    # --- Top Collections by Activity ---
    if result.top_collections:
        lines.append("## Top Collections by Activity")
        lines.append("")
        lines.append("| Namespace | Reads | Read Time | Writes | Write Time |")
        lines.append("|-----------|-------|-----------|--------|------------|")
        # Sort by total activity
        sorted_top = sorted(result.top_collections,
                            key=lambda t: t.get("reads", 0) + t.get("writes", 0),
                            reverse=True)
        for t in sorted_top[:20]:
            ns = t.get("ns", "?")
            reads = _fmt_count(t.get("reads", 0))
            read_time = _fmt_us(t.get("readTimeUs", 0))
            writes = _fmt_count(t.get("writes", 0))
            write_time = _fmt_us(t.get("writeTimeUs", 0))
            lines.append(f"| {ns} | {reads} | {read_time} | {writes} | {write_time} |")
        lines.append("")

    # --- Replication ---
    if result.replication:
        lines.append("## Replication")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        r = result.replication
        if r.get("setName"):
            lines.append(f"| Replica Set | {r['setName']} |")
        lines.append(f"| Is Writable Primary | {r.get('isWritablePrimary', 'N/A')} |")
        if r.get("primary"):
            lines.append(f"| Primary | {r['primary']} |")
        if r.get("hosts"):
            lines.append(f"| Hosts | {', '.join(r['hosts'])} |")
        lines.append("")

    # --- Oplog ---
    if result.oplog:
        lines.append("## Oplog")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        ol = result.oplog
        log_size = ol.get("logSizeMB", 0)
        used = ol.get("usedMB", 0)
        lines.append(f"| Log Size | {log_size:.0f} MB |")
        lines.append(f"| Used | {used:.0f} MB |")
        if log_size > 0:
            lines.append(f"| Usage | {round(100.0 * used / log_size, 1)}% |")
        hours = ol.get("timeDiffHours", 0)
        lines.append(f"| Time Window | {hours:.1f} hours |")
        lines.append("")

    # --- Slow Queries ---
    if result.slow_queries:
        lines.append("## Slow Queries")
        lines.append("")
        lines.append("| Op | Namespace | Duration | Plan |")
        lines.append("|----|-----------|----------|------|")
        for q in result.slow_queries:
            op = q.get("op", "?")
            ns = q.get("ns", "?")
            millis = q.get("millis", 0)
            plan = q.get("planSummary", "")
            lines.append(f"| {op} | {ns} | {millis}ms | {plan} |")
        lines.append("")

    # --- Active Operations ---
    if result.active_ops:
        lines.append("## Active Operations")
        lines.append("")
        lines.append("| OpID | Type | Namespace | Duration |")
        lines.append("|------|------|-----------|----------|")
        sorted_ops = sorted(result.active_ops, key=lambda o: o.get("microsecs_running", 0), reverse=True)
        for op in sorted_ops[:20]:
            opid = op.get("opid", "?")
            op_type = op.get("type", "?")
            ns = op.get("ns", "")
            us = op.get("microsecs_running", 0)
            lines.append(f"| {opid} | {op_type} | {ns} | {_fmt_us(us)} |")
        lines.append("")

    # --- Infrastructure Trends ---
    if result.metrics_history and result.metrics_history.get("windows"):
        windows = result.metrics_history.get("windows", {})
        for window_label, window_data in windows.items():
            mh = window_data.get("metrics", {})
            if not mh:
                continue
            lines.append(f"## Infrastructure Trends ({window_label})")
            lines.append("")
            lines.append("| Metric | Current | Min | Max | Avg | Trend | Change |")
            lines.append("|--------|---------|-----|-----|-----|-------|--------|")
            display_order = [
                ("cpu", "CPU"),
                ("memory", "Memory"),
                ("disk", "Disk"),
                ("network_rx", "Network RX"),
                ("network_tx", "Network TX"),
            ]
            for key, label in display_order:
                if key in mh:
                    m = mh[key]
                    unit = m["unit"]
                    trend = m.get("trend", {})
                    direction = trend.get("direction", "?")
                    change = trend.get("change_pct", 0)
                    arrow = {"increasing": "^", "decreasing": "v", "stable": "~"}.get(direction, "?")
                    spike_note = ""
                    if m.get("spikes"):
                        spike_note = f" ({m['spikes']['count']} spikes)"
                    lines.append(
                        f"| {label} | {m['current']} {unit} | {m['min']} | {m['max']} | "
                        f"{m['avg']} | {arrow} {direction} | {change:+.1f}%{spike_note} |"
                    )
            lines.append("")

    # --- CPU / Memory summary ---
    if result.cpu_memory:
        lines.append("## Resource Usage")
        lines.append("")
        lines.append("| Metric | Value | Status |")
        lines.append("|--------|-------|--------|")
        cm = result.cpu_memory
        if "cpu_percent" in cm:
            cpu = cm["cpu_percent"]
            status = "Critical" if cpu > 85 else "Warning" if cpu > 70 else "Healthy"
            trend_str = _trend_indicator(result.metrics_history, "cpu")
            lines.append(f"| CPU Usage | {cpu} vCPU{trend_str} | {status} |")
            if cm.get("cpu_limit"):
                lines.append(f"| CPU Limit | {cm['cpu_limit']} vCPU | - |")
        if "memory_gb" in cm:
            mem_val = cm["memory_gb"]
            trend_str = _trend_indicator(result.metrics_history, "memory")
            utilization = ""
            if cm.get("memory_limit_gb"):
                pct = round((mem_val / cm["memory_limit_gb"]) * 100, 1)
                status = "Critical" if pct > 90 else "Warning" if pct > 80 else "Healthy"
                utilization = f" ({pct}% of {cm['memory_limit_gb']} GB)"
            else:
                status = "-"
            lines.append(f"| Memory Usage | {mem_val} GB{utilization}{trend_str} | {status} |")
        if result.disk_usage:
            lines.append(f"| Disk Usage | {result.disk_usage.get('used', 'N/A')} | - |")
        lines.append("")

    # --- Recent Errors ---
    if result.recent_errors:
        lines.append("## Recent Errors")
        lines.append("")
        for error in result.recent_errors[:10]:
            lines.append(f"- {error[:150]}...")
        lines.append("")

    # --- Recommendations ---
    if result.recommendations:
        lines.append("## Recommendations")
        lines.append("")
        for i, rec in enumerate(result.recommendations, 1):
            priority = rec["priority"].upper()
            lines.append(f"{i}. **[{priority}]** {rec['issue']}")
            lines.append(f"   **Action:** {rec['action']}")
            if rec.get("explanation"):
                lines.append(f"   **Why:** {rec['explanation']}")
            lines.append("")

    # --- Errors ---
    if result.errors:
        lines.append("## Errors")
        lines.append("")
        for error in result.errors:
            lines.append(f"- {error}")
        lines.append("")

    lines.append("=" * 60)
    lines.append("END OF REPORT")
    lines.append("=" * 60)

    return "\n".join(lines)


# ---------------------------------------------------------------------------
# Single-step debugging
# ---------------------------------------------------------------------------

def run_single_step(args) -> int:
    """Run a single collection step for debugging."""
    service = args.service
    _init_context(args)
    environment_id = dal._ctx.environment_id
    service_id = dal._ctx.service_id

    if args.step == "ssh-test":
        print(f"Testing SSH to service: {service}", file=sys.stderr)
        code, stdout, stderr = run_ssh_query(service, "echo ok", timeout=45)
        print(f"Exit code: {code}")
        print(f"Stdout: {stdout.strip()}")
        if stderr:
            print(f"Stderr: {stderr.strip()}")
        return 0 if (code == 0 and "ok" in stdout) else 1

    elif args.step == "server-status":
        print(f"Running serverStatus on: {service}", file=sys.stderr)
        code, stdout, stderr = run_mongosh_query(service, QUERY_SERVER_STATUS, timeout=30)
        print(f"Exit code: {code}")
        if code == 0 and stdout:
            data = _safe_json(stdout)
            if data:
                print(json.dumps(data, indent=2))
            else:
                print(f"Raw output:\n{stdout}")
        else:
            print(f"Error: {stderr or stdout}")
        return code

    elif args.step == "db-stats":
        print(f"Running db.stats() on: {service}", file=sys.stderr)
        code, stdout, stderr = run_mongosh_query(service, QUERY_DB_STATS, timeout=30)
        print(f"Exit code: {code}")
        if code == 0 and stdout:
            data = _safe_json(stdout)
            if data:
                print(json.dumps(data, indent=2))
            else:
                print(f"Raw output:\n{stdout}")
        else:
            print(f"Error: {stderr or stdout}")
        return code

    elif args.step == "logs":
        print(f"Fetching logs for: {service}", file=sys.stderr)
        logs = get_recent_logs(service, lines=LOG_LINES_DEFAULT,
                               environment_id=environment_id,
                               service_id=service_id)
        print(f"Lines fetched: {len(logs)}")
        for line in logs:
            print(line)
        return 0

    elif args.step == "metrics":
        print(f"Fetching metrics for: {service}", file=sys.stderr)
        if environment_id and service_id:
            metrics = get_all_metrics_from_api(environment_id, service_id)
            if metrics:
                print(json.dumps(metrics, indent=2))
            else:
                print("Metrics API returned no data")
                return 1
        else:
            print("Missing environment_id or service_id from railway config")
            return 1
        return 0

    return 1


# ---------------------------------------------------------------------------
# CLI entry point
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="MongoDB analysis for Railway services.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )

    parser.add_argument("--service", required=True, help="Service name")
    parser.add_argument("--json", action="store_true",
                        help="Output as JSON")
    parser.add_argument("--timeout", type=int, default=300,
                        help="Timeout in seconds (default: 300)")
    parser.add_argument("--quiet", "-q", action="store_true",
                        help="Suppress progress messages")
    parser.add_argument("--skip-logs", action="store_true",
                        help="Skip log fetching for faster analysis")
    parser.add_argument("--metrics-hours", type=int, default=168,
                        help="Hours of metrics history to fetch (default: 168, max: 168)")
    parser.add_argument("--step",
                        choices=["ssh-test", "server-status", "db-stats", "logs", "metrics"],
                        help="Run a single collection step for debugging")
    parser.add_argument("--project-id", help="Project ID (bypasses railway link)")
    parser.add_argument("--environment-id", help="Environment ID (bypasses railway link)")
    parser.add_argument("--service-id", help="Service ID (bypasses railway link)")

    args = parser.parse_args()

    if args.step:
        return run_single_step(args)

    result = analyze_mongo(
        args.service,
        timeout=args.timeout,
        quiet=args.quiet,
        skip_logs=args.skip_logs,
        metrics_hours=min(args.metrics_hours, 168),
        project_id=args.project_id,
        environment_id=args.environment_id,
        service_id=args.service_id,
    )

    if args.json:
        print(json.dumps(asdict(result), indent=2))
    else:
        print(format_report(result))

    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
MySQL analysis for Railway deployments.

Produces a comprehensive report covering:
- Deployment status & resource metrics (CPU, memory, disk)
- Connection overview
- Query throughput & efficiency
- InnoDB buffer pool & row operations
- Lock contention
- Top queries (from performance_schema)
- Table sizes
- Active processes
- Recommendations

Usage:
    analyze-mysql.py --service <name>
    analyze-mysql.py --service <name> --json
    analyze-mysql.py --service <name> --step ssh-test
"""

import argparse
import json
import os
import re
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional, Tuple

import dal
from dal import (
    LOG_LINES_DEFAULT, ProgressTimer, RailwayContext,
    _init_context, progress, run_railway_command, run_ssh_query,
    get_railway_status, get_deployment_status,
    get_all_metrics_from_api, _analyze_window, _build_metrics_history,
    get_recent_logs,
    _safe_int, _safe_float, _format_uptime, _trend_indicator,
)


# ---------------------------------------------------------------------------
# Result container
# ---------------------------------------------------------------------------

@dataclass
class MySQLAnalysisResult:
    """Container for MySQL analysis results."""
    service: str
    db_type: str
    timestamp: str
    deployment_status: str = "UNKNOWN"

    # Resource metrics from Railway API
    disk_usage: Optional[Dict[str, Any]] = None
    cpu_memory: Optional[Dict[str, Any]] = None
    metrics_history: Optional[Dict[str, Any]] = None

    # MySQL data
    overview: Optional[Dict[str, Any]] = None
    query_throughput: Optional[Dict[str, Any]] = None
    innodb_row_ops: Optional[Dict[str, Any]] = None
    query_efficiency: Optional[Dict[str, Any]] = None
    innodb_buffer_pool: Optional[Dict[str, Any]] = None
    innodb_io: Optional[Dict[str, Any]] = None
    network: Optional[Dict[str, Any]] = None
    locks: Optional[Dict[str, Any]] = None
    table_cache: Optional[Dict[str, Any]] = None
    top_queries: List[Dict[str, Any]] = field(default_factory=list)
    top_queries_status: Optional[str] = None
    tables: List[Dict[str, Any]] = field(default_factory=list)
    active_processes: List[Dict[str, Any]] = field(default_factory=list)

    # Logs
    recent_logs: List[str] = field(default_factory=list)
    recent_errors: List[str] = field(default_factory=list)

    # Metadata
    collection_status: Dict[str, Dict[str, Any]] = field(default_factory=dict)
    errors: List[str] = field(default_factory=list)
    recommendations: List[Dict[str, str]] = field(default_factory=list)


# ---------------------------------------------------------------------------
# MySQL-specific helpers
# ---------------------------------------------------------------------------

def run_mysql_query(service: str, query: str, timeout: int = 30) -> Tuple[int, str]:
    """Run a MySQL query via SSH and return (returncode, output).

    Uses -B (batch) mode which produces tab-separated output with headers.
    Filters out the mysql CLI password warning.
    """
    import base64
    query = " ".join(query.split())
    # Base64-encode the query to avoid all shell quoting issues
    # (single quotes in SQL IN clauses break bash -c '...' wrapping)
    encoded = base64.b64encode(query.encode()).decode()
    command = (
        f'''bash +H -c 'echo {encoded} | base64 -d | MYSQL_PWD="$MYSQLPASSWORD" mysql -h localhost -P 3306 '''
        f'''-u "$MYSQLUSER" -D "$MYSQLDATABASE" --default-character-set=utf8mb4 '''
        f'''-B' '''
    )
    code, stdout, stderr = run_ssh_query(service, command, timeout)
    # Filter out the password warning from stdout (mysql sometimes writes it there)
    lines = []
    for line in stdout.split("\n"):
        if "Using a password on the command line" in line:
            continue
        lines.append(line)
    stdout = "\n".join(lines)
    if code != 0:
        # Also filter warning from stderr
        stderr_clean = "\n".join(
            l for l in stderr.split("\n")
            if "Using a password on the command line" not in l
        )
        return code, stderr_clean or stdout
    return 0, stdout


def parse_mysql_batch(output: str) -> List[Dict[str, str]]:
    """Parse MySQL -B (batch/tab-separated) output into list of dicts.

    First line is column headers, subsequent lines are values.
    """
    lines = [l for l in output.strip().split("\n") if l.strip()]
    if len(lines) < 1:
        return []
    headers = lines[0].split("\t")
    rows = []
    for line in lines[1:]:
        values = line.split("\t")
        if len(values) == len(headers):
            rows.append(dict(zip(headers, values)))
    return rows


def parse_mysql_kv(output: str) -> Dict[str, str]:
    """Parse MySQL SHOW output (Variable_name / Value pairs) into a dict."""
    rows = parse_mysql_batch(output)
    result: Dict[str, str] = {}
    for row in rows:
        name = row.get("Variable_name", "")
        value = row.get("Value", "")
        if name:
            result[name] = value
    return result


# ---------------------------------------------------------------------------
# Railway context / status / metrics helpers
# ---------------------------------------------------------------------------

# ---------------------------------------------------------------------------
# MySQL queries
# ---------------------------------------------------------------------------

QUERY_GLOBAL_STATUS = """SHOW GLOBAL STATUS WHERE Variable_name IN ('Threads_connected','Threads_running','Max_used_connections','Questions','Slow_queries','Com_select','Com_insert','Com_update','Com_delete','Innodb_buffer_pool_read_requests','Innodb_buffer_pool_reads','Innodb_buffer_pool_pages_data','Innodb_buffer_pool_pages_free','Innodb_buffer_pool_pages_dirty','Innodb_row_lock_waits','Innodb_row_lock_time','Uptime','Bytes_received','Bytes_sent','Connections','Aborted_clients','Aborted_connects','Innodb_rows_read','Innodb_rows_inserted','Innodb_rows_updated','Innodb_rows_deleted','Innodb_data_reads','Innodb_data_writes','Innodb_buffer_pool_bytes_data','Innodb_buffer_pool_bytes_dirty','Created_tmp_disk_tables','Created_tmp_tables','Handler_read_rnd_next','Handler_read_first','Handler_read_key','Select_full_join','Select_range','Sort_merge_passes','Table_locks_waited','Table_locks_immediate','Open_tables','Opened_tables')"""

QUERY_VARIABLES = """SHOW VARIABLES WHERE Variable_name IN ('max_connections','innodb_buffer_pool_size','long_query_time','version','table_open_cache','performance_schema')"""

QUERY_TABLE_SIZES = """SELECT TABLE_NAME, TABLE_ROWS, DATA_LENGTH, INDEX_LENGTH, DATA_LENGTH + INDEX_LENGTH AS TOTAL_SIZE FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() ORDER BY TOTAL_SIZE DESC LIMIT 15"""

QUERY_PROCESSLIST = """SHOW PROCESSLIST"""

QUERY_TOP_QUERIES = """SELECT DIGEST, LEFT(DIGEST_TEXT, 200) AS DIGEST_TEXT, COUNT_STAR, ROUND(SUM_TIMER_WAIT/1000000000, 2) AS TOTAL_LATENCY_MS, ROUND(AVG_TIMER_WAIT/1000000000, 2) AS AVG_LATENCY_MS, SUM_ROWS_EXAMINED, SUM_ROWS_SENT, SUM_CREATED_TMP_DISK_TABLES, SUM_NO_INDEX_USED FROM performance_schema.events_statements_summary_by_digest WHERE DIGEST IS NOT NULL AND COUNT_STAR > 0 ORDER BY SUM_TIMER_WAIT DESC LIMIT 15"""


# ---------------------------------------------------------------------------
# MySQL data collection
# ---------------------------------------------------------------------------

def collect_mysql_data(service: str, timeout: int = 30) -> Dict[str, Any]:
    """Collect all MySQL metrics via SSH.

    Batches queries into two SSH calls for efficiency:
      1. SHOW GLOBAL STATUS + SHOW VARIABLES
      2. Table sizes + processlist + top queries (performance_schema)

    Returns a dict with raw parsed data keyed by section.
    """
    data: Dict[str, Any] = {
        "global_status": {},
        "variables": {},
        "tables": [],
        "processlist": [],
        "top_queries": [],
        "errors": [],
    }

    # --- Batch 1: SHOW GLOBAL STATUS; SHOW VARIABLES ---
    batch1_query = QUERY_GLOBAL_STATUS + "; " + QUERY_VARIABLES
    code, output = run_mysql_query(service, batch1_query, timeout=timeout)
    if code != 0:
        data["errors"].append(f"Batch 1 (status/variables) failed: {output}")
    else:
        # MySQL concatenates the two result sets; split them by detecting
        # a second header line starting with Variable_name
        sections = _split_mysql_resultsets(output, "Variable_name")
        if len(sections) >= 1:
            data["global_status"] = parse_mysql_kv(sections[0])
        if len(sections) >= 2:
            data["variables"] = parse_mysql_kv(sections[1])

    # --- Batch 2: tables + processlist + top queries ---
    batch2_query = QUERY_TABLE_SIZES + "; " + QUERY_PROCESSLIST + "; " + QUERY_TOP_QUERIES
    code, output = run_mysql_query(service, batch2_query, timeout=timeout)
    if code != 0:
        # Top queries may fail if performance_schema is off; try without
        batch2_fallback = QUERY_TABLE_SIZES + "; " + QUERY_PROCESSLIST
        code2, output2 = run_mysql_query(service, batch2_fallback, timeout=timeout)
        if code2 != 0:
            data["errors"].append(f"Batch 2 (tables/processlist) failed: {output}")
        else:
            sections = _split_mysql_resultsets_multi(output2, [
                "TABLE_NAME",
                "Id",
            ])
            if len(sections) >= 1:
                data["tables"] = parse_mysql_batch(sections[0])
            if len(sections) >= 2:
                data["processlist"] = parse_mysql_batch(sections[1])
    else:
        sections = _split_mysql_resultsets_multi(output, [
            "TABLE_NAME",
            "Id",
            "DIGEST",
        ])
        if len(sections) >= 1:
            data["tables"] = parse_mysql_batch(sections[0])
        if len(sections) >= 2:
            data["processlist"] = parse_mysql_batch(sections[1])
        if len(sections) >= 3:
            data["top_queries"] = parse_mysql_batch(sections[2])

    return data


def _split_mysql_resultsets(output: str, header_key: str) -> List[str]:
    """Split concatenated MySQL batch output into sections by header line."""
    lines = output.strip().split("\n")
    sections: List[List[str]] = []
    current: List[str] = []

    for line in lines:
        if line.startswith(header_key + "\t") or line.strip() == header_key:
            if current:
                sections.append("\n".join(current))
            current = [line]
        else:
            current.append(line)
    if current:
        sections.append("\n".join(current))

    return sections


def _split_mysql_resultsets_multi(output: str, header_keys: List[str]) -> List[str]:
    """Split concatenated MySQL batch output into sections by multiple different header keys."""
    lines = output.strip().split("\n")
    sections: List[List[str]] = []
    current: List[str] = []
    expected_idx = 0

    for line in lines:
        # Check if this line starts a new result set
        matched = False
        if expected_idx < len(header_keys):
            key = header_keys[expected_idx]
            if line.startswith(key + "\t") or line.strip() == key:
                if current:
                    sections.append("\n".join(current))
                current = [line]
                expected_idx += 1
                matched = True
        # Also check remaining headers in case we skipped one
        if not matched:
            for idx in range(expected_idx, len(header_keys)):
                key = header_keys[idx]
                if line.startswith(key + "\t") or line.strip() == key:
                    if current:
                        sections.append("\n".join(current))
                    current = [line]
                    expected_idx = idx + 1
                    matched = True
                    break
        if not matched:
            current.append(line)

    if current:
        sections.append("\n".join(current))

    return sections


# ---------------------------------------------------------------------------
# Parse collected data into result
# ---------------------------------------------------------------------------

def parse_mysql_data(data: Dict[str, Any], result: MySQLAnalysisResult) -> None:
    """Transform raw MySQL data into structured result sections."""
    gs = data.get("global_status", {})
    vs = data.get("variables", {})

    # --- Overview ---
    version = vs.get("version", "unknown")
    uptime_sec = _safe_int(gs.get("Uptime"))
    threads_connected = _safe_int(gs.get("Threads_connected"))
    threads_running = _safe_int(gs.get("Threads_running"))
    max_used_connections = _safe_int(gs.get("Max_used_connections"))
    max_connections = _safe_int(vs.get("max_connections"), 1)
    aborted_clients = _safe_int(gs.get("Aborted_clients"))
    aborted_connects = _safe_int(gs.get("Aborted_connects"))
    connection_usage_pct = round(max_used_connections / max_connections * 100, 1) if max_connections > 0 else 0

    result.overview = {
        "version": version,
        "uptime_seconds": uptime_sec,
        "uptime_human": _format_uptime(uptime_sec),
        "threads_connected": threads_connected,
        "threads_running": threads_running,
        "max_used_connections": max_used_connections,
        "max_connections": max_connections,
        "connection_usage_percent": connection_usage_pct,
        "aborted_clients": aborted_clients,
        "aborted_connects": aborted_connects,
    }

    # --- Query Throughput ---
    questions = _safe_int(gs.get("Questions"))
    slow_queries = _safe_int(gs.get("Slow_queries"))
    long_query_time = vs.get("long_query_time", "10")
    com_select = _safe_int(gs.get("Com_select"))
    com_insert = _safe_int(gs.get("Com_insert"))
    com_update = _safe_int(gs.get("Com_update"))
    com_delete = _safe_int(gs.get("Com_delete"))

    result.query_throughput = {
        "questions": questions,
        "slow_queries": slow_queries,
        "long_query_time": long_query_time,
        "com_select": com_select,
        "com_insert": com_insert,
        "com_update": com_update,
        "com_delete": com_delete,
    }

    # --- InnoDB Row Operations ---
    result.innodb_row_ops = {
        "rows_read": _safe_int(gs.get("Innodb_rows_read")),
        "rows_inserted": _safe_int(gs.get("Innodb_rows_inserted")),
        "rows_updated": _safe_int(gs.get("Innodb_rows_updated")),
        "rows_deleted": _safe_int(gs.get("Innodb_rows_deleted")),
    }

    # --- Query Efficiency ---
    created_tmp_disk = _safe_int(gs.get("Created_tmp_disk_tables"))
    created_tmp = _safe_int(gs.get("Created_tmp_tables"))
    tmp_disk_pct = round(created_tmp_disk / created_tmp * 100, 1) if created_tmp > 0 else 0
    handler_rnd_next = _safe_int(gs.get("Handler_read_rnd_next"))
    handler_first = _safe_int(gs.get("Handler_read_first"))
    handler_key = _safe_int(gs.get("Handler_read_key"))
    scan_total = handler_rnd_next + handler_first + handler_key
    table_scan_pct = round((handler_rnd_next + handler_first) / scan_total * 100, 1) if scan_total > 0 else 0
    select_full_join = _safe_int(gs.get("Select_full_join"))
    select_range = _safe_int(gs.get("Select_range"))
    sort_merge_passes = _safe_int(gs.get("Sort_merge_passes"))

    result.query_efficiency = {
        "created_tmp_disk_tables": created_tmp_disk,
        "created_tmp_tables": created_tmp,
        "tmp_disk_table_percent": tmp_disk_pct,
        "handler_read_rnd_next": handler_rnd_next,
        "handler_read_first": handler_first,
        "handler_read_key": handler_key,
        "table_scan_percent": table_scan_pct,
        "select_full_join": select_full_join,
        "select_range": select_range,
        "sort_merge_passes": sort_merge_passes,
    }

    # --- InnoDB Buffer Pool ---
    read_requests = _safe_int(gs.get("Innodb_buffer_pool_read_requests"))
    reads = _safe_int(gs.get("Innodb_buffer_pool_reads"))
    hit_ratio = round((read_requests - reads) / read_requests * 100, 2) if read_requests > 0 else 0
    pool_size = _safe_int(vs.get("innodb_buffer_pool_size"))
    bytes_data = _safe_int(gs.get("Innodb_buffer_pool_bytes_data"))
    bytes_dirty = _safe_int(gs.get("Innodb_buffer_pool_bytes_dirty"))
    usage_pct = round(bytes_data / pool_size * 100, 1) if pool_size > 0 else 0
    pages_free = _safe_int(gs.get("Innodb_buffer_pool_pages_free"))
    pages_data = _safe_int(gs.get("Innodb_buffer_pool_pages_data"))
    pages_dirty = _safe_int(gs.get("Innodb_buffer_pool_pages_dirty"))

    result.innodb_buffer_pool = {
        "hit_ratio": hit_ratio,
        "read_requests": read_requests,
        "reads": reads,
        "buffer_pool_size": pool_size,
        "bytes_data": bytes_data,
        "bytes_dirty": bytes_dirty,
        "usage_percent": usage_pct,
        "pages_data": pages_data,
        "pages_free": pages_free,
        "pages_dirty": pages_dirty,
    }

    # --- InnoDB I/O ---
    result.innodb_io = {
        "data_reads": _safe_int(gs.get("Innodb_data_reads")),
        "data_writes": _safe_int(gs.get("Innodb_data_writes")),
    }

    # --- Network ---
    result.network = {
        "bytes_received": _safe_int(gs.get("Bytes_received")),
        "bytes_sent": _safe_int(gs.get("Bytes_sent")),
    }

    # --- Locks ---
    row_lock_waits = _safe_int(gs.get("Innodb_row_lock_waits"))
    row_lock_time = _safe_int(gs.get("Innodb_row_lock_time"))
    table_locks_waited = _safe_int(gs.get("Table_locks_waited"))
    table_locks_immediate = _safe_int(gs.get("Table_locks_immediate"))
    lock_total = table_locks_waited + table_locks_immediate
    table_lock_contention = round(table_locks_waited / lock_total * 100, 2) if lock_total > 0 else 0

    result.locks = {
        "row_lock_waits": row_lock_waits,
        "row_lock_time": row_lock_time,
        "table_locks_waited": table_locks_waited,
        "table_locks_immediate": table_locks_immediate,
        "table_lock_contention": table_lock_contention,
    }

    # --- Table Cache ---
    open_tables = _safe_int(gs.get("Open_tables"))
    opened_tables = _safe_int(gs.get("Opened_tables"))
    table_open_cache = _safe_int(vs.get("table_open_cache"))
    cache_utilization_pct = round(open_tables / table_open_cache * 100, 1) if table_open_cache > 0 else 0
    opens_per_sec = round(opened_tables / uptime_sec, 2) if uptime_sec > 0 else 0

    result.table_cache = {
        "open_tables": open_tables,
        "opened_tables": opened_tables,
        "table_open_cache": table_open_cache,
        "cache_utilization_percent": cache_utilization_pct,
        "opens_per_second": opens_per_sec,
    }

    # --- Top Queries ---
    for row in data.get("top_queries", []):
        result.top_queries.append({
            "digest": row.get("DIGEST", ""),
            "digest_text": row.get("DIGEST_TEXT", ""),
            "count_star": _safe_int(row.get("COUNT_STAR")),
            "total_latency_ms": _safe_float(row.get("TOTAL_LATENCY_MS")),
            "avg_latency_ms": _safe_float(row.get("AVG_LATENCY_MS")),
            "rows_examined": _safe_int(row.get("SUM_ROWS_EXAMINED")),
            "rows_sent": _safe_int(row.get("SUM_ROWS_SENT")),
            "tmp_disk_tables": _safe_int(row.get("SUM_CREATED_TMP_DISK_TABLES")),
            "no_index_used": _safe_int(row.get("SUM_NO_INDEX_USED")),
        })

    if result.top_queries:
        result.top_queries_status = "ok"
    else:
        perf_schema = vs.get("performance_schema", "").upper()
        if perf_schema == "OFF":
            result.top_queries_status = "performance_schema_disabled"
        elif perf_schema == "ON":
            result.top_queries_status = "no_queries_recorded"
        else:
            result.top_queries_status = "unknown"

    # --- Tables ---
    for row in data.get("tables", []):
        result.tables.append({
            "name": row.get("TABLE_NAME", ""),
            "rows": _safe_int(row.get("TABLE_ROWS")),
            "data_length": _safe_int(row.get("DATA_LENGTH")),
            "index_length": _safe_int(row.get("INDEX_LENGTH")),
            "total_size": _safe_int(row.get("TOTAL_SIZE")),
        })

    # --- Active Processes ---
    for row in data.get("processlist", []):
        result.active_processes.append({
            "id": row.get("Id", ""),
            "user": row.get("User", ""),
            "db": row.get("db", ""),
            "command": row.get("Command", ""),
            "time": _safe_int(row.get("Time")),
            "state": row.get("State", ""),
            "info": row.get("Info", ""),
        })


# ---------------------------------------------------------------------------
# Formatting helpers
# ---------------------------------------------------------------------------

def _format_count(n: int) -> str:
    """Format large numbers with K/M/G suffix."""
    if n >= 1_000_000_000:
        return f"{n / 1_000_000_000:.1f}G"
    if n >= 1_000_000:
        return f"{n / 1_000_000:.1f}M"
    if n >= 1_000:
        return f"{n / 1_000:.1f}K"
    return str(n)


def _format_bytes(b: int) -> str:
    """Format bytes to human-readable."""
    if b >= 1_073_741_824:
        return f"{b / 1_073_741_824:.1f} GB"
    if b >= 1_048_576:
        return f"{b / 1_048_576:.1f} MB"
    if b >= 1_024:
        return f"{b / 1_024:.1f} KB"
    return f"{b} B"


def _status_ok_warn_crit(value: float, warn_threshold: float, crit_threshold: float) -> str:
    if value >= crit_threshold:
        return "CRITICAL"
    if value >= warn_threshold:
        return "WARN"
    return "OK"


# ---------------------------------------------------------------------------
# Recommendations
# ---------------------------------------------------------------------------

def generate_recommendations(result: MySQLAnalysisResult) -> List[Dict[str, str]]:
    recs: List[Dict[str, str]] = []

    # Collection failures — surface critical issues when SSH/introspection failed
    if result.collection_status:
        failed = {k: v for k, v in result.collection_status.items()
                  if v.get("status") in ("failed", "error")}
        ssh_sources = {"mysql_query"}
        ssh_failed = {k: v for k, v in failed.items() if k in ssh_sources}
        if ssh_failed:
            sources = ", ".join(ssh_failed.keys())
            errors = "; ".join(v.get("error", "unknown") for v in ssh_failed.values())
            recs.append({
                "severity": "critical",
                "category": "collection",
                "message": f"SSH introspection failed — unable to collect {sources}. "
                           f"Error: {errors}. "
                           f"Analysis is incomplete: InnoDB buffer pool, query throughput, "
                           f"locks, and tuning parameters could not be evaluated.",
            })

    def rec(severity: str, message: str):
        recs.append({"severity": severity, "message": message})

    ov = result.overview or {}
    qt = result.query_throughput or {}
    qe = result.query_efficiency or {}
    bp = result.innodb_buffer_pool or {}
    lk = result.locks or {}
    tc = result.table_cache or {}

    # Connection usage
    conn_pct = ov.get("connection_usage_percent", 0)
    max_conn = ov.get("max_connections", 0)
    if conn_pct >= 90:
        rec("critical", f"Connection usage critical at {conn_pct}%. Approaching max_connections ({max_conn}).")
    elif conn_pct >= 70:
        rec("warning", f"Connection usage at {conn_pct}%. Consider increasing max_connections.")

    # Buffer pool hit ratio
    hit_ratio = bp.get("hit_ratio", 100)
    if hit_ratio < 95:
        rec("critical", f"Buffer pool hit ratio at {hit_ratio}%. Increase innodb_buffer_pool_size.")
    elif hit_ratio < 99:
        rec("warning", f"Buffer pool hit ratio at {hit_ratio}% -- room for improvement with more RAM.")

    # Buffer pool usage
    bp_usage = bp.get("usage_percent", 0)
    if bp_usage > 95:
        rec("warning", f"Buffer pool {bp_usage}% full. Data pages may be evicted under load.")

    # Temp tables to disk
    tmp_pct = qe.get("tmp_disk_table_percent", 0)
    created_disk = qe.get("created_tmp_disk_tables", 0)
    created_total = qe.get("created_tmp_tables", 0)
    if tmp_pct > 25:
        rec("warning", f"{tmp_pct}% of temp tables going to disk. Increase tmp_table_size/max_heap_table_size or optimize queries.")
    elif tmp_pct > 10:
        rec("info", f"Temp tables to disk at {tmp_pct}%. Watch for queries creating large temporary results.")

    # Table scan ratio
    scan_pct = qe.get("table_scan_percent", 0)
    if scan_pct > 75:
        rec("critical", f"Table scan ratio at {scan_pct}%. Most reads are full scans -- add indexes.")
    elif scan_pct > 50:
        rec("warning", f"Table scan ratio at {scan_pct}%. Consider indexing frequently queried columns.")

    # Full joins
    full_joins = qe.get("select_full_join", 0)
    if full_joins > 100:
        rec("warning", f"{_format_count(full_joins)} full joins detected. These scan entire tables -- add indexes to join columns.")

    # Sort merge passes
    sort_passes = qe.get("sort_merge_passes", 0)
    if sort_passes > 0:
        rec("info", f"Sort merge passes ({_format_count(sort_passes)}). Increase sort_buffer_size or optimize queries.")

    # Row lock waits
    row_lock_waits = lk.get("row_lock_waits", 0)
    if row_lock_waits > 1000:
        rec("warning", f"InnoDB row lock waits ({_format_count(row_lock_waits)}). Check for lock contention in concurrent writes.")
    elif row_lock_waits > 0:
        rec("info", f"InnoDB row lock waits ({_format_count(row_lock_waits)}). Check for lock contention in concurrent writes.")

    # Table lock contention
    tl_contention = lk.get("table_lock_contention", 0)
    if tl_contention > 5:
        rec("warning", f"Table lock contention at {tl_contention}%. May indicate MyISAM tables -- convert to InnoDB.")

    # Slow queries
    slow = qt.get("slow_queries", 0)
    threshold = qt.get("long_query_time", "10")
    if slow > 0:
        rec("info", f"{_format_count(slow)} slow queries (threshold: {threshold}s). Review with performance_schema or slow query log.")

    # Aborted clients
    aborted_clients = ov.get("aborted_clients", 0)
    if aborted_clients > 0:
        rec("info", f"{_format_count(aborted_clients)} aborted clients. Applications may not be closing connections properly.")

    # Aborted connects
    aborted_connects = ov.get("aborted_connects", 0)
    if aborted_connects > 0:
        rec("info", f"{_format_count(aborted_connects)} aborted connection attempts. Check authentication issues or connection limits.")

    # No index used in top queries
    if result.top_queries:
        no_index_count = sum(1 for q in result.top_queries if q.get("no_index_used", 0) > 0)
        if no_index_count > 0:
            rec("warning", f"Top queries using no index ({no_index_count} of {len(result.top_queries)}). Missing indexes are likely impacting performance.")

    # Table cache
    tc = result.table_cache or {}
    opens_per_sec = tc.get("opens_per_second", 0)
    cache_util = tc.get("cache_utilization_percent", 0)
    if cache_util >= 95:
        rec("warning", f"Table cache {cache_util}% full ({tc.get('open_tables')}/{tc.get('table_open_cache')}). Increase table_open_cache.")
    if opens_per_sec > 5:
        rec("warning", f"Table opens at {opens_per_sec}/sec — cache may be undersized. Increase table_open_cache.")

    # Top queries diagnostic
    if not result.top_queries:
        if result.top_queries_status == "performance_schema_disabled":
            pass  # Off by default on Railway; overhead (~400MB+) is too high to recommend casually
        elif result.top_queries_status == "no_queries_recorded":
            rec("info", "performance_schema is ON but no queries recorded. Database may be idle or recently restarted.")

    return recs


# ---------------------------------------------------------------------------
# Report formatter
# ---------------------------------------------------------------------------

def format_report(result: MySQLAnalysisResult) -> str:
    lines: List[str] = []

    def heading(title: str, level: int = 2):
        prefix = "#" * level
        lines.append(f"\n{prefix} {title}")

    def table_row(*cells: str):
        lines.append("| " + " | ".join(cells) + " |")

    def table_sep(ncols: int):
        lines.append("| " + " | ".join(["---"] * ncols) + " |")

    # Title
    lines.append(f"# MySQL Analysis: {result.service}")
    lines.append(f"Timestamp: {result.timestamp}")
    lines.append(f"Deployment: {result.deployment_status}")

    # --- Resource Overview (from Railway API) ---
    if result.cpu_memory or result.disk_usage:
        heading("Resource Overview")
        table_row("Metric", "Value")
        table_sep(2)
        if result.cpu_memory:
            cm = result.cpu_memory
            if "cpu_percent" in cm:
                trend = _trend_indicator(result.metrics_history, "cpu")
                lines.append(f"| CPU | {cm['cpu_percent']}% vCPU{trend} |")
            if "memory_gb" in cm:
                trend = _trend_indicator(result.metrics_history, "memory")
                mem_str = f"{cm['memory_gb']} GB"
                if "memory_limit_gb" in cm and cm["memory_limit_gb"] > 0:
                    pct = round(cm["memory_gb"] / cm["memory_limit_gb"] * 100, 1)
                    mem_str += f" / {cm['memory_limit_gb']} GB ({pct}%)"
                lines.append(f"| Memory | {mem_str}{trend} |")
        if result.disk_usage:
            trend = _trend_indicator(result.metrics_history, "disk")
            lines.append(f"| Disk | {result.disk_usage.get('used', 'N/A')}{trend} |")

    # --- Overview ---
    ov = result.overview
    if ov:
        heading("Overview")
        table_row("Metric", "Value", "Status")
        table_sep(3)
        table_row("Version", str(ov["version"]), "")
        table_row("Uptime", ov["uptime_human"], "")
        conn_status = _status_ok_warn_crit(ov["connection_usage_percent"], 70, 90)
        table_row(
            "Connections",
            f"{ov['connection_usage_percent']}% ({ov['max_used_connections']}/{ov['max_connections']})",
            conn_status,
        )
        table_row("Threads Running", str(ov["threads_running"]), "")
        aborted_c_status = "WARN" if ov["aborted_clients"] > 0 else "OK"
        table_row("Aborted Clients", _format_count(ov["aborted_clients"]), aborted_c_status)
        aborted_conn_status = "WARN" if ov["aborted_connects"] > 0 else "OK"
        table_row("Aborted Connects", _format_count(ov["aborted_connects"]), aborted_conn_status)

    # --- Query Throughput ---
    qt = result.query_throughput
    if qt:
        heading("Query Throughput")
        table_row("Metric", "Value", "Status")
        table_sep(3)
        table_row("Total Queries", _format_count(qt["questions"]), "")
        slow_status = "WARN" if qt["slow_queries"] > 0 else "OK"
        table_row("Slow Queries", f"{_format_count(qt['slow_queries'])} (> {qt['long_query_time']}s threshold)", slow_status)
        table_row("SELECT", _format_count(qt["com_select"]), "")
        table_row("INSERT", _format_count(qt["com_insert"]), "")
        table_row("UPDATE", _format_count(qt["com_update"]), "")
        table_row("DELETE", _format_count(qt["com_delete"]), "")

    # --- InnoDB Row Operations ---
    ro = result.innodb_row_ops
    if ro:
        heading("InnoDB Row Operations")
        table_row("Operation", "Count")
        table_sep(2)
        table_row("Rows Read", _format_count(ro["rows_read"]))
        table_row("Rows Inserted", _format_count(ro["rows_inserted"]))
        table_row("Rows Updated", _format_count(ro["rows_updated"]))
        table_row("Rows Deleted", _format_count(ro["rows_deleted"]))

    # --- Query Efficiency ---
    qe = result.query_efficiency
    if qe:
        heading("Query Efficiency")
        table_row("Metric", "Value", "Status")
        table_sep(3)
        tmp_status = _status_ok_warn_crit(qe["tmp_disk_table_percent"], 10, 25)
        table_row(
            "Temp Tables to Disk",
            f"{qe['tmp_disk_table_percent']}% ({_format_count(qe['created_tmp_disk_tables'])}/{_format_count(qe['created_tmp_tables'])})",
            tmp_status,
        )
        scan_status = _status_ok_warn_crit(qe["table_scan_percent"], 50, 75)
        table_row("Table Scan Ratio", f"{qe['table_scan_percent']}%", scan_status)
        fj_status = "WARN" if qe["select_full_join"] > 100 else "OK"
        table_row("Full Joins", _format_count(qe["select_full_join"]), fj_status)
        table_row("Sort Merge Passes", _format_count(qe["sort_merge_passes"]), "")

    # --- InnoDB Buffer Pool ---
    bp = result.innodb_buffer_pool
    if bp:
        heading("InnoDB Buffer Pool")
        table_row("Metric", "Value", "Status")
        table_sep(3)
        hr_status = _status_ok_warn_crit(100 - bp["hit_ratio"], 1, 5)  # inverted: lower hit = worse
        table_row("Cache Hit Ratio", f"{bp['hit_ratio']}%", hr_status)
        usage_status = _status_ok_warn_crit(bp["usage_percent"], 90, 95)
        table_row(
            "Pool Usage",
            f"{bp['usage_percent']}% ({_format_bytes(bp['bytes_data'])}/{_format_bytes(bp['buffer_pool_size'])})",
            usage_status,
        )
        table_row("Dirty Pages", _format_bytes(bp["bytes_dirty"]), "")
        table_row("Free Pages", _format_count(bp["pages_free"]), "")

    # --- Network ---
    nw = result.network
    if nw:
        heading("Network")
        table_row("Metric", "Value")
        table_sep(2)
        table_row("Bytes Received", _format_bytes(nw["bytes_received"]))
        table_row("Bytes Sent", _format_bytes(nw["bytes_sent"]))

    # --- Locks ---
    lk = result.locks
    if lk:
        heading("Locks")
        table_row("Metric", "Value", "Status")
        table_sep(3)
        table_row("Row Lock Waits", _format_count(lk["row_lock_waits"]), "")
        table_row("Row Lock Time (ms)", _format_count(lk["row_lock_time"]), "")
        tl_status = _status_ok_warn_crit(lk["table_lock_contention"], 1, 5)
        table_row("Table Lock Contention", f"{lk['table_lock_contention']}%", tl_status)

    # --- Table Cache ---
    tc = result.table_cache
    if tc:
        heading("Table Cache")
        table_row("Metric", "Value")
        table_sep(2)
        table_row("Open Tables", f"{_format_count(tc['open_tables'])} / {_format_count(tc.get('table_open_cache', 0))}")
        table_row("Cache Utilization", f"{tc.get('cache_utilization_percent', 0)}%")
        table_row("Table Opens/sec", f"{tc.get('opens_per_second', 0)}")

    # --- Top Queries ---
    if result.top_queries:
        heading("Top Queries (by total latency)")
        table_row("Query", "Calls", "Avg Latency", "Total Latency", "Rows Examined", "Rows Sent")
        table_sep(6)
        for q in result.top_queries[:15]:
            digest = q["digest_text"][:60] + "..." if len(q["digest_text"]) > 60 else q["digest_text"]
            # Escape pipe chars in query text
            digest = digest.replace("|", "\\|")
            table_row(
                digest,
                _format_count(q["count_star"]),
                f"{q['avg_latency_ms']:.1f}ms",
                f"{q['total_latency_ms']:.1f}ms",
                _format_count(q["rows_examined"]),
                _format_count(q["rows_sent"]),
            )
    else:
        heading("Top Queries (by total latency)")
        if result.top_queries_status == "performance_schema_disabled":
            lines.append("performance_schema is disabled — no query-level data available.")
            lines.append("Note: enabling it requires ~400MB+ additional memory; only advisable on larger instances.")
        elif result.top_queries_status == "no_queries_recorded":
            lines.append("No queries recorded. Database may be idle or recently restarted.")
        else:
            lines.append("No query data available.")
        lines.append("")

    # --- Tables ---
    if result.tables:
        heading("Tables (by size)")
        table_row("Table", "Rows", "Data", "Indexes", "Total")
        table_sep(5)
        for t in result.tables[:15]:
            table_row(
                t["name"],
                _format_count(t["rows"]),
                _format_bytes(t["data_length"]),
                _format_bytes(t["index_length"]),
                _format_bytes(t["total_size"]),
            )

    # --- Active Queries ---
    if result.active_processes:
        heading("Active Queries")
        # Filter out system processes
        user_procs = [p for p in result.active_processes if p["command"] != "Daemon"]
        if user_procs:
            table_row("User", "Database", "Command", "Time (s)", "Query")
            table_sep(5)
            for p in user_procs[:20]:
                info = (p["info"] or "")[:80]
                info = info.replace("|", "\\|")
                table_row(
                    p["user"],
                    p["db"] or "",
                    p["command"],
                    str(p["time"]),
                    info,
                )
        else:
            lines.append("\nNo active user queries.")

    # --- Infrastructure Metrics ---
    if result.metrics_history:
        windows = result.metrics_history.get("windows", {})
        for window_label, window_data in windows.items():
            mh = window_data.get("metrics", {})
            if not mh:
                continue
            lines.append(f"## Infrastructure Metrics ({window_label})")
            lines.append("| Metric | Current | Min | Max | Avg | Trend |")
            lines.append("|--------|---------|-----|-----|-----|-------|")
            for key in ["cpu", "memory", "disk", "network_rx", "network_tx"]:
                if key in mh:
                    entry = mh[key]
                    trend = entry.get("trend", {})
                    trend_str = trend.get("direction", "N/A")
                    change = trend.get("change_pct", 0)
                    if change != 0:
                        trend_str += f" ({change:+.1f}%)"
                    lines.append(
                        f"| {key.replace('_', ' ').title()} "
                        f"| {entry['current']}{entry['unit']} "
                        f"| {entry['min']}{entry['unit']} "
                        f"| {entry['max']}{entry['unit']} "
                        f"| {entry['avg']}{entry['unit']} "
                        f"| {trend_str} |"
                    )
            lines.append("")

    # --- Collection Errors ---
    if result.errors:
        heading("Collection Errors")
        for err in result.errors:
            lines.append(f"- {err}")

    # --- Recommendations ---
    heading("Recommendations")
    if result.recommendations:
        for r in result.recommendations:
            severity = r["severity"].upper()
            lines.append(f"- **[{severity}]** {r['message']}")
    else:
        lines.append("No issues detected.")

    return "\n".join(lines) + "\n"


# ---------------------------------------------------------------------------
# Main analysis function
# ---------------------------------------------------------------------------

def analyze_mysql(service: str, timeout: int = 60, quiet: bool = False,
                  skip_logs: bool = False, metrics_hours: int = 168,
                  project_id: Optional[str] = None,
                  environment_id: Optional[str] = None,
                  service_id: Optional[str] = None) -> MySQLAnalysisResult:
    """Run complete MySQL analysis."""
    if not quiet:
        print(f"Analyzing mysql database: {service}", file=sys.stderr)

    result = MySQLAnalysisResult(
        service=service,
        db_type="mysql",
        timestamp=datetime.now(timezone.utc).isoformat(),
    )

    # === CONTEXT ===
    if not quiet:
        print("  [0/5] Getting Railway context...", file=sys.stderr, flush=True)
    dal._progress_timer.start()

    if environment_id and service_id:
        dal._ctx = RailwayContext(project_id=project_id, environment_id=environment_id, service_id=service_id)
        if not quiet:
            print(f"        using explicit IDs (env={environment_id[:8]}..., svc={service_id[:8]}...)", file=sys.stderr, flush=True)
    else:
        railway_status = get_railway_status()
        if railway_status:
            dal._ctx = RailwayContext(
                project_id=railway_status.get("projectId"),
                environment_id=railway_status.get("environmentId"),
                service_id=railway_status.get("serviceId"),
            )
        environment_id = dal._ctx.environment_id
        service_id = dal._ctx.service_id

    # === DEPLOYMENT STATUS ===
    progress(1, 5, "Fetching deployment status...", quiet)
    result.deployment_status = get_deployment_status(service, service_id=service_id)

    # === SSH PRE-CHECK ===
    progress(2, 5, "Testing SSH connectivity...", quiet)
    ssh_available = False
    ssh_stderr = ""
    ssh_attempts = [30, 60, 90]
    for attempt, attempt_timeout in enumerate(ssh_attempts, 1):
        ssh_code, ssh_stdout, ssh_stderr = run_ssh_query(service, "echo ok", timeout=attempt_timeout)
        if ssh_code == 0 and "ok" in ssh_stdout:
            ssh_available = True
            if not quiet:
                for line in ssh_stderr.splitlines():
                    if line.startswith("Using SSH key:"):
                        print(f"        {line}", file=sys.stderr, flush=True)
                        break
            break
        if not quiet:
            remaining = len(ssh_attempts) - attempt
            if remaining > 0:
                print(f"        SSH attempt {attempt}/{len(ssh_attempts)} failed ({ssh_stderr or 'no response'}), retrying with {ssh_attempts[attempt]}s timeout...", file=sys.stderr, flush=True)
            else:
                print(f"        SSH attempt {attempt}/{len(ssh_attempts)} failed ({ssh_stderr or 'no response'}), giving up", file=sys.stderr, flush=True)

    # === PARALLEL EXECUTION ===
    progress(3, 5, "Running analysis (metrics, queries, logs in parallel)...", quiet)

    def task_metrics():
        if environment_id and service_id:
            return get_all_metrics_from_api(environment_id, service_id, hours=metrics_hours)
        return None

    def task_mysql_queries():
        if not ssh_available:
            return {"errors": [f"SSH not available: {ssh_stderr or 'connection failed'}"]}
        data = collect_mysql_data(service, timeout=timeout)
        if data.get("errors") and not data.get("global_status"):
            # Retry once
            data = collect_mysql_data(service, timeout=timeout)
        return data

    def task_logs():
        if skip_logs:
            return []
        return get_recent_logs(service, lines=LOG_LINES_DEFAULT,
                               environment_id=environment_id,
                               service_id=service_id)

    with ThreadPoolExecutor(max_workers=3) as executor:
        future_metrics = executor.submit(task_metrics)
        future_mysql = executor.submit(task_mysql_queries)
        future_logs = executor.submit(task_logs)

        metrics_result = future_metrics.result()
        mysql_data = future_mysql.result()
        logs_result = future_logs.result()

    # Process metrics
    if metrics_result:
        result.disk_usage = metrics_result.get("disk_usage")
        result.cpu_memory = metrics_result.get("cpu_memory")
        result.metrics_history = metrics_result.get("metrics_history")
        result.collection_status["metrics_api"] = {"status": "success"}
    else:
        result.collection_status["metrics_api"] = {
            "status": "error",
            "error": "Metrics API returned no data",
        }

    # Process MySQL data
    progress(4, 5, "Processing results...", quiet)
    mysql_errors = mysql_data.get("errors", [])
    if mysql_data.get("global_status"):
        parse_mysql_data(mysql_data, result)
        result.collection_status["mysql_query"] = {"status": "success"}
        if mysql_errors:
            result.collection_status["mysql_query"]["warnings"] = mysql_errors
    else:
        error_msg = "; ".join(mysql_errors) if mysql_errors else "No data returned"
        if not ssh_available:
            error_msg = f"SSH failed after {len(ssh_attempts)} attempts: {ssh_stderr or 'connection failed'}"
        result.errors.append(f"MySQL data collection failed: {error_msg}")
        result.collection_status["mysql_query"] = {
            "status": "error",
            "error": error_msg,
        }

    # Process logs
    if skip_logs:
        result.collection_status["logs_api"] = {"status": "skipped", "reason": "skip_logs flag set"}
    elif logs_result:
        result.recent_logs = logs_result
        result.collection_status["logs_api"] = {"status": "success", "lines": len(logs_result)}
        result.recent_errors = [
            line for line in result.recent_logs
            if "ERROR" in line.upper() or "FATAL" in line.upper()
        ][:100]
    else:
        result.recent_logs = []
        result.collection_status["logs_api"] = {
            "status": "error",
            "error": "Logs API returned no data",
        }

    # === RECOMMENDATIONS ===
    progress(5, 5, "Generating recommendations...", quiet)
    result.recommendations = generate_recommendations(result)

    if not quiet:
        total = dal._progress_timer.total_elapsed()
        print(f"Done.{total}", file=sys.stderr)

    return result


# ---------------------------------------------------------------------------
# Single-step debugging
# ---------------------------------------------------------------------------

def run_single_step(args) -> int:
    """Run a single collection step for debugging."""
    service = args.service
    _init_context(args)
    environment_id = dal._ctx.environment_id
    service_id = dal._ctx.service_id

    if args.step == "ssh-test":
        print(f"Testing SSH to service: {service}", file=sys.stderr)
        code, stdout, stderr = run_ssh_query(service, "echo ok", timeout=45)
        print(f"Exit code: {code}")
        print(f"Stdout: {stdout.strip()}")
        if stderr:
            print(f"Stderr: {stderr.strip()}")
        return 0 if (code == 0 and "ok" in stdout) else 1

    elif args.step == "query":
        print(f"Running MySQL queries on: {service}", file=sys.stderr)
        data = collect_mysql_data(service, timeout=args.timeout)
        print(json.dumps(data, indent=2, default=str))
        return 0 if data.get("global_status") else 1

    elif args.step == "logs":
        print(f"Fetching logs for: {service}", file=sys.stderr)
        logs = get_recent_logs(service, lines=LOG_LINES_DEFAULT,
                               environment_id=environment_id,
                               service_id=service_id)
        print(f"Lines fetched: {len(logs)}")
        for line in logs:
            print(line)
        return 0

    elif args.step == "metrics":
        print(f"Fetching metrics for: {service}", file=sys.stderr)
        if environment_id and service_id:
            metrics = get_all_metrics_from_api(environment_id, service_id)
            if metrics:
                print(json.dumps(metrics, indent=2))
            else:
                print("Metrics API returned no data")
                return 1
        else:
            print("Missing environment_id or service_id from railway config")
            return 1
        return 0

    return 1


# ---------------------------------------------------------------------------
# Main entry point
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="MySQL analysis for Railway services.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )

    parser.add_argument("--service", required=True, help="Service name")
    parser.add_argument("--json", action="store_true",
                        help="Output as JSON")
    parser.add_argument("--timeout", type=int, default=60,
                        help="Timeout in seconds for SSH queries (default: 60)")
    parser.add_argument("--quiet", "-q", action="store_true",
                        help="Suppress progress messages")
    parser.add_argument("--skip-logs", action="store_true",
                        help="Skip log fetching for faster analysis")
    parser.add_argument("--metrics-hours", type=int, default=168,
                        help="Hours of metrics history to fetch (default: 168, max: 168)")
    parser.add_argument("--step", choices=["ssh-test", "query", "logs", "metrics"],
                        help="Run a single collection step for debugging")
    parser.add_argument("--project-id", help="Project ID (bypasses railway link)")
    parser.add_argument("--environment-id", help="Environment ID (bypasses railway link)")
    parser.add_argument("--service-id", help="Service ID (bypasses railway link)")

    args = parser.parse_args()

    # Single-step debugging mode
    if args.step:
        return run_single_step(args)

    # Run analysis
    result = analyze_mysql(
        args.service,
        timeout=args.timeout,
        quiet=args.quiet,
        skip_logs=args.skip_logs,
        metrics_hours=min(args.metrics_hours, 168),
        project_id=args.project_id,
        environment_id=args.environment_id,
        service_id=args.service_id,
    )

    # Output
    if args.json:
        print(json.dumps(asdict(result), indent=2))
    else:
        print(format_report(result))

    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
Complete database analysis for Railway deployments.

Produces a comprehensive report covering:
- Deployment status
- Resource overview (disk, connections)
- Memory configuration
- Cache efficiency (overall and per-table)
- Vacuum health
- Query performance (with --deep)
- Index health
- Recommendations

Usage:
    analyze-postgres.py --service <name>
    analyze-postgres.py --service <name> --deep
    analyze-postgres.py --service <name> --json
"""

import argparse
import base64
import json
import os
import subprocess
import sys
import re
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from typing import Dict, List, Optional, Any, Tuple
from dataclasses import dataclass, field, asdict

import dal
from dal import (
    LOG_LINES_DEFAULT, ProgressTimer, RailwayContext,
    _init_context, progress, run_railway_command, run_ssh_query, run_psql_query,
    get_railway_status, get_deployment_status,
    get_all_metrics_from_api, _analyze_window, _build_metrics_history,
    get_recent_logs,
    _trend_indicator,
)


@dataclass
class AnalysisResult:
    """Container for analysis results."""
    service: str
    db_type: str
    timestamp: str
    deployment_status: str = "UNKNOWN"
    disk_usage: Optional[Dict[str, Any]] = None
    cpu_memory: Optional[Dict[str, Any]] = None
    connections: Optional[Dict[str, Any]] = None
    connection_states: List[Dict[str, Any]] = field(default_factory=list)
    connections_by_app: List[Dict[str, Any]] = field(default_factory=list)
    connections_by_age: List[Dict[str, Any]] = field(default_factory=list)
    oldest_connection_sec: Optional[int] = None
    oldest_connections: List[Dict[str, Any]] = field(default_factory=list)
    memory_config: Optional[Dict[str, Any]] = None
    cache_hit: Optional[Dict[str, Any]] = None
    cache_per_table: List[Dict[str, Any]] = field(default_factory=list)
    table_sizes: List[Dict[str, Any]] = field(default_factory=list)
    database_stats: Optional[Dict[str, Any]] = None
    size_breakdown: Optional[Dict[str, Any]] = None
    vacuum_health: List[Dict[str, Any]] = field(default_factory=list)
    xid_age: Optional[Dict[str, Any]] = None
    pg_stat_statements_installed: bool = False
    top_queries: List[Dict[str, Any]] = field(default_factory=list)
    long_running_queries: List[Dict[str, Any]] = field(default_factory=list)
    idle_in_transaction: List[Dict[str, Any]] = field(default_factory=list)
    blocked_queries: List[Dict[str, Any]] = field(default_factory=list)
    locks: List[Dict[str, Any]] = field(default_factory=list)
    unused_indexes: List[Dict[str, Any]] = field(default_factory=list)
    invalid_indexes: List[Dict[str, Any]] = field(default_factory=list)
    seq_scan_tables: List[Dict[str, Any]] = field(default_factory=list)
    replication: List[Dict[str, Any]] = field(default_factory=list)
    bgwriter: Optional[Dict[str, Any]] = None
    archiver: Optional[Dict[str, Any]] = None
    progress_vacuum: List[Dict[str, Any]] = field(default_factory=list)
    ssl_stats: Optional[Dict[str, Any]] = None
    ha_cluster: Optional[Dict[str, Any]] = None
    cluster_logs: List[Dict[str, Any]] = field(default_factory=list)
    recent_logs: List[str] = field(default_factory=list)  # Raw unfiltered logs for LLM analysis
    recent_errors: List[str] = field(default_factory=list)  # Legacy: filtered error logs
    metrics_history: Optional[Dict[str, Any]] = None  # Multi-window time series + trend analysis for CPU, memory, disk, network
    collection_status: Dict[str, Dict[str, Any]] = field(default_factory=dict)  # Status of each data source
    errors: List[str] = field(default_factory=list)
    recommendations: List[Dict[str, str]] = field(default_factory=list)



def run_psql_query_safe(service: str, query: str, timeout: int = 60) -> Tuple[int, str, str]:
    """Run a psql query using base64 encoding to avoid shell quoting issues."""
    encoded = base64.b64encode(query.encode()).decode()
    # 2>/dev/null suppresses psql warnings (e.g., collation version mismatch) that pollute stdout
    command = f"echo '{encoded}' | base64 -d | psql $DATABASE_URL -P pager=off -t -A 2>/dev/null"
    return run_ssh_query(service, command, timeout)


def build_analysis_query() -> str:
    """Build a single SQL query that returns all analysis data as JSON."""
    return """
SELECT json_build_object(
    'connections', (
        SELECT json_build_object(
            'current', (SELECT count(*) FROM pg_stat_activity WHERE datname = current_database()),
            'max', (SELECT setting::int FROM pg_settings WHERE name = 'max_connections'),
            'reserved', (SELECT setting::int FROM pg_settings WHERE name = 'superuser_reserved_connections'),
            'active', (SELECT count(*) FROM pg_stat_activity WHERE datname = current_database() AND state = 'active'),
            'idle', (SELECT count(*) FROM pg_stat_activity WHERE datname = current_database() AND state = 'idle'),
            'idle_in_transaction', (SELECT count(*) FROM pg_stat_activity WHERE datname = current_database() AND state = 'idle in transaction')
        )
    ),
    'memory_config', (
        SELECT json_agg(json_build_object(
            'name', name,
            'setting', setting,
            'unit', unit
        ))
        FROM pg_settings
        WHERE name IN (
            'shared_buffers', 'effective_cache_size', 'work_mem', 'maintenance_work_mem',
            'wal_buffers', 'checkpoint_completion_target', 'min_wal_size', 'max_wal_size',
            'max_parallel_workers', 'max_parallel_workers_per_gather', 'random_page_cost',
            'default_statistics_target', 'synchronous_commit', 'max_connections',
            'autovacuum', 'autovacuum_vacuum_scale_factor', 'autovacuum_analyze_scale_factor',
            'track_activity_query_size', 'log_min_duration_statement',
            'idle_in_transaction_session_timeout', 'statement_timeout',
            'track_io_timing'
        )
    ),
    'cache_hit', (
        SELECT json_build_object(
            'table_hit_pct', ROUND(100.0 * sum(heap_blks_hit) / NULLIF(sum(heap_blks_hit) + sum(heap_blks_read), 0), 2),
            'index_hit_pct', ROUND(100.0 * sum(idx_blks_hit) / NULLIF(sum(idx_blks_hit) + sum(idx_blks_read), 0), 2)
        )
        FROM pg_statio_user_tables
    ),
    'database_stats', (
        SELECT json_build_object(
            'deadlocks', deadlocks,
            'temp_files', temp_files,
            'temp_bytes', temp_bytes,
            'stats_reset', COALESCE(stats_reset::text, 'never'),
            'blks_read', blks_read,
            'blks_hit', blks_hit,
            'tup_returned', tup_returned,
            'tup_fetched', tup_fetched,
            'tup_inserted', tup_inserted,
            'tup_updated', tup_updated,
            'tup_deleted', tup_deleted,
            'conflicts', conflicts,
            'checksum_failures', COALESCE(checksum_failures, 0)
        )
        FROM pg_stat_database
        WHERE datname = current_database()
    ),
    'cache_per_table', (
        SELECT COALESCE(json_agg(t ORDER BY t.disk_reads DESC), '[]'::json)
        FROM (
            SELECT
                relname as table_name,
                heap_blks_read as disk_reads,
                heap_blks_hit as cache_hits,
                ROUND(100.0 * heap_blks_hit / NULLIF(heap_blks_hit + heap_blks_read, 0), 2) as hit_pct
            FROM pg_statio_user_tables
            WHERE heap_blks_read > 10000
            ORDER BY heap_blks_read DESC LIMIT 1000
        ) t
    ),
    'table_sizes', (
        SELECT COALESCE(json_agg(t ORDER BY t.total_bytes DESC), '[]'::json)
        FROM (
            SELECT
                schemaname as schema,
                relname as table_name,
                pg_size_pretty(pg_total_relation_size(relid)) as size,
                pg_total_relation_size(relid) as total_bytes,
                pg_table_size(relid) as data_bytes,
                pg_indexes_size(relid) as index_bytes,
                n_live_tup as row_count
            FROM pg_stat_user_tables
            ORDER BY pg_total_relation_size(relid) DESC LIMIT 1000
        ) t
    ),
    'size_breakdown', (
        SELECT json_build_object(
            'database_bytes', pg_database_size(current_database()),
            'wal_bytes', COALESCE((SELECT sum(size) FROM pg_ls_waldir()), 0),
            'user_tables_bytes', COALESCE((SELECT sum(pg_table_size(relid)) FROM pg_stat_user_tables), 0),
            'user_indexes_bytes', COALESCE((SELECT sum(pg_indexes_size(relid)) FROM pg_stat_user_tables), 0),
            'system_bytes', COALESCE((
                SELECT sum(pg_total_relation_size(c.oid))
                FROM pg_class c
                JOIN pg_namespace n ON n.oid = c.relnamespace
                WHERE n.nspname IN ('pg_catalog', 'information_schema') AND NOT c.relisshared
            ), 0)
        )
    ),
    'vacuum_health', (
        SELECT COALESCE(json_agg(t ORDER BY t.dead_rows DESC), '[]'::json)
        FROM (
            SELECT
                s.schemaname as schema,
                s.relname as table_name,
                n_live_tup as live_rows,
                n_dead_tup as dead_rows,
                CASE WHEN n_live_tup > 0 THEN ROUND(100.0 * n_dead_tup / n_live_tup, 2) ELSE 0 END as dead_pct,
                vacuum_count,
                autovacuum_count,
                COALESCE(last_vacuum::text, 'never') as last_vacuum,
                COALESCE(last_autovacuum::text, 'never') as last_autovacuum,
                COALESCE(last_analyze::text, 'never') as last_analyze,
                age(c.relfrozenxid) as xid_age,
                CASE WHEN n_dead_tup > 1000 AND (n_live_tup = 0 OR n_dead_tup::float / NULLIF(n_live_tup, 0) > 0.1) THEN true ELSE false END as needs_vacuum,
                CASE WHEN age(c.relfrozenxid) > 150000000 THEN true ELSE false END as needs_freeze
            FROM pg_stat_user_tables s
            JOIN pg_class c ON c.oid = s.relid
            WHERE n_dead_tup > 100
            ORDER BY n_dead_tup DESC LIMIT 1000
        ) t
    ),
    'xid_age', (
        SELECT json_build_object(
            'value', age(datfrozenxid)
        )
        FROM pg_database WHERE datname = current_database()
    ),
    'unused_indexes', (
        SELECT COALESCE(json_agg(t ORDER BY t.size_bytes DESC), '[]'::json)
        FROM (
            SELECT
                s.schemaname as schema,
                s.relname as table_name,
                s.indexrelname as index_name,
                pg_size_pretty(pg_relation_size(s.indexrelid)) as size,
                pg_relation_size(s.indexrelid) as size_bytes,
                s.idx_scan as scans,
                t.seq_scan as table_seq_scans,
                t.idx_scan as table_idx_scans,
                t.n_live_tup as table_rows,
                i.indisprimary as is_primary,
                i.indisunique as is_unique,
                CASE WHEN t.seq_scan > 0 AND s.idx_scan = 0 AND t.n_live_tup > 1000
                    THEN t.seq_scan ELSE 0 END as missing_index_score
            FROM pg_stat_user_indexes s
            JOIN pg_stat_user_tables t ON s.relid = t.relid
            JOIN pg_index i ON s.indexrelid = i.indexrelid
            WHERE s.idx_scan = 0 AND pg_relation_size(s.indexrelid) > 8192
            ORDER BY pg_relation_size(s.indexrelid) DESC LIMIT 1000
        ) t
    ),
    'connection_states', (
        SELECT COALESCE(json_agg(t ORDER BY t.count DESC), '[]'::json)
        FROM (
            SELECT state, count(*) as count
            FROM pg_stat_activity
            WHERE datname = current_database()
            GROUP BY state
            ORDER BY count DESC
        ) t
    ),
    'connections_by_app', (
        SELECT COALESCE(json_agg(t ORDER BY t.count DESC), '[]'::json)
        FROM (
            SELECT COALESCE(application_name, '') as app, COUNT(*) as count
            FROM pg_stat_activity
            WHERE datname = current_database()
            GROUP BY application_name
            ORDER BY count DESC LIMIT 100
        ) t
    ),
    'connections_by_age', (
        SELECT COALESCE(json_agg(t), '[]'::json)
        FROM (
            SELECT
                CASE
                    WHEN age_seconds < 60 THEN '< 1 min'
                    WHEN age_seconds < 300 THEN '1-5 min'
                    WHEN age_seconds < 3600 THEN '5-60 min'
                    WHEN age_seconds < 86400 THEN '1-24 hr'
                    ELSE '> 24 hr'
                END as range,
                count(*) as count
            FROM (
                SELECT EXTRACT(EPOCH FROM (now() - backend_start)) as age_seconds
                FROM pg_stat_activity WHERE datname = current_database()
            ) sub
            GROUP BY 1
            ORDER BY MIN(age_seconds)
        ) t
    ),
    'oldest_connection_sec', (
        SELECT COALESCE(MAX(EXTRACT(EPOCH FROM (now() - backend_start)))::int, 0)
        FROM pg_stat_activity
        WHERE datname = current_database()
    ),
    'oldest_connections', (
        SELECT COALESCE(json_agg(t), '[]'::json)
        FROM (
            SELECT
                COALESCE(application_name, '') as application_name,
                state,
                LEFT(query, 100) as query_preview,
                ROUND(EXTRACT(EPOCH FROM (now() - backend_start)) / 3600)::int as age_hours,
                ROUND(EXTRACT(EPOCH FROM (now() - backend_start)) / 86400, 1) as age_days,
                client_addr::text,
                wait_event_type,
                wait_event
            FROM pg_stat_activity
            WHERE datname = current_database()
              AND EXTRACT(EPOCH FROM (now() - backend_start)) > 86400
            ORDER BY backend_start ASC
            LIMIT 5
        ) t
    ),
    'seq_scan_tables', (
        SELECT COALESCE(json_agg(t ORDER BY t.seq_scans DESC), '[]'::json)
        FROM (
            SELECT
                relname as table_name,
                seq_scan as seq_scans,
                idx_scan as idx_scans,
                n_live_tup as rows
            FROM pg_stat_user_tables
            WHERE seq_scan > 100 AND n_live_tup > 1000
            ORDER BY seq_scan DESC LIMIT 100
        ) t
    ),
    'pg_stat_statements_installed', (
        SELECT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements')
    ),
    'top_queries', (
        SELECT COALESCE(json_agg(t), '[]'::json)
        FROM (
            SELECT
                query,
                calls,
                ROUND(total_exec_time::numeric/1000/60, 1) as total_min,
                ROUND(mean_exec_time::numeric, 1) as mean_ms,
                ROUND(min_exec_time::numeric, 1) as min_ms,
                ROUND(max_exec_time::numeric, 1) as max_ms,
                ROUND(stddev_exec_time::numeric, 1) as stddev_ms,
                rows,
                CASE WHEN calls > 0 THEN ROUND(rows::numeric / calls, 2) ELSE 0 END as rows_per_call,
                total_exec_time,
                ROUND(total_plan_time::numeric, 1) as total_plan_ms,
                ROUND(mean_plan_time::numeric, 1) as mean_plan_ms,
                shared_blks_hit,
                shared_blks_read,
                shared_blks_dirtied,
                shared_blks_written,
                ROUND(100.0 * shared_blks_hit / NULLIF(shared_blks_hit + shared_blks_read, 0), 2) as cache_hit_pct,
                local_blks_hit,
                local_blks_read,
                temp_blks_read,
                temp_blks_written,
                ROUND(blk_read_time::numeric, 1) as blk_read_time_ms,
                ROUND(blk_write_time::numeric, 1) as blk_write_time_ms,
                wal_records,
                wal_bytes
            FROM pg_stat_statements s
            JOIN pg_database d ON s.dbid = d.oid
            WHERE d.datname = current_database()
            ORDER BY total_exec_time DESC LIMIT 100
        ) t
    ),
    'long_running_queries', (
        SELECT COALESCE(json_agg(t ORDER BY t.duration_sec DESC), '[]'::json)
        FROM (
            SELECT
                pid,
                EXTRACT(EPOCH FROM (now() - query_start))::int as duration_sec,
                query
            FROM pg_stat_activity
            WHERE state = 'active'
                AND now() - query_start > interval '5 seconds'
            ORDER BY query_start LIMIT 100
        ) t
    ),
    'idle_in_transaction', (
        SELECT COALESCE(json_agg(t ORDER BY t.idle_sec DESC), '[]'::json)
        FROM (
            SELECT
                pid,
                EXTRACT(EPOCH FROM (now() - state_change))::int as idle_sec,
                COALESCE(usename, '') as username,
                COALESCE(application_name, '') as app,
                query as last_query
            FROM pg_stat_activity
            WHERE state = 'idle in transaction'
                AND now() - state_change > interval '30 seconds'
            ORDER BY state_change LIMIT 100
        ) t
    ),
    'blocked_queries', (
        SELECT COALESCE(json_agg(t ORDER BY t.wait_sec DESC), '[]'::json)
        FROM (
            SELECT
                blocked.pid,
                EXTRACT(EPOCH FROM (now() - blocked.query_start))::int as wait_sec,
                COALESCE(blocked.usename, '') as username,
                COALESCE(blocking.pid::text, '') as blocking_pid,
                left(blocked.query, 60) as query
            FROM pg_stat_activity blocked
            JOIN pg_locks blocked_locks ON blocked.pid = blocked_locks.pid
            JOIN pg_locks blocking_locks ON blocked_locks.locktype = blocking_locks.locktype
                AND blocked_locks.database IS NOT DISTINCT FROM blocking_locks.database
                AND blocked_locks.relation IS NOT DISTINCT FROM blocking_locks.relation
                AND blocked_locks.page IS NOT DISTINCT FROM blocking_locks.page
                AND blocked_locks.tuple IS NOT DISTINCT FROM blocking_locks.tuple
                AND blocked_locks.virtualxid IS NOT DISTINCT FROM blocking_locks.virtualxid
                AND blocked_locks.transactionid IS NOT DISTINCT FROM blocking_locks.transactionid
                AND blocked_locks.classid IS NOT DISTINCT FROM blocking_locks.classid
                AND blocked_locks.objid IS NOT DISTINCT FROM blocking_locks.objid
                AND blocked_locks.objsubid IS NOT DISTINCT FROM blocking_locks.objsubid
                AND blocked_locks.pid != blocking_locks.pid
            JOIN pg_stat_activity blocking ON blocking_locks.pid = blocking.pid
            WHERE NOT blocked_locks.granted
            ORDER BY blocked.query_start LIMIT 100
        ) t
    ),
    'locks', (
        SELECT COALESCE(json_agg(t), '[]'::json)
        FROM (
            SELECT
                l.locktype,
                l.mode,
                COALESCE(a.usename, '') as username,
                COALESCE(a.application_name, '') as app,
                left(COALESCE(a.query, ''), 50) as query
            FROM pg_locks l
            JOIN pg_stat_activity a ON l.pid = a.pid
            WHERE a.datname = current_database() AND NOT l.granted
            LIMIT 100
        ) t
    ),
    'replication', (
        SELECT COALESCE(json_agg(t), '[]'::json)
        FROM (
            SELECT
                COALESCE(client_addr::text, 'local') as client,
                state,
                sent_lsn::text as sent_lsn,
                replay_lsn::text as replay_lsn
            FROM pg_stat_replication
        ) t
    ),
    'bgwriter', (
        SELECT json_build_object(
            'checkpoints_timed', checkpoints_timed,
            'checkpoints_req', checkpoints_req,
            'buffers_checkpoint', buffers_checkpoint,
            'buffers_clean', buffers_clean,
            'buffers_backend', buffers_backend,
            'buffers_backend_fsync', buffers_backend_fsync,
            'maxwritten_clean', maxwritten_clean,
            'stats_reset', COALESCE(stats_reset::text, 'never')
        )
        FROM pg_stat_bgwriter
    ),
    'invalid_indexes', (
        SELECT COALESCE(json_agg(json_build_object(
            'schema', n.nspname,
            'table', c.relname,
            'index', i.relname
        )), '[]'::json)
        FROM pg_index x
        JOIN pg_class c ON c.oid = x.indrelid
        JOIN pg_class i ON i.oid = x.indexrelid
        JOIN pg_namespace n ON n.oid = c.relnamespace
        WHERE NOT x.indisvalid
    ),
    'archiver', (
        SELECT json_build_object(
            'archived_count', archived_count,
            'failed_count', failed_count,
            'last_archived_wal', last_archived_wal,
            'last_archived_time', COALESCE(last_archived_time::text, 'never'),
            'last_failed_wal', last_failed_wal,
            'last_failed_time', COALESCE(last_failed_time::text, 'never'),
            'stats_reset', COALESCE(stats_reset::text, 'never')
        )
        FROM pg_stat_archiver
    ),
    'progress_vacuum', (
        SELECT COALESCE(json_agg(json_build_object(
            'pid', p.pid,
            'datname', d.datname,
            'relname', c.relname,
            'phase', p.phase,
            'heap_blks_total', p.heap_blks_total,
            'heap_blks_scanned', p.heap_blks_scanned,
            'heap_blks_vacuumed', p.heap_blks_vacuumed,
            'index_vacuum_count', p.index_vacuum_count,
            'max_dead_tuples', p.max_dead_tuples,
            'num_dead_tuples', p.num_dead_tuples
        )), '[]'::json)
        FROM pg_stat_progress_vacuum p
        JOIN pg_database d ON p.datid = d.oid
        LEFT JOIN pg_class c ON p.relid = c.oid
    ),
    'ssl_stats', (
        SELECT json_build_object(
            'ssl_connections', (SELECT count(*) FROM pg_stat_ssl WHERE ssl = true),
            'non_ssl_connections', (SELECT count(*) FROM pg_stat_ssl WHERE ssl = false),
            'ssl_versions', (
                SELECT COALESCE(json_agg(json_build_object('version', version, 'count', cnt)), '[]'::json)
                FROM (SELECT version, count(*) as cnt FROM pg_stat_ssl WHERE ssl = true GROUP BY version) v
            )
        )
    )
)::text;
"""


def parse_batched_analysis(data: Dict[str, Any], result: AnalysisResult) -> None:
    """Parse the batched JSON analysis data into the result object."""

    # Connections
    conn = data.get("connections")
    if conn:
        current = conn.get("current", 0)
        max_conn = conn.get("max", 1)
        reserved = conn.get("reserved", 3)
        result.connections = {
            "current": current,
            "max": max_conn,
            "reserved": reserved,
            "available": max_conn - current - reserved,
            "percent": round(current / max_conn * 100, 1) if max_conn > 0 else 0,
            "active": conn.get("active", 0),
            "idle": conn.get("idle", 0),
            "idle_in_transaction": conn.get("idle_in_transaction", 0),
        }

    # Memory config (expanded for tuning analysis)
    mem_config = data.get("memory_config")
    if mem_config:
        result.memory_config = {}
        for row in mem_config:
            name = row["name"]
            setting = row["setting"]
            unit = row["unit"]

            # Handle different value types
            if unit == "8kB":
                # Convert 8kB pages to MB
                mb = int(setting) * 8 / 1024 if str(setting).isdigit() else 0
                result.memory_config[name] = {"value": int(setting) if str(setting).isdigit() else 0, "unit": unit, "mb": round(mb, 1)}
            elif unit == "kB":
                mb = int(setting) / 1024 if str(setting).isdigit() else 0
                result.memory_config[name] = {"value": int(setting) if str(setting).isdigit() else 0, "unit": unit, "mb": round(mb, 1)}
            elif unit == "MB":
                result.memory_config[name] = {"value": int(setting) if str(setting).isdigit() else 0, "unit": unit, "mb": int(setting) if str(setting).isdigit() else 0}
            elif unit in ("ms", "s", "min"):
                # Time-based settings
                result.memory_config[name] = {"value": setting, "unit": unit}
            elif name in ("random_page_cost", "checkpoint_completion_target", "autovacuum_vacuum_scale_factor", "autovacuum_analyze_scale_factor"):
                # Float settings
                result.memory_config[name] = {"value": float(setting) if setting else 0}
            elif name in ("synchronous_commit", "autovacuum", "track_io_timing"):
                # On/off settings
                result.memory_config[name] = {"value": setting}
            else:
                # Integer settings (max_connections, max_parallel_workers, etc.)
                result.memory_config[name] = {"value": int(setting) if str(setting).isdigit() else setting}

    # Cache hit
    cache = data.get("cache_hit")
    if cache:
        result.cache_hit = {
            "table_hit_pct": cache.get("table_hit_pct"),
            "index_hit_pct": cache.get("index_hit_pct"),
        }

    # Database stats
    db_stats = data.get("database_stats")
    if db_stats:
        result.database_stats = {
            "deadlocks": db_stats.get("deadlocks", 0),
            "temp_files": db_stats.get("temp_files", 0),
            "temp_bytes": db_stats.get("temp_bytes", 0),
            "stats_reset": db_stats.get("stats_reset", "unknown"),
            "blks_read": db_stats.get("blks_read", 0),
            "blks_hit": db_stats.get("blks_hit", 0),
            "tup_returned": db_stats.get("tup_returned", 0),
            "tup_fetched": db_stats.get("tup_fetched", 0),
            "tup_inserted": db_stats.get("tup_inserted", 0),
            "tup_updated": db_stats.get("tup_updated", 0),
            "tup_deleted": db_stats.get("tup_deleted", 0),
            "conflicts": db_stats.get("conflicts", 0),
            "checksum_failures": db_stats.get("checksum_failures", 0),
        }

    # Cache per table
    cache_per_table = data.get("cache_per_table", [])
    result.cache_per_table = [
        {
            "table": t.get("table_name"),
            "disk_reads": str(t.get("disk_reads", 0)),
            "cache_hits": str(t.get("cache_hits", 0)),
            "hit_pct": str(t.get("hit_pct", 0)),
        }
        for t in cache_per_table
    ]

    # Table sizes
    table_sizes = data.get("table_sizes", [])
    result.table_sizes = [
        {
            "schema": t.get("schema"),
            "table": t.get("table_name"),
            "size": t.get("size"),
            "total_bytes": str(t.get("total_bytes", 0)),
            "data_bytes": str(t.get("data_bytes", 0)),
            "index_bytes": str(t.get("index_bytes", 0)),
            "row_count": str(t.get("row_count", 0)),
        }
        for t in table_sizes
    ]

    # Size breakdown
    size = data.get("size_breakdown")
    if size:
        result.size_breakdown = {
            "database_bytes": size.get("database_bytes", 0),
            "wal_bytes": size.get("wal_bytes", 0),
            "user_tables_bytes": size.get("user_tables_bytes", 0),
            "user_indexes_bytes": size.get("user_indexes_bytes", 0),
            "system_bytes": size.get("system_bytes", 0),
        }

    # Vacuum health
    vacuum = data.get("vacuum_health", [])
    result.vacuum_health = [
        {
            "schema": t.get("schema"),
            "table": t.get("table_name"),
            "live_rows": str(t.get("live_rows", 0)),
            "dead_rows": str(t.get("dead_rows", 0)),
            "dead_pct": str(t.get("dead_pct", 0)),
            "vacuum_count": str(t.get("vacuum_count", 0)),
            "autovacuum_count": str(t.get("autovacuum_count", 0)),
            "last_vacuum": t.get("last_vacuum", "never"),
            "last_autovacuum": t.get("last_autovacuum", "never"),
            "last_analyze": t.get("last_analyze", "never"),
            "xid_age": str(t.get("xid_age", 0)),
            "needs_vacuum": "true" if t.get("needs_vacuum") else "false",
            "needs_freeze": "true" if t.get("needs_freeze") else "false",
        }
        for t in vacuum
    ]

    # XID age
    xid = data.get("xid_age")
    if xid and xid.get("value") is not None:
        xid_val = xid["value"]
        result.xid_age = {
            "value": xid_val,
            "millions": round(xid_val / 1_000_000, 1),
            "pct_to_wraparound": round(xid_val / 2_147_483_647 * 100, 2)
        }

    # Unused indexes
    unused = data.get("unused_indexes", [])
    result.unused_indexes = [
        {
            "schema": t.get("schema"),
            "table": t.get("table_name"),
            "index": t.get("index_name"),
            "size": t.get("size"),
            "size_bytes": str(t.get("size_bytes", 0)),
            "scans": str(t.get("scans", 0)),
            "table_seq_scans": str(t.get("table_seq_scans", 0)),
            "table_idx_scans": str(t.get("table_idx_scans", 0)),
            "table_rows": str(t.get("table_rows", 0)),
            "is_primary": t.get("is_primary", False),
            "is_unique": t.get("is_unique", False),
            "missing_index_score": str(t.get("missing_index_score", 0)),
        }
        for t in unused
    ]

    # Connection states
    conn_states = data.get("connection_states", [])
    result.connection_states = [
        {"state": t.get("state"), "count": str(t.get("count", 0))}
        for t in conn_states
    ]

    # Connections by app
    conn_app = data.get("connections_by_app", [])
    result.connections_by_app = [
        {"app": t.get("app", ""), "count": str(t.get("count", 0))}
        for t in conn_app
    ]

    # Connections by age
    conn_age = data.get("connections_by_age", [])
    result.connections_by_age = [
        {"range": t.get("range"), "count": str(t.get("count", 0))}
        for t in conn_age
    ]

    # Oldest connection
    oldest = data.get("oldest_connection_sec")
    if oldest is not None:
        result.oldest_connection_sec = oldest

    # Details of old connections (>24 hours)
    oldest_conns = data.get("oldest_connections", [])
    result.oldest_connections = [
        {
            "application_name": c.get("application_name", ""),
            "state": c.get("state"),
            "query_preview": c.get("query_preview"),
            "age_hours": c.get("age_hours"),
            "age_days": c.get("age_days"),
            "client_addr": c.get("client_addr"),
            "wait_event_type": c.get("wait_event_type"),
            "wait_event": c.get("wait_event"),
        }
        for c in oldest_conns
    ]

    # Seq scan tables
    seq_tables = data.get("seq_scan_tables", [])
    result.seq_scan_tables = [
        {
            "table": t.get("table_name"),
            "seq_scans": str(t.get("seq_scans", 0)),
            "idx_scans": str(t.get("idx_scans", 0)),
            "rows": str(t.get("rows", 0)),
        }
        for t in seq_tables
    ]

    # Top queries
    top_q = data.get("top_queries", [])
    result.top_queries = [
        {
            "query": t.get("query"),
            "calls": str(t.get("calls", 0)),
            "total_min": str(t.get("total_min", 0)),
            "mean_ms": str(t.get("mean_ms", 0)),
            "min_ms": str(t.get("min_ms", 0)),
            "max_ms": str(t.get("max_ms", 0)),
            "stddev_ms": str(t.get("stddev_ms", 0)),
            "rows": str(t.get("rows", 0)),
            "rows_per_call": str(t.get("rows_per_call", 0)),
            "total_plan_ms": str(t.get("total_plan_ms", 0)),
            "mean_plan_ms": str(t.get("mean_plan_ms", 0)),
            "shared_blks_hit": t.get("shared_blks_hit", 0),
            "shared_blks_read": t.get("shared_blks_read", 0),
            "shared_blks_dirtied": t.get("shared_blks_dirtied", 0),
            "shared_blks_written": t.get("shared_blks_written", 0),
            "cache_hit_pct": t.get("cache_hit_pct"),
            "local_blks_hit": t.get("local_blks_hit", 0),
            "local_blks_read": t.get("local_blks_read", 0),
            "temp_blks_read": t.get("temp_blks_read", 0),
            "temp_blks_written": t.get("temp_blks_written", 0),
            "blk_read_time_ms": str(t.get("blk_read_time_ms", 0)),
            "blk_write_time_ms": str(t.get("blk_write_time_ms", 0)),
            "wal_records": t.get("wal_records", 0),
            "wal_bytes": t.get("wal_bytes", 0),
        }
        for t in top_q
    ]

    # Long running queries
    long_q = data.get("long_running_queries", [])
    result.long_running_queries = [
        {
            "pid": str(t.get("pid")),
            "duration_sec": str(t.get("duration_sec", 0)),
            "query": t.get("query"),
        }
        for t in long_q
    ]

    # Idle in transaction
    idle_txn = data.get("idle_in_transaction", [])
    result.idle_in_transaction = [
        {
            "pid": str(t.get("pid")),
            "idle_sec": str(t.get("idle_sec", 0)),
            "user": t.get("username", ""),
            "app": t.get("app", ""),
            "last_query": t.get("last_query"),
        }
        for t in idle_txn
    ]

    # Blocked queries
    blocked = data.get("blocked_queries", [])
    result.blocked_queries = [
        {
            "pid": str(t.get("pid")),
            "wait_sec": str(t.get("wait_sec", 0)),
            "user": t.get("username", ""),
            "blocking_pid": t.get("blocking_pid", ""),
            "query": t.get("query"),
        }
        for t in blocked
    ]

    # Locks
    locks = data.get("locks", [])
    result.locks = [
        {
            "locktype": t.get("locktype"),
            "mode": t.get("mode"),
            "user": t.get("username", ""),
            "app": t.get("app", ""),
            "query": t.get("query"),
        }
        for t in locks
    ]

    # Replication
    repl = data.get("replication", [])
    result.replication = [
        {
            "client": t.get("client"),
            "state": t.get("state"),
            "sent_lsn": t.get("sent_lsn"),
            "replay_lsn": t.get("replay_lsn"),
        }
        for t in repl
    ]

    # pg_stat_statements installed flag
    result.pg_stat_statements_installed = data.get("pg_stat_statements_installed", False)

    # Background writer stats
    bgwriter = data.get("bgwriter")
    if bgwriter:
        result.bgwriter = {
            "checkpoints_timed": bgwriter.get("checkpoints_timed", 0),
            "checkpoints_req": bgwriter.get("checkpoints_req", 0),
            "buffers_checkpoint": bgwriter.get("buffers_checkpoint", 0),
            "buffers_clean": bgwriter.get("buffers_clean", 0),
            "buffers_backend": bgwriter.get("buffers_backend", 0),
            "buffers_backend_fsync": bgwriter.get("buffers_backend_fsync", 0),
            "maxwritten_clean": bgwriter.get("maxwritten_clean", 0),
            "stats_reset": bgwriter.get("stats_reset", "never"),
        }

    # Invalid indexes
    invalid_idx = data.get("invalid_indexes", [])
    result.invalid_indexes = [
        {
            "schema": t.get("schema"),
            "table": t.get("table"),
            "index": t.get("index"),
        }
        for t in invalid_idx
    ]

    # WAL archiver stats
    archiver = data.get("archiver")
    if archiver:
        result.archiver = {
            "archived_count": archiver.get("archived_count", 0),
            "failed_count": archiver.get("failed_count", 0),
            "last_archived_wal": archiver.get("last_archived_wal"),
            "last_archived_time": archiver.get("last_archived_time", "never"),
            "last_failed_wal": archiver.get("last_failed_wal"),
            "last_failed_time": archiver.get("last_failed_time", "never"),
            "stats_reset": archiver.get("stats_reset", "never"),
        }

    # Vacuum progress
    progress_vac = data.get("progress_vacuum", [])
    result.progress_vacuum = [
        {
            "pid": t.get("pid"),
            "datname": t.get("datname"),
            "relname": t.get("relname"),
            "phase": t.get("phase"),
            "heap_blks_total": t.get("heap_blks_total", 0),
            "heap_blks_scanned": t.get("heap_blks_scanned", 0),
            "heap_blks_vacuumed": t.get("heap_blks_vacuumed", 0),
            "index_vacuum_count": t.get("index_vacuum_count", 0),
            "max_dead_tuples": t.get("max_dead_tuples", 0),
            "num_dead_tuples": t.get("num_dead_tuples", 0),
        }
        for t in progress_vac
    ]

    # SSL connection stats
    ssl = data.get("ssl_stats")
    if ssl:
        result.ssl_stats = {
            "ssl_connections": ssl.get("ssl_connections", 0),
            "non_ssl_connections": ssl.get("non_ssl_connections", 0),
            "ssl_versions": ssl.get("ssl_versions", []),
        }


def parse_psql_output(output: str, columns: List[str]) -> List[Dict[str, str]]:
    """Parse psql -t -A output (pipe-separated) into list of dicts."""
    rows = []
    for line in output.strip().split("\n"):
        if not line or line.startswith("("):
            continue
        values = line.split("|")
        if len(values) == len(columns):
            rows.append(dict(zip(columns, [v.strip() for v in values])))
    return rows


def get_disk_usage_from_api(environment_id: str, service_id: str) -> Optional[Dict[str, Any]]:
    """Get disk usage from Railway metrics API."""
    from datetime import timedelta

    # Build the API query
    start_date = (datetime.now(timezone.utc) - timedelta(hours=1)).isoformat()

    # Use railway-api.sh script
    import os
    script_dir = os.path.dirname(os.path.abspath(__file__))
    api_script = os.path.join(script_dir, "railway-api.sh")

    if not os.path.exists(api_script):
        return None

    query = '''query metrics($environmentId: String!, $serviceId: String, $startDate: DateTime!, $measurements: [MetricMeasurement!]!) {
        metrics(environmentId: $environmentId, serviceId: $serviceId, startDate: $startDate, measurements: $measurements) {
            measurement values { ts value }
        }
    }'''

    variables = json.dumps({
        "environmentId": environment_id,
        "serviceId": service_id,
        "startDate": start_date,
        "measurements": ["DISK_USAGE_GB"]
    })

    try:
        result = subprocess.run(
            [api_script, query, variables],
            capture_output=True,
            text=True,
            timeout=30
        )
        if result.returncode != 0:
            return None

        data = json.loads(result.stdout)
        metrics = data.get("data", {}).get("metrics", [])

        for metric in metrics:
            if metric.get("measurement") == "DISK_USAGE_GB":
                values = metric.get("values", [])
                if values:
                    # Get latest value
                    latest = values[-1].get("value", 0)
                    return {
                        "used_gb": round(latest, 2),
                        "used": f"{latest:.1f} GB",
                    }
    except (subprocess.TimeoutExpired, json.JSONDecodeError, FileNotFoundError):
        pass

    return None


def get_disk_usage(service: str, environment_id: Optional[str] = None, service_id: Optional[str] = None) -> Optional[Dict[str, Any]]:
    """Get disk usage - try API first, fall back to SSH."""
    # Try Railway API first
    if environment_id and service_id:
        api_result = get_disk_usage_from_api(environment_id, service_id)
        if api_result:
            return api_result

    # Fall back to SSH
    command = "df -h /var/lib/postgresql/data 2>/dev/null || df -h / | tail -1"
    code, stdout, stderr = run_ssh_query(service, command)
    if code != 0 or not stdout:
        return None

    # Parse df output: Filesystem Size Used Avail Use% Mounted
    lines = stdout.strip().split("\n")
    for line in lines:
        if line and not line.startswith("Filesystem"):
            parts = line.split()
            if len(parts) >= 5:
                return {
                    "total": parts[1],
                    "used": parts[2],
                    "available": parts[3],
                    "use_percent": parts[4].rstrip("%"),
                }
    return None


def get_cpu_memory_from_api(environment_id: str, service_id: str) -> Optional[Dict[str, Any]]:
    """Get CPU and memory usage from Railway metrics API.

    DEPRECATED: Use get_all_metrics_from_api() instead for combined disk/cpu/memory.
    """
    result = get_all_metrics_from_api(environment_id, service_id)
    if result:
        return result.get("cpu_memory")
    return None


def get_recent_errors(service: str, limit: int = 10) -> List[str]:
    """Get recent error logs (legacy - kept for backwards compat)."""
    code, stdout, stderr = run_railway_command(
        ["logs", "--service", service, "--lines", "100", "--filter", "@level:error"],
        timeout=30
    )
    if code != 0:
        return []

    errors = []
    for line in stdout.strip().split("\n")[:limit]:
        if line.strip():
            errors.append(line.strip())
    return errors


def get_cluster_logs(
    ha_cluster: Optional[Dict[str, Any]],
    environment_id: Optional[str],
    limit: int = 100
) -> List[Dict[str, Any]]:
    """Get logs from all HA cluster members via Railway API.

    For HA clusters, each member may be a separate deployment.
    This function fetches recent logs from each cluster member.
    """
    if not ha_cluster or not environment_id:
        return []

    members = ha_cluster.get("members", [])
    if not members:
        return []

    import os
    script_dir = os.path.dirname(os.path.abspath(__file__))
    api_script = os.path.join(script_dir, "railway-api.sh")

    if not os.path.exists(api_script):
        return []

    cluster_logs = []

    # Query to get deployments for the environment
    deployment_query = '''query deployments($environmentId: String!) {
        deployments(input: { environmentId: $environmentId }) {
            edges { node { id status staticUrl service { id name } } }
        }
    }'''

    try:
        result = subprocess.run(
            [api_script, deployment_query, json.dumps({"environmentId": environment_id})],
            capture_output=True,
            text=True,
            timeout=30
        )
        if result.returncode != 0:
            return []

        data = json.loads(result.stdout)
        deployments = data.get("data", {}).get("deployments", {}).get("edges", [])

        # Find deployments that match cluster member names
        member_names = {m.get("name", "").lower() for m in members}

        for edge in deployments:
            deployment = edge.get("node", {})
            deployment_id = deployment.get("id")
            service_name = deployment.get("service", {}).get("name", "").lower()
            status = deployment.get("status")

            # Check if this deployment corresponds to a cluster member
            is_member = any(
                member_name in service_name or service_name in member_name
                for member_name in member_names
            )

            if not is_member and status != "SUCCESS":
                continue

            if not deployment_id:
                continue

            # Fetch logs for this deployment
            log_query = '''query deploymentLogs($deploymentId: String!, $limit: Int) {
                deploymentLogs(deploymentId: $deploymentId, limit: $limit) {
                    timestamp message severity
                }
            }'''

            log_result = subprocess.run(
                [api_script, log_query, json.dumps({
                    "deploymentId": deployment_id,
                    "limit": limit
                })],
                capture_output=True,
                text=True,
                timeout=30
            )

            if log_result.returncode == 0:
                log_data = json.loads(log_result.stdout)
                logs = log_data.get("data", {}).get("deploymentLogs", [])
                if logs:
                    cluster_logs.append({
                        "member": service_name,
                        "deployment_id": deployment_id,
                        "status": status,
                        "logs": logs[-limit:],  # Last N logs
                    })

    except (subprocess.TimeoutExpired, json.JSONDecodeError, FileNotFoundError):
        pass

    return cluster_logs


def is_postgres_ha_service(service_id: Optional[str]) -> bool:
    """Check if service is from postgres-ha template.

    Returns True if the service source repo contains 'postgres-ha',
    indicating this is part of an HA cluster that uses Patroni.
    """
    if not service_id:
        return False

    script_dir = os.path.dirname(os.path.abspath(__file__))
    api_script = os.path.join(script_dir, "railway-api.sh")

    if not os.path.exists(api_script):
        return False

    query = '''query service($id: String!) {
        service(id: $id) {
            source { repo }
        }
    }'''

    try:
        result = subprocess.run(
            [api_script, query, json.dumps({"id": service_id})],
            capture_output=True,
            text=True,
            timeout=10
        )
        if result.returncode != 0:
            return False

        data = json.loads(result.stdout)
        repo = data.get("data", {}).get("service", {}).get("source", {}).get("repo", "")
        return "postgres-ha" in repo.lower() if repo else False
    except (subprocess.TimeoutExpired, json.JSONDecodeError, FileNotFoundError):
        return False


def analyze_postgres(service: str, timeout: int = 300, quiet: bool = False,
                     skip_logs: bool = False,
                     metrics_hours: int = 168,
                     project_id: Optional[str] = None,
                     environment_id: Optional[str] = None,
                     service_id: Optional[str] = None) -> AnalysisResult:
    """Run complete Postgres analysis with maximum data collection.

    Uses a single batched SQL query to collect all database metrics,
    minimizing SSH connections (~3 total instead of ~22).

    Args:
        skip_logs: Skip log fetching for faster analysis (~60s saved)
        metrics_hours: Hours of metrics history to fetch (default: 168, max: 168)
        project_id: Project ID (bypasses railway link config)
        environment_id: Environment ID (bypasses railway link config)
        service_id: Service ID (bypasses railway link config)
    """
    if not quiet:
        print(f"Analyzing postgres database: {service}", file=sys.stderr)

    result = AnalysisResult(
        service=service,
        db_type="postgres",
        timestamp=datetime.now(timezone.utc).isoformat(),
    )

    # === FAST CONTEXT LOADING ===
    # Use explicit IDs if provided, otherwise read from config file (instant)
    if not quiet:
        print("  [0/5] Getting Railway context...", file=sys.stderr, flush=True)
    dal._progress_timer.start()

    if environment_id and service_id:
        # IDs passed directly — no need to read config or link
        dal._ctx = RailwayContext(project_id=project_id, environment_id=environment_id, service_id=service_id)
        if not quiet:
            print(f"        using explicit IDs (env={environment_id[:8]}..., svc={service_id[:8]}...)", file=sys.stderr, flush=True)
    else:
        # Fall back to reading railway context from local config (instant, no API call)
        railway_status = get_railway_status()
        if railway_status:
            dal._ctx = RailwayContext(
                project_id=railway_status.get("projectId"),
                environment_id=railway_status.get("environmentId"),
                service_id=railway_status.get("serviceId"),
            )
        environment_id = dal._ctx.environment_id
        service_id = dal._ctx.service_id

    # Check if this is an HA service - only call API if name suggests HA
    is_ha_service = False
    if any(hint in service.lower() for hint in ["postgres-ha", "patroni", "-ha"]):
        is_ha_service = is_postgres_ha_service(service_id)

    # Get deployment status via API (~1s) instead of CLI (~15s)
    progress(1, 5, "Fetching deployment status...", quiet)
    result.deployment_status = get_deployment_status(service, service_id=service_id)

    # === SSH PRE-CHECK WITH RETRY ===
    # SSH can be flaky — retry with increasing timeouts before giving up
    progress(2, 4, "Testing SSH connectivity...", quiet)
    ssh_available = False
    ssh_stderr = ""
    ssh_attempts = [30, 60, 90]
    for attempt, attempt_timeout in enumerate(ssh_attempts, 1):
        ssh_code, ssh_stdout, ssh_stderr = run_ssh_query(service, "echo ok", timeout=attempt_timeout)
        if ssh_code == 0 and "ok" in ssh_stdout:
            ssh_available = True
            if not quiet:
                for line in ssh_stderr.splitlines():
                    if line.startswith("Using SSH key:"):
                        print(f"        {line}", file=sys.stderr, flush=True)
                        break
            break
        if not quiet:
            remaining = len(ssh_attempts) - attempt
            if remaining > 0:
                print(f"        SSH attempt {attempt}/{len(ssh_attempts)} failed ({ssh_stderr or 'no response'}), retrying with {ssh_attempts[attempt]}s timeout...", file=sys.stderr, flush=True)
            else:
                print(f"        SSH attempt {attempt}/{len(ssh_attempts)} failed ({ssh_stderr or 'no response'}), giving up", file=sys.stderr, flush=True)

    # === PARALLEL EXECUTION OF SLOW OPERATIONS ===
    # Run metrics API, database query, and logs in parallel (~17-27s down to ~max of the three)
    progress(3, 4, "Running analysis (metrics, query, logs in parallel)...", quiet)

    analysis_query = build_analysis_query()

    # Define parallel tasks
    def task_metrics():
        """Fetch all metrics (disk, CPU, memory) in one API call."""
        if environment_id and service_id:
            return get_all_metrics_from_api(environment_id, service_id, hours=metrics_hours)
        return None

    def task_database_query():
        """Run the batched database analysis query with retry."""
        if not ssh_available:
            return (1, "", f"SSH not available: {ssh_stderr or 'connection failed'}")
        code, stdout, stderr = run_psql_query_safe(service, analysis_query, timeout=timeout)
        if code != 0:
            # Retry once — SSH sessions can drop mid-query
            if not quiet:
                print(f"        Database query failed ({stderr or 'unknown'}), retrying...", file=sys.stderr, flush=True)
            code, stdout, stderr = run_psql_query_safe(service, analysis_query, timeout=timeout)
        return (code, stdout, stderr)

    def task_logs():
        """Fetch recent logs via API (~3s)."""
        if skip_logs:
            return []
        return get_recent_logs(service, lines=LOG_LINES_DEFAULT,
                               environment_id=environment_id,
                               service_id=service_id)

    def task_ha_cluster():
        """Check HA cluster status (Patroni)."""
        if not is_ha_service:
            return "skipped_not_ha"
        if not ssh_available:
            return "skipped_no_ssh"
        code, stdout, stderr = run_ssh_query(service, "curl -s localhost:8008/cluster 2>/dev/null || echo '{}'")
        if code == 0 and stdout and stdout.strip() != "{}":
            try:
                patroni_data = json.loads(stdout)
                members = patroni_data.get("members", [])
                if members:
                    return {
                        "members": [
                            {
                                "name": m.get("name"),
                                "role": m.get("role"),
                                "state": m.get("state"),
                                "timeline": m.get("timeline"),
                                "lag": m.get("lag"),
                            }
                            for m in members
                        ]
                    }
            except json.JSONDecodeError:
                pass
        return None

    # Run all tasks in parallel
    with ThreadPoolExecutor(max_workers=4) as executor:
        future_metrics = executor.submit(task_metrics)
        future_db = executor.submit(task_database_query)
        future_logs = executor.submit(task_logs)
        future_ha = executor.submit(task_ha_cluster)

        # Collect results
        metrics_result = future_metrics.result()
        db_result = future_db.result()
        logs_result = future_logs.result()
        ha_result = future_ha.result()

    # Process metrics result (combined disk + cpu/memory + 24h history)
    if metrics_result:
        result.disk_usage = metrics_result.get("disk_usage")
        result.cpu_memory = metrics_result.get("cpu_memory")
        result.metrics_history = metrics_result.get("metrics_history")
        result.collection_status["metrics_api"] = {"status": "success"}
    else:
        result.collection_status["metrics_api"] = {
            "status": "error",
            "error": "Metrics API returned no data"
        }

    # Process database query result
    code, stdout, stderr = db_result
    if code == 0 and stdout:
        try:
            data = json.loads(stdout.strip())
            parse_batched_analysis(data, result)
            result.collection_status["database_query"] = {"status": "success"}
        except json.JSONDecodeError as e:
            result.errors.append(f"Failed to parse batched analysis JSON: {e}")
            result.collection_status["database_query"] = {
                "status": "error",
                "error": f"JSON parse error: {e}"
            }
    else:
        error_msg = stderr or stdout or "Unknown error"
        if not ssh_available:
            error_msg = f"SSH failed after {len(ssh_attempts)} attempts: {ssh_stderr or 'connection failed'}"
        result.errors.append(f"Batched analysis query failed: {error_msg}")
        result.collection_status["database_query"] = {
            "status": "error",
            "error": error_msg
        }

    # Process HA cluster result
    if ha_result == "skipped_not_ha":
        result.ha_cluster = None
        result.collection_status["ha_cluster"] = {"status": "skipped", "reason": "not an HA service"}
    elif ha_result == "skipped_no_ssh":
        result.ha_cluster = None
        result.collection_status["ha_cluster"] = {"status": "skipped", "reason": "SSH not available"}
    elif ha_result is not None:
        result.ha_cluster = ha_result
        result.collection_status["ha_cluster"] = {"status": "success"}
    else:
        result.ha_cluster = None
        result.collection_status["ha_cluster"] = {
            "status": "error" if is_ha_service else "skipped",
            "error": "Failed to retrieve Patroni cluster data" if is_ha_service else "not an HA service"
        }

    # Process logs result
    if skip_logs:
        result.collection_status["logs_api"] = {"status": "skipped", "reason": "skip_logs flag set"}
    elif logs_result:
        result.recent_logs = logs_result
        result.collection_status["logs_api"] = {
            "status": "success",
            "lines": len(logs_result)
        }

        # Extract error logs locally
        result.recent_errors = [
            line for line in result.recent_logs
            if 'ERROR' in line.upper() or 'FATAL' in line.upper() or 'PANIC' in line.upper()
        ][:100]

        # HA cluster logs (API call) - done after parallel since it depends on ha_cluster
        if result.ha_cluster and environment_id:
            progress(4, 5, "Fetching HA cluster logs...", quiet)
            result.cluster_logs = get_cluster_logs(result.ha_cluster, environment_id, limit=5000)
    else:
        result.recent_logs = []
        result.collection_status["logs_api"] = {
            "status": "error",
            "error": "Logs API returned no data"
        }

    # Generate recommendations
    progress(5, 5, "Generating recommendations...", quiet)
    result.recommendations = generate_recommendations(result)

    if not quiet:
        total = dal._progress_timer.total_elapsed()
        print(f"Done.{total}", file=sys.stderr)

    return result


def generate_recommendations(result: AnalysisResult) -> List[Dict[str, str]]:
    """Generate recommendations based on analysis results."""
    recommendations = []

    # Collection failures — surface critical issues when SSH/introspection failed
    if result.collection_status:
        failed = {k: v for k, v in result.collection_status.items()
                  if v.get("status") in ("failed", "error")}
        ssh_sources = {"database_query", "ha_cluster"}
        ssh_failed = {k: v for k, v in failed.items() if k in ssh_sources}
        if ssh_failed:
            sources = ", ".join(ssh_failed.keys())
            errors = "; ".join(v.get("error", "unknown") for v in ssh_failed.values())
            recommendations.append({
                "severity": "critical",
                "category": "collection",
                "message": f"SSH introspection failed — unable to collect {sources}. "
                           f"Error: {errors}. "
                           f"Analysis is incomplete: connection stats, query performance, "
                           f"table bloat, and tuning parameters could not be evaluated.",
            })

    # === POSTGRESQL TUNING RECOMMENDATIONS ===
    # Based on best practices from PostgreSQL wiki and community
    if result.memory_config:
        mem = result.memory_config

        # Get system memory from CPU/memory metrics if available
        system_memory_gb = None
        if result.cpu_memory and "memory_limit_gb" in result.cpu_memory:
            # Use actual memory limit from Railway API
            system_memory_gb = result.cpu_memory["memory_limit_gb"]
        elif result.cpu_memory and "memory_gb" in result.cpu_memory:
            # Fallback: estimate total as ~2x current usage
            system_memory_gb = result.cpu_memory["memory_gb"] * 2  # rough estimate

        # shared_buffers check (should be ~25% of RAM, max ~40%)
        shared_buffers = mem.get("shared_buffers", {})
        if shared_buffers and shared_buffers.get("mb"):
            sb_mb = shared_buffers["mb"]
            # Flag if shared_buffers is very low (< 128MB) - likely default
            if sb_mb < 128:
                # Calculate recommended value based on system memory or default to 1GB
                rec_sb = "1GB"
                if system_memory_gb:
                    rec_sb_mb = int(system_memory_gb * 1024 * 0.25)
                    rec_sb = f"{rec_sb_mb}MB" if rec_sb_mb < 1024 else f"{round(rec_sb_mb/1024, 1)}GB"
                recommendations.append({
                    "priority": "immediate",
                    "issue": f"shared_buffers is only {sb_mb}MB (likely default)",
                    "action": f"Increase shared_buffers to {rec_sb} (25% of RAM)",
                    "explanation": "shared_buffers is PostgreSQL's main data cache - pages read from disk are stored here. "
                                   f"At {sb_mb}MB, your entire working set cannot fit in memory, forcing repeated disk reads. "
                                   "The rule of thumb is 25% of total RAM, up to 40% for read-heavy workloads.",
                    "commands": [
                        f"ALTER SYSTEM SET shared_buffers = '{rec_sb}';",
                        "-- Requires database restart to take effect"
                    ],
                    "restart_required": True,
                })
            elif sb_mb < 256:
                rec_sb = "512MB"
                if system_memory_gb:
                    rec_sb_mb = int(system_memory_gb * 1024 * 0.25)
                    rec_sb = f"{rec_sb_mb}MB" if rec_sb_mb < 1024 else f"{round(rec_sb_mb/1024, 1)}GB"
                recommendations.append({
                    "priority": "short-term",
                    "issue": f"shared_buffers is {sb_mb}MB - may be undersized",
                    "action": f"Consider increasing shared_buffers to {rec_sb} (25% of RAM)",
                    "explanation": "shared_buffers holds cached data pages. A larger buffer pool means more data stays in memory, "
                                   "reducing disk I/O. Current size may be limiting cache hit ratio.",
                    "commands": [
                        f"ALTER SYSTEM SET shared_buffers = '{rec_sb}';",
                        "-- Requires database restart to take effect"
                    ],
                    "restart_required": True,
                })

        # effective_cache_size check (should be 50-75% of RAM)
        effective_cache = mem.get("effective_cache_size", {})
        if effective_cache and effective_cache.get("mb"):
            ec_mb = effective_cache["mb"]
            # Flag if effective_cache_size seems low
            if ec_mb < 512:
                rec_ec = "3GB"
                if system_memory_gb:
                    rec_ec_mb = int(system_memory_gb * 1024 * 0.75)
                    rec_ec = f"{rec_ec_mb}MB" if rec_ec_mb < 1024 else f"{round(rec_ec_mb/1024, 1)}GB"
                recommendations.append({
                    "priority": "short-term",
                    "issue": f"effective_cache_size is {ec_mb}MB - may cause poor query plans",
                    "action": f"Set effective_cache_size to {rec_ec} (75% of RAM)",
                    "explanation": "effective_cache_size is a hint to the query planner about how much memory is available for caching "
                                   "(shared_buffers + OS cache). It does NOT allocate memory - it just helps PostgreSQL estimate "
                                   "whether data is likely to be cached. A low value makes the planner pessimistic, avoiding efficient "
                                   "index scans in favor of sequential scans.",
                    "commands": [
                        f"ALTER SYSTEM SET effective_cache_size = '{rec_ec}';",
                        "SELECT pg_reload_conf();  -- Takes effect immediately"
                    ],
                    "restart_required": False,
                })

        # work_mem check (per-operation memory for sorts/hashes)
        work_mem = mem.get("work_mem", {})
        if work_mem and work_mem.get("mb"):
            wm_mb = work_mem["mb"]
            # Calculate recommended work_mem based on connections and RAM
            max_conns = result.connections.get("max", 100) if result.connections else 100
            rec_wm = "32MB"
            if system_memory_gb:
                # Formula: (RAM / max_connections) / 4
                rec_wm_mb = int((system_memory_gb * 1024 / max_conns) / 4)
                rec_wm_mb = max(16, min(rec_wm_mb, 128))  # Clamp between 16-128MB
                rec_wm = f"{rec_wm_mb}MB"

            # Flag if work_mem is at default (4MB) with high temp file usage
            if result.database_stats:
                temp_files = result.database_stats.get("temp_files", 0)
                temp_bytes = result.database_stats.get("temp_bytes", 0)
                temp_gb = round(temp_bytes / 1024 / 1024 / 1024, 1) if temp_bytes > 0 else 0
                if wm_mb <= 4 and temp_files > 1000:
                    recommendations.append({
                        "priority": "immediate",
                        "issue": f"work_mem is only {wm_mb}MB with {temp_files:,} temp files ({temp_gb} GB) spilled to disk",
                        "action": f"Increase work_mem to {rec_wm}",
                        "explanation": f"work_mem controls how much memory each sort, hash, or join operation can use BEFORE "
                                       f"spilling to disk (temp files). At {wm_mb}MB, your queries are constantly spilling. "
                                       f"The {temp_files:,} temp files mean disk I/O instead of fast memory operations. "
                                       f"CAUTION: A query can use multiple work_mem allocations (one per sort node), "
                                       f"so don't set this too high. Formula: (RAM / max_connections) / 4.",
                        "commands": [
                            f"ALTER SYSTEM SET work_mem = '{rec_wm}';",
                            "SELECT pg_reload_conf();  -- Takes effect for new connections"
                        ],
                        "restart_required": False,
                    })
                elif wm_mb <= 4:
                    recommendations.append({
                        "priority": "long-term",
                        "issue": f"work_mem is at default ({wm_mb}MB)",
                        "action": f"Consider increasing work_mem to {rec_wm} for complex queries",
                        "explanation": "work_mem is memory per sort/hash operation. The default 4MB is conservative. "
                                       "Increasing it can speed up complex queries but uses more memory per operation.",
                        "commands": [
                            f"ALTER SYSTEM SET work_mem = '{rec_wm}';",
                            "SELECT pg_reload_conf();  -- Takes effect for new connections"
                        ],
                        "restart_required": False,
                    })

        # maintenance_work_mem check
        maint_mem = mem.get("maintenance_work_mem", {})
        if maint_mem and maint_mem.get("mb"):
            mm_mb = maint_mem["mb"]
            if mm_mb < 64:
                rec_mm = "256MB"
                if system_memory_gb and system_memory_gb >= 8:
                    rec_mm = "512MB"
                recommendations.append({
                    "priority": "short-term",
                    "issue": f"maintenance_work_mem is {mm_mb}MB - VACUUM and CREATE INDEX will be slow",
                    "action": f"Increase maintenance_work_mem to {rec_mm}",
                    "explanation": "maintenance_work_mem is used by VACUUM, CREATE INDEX, and ALTER TABLE operations. "
                                   f"At {mm_mb}MB, these maintenance operations process data in small batches, making them slow. "
                                   "Unlike work_mem, only one maintenance operation runs per session, so this can safely be higher.",
                    "commands": [
                        f"ALTER SYSTEM SET maintenance_work_mem = '{rec_mm}';",
                        "SELECT pg_reload_conf();  -- Takes effect immediately"
                    ],
                    "restart_required": False,
                })

        # random_page_cost check (default 4.0, should be 1.1-2.0 for SSD)
        rpc = mem.get("random_page_cost", {})
        if rpc and rpc.get("value"):
            rpc_val = float(rpc["value"])
            if rpc_val >= 4.0:
                recommendations.append({
                    "priority": "short-term",
                    "issue": f"random_page_cost is {rpc_val} (HDD default) - Railway uses SSDs",
                    "action": "Set random_page_cost to 1.5 for SSD storage",
                    "explanation": "random_page_cost tells the query planner how expensive random disk access is compared to "
                                   "sequential access. The default 4.0 assumes slow HDDs where random reads are 4x more expensive. "
                                   "Railway uses fast SSDs where random reads are almost as fast as sequential. At 4.0, "
                                   "the planner avoids index scans (random access) in favor of slower sequential scans.",
                    "commands": [
                        "ALTER SYSTEM SET random_page_cost = 1.5;",
                        "SELECT pg_reload_conf();  -- Takes effect immediately"
                    ],
                    "restart_required": False,
                })

        # checkpoint_completion_target check (should be 0.9)
        cct = mem.get("checkpoint_completion_target", {})
        if cct and cct.get("value"):
            cct_val = float(cct["value"])
            if cct_val < 0.9:
                recommendations.append({
                    "priority": "long-term",
                    "issue": f"checkpoint_completion_target is {cct_val} - I/O may be spiky",
                    "action": "Set checkpoint_completion_target to 0.9",
                    "explanation": f"PostgreSQL periodically writes dirty buffers to disk (checkpoints). At {cct_val}, "
                                   f"it tries to complete this in {int(cct_val*100)}% of the checkpoint interval, causing I/O spikes. "
                                   "At 0.9, writes spread over 90% of the interval, smoothing disk I/O. "
                                   "WHY: Spiky I/O can cause query latency spikes during checkpoints. "
                                   "SIDE EFFECT: Slightly more consistent (but spread out) disk writes. No downside in practice.",
                    "commands": [
                        "ALTER SYSTEM SET checkpoint_completion_target = 0.9;",
                        "SELECT pg_reload_conf();  -- Takes effect immediately"
                    ],
                    "restart_required": False,
                })

        # max_parallel_workers check
        mpw = mem.get("max_parallel_workers", {})
        mpwpg = mem.get("max_parallel_workers_per_gather", {})
        if mpw and mpw.get("value") == 0:
            recommendations.append({
                "priority": "short-term",
                "issue": "max_parallel_workers is 0 - parallel queries disabled",
                "action": "Set max_parallel_workers to number of CPU cores",
                "explanation": "PostgreSQL can use multiple CPU cores for large sequential scans, aggregates, and joins. "
                               "With max_parallel_workers=0, all queries run single-threaded regardless of table size. "
                               "WHY: Large analytical queries (COUNT, SUM, scans of big tables) could run 2-8x faster with parallelism. "
                               "SIDE EFFECT: Parallel queries use more CPU and memory simultaneously. For OLTP workloads with many "
                               "small queries, this rarely triggers. For analytical queries, it's a significant speedup. "
                               "IF NOT CHANGED: Large table scans will always be slow, even with idle CPU cores.",
                "commands": [
                    "ALTER SYSTEM SET max_parallel_workers = 4;  -- Adjust to your CPU count",
                    "ALTER SYSTEM SET max_parallel_workers_per_gather = 2;",
                    "SELECT pg_reload_conf();"
                ],
                "restart_required": False,
            })
        elif mpwpg and mpwpg.get("value") == 0:
            recommendations.append({
                "priority": "long-term",
                "issue": "max_parallel_workers_per_gather is 0 - parallel queries won't use workers",
                "action": "Set max_parallel_workers_per_gather to 2-4",
                "explanation": "Even though max_parallel_workers allows parallel execution, max_parallel_workers_per_gather=0 "
                               "means each query can use 0 parallel workers (i.e., none). "
                               "WHY: This effectively disables parallelism for all queries. "
                               "SIDE EFFECT: Each parallel query can use up to this many additional workers. "
                               "Setting to 2 means a query could use 3 total processes (1 leader + 2 workers). "
                               "IF NOT CHANGED: You have parallel infrastructure configured but no queries will use it.",
                "commands": [
                    "ALTER SYSTEM SET max_parallel_workers_per_gather = 2;",
                    "SELECT pg_reload_conf();"
                ],
                "restart_required": False,
            })

        # autovacuum check
        autovac = mem.get("autovacuum", {})
        if autovac and autovac.get("value") == "off":
            recommendations.append({
                "priority": "immediate",
                "issue": "autovacuum is DISABLED - database will bloat and eventually fail",
                "action": "Enable autovacuum immediately",
                "explanation": "Autovacuum is PostgreSQL's background process that reclaims space from deleted/updated rows "
                               "and prevents transaction ID wraparound. With autovacuum OFF: "
                               "1) Tables bloat indefinitely - deleted rows waste space and slow queries. "
                               "2) Transaction IDs (XIDs) are never frozen - the database WILL shut down when XIDs wrap (~2 billion transactions). "
                               "3) Table statistics become stale - query planner makes bad decisions. "
                               "WHY IT WAS DISABLED: Sometimes disabled for bulk loads, but must be re-enabled after. "
                               "IF NOT CHANGED: Database will eventually refuse all writes to prevent corruption. This is not recoverable without maintenance.",
                "commands": [
                    "ALTER SYSTEM SET autovacuum = on;",
                    "SELECT pg_reload_conf();"
                ],
                "restart_required": False,
            })

        # synchronous_commit info (not a warning, just info)
        sync = mem.get("synchronous_commit", {})
        if sync and sync.get("value") == "off":
            recommendations.append({
                "priority": "long-term",
                "issue": "synchronous_commit is off - faster writes but risk of data loss on crash",
                "action": "Evaluate if this is acceptable for your data",
                "explanation": "With synchronous_commit=off, PostgreSQL returns 'success' to clients BEFORE data is flushed to disk. "
                               "BENEFIT: Write transactions are 2-10x faster because they don't wait for disk. "
                               "RISK: If the server crashes, the last ~100-800ms of committed transactions may be lost. "
                               "The database will NOT be corrupted - it will be consistent, just missing recent commits. "
                               "ACCEPTABLE FOR: Session data, analytics, caches, logs - anything you can afford to lose. "
                               "NOT ACCEPTABLE FOR: Financial transactions, user data, anything where 'committed' must mean 'durable'. "
                               "IF NOT CHANGED: You keep the performance benefit but accept the crash-loss risk.",
            })

    # pg_stat_statements not available
    if not result.top_queries:
        recommendations.append({
            "priority": "short-term",
            "issue": "pg_stat_statements extension not available - cannot analyze query performance",
            "action": "Enable pg_stat_statements extension",
            "explanation": "pg_stat_statements tracks execution statistics for all SQL queries: call count, total time, "
                           "rows returned, cache hits, temp file usage. Without it, you cannot identify slow queries or optimization targets. "
                           "WHY: This analysis found memory/vacuum issues but cannot pinpoint which QUERIES cause problems. "
                           "SIDE EFFECT: Minor overhead (~1-5%) for tracking. Stores stats in shared memory. "
                           "IF NOT ENABLED: You're flying blind - you can see symptoms (high I/O, temp files) but not the queries causing them. "
                           "To enable, run: python3 scripts/enable-pg-stats.py --service <name> (may require brief restart).",
        })

    # Cache hit ratio
    if result.cache_hit:
        table_hit = result.cache_hit.get("table_hit_pct")
        if table_hit is not None and table_hit < 95:
            priority = "immediate" if table_hit < 90 else "short-term"
            # Find the worst offending tables for context
            worst_tables = []
            for t in result.cache_per_table[:3]:
                if float(t.get("hit_pct") or 100) < 90:
                    worst_tables.append(f"{t['table']} ({t['hit_pct']}%)")
            context = f" Worst tables: {', '.join(worst_tables)}." if worst_tables else ""
            recommendations.append({
                "priority": priority,
                "issue": f"Table cache hit ratio is {table_hit}% (should be >95%)",
                "action": "Increase shared_buffers - data is being read from disk instead of memory cache",
                "explanation": f"Cache hit ratio measures how often PostgreSQL finds requested data in memory (shared_buffers) "
                               f"vs reading from disk. At {table_hit}%, roughly {100-table_hit}% of data requests hit disk.{context}",
            })

    # Per-table cache - check for low hit rates with high disk reads
    for table in result.cache_per_table:
        try:
            hit_pct = float(table.get("hit_pct") or 100)
            disk_reads = int(table.get("disk_reads") or 0)
            table_size = table.get("size", "unknown")
        except (ValueError, TypeError):
            continue

        if hit_pct < 50 and disk_reads > 1_000_000:
            recommendations.append({
                "priority": "immediate",
                "issue": f"Table '{table['table']}' has {hit_pct}% cache hit with {disk_reads:,} disk reads",
                "action": "Increase shared_buffers to fit this table in memory",
                "explanation": f"The '{table['table']}' table ({table_size}) is almost never found in cache. "
                               f"With {disk_reads:,} disk reads, every query touching this table causes disk I/O. "
                               f"This is likely because the table is larger than shared_buffers.",
            })
        elif hit_pct < 80 and disk_reads > 10_000_000:
            recommendations.append({
                "priority": "short-term",
                "issue": f"Table '{table['table']}' has {hit_pct}% cache hit with {disk_reads:,} disk reads",
                "action": "Consider increasing shared_buffers for better caching",
                "explanation": f"The '{table['table']}' table has a low cache hit rate, causing frequent disk reads. "
                               f"Increasing shared_buffers would allow more of this table to stay in memory.",
            })

    # Memory config
    if result.memory_config and result.table_sizes:
        shared_buffers_mb = result.memory_config.get("shared_buffers", {}).get("mb", 0)
        total_table_bytes = sum(int(t.get("bytes", 0)) for t in result.table_sizes)
        total_table_mb = total_table_bytes / 1024 / 1024

        if shared_buffers_mb > 0 and total_table_mb > shared_buffers_mb * 4:
            largest_table = result.table_sizes[0] if result.table_sizes else None
            context = ""
            if largest_table:
                lt_mb = int(largest_table.get("bytes", 0)) / 1024 / 1024
                context = f" Your largest table ({largest_table['table']}) is {round(lt_mb)}MB alone."
            recommendations.append({
                "priority": "immediate",
                "issue": f"shared_buffers ({shared_buffers_mb}MB) is much smaller than working set (~{round(total_table_mb)}MB)",
                "action": f"Increase shared_buffers to at least {round(total_table_mb / 4)}MB",
                "explanation": f"Your database has ~{round(total_table_mb)}MB of table data but only {shared_buffers_mb}MB of buffer cache.{context} "
                               f"PostgreSQL cannot keep frequently-accessed data in memory, causing constant disk I/O.",
            })

    # Vacuum health (using enhanced flags)
    for table in result.vacuum_health:
        dead_pct = float(table.get("dead_pct", 0))
        dead_rows = int(table.get("dead_rows", 0))
        needs_vacuum = table.get("needs_vacuum") == "true"
        needs_freeze = table.get("needs_freeze") == "true"
        last_vacuum = table.get("last_vacuum", "never")
        last_analyze = table.get("last_analyze", "never")

        # Check needs_freeze flag first (more urgent)
        if needs_freeze:
            recommendations.append({
                "priority": "immediate",
                "issue": f"Table '{table['table']}' needs FREEZE (XID age > 150M)",
                "action": f"Run: VACUUM FREEZE \"{table['table']}\";",
                "explanation": "PostgreSQL uses transaction IDs (XIDs) that can wrap around after ~2 billion transactions. "
                               "VACUUM FREEZE marks old rows as 'frozen' so they don't need XID checking. "
                               "If XIDs wrap around without freezing, the database will shut down to prevent data corruption.",
                "commands": [f"VACUUM FREEZE \"{table['table']}\";"],
            })
        elif needs_vacuum:
            recommendations.append({
                "priority": "immediate",
                "issue": f"Table '{table['table']}' needs VACUUM ({dead_pct}% dead rows, {dead_rows:,} rows)",
                "action": f"Run: VACUUM ANALYZE \"{table['table']}\";",
                "explanation": f"This table has {dead_rows:,} dead rows ({dead_pct}% of table) from UPDATE/DELETE operations. "
                               "Dead rows waste disk space and slow down queries by making them scan more pages. "
                               f"Last vacuum: {last_vacuum}. Last analyze: {last_analyze}. "
                               "ANALYZE also updates statistics for better query plans.",
                "commands": [f"VACUUM ANALYZE \"{table['table']}\";"],
            })
        elif dead_pct > 20:
            recommendations.append({
                "priority": "immediate",
                "issue": f"Table '{table['table']}' has {dead_pct}% dead rows ({dead_rows:,} rows)",
                "action": f"Run: VACUUM ANALYZE \"{table['table']}\";",
                "explanation": f"Over 20% of this table is dead rows from UPDATEs and DELETEs. "
                               f"This bloat forces queries to scan many useless rows. Last vacuum: {last_vacuum}.",
                "commands": [f"VACUUM ANALYZE \"{table['table']}\";"],
            })
        elif dead_pct > 10:
            recommendations.append({
                "priority": "short-term",
                "issue": f"Table '{table['table']}' has {dead_pct}% dead rows ({dead_rows:,} rows)",
                "action": f"Run: VACUUM ANALYZE \"{table['table']}\";",
                "explanation": f"This table has accumulated {dead_rows:,} dead rows. While autovacuum should handle this, "
                               f"it may be falling behind. Last vacuum: {last_vacuum}.",
                "commands": [f"VACUUM ANALYZE \"{table['table']}\";"],
            })

    # XID age
    if result.xid_age:
        xid_millions = result.xid_age.get("millions", 0)
        if xid_millions > 150:
            recommendations.append({
                "priority": "immediate",
                "issue": f"XID age is {xid_millions}M (wraparound risk at 2147M)",
                "action": "Run VACUUM FREEZE on all high-XID tables",
                "explanation": "PostgreSQL's transaction ID counter wraps around at ~2.1 billion. At 150M+, you're using ~7% of "
                               "the available space. If this reaches 2 billion without VACUUM FREEZE, PostgreSQL will "
                               "shut down to prevent data corruption. This is a critical issue requiring immediate action.",
                "commands": ["VACUUM FREEZE;  -- Run on affected tables"],
            })
        elif xid_millions > 100:
            recommendations.append({
                "priority": "short-term",
                "issue": f"XID age is {xid_millions}M (approaching wraparound risk)",
                "action": "Monitor autovacuum and consider manual VACUUM FREEZE",
                "explanation": "XID age is elevated. Autovacuum should handle this, but verify it's running. "
                               "If tables are being vacuumed but XID age stays high, long-running transactions may be blocking freezing.",
            })

    # Database stats (deadlocks, temp files)
    if result.database_stats:
        deadlocks = result.database_stats.get("deadlocks", 0)
        if deadlocks > 0:
            recommendations.append({
                "priority": "short-term",
                "issue": f"{deadlocks} deadlock(s) detected since last stats reset",
                "action": "Review application transaction logic and lock ordering",
                "explanation": f"A deadlock occurs when two transactions each hold a lock the other needs, creating a cycle. "
                               f"PostgreSQL detects this and kills one transaction (the 'victim') so the other can proceed. "
                               f"WHY THIS MATTERS: {deadlocks} deadlocks means {deadlocks} transactions were aborted and had to retry. "
                               f"COMMON CAUSES: 1) Transactions locking rows in different orders. 2) Long transactions holding locks. "
                               f"3) Hot rows updated by many concurrent transactions. "
                               f"FIX: Ensure all code paths lock tables/rows in the same order. Keep transactions short. "
                               f"IF NOT FIXED: Deadlocks will continue, causing random transaction failures and retries.",
            })

        # Temp files - flag with description based on daily rate
        temp_files = result.database_stats.get("temp_files", 0)
        temp_bytes = result.database_stats.get("temp_bytes", 0)
        temp_gb = round(temp_bytes / 1024 / 1024 / 1024, 1) if temp_bytes > 0 else 0
        stats_reset = result.database_stats.get("stats_reset", "unknown")
        # Calculate days since reset for rate-based thresholds
        days_since_reset = None
        if stats_reset and stats_reset not in ("unknown", "never"):
            try:
                reset_date = datetime.fromisoformat(stats_reset.replace('Z', '+00:00'))
                days_since_reset = (datetime.now(timezone.utc) - reset_date).days
                days_since_reset = max(days_since_reset, 1)  # Avoid division by zero
            except (ValueError, TypeError):
                pass
        # Use rate-based threshold if we have time period data
        if days_since_reset and days_since_reset > 0:
            gb_per_day = temp_gb / days_since_reset
            files_per_day = round(temp_files / days_since_reset)
            if gb_per_day > 5:  # More than 5GB/day is concerning
                # Get current work_mem for context
                wm_mb = result.memory_config.get("work_mem", {}).get("mb", 4) if result.memory_config else 4
                recommendations.append({
                    "priority": "short-term",
                    "issue": f"High temp file usage: ~{files_per_day:,} files/day ({round(gb_per_day, 1)} GB/day)",
                    "action": "Increase work_mem from {wm_mb}MB to 32-64MB",
                    "explanation": f"When a query needs to sort or hash more data than work_mem allows ({wm_mb}MB), "
                                   f"PostgreSQL spills to temp files on disk. Your queries are creating ~{files_per_day:,} temp files daily, "
                                   f"writing {round(gb_per_day, 1)}GB to disk. This is slower than in-memory operations.",
                    "commands": [
                        "ALTER SYSTEM SET work_mem = '32MB';",
                        "SELECT pg_reload_conf();  -- Takes effect for new connections"
                    ],
                    "restart_required": False,
                })
        elif temp_files > 10000 or temp_gb > 10:  # Fallback if no date
            wm_mb = result.memory_config.get("work_mem", {}).get("mb", 4) if result.memory_config else 4
            recommendations.append({
                "priority": "short-term",
                "issue": f"High temp file usage: {temp_files:,} files, {temp_gb} GB written since stats reset",
                "action": f"Increase work_mem from {wm_mb}MB to 32-64MB",
                "explanation": f"Queries are spilling to disk because work_mem ({wm_mb}MB) is too small for sort/hash operations. "
                               f"Each temp file represents a query that couldn't fit its working data in memory.",
                "commands": [
                    "ALTER SYSTEM SET work_mem = '32MB';",
                    "SELECT pg_reload_conf();  -- Takes effect for new connections"
                ],
                "restart_required": False,
            })

    # Connection usage
    if result.connections:
        pct = result.connections.get("percent", 0)
        current = result.connections.get("current", 0)
        max_conn = result.connections.get("max", 100)
        available = result.connections.get("available", max_conn - current)
        if pct > 90:
            recommendations.append({
                "priority": "immediate",
                "issue": f"Connection usage is {pct}% ({current}/{max_conn}, only {available} available)",
                "action": "Use connection pooling (PgBouncer) or increase max_connections",
                "explanation": f"You're using {current} of {max_conn} connections. Each PostgreSQL connection uses memory "
                               f"(~10MB each). Rather than increasing max_connections, use connection pooling (PgBouncer) "
                               f"to multiplex many app connections over fewer database connections.",
            })
        elif pct > 70:
            recommendations.append({
                "priority": "short-term",
                "issue": f"Connection usage is {pct}% ({current}/{max_conn})",
                "action": "Consider connection pooling for scalability",
                "explanation": "Connection usage is elevated. Connection pooling (PgBouncer) helps applications share "
                               "database connections efficiently, especially during traffic spikes.",
            })

    # Old connections
    if result.oldest_connection_sec is not None:
        age_hours = result.oldest_connection_sec / 3600
        age_days = round(age_hours / 24, 1)
        if age_hours > 48:
            # Include details about what the old connections are
            conn_details = ""
            if result.oldest_connections:
                details_list = []
                for c in result.oldest_connections[:3]:
                    app = c.get("application_name") or "(unnamed)"
                    state = c.get("state", "unknown")
                    days = c.get("age_days", "?")
                    details_list.append(f"{app} ({state}, {days} days)")
                conn_details = f" Old connections: {'; '.join(details_list)}."

            recommendations.append({
                "priority": "short-term",
                "issue": f"Oldest connection is ~{age_days} days old ({round(age_hours)} hours)",
                "action": "Review connection pooling settings and application connection management",
                "explanation": f"Long-lived connections can indicate connection pool misconfiguration or connection leaks. "
                               f"They can also hold locks or prevent autovacuum from cleaning up. "
                               f"If using connection pooling, ensure idle connections are recycled.{conn_details}",
            })

    # Disk usage
    if result.disk_usage:
        use_pct = int(result.disk_usage.get("use_percent", 0))
        used = result.disk_usage.get("used", "unknown")
        total = result.disk_usage.get("total", "unknown")
        if use_pct > 85:
            recommendations.append({
                "priority": "immediate",
                "issue": f"Disk usage is {use_pct}% ({used} / {total})",
                "action": "Increase volume size or clean up data",
                "explanation": "PostgreSQL needs free disk space for WAL files, temp files, and VACUUM operations. "
                               "Running out of disk space can cause database crashes. Consider: "
                               "1) Increasing volume size, 2) Dropping unused indexes, 3) VACUUM FULL on bloated tables, "
                               "4) Archiving old data.",
            })
        elif use_pct > 70:
            recommendations.append({
                "priority": "short-term",
                "issue": f"Disk usage is {use_pct}% ({used} / {total})",
                "action": "Plan for volume expansion",
                "explanation": f"Disk is at {use_pct}%, approaching the danger zone. PostgreSQL needs free space for: "
                               f"1) WAL files - write-ahead logs that ensure durability. "
                               f"2) Temp files - sorts and hashes spill here when work_mem is exceeded. "
                               f"3) VACUUM operations - need space to rewrite tables during VACUUM FULL. "
                               f"IF NOT ADDRESSED: At 85%+ you risk write failures. At 100%, database crashes and may not restart. "
                               f"ACTIONS: Increase volume size in Railway, or identify large unused tables/indexes to drop.",
            })

    # Unused indexes - only flag non-PK, non-unique indexes >100MB
    droppable_indexes = [
        idx for idx in result.unused_indexes
        if not idx.get("is_primary") and not idx.get("is_unique")
        and int(idx.get("size_bytes", 0)) > 100 * 1024 * 1024
    ]
    if droppable_indexes:
        total_size = sum_index_sizes(droppable_indexes)
        index_names = [idx['index'] for idx in droppable_indexes[:3]]
        recommendations.append({
            "priority": "long-term",
            "issue": f"{len(droppable_indexes)} unused non-constraint indexes >100MB ({total_size})",
            "action": "Review and drop unused indexes to save space and improve write performance",
            "explanation": f"These indexes have 0 scans since stats reset, meaning no queries are using them. "
                           f"Each index costs disk space AND slows down writes (INSERT/UPDATE/DELETE must update all indexes). "
                           f"Examples: {', '.join(index_names)}{'...' if len(droppable_indexes) > 3 else ''}",
            "commands": [f"DROP INDEX IF EXISTS \"{idx['index']}\";  -- saves {idx['size']}" for idx in droppable_indexes[:3]],
        })

    # Tables with high missing index score (lots of seq scans, no index usage)
    for idx in result.unused_indexes:
        try:
            missing_score = int(idx.get("missing_index_score", 0))
            if missing_score > 1000:
                table_rows = idx.get("table_rows", "unknown")
                recommendations.append({
                    "priority": "short-term",
                    "issue": f"Table '{idx['table']}' has {missing_score:,} sequential scans with no index usage",
                    "action": f"Consider adding an index on commonly filtered columns of '{idx['table']}'",
                    "explanation": f"Sequential scans read the entire table ({table_rows} rows) for each query. "
                                   f"With {missing_score:,} sequential scans, queries are repeatedly scanning all rows. "
                                   f"An index on commonly filtered columns (WHERE clauses) would dramatically speed this up.",
                })
        except (ValueError, TypeError):
            pass

    # Long-running queries
    if result.long_running_queries:
        for q in result.long_running_queries[:3]:
            try:
                duration = int(q.get("duration_sec", 0))
                query_preview = q.get("query", "")[:80]
                if duration > 60:
                    recommendations.append({
                        "priority": "immediate",
                        "issue": f"Query running for {duration}s (PID {q.get('pid')})",
                        "action": "Investigate and potentially cancel",
                        "explanation": f"This query has been running for {duration} seconds. "
                                       f"QUERY: {query_preview}... "
                                       f"WHY THIS MATTERS: Long queries hold locks, consume memory, and may indicate missing indexes or inefficient queries. "
                                       f"TO CANCEL (graceful): SELECT pg_cancel_backend({q.get('pid')}); "
                                       f"TO TERMINATE (force): SELECT pg_terminate_backend({q.get('pid')}); "
                                       f"SIDE EFFECT OF CANCEL: The query's transaction will be rolled back. The application will receive an error.",
                        "commands": [f"SELECT pg_cancel_backend({q.get('pid')});  -- Graceful cancel"],
                    })
            except (ValueError, TypeError):
                pass

    # Idle in transaction (stuck transactions)
    if result.idle_in_transaction:
        for txn in result.idle_in_transaction[:3]:
            try:
                idle_sec = int(txn.get("idle_sec", 0))
                app_name = txn.get("application_name", "unknown app")
                if idle_sec > 300:  # 5 minutes
                    recommendations.append({
                        "priority": "immediate",
                        "issue": f"Transaction idle for {idle_sec}s (PID {txn.get('pid')}, user: {txn.get('user', 'unknown')}, app: {app_name})",
                        "action": "Terminate the stuck transaction",
                        "explanation": f"This connection started a transaction (BEGIN) but hasn't done anything for {idle_sec}s. "
                                       f"WHY THIS IS BAD: 1) Holds row-level locks that block other queries. "
                                       f"2) Prevents VACUUM from cleaning dead rows in any table it touched. "
                                       f"3) Holds a transaction ID slot, contributing to XID bloat. "
                                       f"COMMON CAUSES: Application bug, network timeout without cleanup, abandoned connection. "
                                       f"TO FIX: SELECT pg_terminate_backend({txn.get('pid')}); (terminates connection). "
                                       f"PREVENTION: Set idle_in_transaction_session_timeout to auto-kill stuck transactions.",
                        "commands": [
                            f"SELECT pg_terminate_backend({txn.get('pid')});  -- Kill this connection",
                            "ALTER SYSTEM SET idle_in_transaction_session_timeout = '5min';  -- Auto-kill in future",
                        ],
                    })
                elif idle_sec > 60:
                    recommendations.append({
                        "priority": "short-term",
                        "issue": f"Transaction idle for {idle_sec}s (PID {txn.get('pid')}, app: {app_name})",
                        "action": "Review application transaction handling",
                        "explanation": f"This transaction has been idle for {idle_sec}s. While not critical yet, "
                                       f"transactions should be short-lived. Long idle transactions hold locks and block VACUUM. "
                                       f"COMMON CAUSES: Missing COMMIT/ROLLBACK, waiting for user input inside transaction, connection pool issues. "
                                       f"PREVENTION: Use idle_in_transaction_session_timeout to auto-terminate stuck transactions.",
                    })
            except (ValueError, TypeError):
                pass

    # Blocked queries
    if result.blocked_queries:
        for q in result.blocked_queries[:3]:
            try:
                wait_sec = int(q.get("wait_sec", 0))
                if wait_sec > 30:
                    recommendations.append({
                        "priority": "immediate",
                        "issue": f"Query waiting {wait_sec}s for lock (PID {q.get('pid')} blocked by {q.get('blocking_pid')})",
                        "action": "Investigate the blocking query and terminate if appropriate",
                        "explanation": f"PID {q.get('pid')} has been waiting {wait_sec}s for a lock held by PID {q.get('blocking_pid')}. "
                                       f"WHY: The blocking query/transaction is holding a lock (row, table, or advisory) that this query needs. "
                                       f"COMMON CAUSES: Long-running transaction, idle-in-transaction, DDL operations (ALTER TABLE). "
                                       f"TO INVESTIGATE: SELECT query FROM pg_stat_activity WHERE pid = {q.get('blocking_pid')}; "
                                       f"TO UNBLOCK: Cancel or terminate the blocking PID if it's stuck. "
                                       f"SIDE EFFECT: Terminating the blocker will rollback its transaction, but unblock waiting queries.",
                        "commands": [
                            f"-- See what {q.get('blocking_pid')} is doing:",
                            f"SELECT pid, state, query FROM pg_stat_activity WHERE pid = {q.get('blocking_pid')};",
                            f"-- To terminate (if stuck): SELECT pg_terminate_backend({q.get('blocking_pid')});",
                        ],
                    })
            except (ValueError, TypeError):
                pass

    # Lock contention
    if result.locks:
        lock_types = set(lock.get("locktype", "unknown") for lock in result.locks)
        recommendations.append({
            "priority": "immediate",
            "issue": f"{len(result.locks)} blocked lock(s) detected ({', '.join(lock_types)})",
            "action": "Investigate lock contention - may indicate long transactions or deadlocks",
            "explanation": "Queries are waiting for locks held by other transactions. Common causes: "
                           "1) Long-running transactions holding locks, 2) Deadlocks (PostgreSQL will resolve these automatically), "
                           "3) DDL operations (ALTER TABLE) blocking normal queries. "
                           "Check blocked_queries and idle_in_transaction sections for details.",
        })

    # Sequential scans on large tables
    for table in result.seq_scan_tables:
        try:
            seq_scans = int(table.get("seq_scans", 0))
            idx_scans = int(table.get("idx_scans", 0))
            rows = int(table.get("rows", 0))
            if seq_scans > 1000 and idx_scans == 0 and rows > 10000:
                recommendations.append({
                    "priority": "short-term",
                    "issue": f"Table '{table['table']}' has {seq_scans:,} sequential scans with 0 index scans ({rows:,} rows)",
                    "action": "Add indexes on columns used in WHERE, JOIN, and ORDER BY clauses",
                    "explanation": f"Every query on '{table['table']}' scans all {rows:,} rows instead of using an index. "
                                   f"With {seq_scans:,} sequential scans, this table is a performance hotspot. "
                                   f"To find which columns to index, run: EXPLAIN ANALYZE on slow queries touching this table, "
                                   f"or check pg_stat_statements for common query patterns.",
                })
        except (ValueError, TypeError):
            pass

    # HA cluster issues
    if result.ha_cluster:
        members = result.ha_cluster.get("members", [])
        for m in members:
            state = m.get("state", "")
            if state == "start failed":
                recommendations.append({
                    "priority": "immediate",
                    "issue": f"HA replica '{m.get('name')}' is in 'start failed' state",
                    "action": "Resync the replica",
                    "explanation": f"The replica '{m.get('name')}' failed to start, typically due to timeline divergence. "
                                   f"This happens when the replica's WAL history diverges from the primary (e.g., after failover). "
                                   f"WHY THIS MATTERS: This replica cannot be used for failover or read scaling until fixed. "
                                   f"FIX: The replica needs a fresh base backup (pg_basebackup) from the primary. "
                                   f"IF NOT FIXED: You're running without redundancy - if the primary fails, no automatic failover is possible.",
                })
            elif state not in ("running", "streaming"):
                recommendations.append({
                    "priority": "short-term",
                    "issue": f"HA replica '{m.get('name')}' is in '{state}' state",
                    "action": "Investigate replica health",
                    "explanation": f"Expected state is 'running' or 'streaming', but replica is '{state}'. "
                                   f"POSSIBLE STATES: 'creating' (initializing), 'stopped' (manually stopped), 'start failed' (broken). "
                                   f"WHY THIS MATTERS: Non-streaming replicas may have stale data and can't be used for failover. "
                                   f"CHECK: Replica logs for specific errors. Network connectivity to primary. WAL lag.",
                })

    # Recent errors
    if result.recent_errors and len(result.recent_errors) > 5:
        # Summarize error types (recent_errors is a list of strings)
        error_samples = [e[:60] if isinstance(e, str) else str(e)[:60] for e in result.recent_errors[:3]]
        recommendations.append({
            "priority": "short-term",
            "issue": f"{len(result.recent_errors)} recent errors in logs",
            "action": "Review error logs for patterns",
            "explanation": f"Multiple errors detected in recent logs. Sample messages: {'; '.join(error_samples)}... "
                           f"WHY THIS MATTERS: Frequent errors may indicate application bugs, configuration issues, or resource constraints. "
                           f"CHECK: Look for patterns - are errors from one app? One query? Specific time periods? "
                           f"COMMON TYPES: Connection errors (app/network issue), query errors (syntax/permissions), "
                           f"out-of-memory errors (need more RAM or lower work_mem).",
        })

    # Invalid indexes
    if result.invalid_indexes:
        for idx in result.invalid_indexes:
            recommendations.append({
                "priority": "immediate",
                "issue": f"Invalid index '{idx.get('index')}' on {idx.get('schema')}.{idx.get('table')}",
                "action": "Drop and recreate the index",
                "explanation": f"This index is marked as invalid - PostgreSQL will NOT use it for queries. "
                               f"CAUSE: Usually a CREATE INDEX CONCURRENTLY that failed partway through (e.g., due to constraint violation, "
                               f"out of disk space, or duplicate key). "
                               f"WHY THIS MATTERS: The index takes up disk space and slows writes, but provides zero query benefit. "
                               f"FIX: Drop it and recreate. Use CONCURRENTLY to avoid locking the table.",
                "commands": [
                    f"DROP INDEX CONCURRENTLY IF EXISTS \"{idx.get('index')}\";",
                    f"-- Then recreate with: CREATE INDEX CONCURRENTLY ...",
                ],
            })

    # WAL archiver failures
    if result.archiver and result.archiver.get("failed_count", 0) > 0:
        last_failed_wal = result.archiver.get("last_failed_wal", "unknown")
        last_failed_time = result.archiver.get("last_failed_time", "unknown")
        recommendations.append({
            "priority": "immediate",
            "issue": f"WAL archiver has {result.archiver['failed_count']} failed archival(s)",
            "action": "Check archive_command configuration and destination storage",
            "explanation": f"WAL (Write-Ahead Log) archiving is failing. Last failed WAL: {last_failed_wal} at {last_failed_time}. "
                           f"This affects point-in-time recovery capability. Common causes: "
                           f"1) Archive destination full or unreachable, 2) Permissions issues, 3) archive_command misconfiguration.",
        })

    # Background writer issues
    if result.bgwriter:
        bg = result.bgwriter
        # High backend fsync indicates shared_buffers pressure
        if bg.get("buffers_backend_fsync", 0) > 0:
            recommendations.append({
                "priority": "short-term",
                "issue": f"Backend processes forced {bg['buffers_backend_fsync']:,} fsync operations",
                "action": "Increase shared_buffers to reduce dirty buffer pressure",
                "explanation": "Normally, the background writer or checkpointer flushes dirty buffers to disk. "
                               f"When shared_buffers is too small, backends must flush dirty buffers themselves "
                               f"(buffers_backend_fsync > 0). This forces query processes to do I/O, causing latency spikes.",
            })

        # Check if most checkpoints are requested (not timed)
        timed = bg.get("checkpoints_timed", 0)
        req = bg.get("checkpoints_req", 0)
        total = timed + req
        if total > 10 and req > timed:
            req_pct = round(100.0 * req / total, 1)
            recommendations.append({
                "priority": "short-term",
                "issue": f"{req_pct}% of checkpoints are requested (not timed) - WAL is filling up",
                "action": "Increase max_wal_size to 2-4GB",
                "explanation": f"Checkpoints should happen on a timer (checkpoint_timeout), not because WAL fills up. "
                               f"With {req_pct}% requested checkpoints, WAL segments are filling faster than expected. "
                               f"This causes I/O spikes. Increasing max_wal_size gives more headroom before forced checkpoints.",
                "commands": [
                    "ALTER SYSTEM SET max_wal_size = '2GB';",
                    "SELECT pg_reload_conf();  -- Takes effect immediately"
                ],
                "restart_required": False,
            })

        # High maxwritten_clean indicates bgwriter can't keep up
        if bg.get("maxwritten_clean", 0) > 100:
            recommendations.append({
                "priority": "long-term",
                "issue": f"Background writer hit max write limit {bg['maxwritten_clean']:,} times",
                "action": "Increase bgwriter_lru_maxpages to let bgwriter flush more buffers per round",
                "explanation": "The background writer proactively flushes dirty buffers before they're needed. "
                               f"It hit the per-round limit {bg['maxwritten_clean']:,} times, meaning it couldn't "
                               f"keep up with the write rate. Increasing bgwriter_lru_maxpages allows more buffer "
                               f"flushes per round.",
            })

    return recommendations


def sum_index_sizes(indexes: List[Dict[str, Any]]) -> str:
    """Sum up index sizes and return human-readable string."""
    total_bytes = 0
    for idx in indexes:
        size_str = idx.get("size", "0")
        # Parse sizes like "23 MB", "8448 kB", etc.
        match = re.match(r"(\d+)\s*(MB|kB|GB|bytes?)?", size_str, re.IGNORECASE)
        if match:
            value = int(match.group(1))
            unit = (match.group(2) or "bytes").upper()
            if unit in ("KB", "KB"):
                total_bytes += value * 1024
            elif unit == "MB":
                total_bytes += value * 1024 * 1024
            elif unit == "GB":
                total_bytes += value * 1024 * 1024 * 1024
            else:
                total_bytes += value

    if total_bytes >= 1024 * 1024 * 1024:
        return f"{total_bytes / 1024 / 1024 / 1024:.1f} GB"
    elif total_bytes >= 1024 * 1024:
        return f"{total_bytes / 1024 / 1024:.1f} MB"
    elif total_bytes >= 1024:
        return f"{total_bytes / 1024:.1f} KB"
    return f"{total_bytes} bytes"


def format_report(result: AnalysisResult) -> str:
    """Format analysis result as human-readable report."""
    lines = []
    lines.append("=" * 60)
    lines.append(f"Database Analysis: {result.service}")
    lines.append("=" * 60)
    lines.append(f"Type: {result.db_type}")
    lines.append(f"Generated: {result.timestamp}")
    lines.append(f"Status: {result.deployment_status}")
    lines.append("")

    # Collection status table
    if result.collection_status:
        lines.append("## Data Collection Status")
        lines.append("")
        lines.append("| Source | Status | Details |")
        lines.append("|--------|--------|---------|")
        source_labels = {
            "database_query": "Database Query (SSH)",
            "metrics_api": "Metrics API",
            "logs_api": "Logs API",
            "ha_cluster": "HA Cluster (Patroni)",
        }
        for source in ["database_query", "metrics_api", "logs_api", "ha_cluster"]:
            if source in result.collection_status:
                info = result.collection_status[source]
                status = info["status"].upper()
                details = ""
                if info.get("error"):
                    details = info["error"]
                elif info.get("reason"):
                    details = info["reason"]
                elif info.get("lines"):
                    details = f"{info['lines']} lines collected"
                elif status == "SUCCESS":
                    details = "OK"
                label = source_labels.get(source, source)
                lines.append(f"| {label} | {status} | {details} |")
        lines.append("")

    # Summary table
    lines.append("## Summary")
    lines.append("")
    lines.append("| Metric | Value | Status |")
    lines.append("|--------|-------|--------|")

    # Deployment
    status_icon = "Healthy" if result.deployment_status == "SUCCESS" else "Warning"
    lines.append(f"| Deployment | {result.deployment_status} | {status_icon} |")

    # Connections
    if result.connections:
        pct = result.connections["percent"]
        current = result.connections["current"]
        max_conn = result.connections["max"]
        reserved = result.connections.get("reserved", 3)
        available = result.connections.get("available", max_conn - current)
        active = result.connections.get("active", 0)
        idle = result.connections.get("idle", 0)
        idle_in_txn = result.connections.get("idle_in_transaction", 0)
        status = "Critical" if pct > 90 else "Warning" if pct > 70 else "Healthy"
        lines.append(f"| Connections | {current} / {max_conn} ({pct}%) | {status} |")
        lines.append(f"| - Active/Idle/IdleTxn | {active} / {idle} / {idle_in_txn} | {'Warning' if idle_in_txn > 5 else '-'} |")
        lines.append(f"| - Available | {available} (reserved: {reserved}) | - |")

    # Database size
    if result.size_breakdown and result.size_breakdown.get("database_bytes"):
        db_bytes = result.size_breakdown["database_bytes"]
        db_gb = round(db_bytes / 1024 / 1024 / 1024, 2)
        lines.append(f"| Database Size | {db_gb} GB | - |")

    # Disk
    if result.disk_usage:
        pct = int(result.disk_usage["use_percent"])
        status = "Critical" if pct > 85 else "Warning" if pct > 70 else "Healthy"
        lines.append(f"| Disk | {result.disk_usage['used']} / {result.disk_usage['total']} ({pct}%) | {status} |")

    # Cache hit
    if result.cache_hit:
        table_hit = result.cache_hit.get("table_hit_pct")
        if table_hit is not None:
            status = "Healthy" if table_hit >= 99 else "OK" if table_hit >= 95 else "Warning" if table_hit >= 90 else "Critical"
            lines.append(f"| Table Cache Hit | {table_hit}% | {status} |")

        index_hit = result.cache_hit.get("index_hit_pct")
        if index_hit is not None:
            status = "Healthy" if index_hit >= 99 else "OK" if index_hit >= 95 else "Warning"
            lines.append(f"| Index Cache Hit | {index_hit}% | {status} |")

    # Memory config summary
    if result.memory_config:
        if "shared_buffers" in result.memory_config:
            sb = result.memory_config["shared_buffers"]
            mb = sb.get("mb", 0)
            status = "Warning" if mb < 128 else "OK" if mb < 256 else "Healthy"
            lines.append(f"| shared_buffers | {mb} MB | {status} |")
        if "work_mem" in result.memory_config:
            wm = result.memory_config["work_mem"]
            mb = wm.get("mb", 0)
            status = "Default" if mb <= 4 else "OK"
            lines.append(f"| work_mem | {mb} MB | {status} |")

    # XID age
    if result.xid_age:
        millions = result.xid_age["millions"]
        status = "Critical" if millions > 150 else "Warning" if millions > 100 else "Healthy"
        lines.append(f"| XID Age | {millions}M | {status} |")

    # CPU/Memory (with trend indicators from 24h history)
    if result.cpu_memory:
        if "cpu_percent" in result.cpu_memory:
            cpu = result.cpu_memory["cpu_percent"]
            status = "Critical" if cpu > 85 else "Warning" if cpu > 70 else "Healthy"
            trend_str = _trend_indicator(result.metrics_history, "cpu")
            lines.append(f"| CPU Usage | {cpu} vCPU{trend_str} | {status} |")
            if result.cpu_memory.get("cpu_limit"):
                lines.append(f"| CPU Limit | {result.cpu_memory['cpu_limit']} vCPU | - |")
        if "memory_gb" in result.cpu_memory:
            mem = result.cpu_memory["memory_gb"]
            trend_str = _trend_indicator(result.metrics_history, "memory")
            utilization = ""
            if result.cpu_memory.get("memory_limit_gb"):
                pct = round((mem / result.cpu_memory["memory_limit_gb"]) * 100, 1)
                status = "Critical" if pct > 90 else "Warning" if pct > 80 else "Healthy"
                utilization = f" ({pct}% of {result.cpu_memory['memory_limit_gb']} GB)"
            else:
                status = "-"
            lines.append(f"| Memory Usage | {mem} GB{utilization}{trend_str} | {status} |")

    # Database stats
    if result.database_stats:
        stats_reset = result.database_stats.get("stats_reset", "unknown")
        # Calculate days since stats reset for rate calculations
        days_since_reset = None
        if stats_reset and stats_reset not in ("unknown", "never"):
            try:
                reset_date = datetime.fromisoformat(stats_reset.replace('Z', '+00:00'))
                days_since_reset = (datetime.now(timezone.utc) - reset_date).days
                days_since_reset = max(days_since_reset, 1)  # Avoid division by zero
            except (ValueError, TypeError):
                pass
        # Shorten timestamp to just date if it's a full timestamp
        stats_reset_display = stats_reset
        if stats_reset and stats_reset != "unknown" and stats_reset != "never" and len(stats_reset) > 10:
            stats_reset_display = stats_reset[:10]
        lines.append(f"| Stats Reset | {stats_reset_display} | - |")
        deadlocks = result.database_stats.get("deadlocks", 0)
        temp_files = result.database_stats.get("temp_files", 0)
        temp_bytes = result.database_stats.get("temp_bytes", 0)
        temp_gb = round(temp_bytes / 1024 / 1024 / 1024, 2) if temp_bytes > 0 else 0
        status = "Warning" if deadlocks > 0 else "Healthy"
        lines.append(f"| Deadlocks | {deadlocks} (since reset) | {status} |")
        # Show temp files with daily rate if we have time period data
        if days_since_reset:
            files_per_day = round(temp_files / days_since_reset)
            gb_per_day = round(temp_gb / days_since_reset, 2)
            # Status based on daily rate, not totals
            if gb_per_day > 5:
                temp_status = "High"
            elif gb_per_day > 1:
                temp_status = "Moderate"
            else:
                temp_status = "OK"
            lines.append(f"| Temp Files | {temp_files:,} ({temp_gb} GB) over {days_since_reset}d (~{files_per_day}/day, {gb_per_day} GB/day) | {temp_status} |")
        else:
            lines.append(f"| Temp Files | {temp_files:,} ({temp_gb} GB since reset) | - |")

    # Size breakdown
    if result.size_breakdown:
        wal_bytes = result.size_breakdown.get("wal_bytes", 0)
        wal_mb = round(wal_bytes / 1024 / 1024, 1)
        lines.append(f"| WAL Size | {wal_mb} MB | - |")

    # Oldest connection
    if result.oldest_connection_sec is not None:
        age_hrs = round(result.oldest_connection_sec / 3600, 1)
        status = "Warning" if age_hrs > 24 else "Healthy"
        lines.append(f"| Oldest Connection | {age_hrs} hrs | {status} |")

    # pg_stat_statements extension
    pss_status = "Installed" if result.pg_stat_statements_installed else "Not installed"
    pss_icon = "OK" if result.pg_stat_statements_installed else "Info"
    lines.append(f"| pg_stat_statements | {pss_status} | {pss_icon} |")

    lines.append("")

    # Infrastructure Trends (multi-window)
    if result.metrics_history and result.metrics_history.get("windows"):
        windows = result.metrics_history.get("windows", {})
        for window_label, window_data in windows.items():
            mh = window_data.get("metrics", {})
            if not mh:
                continue
            lines.append(f"## Infrastructure Trends ({window_label})")
            lines.append("")
            lines.append("| Metric | Current | Min | Max | Avg | Trend | Change |")
            lines.append("|--------|---------|-----|-----|-----|-------|--------|")
            display_order = [
                ("cpu", "CPU"),
                ("memory", "Memory"),
                ("disk", "Disk"),
                ("network_rx", "Network RX"),
                ("network_tx", "Network TX"),
            ]
            for key, label in display_order:
                if key in mh:
                    m = mh[key]
                    unit = m["unit"]
                    trend = m.get("trend", {})
                    direction = trend.get("direction", "?")
                    change = trend.get("change_pct", 0)
                    arrow = {"increasing": "^", "decreasing": "v", "stable": "~"}.get(direction, "?")
                    spike_note = ""
                    if m.get("spikes"):
                        spike_note = f" ({m['spikes']['count']} spikes)"
                    lines.append(
                        f"| {label} | {m['current']} {unit} | {m['min']} | {m['max']} | {m['avg']} | "
                        f"{arrow} {direction} | {change:+.1f}%{spike_note} |"
                    )
            lines.append("")

    # PostgreSQL Configuration (tuning parameters)
    if result.memory_config:
        lines.append("## PostgreSQL Configuration")
        lines.append("")
        lines.append("| Parameter | Value | Recommended | Status |")
        lines.append("|-----------|-------|-------------|--------|")

        mem = result.memory_config

        # Memory settings
        if "shared_buffers" in mem:
            sb = mem["shared_buffers"]
            mb = sb.get("mb", 0)
            status = "Low" if mb < 128 else "Default" if mb < 256 else "OK"
            lines.append(f"| shared_buffers | {mb} MB | 25% of RAM | {status} |")

        if "effective_cache_size" in mem:
            ec = mem["effective_cache_size"]
            mb = ec.get("mb", 0)
            status = "Low" if mb < 512 else "OK"
            lines.append(f"| effective_cache_size | {mb} MB | 50-75% of RAM | {status} |")

        if "work_mem" in mem:
            wm = mem["work_mem"]
            mb = wm.get("mb", 0)
            status = "Default" if mb <= 4 else "OK"
            lines.append(f"| work_mem | {mb} MB | 16-64 MB | {status} |")

        if "maintenance_work_mem" in mem:
            mm = mem["maintenance_work_mem"]
            mb = mm.get("mb", 0)
            status = "Low" if mb < 64 else "OK"
            lines.append(f"| maintenance_work_mem | {mb} MB | 256-1024 MB | {status} |")

        # WAL settings
        if "wal_buffers" in mem:
            wb = mem["wal_buffers"]
            mb = wb.get("mb", 0)
            lines.append(f"| wal_buffers | {mb} MB | 16 MB | OK |")

        if "checkpoint_completion_target" in mem:
            cct = mem["checkpoint_completion_target"]
            val = cct.get("value", 0)
            status = "Low" if float(val) < 0.9 else "OK"
            lines.append(f"| checkpoint_completion_target | {val} | 0.9 | {status} |")

        # Parallelism
        if "max_parallel_workers" in mem:
            mpw = mem["max_parallel_workers"]
            val = mpw.get("value", 0)
            status = "Disabled" if val == 0 else "OK"
            lines.append(f"| max_parallel_workers | {val} | CPU cores | {status} |")

        if "max_parallel_workers_per_gather" in mem:
            mpwpg = mem["max_parallel_workers_per_gather"]
            val = mpwpg.get("value", 0)
            status = "Disabled" if val == 0 else "OK"
            lines.append(f"| max_parallel_workers_per_gather | {val} | 2-4 | {status} |")

        # Planner
        if "random_page_cost" in mem:
            rpc = mem["random_page_cost"]
            val = rpc.get("value", 4.0)
            status = "HDD default" if float(val) >= 4.0 else "SSD optimized" if float(val) <= 2.0 else "OK"
            lines.append(f"| random_page_cost | {val} | 1.1-2.0 (SSD) | {status} |")

        if "default_statistics_target" in mem:
            dst = mem["default_statistics_target"]
            val = dst.get("value", 100)
            lines.append(f"| default_statistics_target | {val} | 100-500 | OK |")

        # Autovacuum
        if "autovacuum" in mem:
            av = mem["autovacuum"]
            val = av.get("value", "on")
            status = "CRITICAL" if val == "off" else "OK"
            lines.append(f"| autovacuum | {val} | on | {status} |")

        # Durability
        if "synchronous_commit" in mem:
            sc = mem["synchronous_commit"]
            val = sc.get("value", "on")
            status = "Faster (risk)" if val == "off" else "Safe"
            lines.append(f"| synchronous_commit | {val} | on (safe) | {status} |")

        lines.append("")

    # Connection states
    if result.connection_states:
        lines.append("## Connection States")
        lines.append("")
        lines.append("| State | Count |")
        lines.append("|-------|-------|")
        for s in result.connection_states:
            lines.append(f"| {s.get('state', 'unknown')} | {s.get('count', 0)} |")
        lines.append("")

    # Connections by application
    if result.connections_by_app:
        lines.append("## Connections by Application")
        lines.append("")
        lines.append("| Application | Count |")
        lines.append("|-------------|-------|")
        for c in result.connections_by_app[:10]:
            app = c.get('app', '') or '(empty)'
            lines.append(f"| {app} | {c.get('count', 0)} |")
        lines.append("")

    # Connections by age
    if result.connections_by_age:
        lines.append("## Connections by Age")
        lines.append("")
        lines.append("| Age Range | Count |")
        lines.append("|-----------|-------|")
        for c in result.connections_by_age:
            lines.append(f"| {c.get('range', '')} | {c.get('count', 0)} |")
        lines.append("")

    # Per-table cache
    if result.cache_per_table:
        lines.append("## Per-Table Cache Hit Rates")
        lines.append("")
        lines.append("| Table | Hit % | Disk Reads | Status |")
        lines.append("|-------|-------|------------|--------|")
        for t in result.cache_per_table[:10]:
            hit_pct = float(t.get("hit_pct", 0))
            status = "OK" if hit_pct >= 95 else "Warning" if hit_pct >= 80 else "Critical"
            lines.append(f"| {t['table']} | {t['hit_pct']}% | {int(t['disk_reads']):,} | {status} |")
        lines.append("")

    # Table sizes
    if result.table_sizes:
        lines.append("## Table Sizes")
        lines.append("")
        lines.append("| Schema.Table | Total | Data | Indexes | Rows |")
        lines.append("|--------------|-------|------|---------|------|")
        for t in result.table_sizes[:10]:
            schema = t.get('schema', 'public')
            table = t.get('table', '')
            full_name = f"{schema}.{table}" if schema != 'public' else table
            data_bytes = int(t.get('data_bytes', 0))
            index_bytes = int(t.get('index_bytes', 0))
            data_mb = round(data_bytes / 1024 / 1024, 1)
            index_mb = round(index_bytes / 1024 / 1024, 1)
            row_count = t.get('row_count', '0')
            lines.append(f"| {full_name} | {t['size']} | {data_mb}MB | {index_mb}MB | {row_count} |")
        lines.append("")

    # Vacuum health
    if result.vacuum_health:
        lines.append("## Vacuum Health")
        lines.append("")
        lines.append("| Schema.Table | Dead Rows | Dead % | V/AV Count | Last Analyze | XID Age | Flags |")
        lines.append("|--------------|-----------|--------|------------|--------------|---------|-------|")
        for t in result.vacuum_health[:10]:
            schema = t.get('schema', 'public')
            table = t.get('table', '')
            vacuum_count = t.get('vacuum_count', '0')
            autovacuum_count = t.get('autovacuum_count', '0')
            last_analyze = t.get('last_analyze', 'never')
            # Shorten timestamp to just date
            if last_analyze and last_analyze != 'never' and len(last_analyze) > 10:
                last_analyze = last_analyze[:10]
            xid_age = t.get('xid_age', '0')
            xid_millions = round(int(xid_age) / 1_000_000, 1) if xid_age.isdigit() else 0
            flags = []
            if t.get('needs_vacuum') == 'true':
                flags.append('VACUUM')
            if t.get('needs_freeze') == 'true':
                flags.append('FREEZE')
            flags_str = ', '.join(flags) if flags else '-'
            full_name = f"{schema}.{table}" if schema != 'public' else table
            lines.append(f"| {full_name} | {int(t['dead_rows']):,} | {t['dead_pct']}% | {vacuum_count}/{autovacuum_count} | {last_analyze} | {xid_millions}M | {flags_str} |")
        lines.append("")

    # Unused indexes
    if result.unused_indexes:
        lines.append("## Unused Indexes (0 scans since stats reset)")
        lines.append("")
        lines.append("| Schema.Table | Index | Size | Type | Table Idx Scans |")
        lines.append("|--------------|-------|------|------|-----------------|")
        for t in result.unused_indexes[:20]:
            schema = t.get('schema', 'public')
            table = t.get('table', '')
            full_name = f"{schema}.{table}" if schema != 'public' else table
            table_idx_scans = t.get('table_idx_scans', '0')
            # Show index type
            idx_type = "PK" if t.get('is_primary') else "UNIQUE" if t.get('is_unique') else "idx"
            lines.append(f"| {full_name} | {t['index']} | {t['size']} | {idx_type} | {table_idx_scans} |")
        lines.append("")

    # Invalid indexes (failed concurrent index builds)
    if result.invalid_indexes:
        lines.append("## Invalid Indexes (require rebuild)")
        lines.append("")
        lines.append("| Schema | Table | Index |")
        lines.append("|--------|-------|-------|")
        for t in result.invalid_indexes:
            lines.append(f"| {t.get('schema', '')} | {t.get('table', '')} | {t.get('index', '')} |")
        lines.append("")

    # Top queries
    if result.top_queries:
        lines.append("## Top Queries by Execution Time")
        lines.append("")
        lines.append("| Query | Calls | Total (min) | Mean (ms) | Min/Max (ms) | Stddev | Rows/Call | Cache Hit % | Temp R/W | Plan (ms) | I/O Time (ms) |")
        lines.append("|-------|-------|-------------|-----------|--------------|--------|-----------|-------------|----------|-----------|---------------|")
        for t in result.top_queries[:15]:
            query = t.get('query', '')[:50]
            cache_pct = t.get('cache_hit_pct')
            cache_str = f"{cache_pct}%" if cache_pct is not None else "-"
            temp_r = t.get('temp_blks_read', 0)
            temp_w = t.get('temp_blks_written', 0)
            temp_str = f"{temp_r:,}/{temp_w:,}" if (temp_r or temp_w) else "-"
            min_max = f"{t.get('min_ms', '-')}/{t.get('max_ms', '-')}"
            stddev = t.get('stddev_ms', '-')
            rows_per_call = t.get('rows_per_call', '-')
            plan_ms = t.get('mean_plan_ms', '-')
            blk_read = float(t.get('blk_read_time_ms', 0) or 0)
            blk_write = float(t.get('blk_write_time_ms', 0) or 0)
            io_time = f"{blk_read + blk_write:.0f}" if (blk_read + blk_write) > 0 else "-"
            lines.append(f"| {query}... | {t.get('calls', '')} | {t.get('total_min', '')} | {t.get('mean_ms', '')} | {min_max} | {stddev} | {rows_per_call} | {cache_str} | {temp_str} | {plan_ms} | {io_time} |")
        lines.append("")

    # Long-running queries
    if result.long_running_queries:
        lines.append("## Long-Running Queries (>5s)")
        lines.append("")
        lines.append("| PID | Duration (s) | Query |")
        lines.append("|-----|--------------|-------|")
        for q in result.long_running_queries:
            lines.append(f"| {q.get('pid', '')} | {q.get('duration_sec', '')} | {q.get('query', '')[:40]}... |")
        lines.append("")

    # Idle in transaction (stuck transactions)
    if result.idle_in_transaction:
        lines.append("## Idle In Transaction (>30s)")
        lines.append("")
        lines.append("| PID | Idle (s) | User | App | Last Query |")
        lines.append("|-----|----------|------|-----|------------|")
        for txn in result.idle_in_transaction:
            lines.append(f"| {txn.get('pid', '')} | {txn.get('idle_sec', '')} | {txn.get('user', '')} | {txn.get('app', '')[:15]} | {txn.get('last_query', '')[:30]}... |")
        lines.append("")

    # Blocked queries
    if result.blocked_queries:
        lines.append("## Blocked Queries (waiting on locks)")
        lines.append("")
        lines.append("| PID | Wait (s) | User | Blocked By | Query |")
        lines.append("|-----|----------|------|------------|-------|")
        for q in result.blocked_queries:
            lines.append(f"| {q.get('pid', '')} | {q.get('wait_sec', '')} | {q.get('user', '')} | PID {q.get('blocking_pid', '')} | {q.get('query', '')[:30]}... |")
        lines.append("")

    # Lock contention
    if result.locks:
        lines.append("## Lock Contention")
        lines.append("")
        lines.append("| Lock Type | Mode | User | App | Query |")
        lines.append("|-----------|------|------|-----|-------|")
        for lock in result.locks:
            lines.append(f"| {lock.get('locktype', '')} | {lock.get('mode', '')} | {lock.get('user', '')} | {lock.get('app', '')[:15]} | {lock.get('query', '')[:25]}... |")
        lines.append("")

    # Sequential scan patterns
    if result.seq_scan_tables:
        lines.append("## Tables with High Sequential Scans")
        lines.append("")
        lines.append("| Table | Seq Scans | Index Scans | Rows |")
        lines.append("|-------|-----------|-------------|------|")
        for t in result.seq_scan_tables[:10]:
            lines.append(f"| {t.get('table', '')} | {t.get('seq_scans', '')} | {t.get('idx_scans', '')} | {t.get('rows', '')} |")
        lines.append("")

    # Replication
    if result.replication:
        lines.append("## Replication Status")
        lines.append("")
        lines.append("| Client | State | Sent LSN | Replay LSN |")
        lines.append("|--------|-------|----------|------------|")
        for r in result.replication:
            lines.append(f"| {r.get('client', '')} | {r.get('state', '')} | {r.get('sent_lsn', '')} | {r.get('replay_lsn', '')} |")
        lines.append("")

    # HA Cluster
    if result.ha_cluster:
        lines.append("## HA Cluster (Patroni)")
        lines.append("")
        members = result.ha_cluster.get("members", [])
        if members:
            lines.append("| Name | Role | State | Timeline | Lag |")
            lines.append("|------|------|-------|----------|-----|")
            for m in members:
                lag = m.get('lag', 0) or 0
                lines.append(f"| {m.get('name', '')} | {m.get('role', '')} | {m.get('state', '')} | {m.get('timeline', '')} | {lag} |")
        lines.append("")

    # Cluster logs (for HA clusters) - raw output for LLM analysis
    if result.cluster_logs:
        lines.append("## Cluster Member Logs")
        lines.append("")
        lines.append("(Use --json for full log data. LLM will analyze patterns.)")
        lines.append("")
        for member_log in result.cluster_logs:
            member = member_log.get('member', 'unknown')
            status = member_log.get('status', 'unknown')
            logs = member_log.get('logs', [])
            lines.append(f"### {member} ({status}) - {len(logs)} log entries collected")
            lines.append("")

    # Background writer stats
    if result.bgwriter:
        lines.append("## Background Writer Stats")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        bg = result.bgwriter
        total_checkpoints = bg.get('checkpoints_timed', 0) + bg.get('checkpoints_req', 0)
        timed_pct = round(100.0 * bg.get('checkpoints_timed', 0) / total_checkpoints, 1) if total_checkpoints > 0 else 0
        lines.append(f"| Checkpoints (timed/requested) | {bg.get('checkpoints_timed', 0):,} / {bg.get('checkpoints_req', 0):,} ({timed_pct}% timed) |")
        lines.append(f"| Buffers: checkpoint | {bg.get('buffers_checkpoint', 0):,} |")
        lines.append(f"| Buffers: bgwriter clean | {bg.get('buffers_clean', 0):,} |")
        lines.append(f"| Buffers: backend direct | {bg.get('buffers_backend', 0):,} |")
        lines.append(f"| Buffers: backend fsync | {bg.get('buffers_backend_fsync', 0):,} |")
        lines.append(f"| Max written clean | {bg.get('maxwritten_clean', 0):,} |")
        stats_reset = bg.get('stats_reset', 'never')
        if stats_reset and stats_reset != 'never' and len(stats_reset) > 10:
            stats_reset = stats_reset[:10]
        lines.append(f"| Stats reset | {stats_reset} |")
        lines.append("")

    # WAL archiver stats
    if result.archiver:
        arch = result.archiver
        lines.append("## WAL Archiver Status")
        lines.append("")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Archived WAL count | {arch.get('archived_count', 0):,} |")
        lines.append(f"| Failed archival count | {arch.get('failed_count', 0):,} |")
        last_wal = arch.get('last_archived_wal') or 'none'
        last_time = arch.get('last_archived_time', 'never')
        if last_time and last_time != 'never' and len(last_time) > 19:
            last_time = last_time[:19]
        lines.append(f"| Last archived WAL | {last_wal} |")
        lines.append(f"| Last archived time | {last_time} |")
        if arch.get('failed_count', 0) > 0:
            failed_wal = arch.get('last_failed_wal') or 'none'
            failed_time = arch.get('last_failed_time', 'never')
            if failed_time and failed_time != 'never' and len(failed_time) > 19:
                failed_time = failed_time[:19]
            lines.append(f"| Last failed WAL | {failed_wal} |")
            lines.append(f"| Last failed time | {failed_time} |")
        lines.append("")

    # Vacuum progress (ongoing vacuums)
    if result.progress_vacuum:
        lines.append("## Ongoing Vacuum Operations")
        lines.append("")
        lines.append("| PID | Table | Phase | Progress |")
        lines.append("|-----|-------|-------|----------|")
        for vac in result.progress_vacuum:
            total = vac.get('heap_blks_total', 0)
            scanned = vac.get('heap_blks_scanned', 0)
            pct = round(100.0 * scanned / total, 1) if total > 0 else 0
            lines.append(f"| {vac.get('pid', '')} | {vac.get('relname', '')} | {vac.get('phase', '')} | {pct}% ({scanned:,}/{total:,} blks) |")
        lines.append("")

    # SSL connection stats
    if result.ssl_stats:
        ssl = result.ssl_stats
        lines.append("## SSL Connection Stats")
        lines.append("")
        total = ssl.get('ssl_connections', 0) + ssl.get('non_ssl_connections', 0)
        ssl_pct = round(100.0 * ssl.get('ssl_connections', 0) / total, 1) if total > 0 else 0
        lines.append(f"- SSL connections: {ssl.get('ssl_connections', 0)} ({ssl_pct}%)")
        lines.append(f"- Non-SSL connections: {ssl.get('non_ssl_connections', 0)}")
        versions = ssl.get('ssl_versions', [])
        if versions:
            lines.append("- SSL versions in use:")
            for v in versions:
                lines.append(f"  - {v.get('version', 'unknown')}: {v.get('count', 0)} connections")
        lines.append("")

    # Recent errors
    if result.recent_errors:
        lines.append("## Recent Errors")
        lines.append("")
        for error in result.recent_errors[:5]:
            lines.append(f"- {error[:100]}...")
        lines.append("")

    # Recommendations
    if result.recommendations:
        lines.append("## Recommendations")
        lines.append("")
        for i, rec in enumerate(result.recommendations, 1):
            priority = rec["priority"].upper()
            lines.append(f"{i}. **[{priority}]** {rec['issue']}")
            lines.append(f"   **Action:** {rec['action']}")

            # Show explanation if available
            if rec.get("explanation"):
                lines.append(f"   **Why:** {rec['explanation']}")

            # Show commands if available
            if rec.get("commands"):
                lines.append("   **Commands:**")
                for cmd in rec["commands"]:
                    lines.append(f"   ```sql")
                    lines.append(f"   {cmd}")
                    lines.append(f"   ```")

            # Note if restart is required
            if rec.get("restart_required"):
                lines.append("   ⚠️ *Requires database restart*")

            lines.append("")

    # Errors
    if result.errors:
        lines.append("## Errors")
        lines.append("")
        for error in result.errors:
            lines.append(f"- {error}")
        lines.append("")

    lines.append("=" * 60)
    lines.append("END OF REPORT")
    lines.append("=" * 60)

    return "\n".join(lines)


def main():
    parser = argparse.ArgumentParser(
        description="Complete database analysis for Railway services.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )

    parser.add_argument("--service", required=True, help="Service name")
    parser.add_argument("--json", action="store_true",
                       help="Output as JSON")
    parser.add_argument("--timeout", type=int, default=300,
                       help="Timeout in seconds for analysis query (default: 300)")
    parser.add_argument("--quiet", "-q", action="store_true",
                       help="Suppress progress messages")
    parser.add_argument("--skip-logs", action="store_true",
                       help="Skip log fetching for faster analysis")
    parser.add_argument("--metrics-hours", type=int, default=168,
                       help="Hours of metrics history to fetch (default: 168, max: 168)")
    parser.add_argument("--step", choices=["ssh-test", "query", "logs", "metrics"],
                       help="Run a single collection step for debugging")
    parser.add_argument("--project-id", help="Project ID (bypasses railway link)")
    parser.add_argument("--environment-id", help="Environment ID (bypasses railway link)")
    parser.add_argument("--service-id", help="Service ID (bypasses railway link)")

    args = parser.parse_args()

    # Single-step debugging mode
    if args.step:
        return run_single_step(args)

    # Run analysis
    result = analyze_postgres(args.service, timeout=args.timeout, quiet=args.quiet,
                              skip_logs=args.skip_logs,
                              metrics_hours=min(args.metrics_hours, 168),
                              project_id=args.project_id,
                              environment_id=args.environment_id,
                              service_id=args.service_id)

    # Output
    if args.json:
        print(json.dumps(asdict(result), indent=2))
    else:
        print(format_report(result))

    return 0


def run_single_step(args) -> int:
    """Run a single collection step for debugging."""
    service = args.service
    _init_context(args)
    environment_id = dal._ctx.environment_id
    service_id = dal._ctx.service_id

    if args.step == "ssh-test":
        print(f"Testing SSH to service: {service}", file=sys.stderr)
        code, stdout, stderr = run_ssh_query(service, "echo ok", timeout=45)
        print(f"Exit code: {code}")
        print(f"Stdout: {stdout.strip()}")
        if stderr:
            print(f"Stderr: {stderr.strip()}")
        return 0 if (code == 0 and "ok" in stdout) else 1

    elif args.step == "query":
        print(f"Running analysis query on: {service}", file=sys.stderr)
        query = build_analysis_query()
        code, stdout, stderr = run_psql_query_safe(service, query, timeout=args.timeout)
        print(f"Exit code: {code}")
        if code == 0 and stdout:
            try:
                data = json.loads(stdout.strip())
                print(json.dumps(data, indent=2))
            except json.JSONDecodeError:
                print(f"Raw output:\n{stdout}")
        else:
            print(f"Error: {stderr or stdout}")
        return code

    elif args.step == "logs":
        print(f"Fetching logs for: {service}", file=sys.stderr)
        logs = get_recent_logs(service, lines=LOG_LINES_DEFAULT,
                               environment_id=environment_id,
                               service_id=service_id)
        print(f"Lines fetched: {len(logs)}")
        for line in logs:
            print(line)
        return 0

    elif args.step == "metrics":
        print(f"Fetching metrics for: {service}", file=sys.stderr)
        if environment_id and service_id:
            metrics = get_all_metrics_from_api(environment_id, service_id)
            if metrics:
                print(json.dumps(metrics, indent=2))
            else:
                print("Metrics API returned no data")
                return 1
        else:
            print("Missing environment_id or service_id from railway config")
            return 1
        return 0

    return 1


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
Redis analysis for Railway deployments.

Produces a comprehensive report covering:
- Server overview (version, uptime, clients)
- Memory usage and fragmentation
- Throughput and command stats
- Cache performance (hit/miss ratio)
- Persistence status
- Keyspace summary
- Railway infrastructure metrics (CPU, memory, disk, network)
- Recent logs
- Recommendations

Usage:
    analyze-redis.py --service <name>
    analyze-redis.py --service <name> --json
    analyze-redis.py --service <name> --step ssh-test
"""

import argparse
import json
import os
import subprocess
import sys
import re
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from typing import Dict, List, Optional, Any, Tuple
from dataclasses import dataclass, field, asdict

import dal
from dal import (
    LOG_LINES_DEFAULT, ProgressTimer, RailwayContext,
    _init_context, progress, run_railway_command, run_ssh_query,
    get_railway_status, get_deployment_status,
    get_all_metrics_from_api, _analyze_window, _build_metrics_history,
    get_recent_logs,
    _safe_int, _safe_float, _format_uptime,
)


# ---------------------------------------------------------------------------
# Data model
# ---------------------------------------------------------------------------

@dataclass
class RedisAnalysisResult:
    """Container for Redis analysis results."""
    service: str
    db_type: str
    timestamp: str
    deployment_status: str = "UNKNOWN"

    # Redis INFO sections
    overview: Optional[Dict[str, Any]] = None
    memory: Optional[Dict[str, Any]] = None
    throughput: Optional[Dict[str, Any]] = None
    cache: Optional[Dict[str, Any]] = None
    persistence: Optional[Dict[str, Any]] = None
    keyspace: List[Dict[str, Any]] = field(default_factory=list)
    total_keys: int = 0
    command_stats: List[Dict[str, Any]] = field(default_factory=list)
    slowlog_len: Optional[int] = None
    slowlog_entries: List[Dict[str, Any]] = field(default_factory=list)
    big_keys: List[Dict[str, Any]] = field(default_factory=list)

    # Railway infrastructure
    metrics_history: Optional[Dict[str, Any]] = None
    recent_logs: List[str] = field(default_factory=list)

    # Status tracking
    collection_status: Dict[str, Dict[str, Any]] = field(default_factory=dict)
    errors: List[str] = field(default_factory=list)
    recommendations: List[Dict[str, str]] = field(default_factory=list)


# ---------------------------------------------------------------------------
# Redis data collection
# ---------------------------------------------------------------------------

def parse_redis_info(raw: str) -> Dict[str, str]:
    """Parse Redis INFO output into a flat key:value dict.

    Lines starting with # are section headers and are skipped.
    """
    info: Dict[str, str] = {}
    for line in raw.splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        if ":" in line:
            key, _, value = line.partition(":")
            info[key.strip()] = value.strip()
    return info


def extract_overview(info: Dict[str, str]) -> Dict[str, Any]:
    """Extract overview metrics from INFO dict."""
    return {
        "redis_version": info.get("redis_version", "unknown"),
        "uptime_in_seconds": _safe_int(info.get("uptime_in_seconds")),
        "connected_clients": _safe_int(info.get("connected_clients")),
        "blocked_clients": _safe_int(info.get("blocked_clients")),
        "rejected_connections": _safe_int(info.get("rejected_connections")),
    }


def extract_memory(info: Dict[str, str]) -> Dict[str, Any]:
    """Extract memory metrics from INFO dict."""
    return {
        "used_memory_human": info.get("used_memory_human", "N/A"),
        "used_memory_rss_human": info.get("used_memory_rss_human", "N/A"),
        "used_memory_peak_human": info.get("used_memory_peak_human", "N/A"),
        "mem_fragmentation_ratio": _safe_float(info.get("mem_fragmentation_ratio")),
        "maxmemory": _safe_int(info.get("maxmemory")),
        "maxmemory_human": info.get("maxmemory_human", "N/A"),
        "maxmemory_policy": info.get("maxmemory_policy", "unknown"),
    }


def extract_throughput(info: Dict[str, str]) -> Dict[str, Any]:
    """Extract throughput metrics from INFO dict."""
    return {
        "instantaneous_ops_per_sec": _safe_int(info.get("instantaneous_ops_per_sec")),
        "total_commands_processed": _safe_int(info.get("total_commands_processed")),
        "total_connections_received": _safe_int(info.get("total_connections_received")),
    }


def extract_cache(info: Dict[str, str]) -> Dict[str, Any]:
    """Extract cache performance metrics from INFO dict."""
    hits = _safe_int(info.get("keyspace_hits"))
    misses = _safe_int(info.get("keyspace_misses"))
    total = hits + misses
    hit_rate = round(hits / total * 100, 2) if total > 0 else 0.0
    return {
        "keyspace_hits": hits,
        "keyspace_misses": misses,
        "hit_rate": hit_rate,
        "expired_keys": _safe_int(info.get("expired_keys")),
        "evicted_keys": _safe_int(info.get("evicted_keys")),
    }


def extract_persistence(info: Dict[str, str]) -> Dict[str, Any]:
    """Extract persistence metrics from INFO dict."""
    return {
        "rdb_last_save_time": _safe_int(info.get("rdb_last_save_time")),
        "rdb_last_bgsave_status": info.get("rdb_last_bgsave_status", "unknown"),
        "rdb_current_bgsave_time_sec": _safe_int(info.get("rdb_current_bgsave_time_sec")),
        "aof_enabled": info.get("aof_enabled", "0") == "1",
        "aof_last_rewrite_status": info.get("aof_last_rewrite_status", "unknown"),
    }


def extract_keyspace(info: Dict[str, str]) -> Tuple[List[Dict[str, Any]], int]:
    """Extract keyspace metrics from INFO dict.

    Keyspace entries look like: db0:keys=1234,expires=567,avg_ttl=12345
    Returns (list of db dicts, total_keys).
    """
    databases: List[Dict[str, Any]] = []
    total_keys = 0
    for key, value in info.items():
        if not re.match(r'^db\d+$', key):
            continue
        # Parse keys=X,expires=Y,avg_ttl=Z
        parts = {}
        for item in value.split(","):
            k, _, v = item.partition("=")
            parts[k] = v
        keys = _safe_int(parts.get("keys"))
        total_keys += keys
        databases.append({
            "db": key,
            "keys": keys,
            "expires": _safe_int(parts.get("expires")),
            "avg_ttl": _safe_int(parts.get("avg_ttl")),
        })
    return databases, total_keys


def extract_command_stats(info: Dict[str, str]) -> List[Dict[str, Any]]:
    """Extract command statistics from INFO dict.

    Command stat entries look like: cmdstat_GET:calls=123,usec=456,usec_per_call=3.71
    Returns list sorted by calls descending.
    """
    stats: List[Dict[str, Any]] = []
    for key, value in info.items():
        if not key.startswith("cmdstat_"):
            continue
        cmd_name = key[len("cmdstat_"):]
        parts = {}
        for item in value.split(","):
            k, _, v = item.partition("=")
            parts[k] = v
        stats.append({
            "command": cmd_name,
            "calls": _safe_int(parts.get("calls")),
            "usec": _safe_int(parts.get("usec")),
            "usec_per_call": _safe_float(parts.get("usec_per_call")),
        })
    stats.sort(key=lambda x: x["calls"], reverse=True)
    return stats


_CLIENT_IP_RE = re.compile(r'^(\[.+\]|(?:\d{1,3}\.){3}\d{1,3}):\d+$')


def parse_slowlog_get(raw: str) -> List[Dict[str, Any]]:
    """Parse SLOWLOG GET output into structured entries.

    redis-cli --raw SLOWLOG GET format (Redis 4.0+):
      <id>
      <timestamp_unix>
      <duration_us>
      <cmd>
      <arg1>
      ...
      <argN>
      <client_ip:port>
      [<client_name>]   (optional, may be absent)

    There is no num_args field in the raw output. The client IP line
    (IPv4 or IPv6 with port) marks the end of each entry's arguments.
    """
    entries: List[Dict[str, Any]] = []
    lines = [l.strip() for l in raw.strip().splitlines() if l.strip()]
    if not lines:
        return entries

    i = 0
    while i < len(lines):
        try:
            entry_id = int(lines[i])
        except (ValueError, IndexError):
            i += 1
            continue

        if i + 3 >= len(lines):
            break

        try:
            timestamp = int(lines[i + 1])
            duration_us = int(lines[i + 2])
        except (ValueError, IndexError):
            i += 1
            continue

        # lines[i+3] is the command name; scan forward for client IP
        cmd_start = i + 3
        client_ip_pos = None
        for k in range(cmd_start, min(cmd_start + 30, len(lines))):
            if _CLIENT_IP_RE.match(lines[k]):
                client_ip_pos = k
                break

        if client_ip_pos is not None:
            cmd_parts = lines[cmd_start:client_ip_pos]
            # Advance past client IP and optional client name
            next_i = client_ip_pos + 1
            if next_i < len(lines):
                try:
                    int(lines[next_i])
                except ValueError:
                    next_i += 1  # skip client name
        else:
            # No client IP found — take command + first arg only and advance
            cmd_parts = lines[cmd_start:cmd_start + 2]
            next_i = cmd_start + 2

        command = " ".join(cmd_parts) if cmd_parts else "unknown"
        if len(command) > 120:
            command = command[:117] + "..."

        entries.append({
            "id": entry_id,
            "timestamp_unix": timestamp,
            "duration_us": duration_us,
            "command": command,
        })

        i = next_i

    return entries


def parse_bigkeys(raw: str) -> List[Dict[str, Any]]:
    """Parse redis-cli --bigkeys output into structured entries.

    Looks for lines like:
      Biggest string found "cache:render:page/dashboard" has 2145832 bytes
      Biggest hash found "user:sessions" has 14291 fields
      Biggest list found "queue:notifications" has 8402 items
      Biggest set found "tags:all" has 291 members
      Biggest zset found "leaderboard:global" has 10042 members
      Biggest stream found "events:main" has 5012 entries
    """
    entries: List[Dict[str, Any]] = []
    # Match: Biggest <type> found "<key>" has <count> <unit>
    # Redis 8+ uses double quotes; older versions used single quotes
    pattern = re.compile(
        r'Biggest\s+(\w+)\s+found\s+["\']([^"\']+)["\']\s+has\s+([\d,]+)\s+(\w+)',
        re.IGNORECASE,
    )
    for line in raw.splitlines():
        m = pattern.search(line)
        if m:
            key_type = m.group(1).lower()
            key_name = m.group(2)
            size_str = m.group(3).replace(",", "")
            unit = m.group(4).lower()
            size = _safe_int(size_str)

            # Format size for display
            if unit == "bytes":
                detail = _format_bytes_human(size)
            else:
                detail = f"{size:,} {unit}"

            entries.append({
                "type": key_type,
                "key": key_name,
                "size_or_count": size,
                "detail": detail,
            })
    return entries


# ---------------------------------------------------------------------------
# Formatting helpers
# ---------------------------------------------------------------------------

def _format_number(n: int) -> str:
    """Format a large number with K/M/B suffixes."""
    if n >= 1_000_000_000:
        return f"{n / 1_000_000_000:.1f}B"
    if n >= 1_000_000:
        return f"{n / 1_000_000:.1f}M"
    if n >= 1_000:
        return f"{n / 1_000:.1f}K"
    return f"{n:,}"


def _format_duration(seconds: int) -> str:
    """Format a duration in seconds to a human-readable relative string."""
    if seconds <= 0:
        return "N/A"
    if seconds < 60:
        return f"{seconds}s ago"
    if seconds < 3600:
        return f"{seconds // 60}m ago"
    if seconds < 86400:
        return f"{seconds // 3600}h ago"
    return f"{seconds // 86400}d ago"


def _format_ttl(ms: int) -> str:
    """Format average TTL in milliseconds to a human-readable string."""
    if ms <= 0:
        return "none"
    seconds = ms // 1000
    if seconds < 60:
        return f"{seconds}s"
    if seconds < 3600:
        return f"{seconds // 60}m"
    if seconds < 86400:
        return f"{seconds // 3600}h"
    return f"{seconds // 86400}d"


def _format_usec(usec: float) -> str:
    """Format microseconds to a human-readable string."""
    if usec < 1000:
        return f"{usec:.1f}us"
    if usec < 1_000_000:
        return f"{usec / 1000:.1f}ms"
    return f"{usec / 1_000_000:.2f}s"


def _format_total_time(usec: int) -> str:
    """Format total microseconds to a readable time string."""
    seconds = usec / 1_000_000
    if seconds < 1:
        return f"{usec / 1000:.1f}ms"
    if seconds < 60:
        return f"{seconds:.1f}s"
    if seconds < 3600:
        return f"{seconds / 60:.1f}m"
    return f"{seconds / 3600:.1f}h"


def _format_bytes_human(nbytes: int) -> str:
    """Format bytes into human-readable string."""
    if nbytes <= 0:
        return "0"
    for unit in ["B", "K", "M", "G", "T"]:
        if nbytes < 1024:
            return f"{nbytes:.1f}{unit}" if nbytes != int(nbytes) else f"{int(nbytes)}{unit}"
        nbytes /= 1024
    return f"{nbytes:.1f}P"


# ---------------------------------------------------------------------------
# Recommendations engine
# ---------------------------------------------------------------------------

def generate_recommendations(result: RedisAnalysisResult) -> List[Dict[str, str]]:
    """Generate recommendations based on collected metrics."""
    recs: List[Dict[str, str]] = []

    # Collection failures — surface critical issues when SSH/introspection failed
    if result.collection_status:
        failed = {k: v for k, v in result.collection_status.items() if v.get("status") == "failed"}
        ssh_sources = {"redis_info", "slowlog", "slowlog_entries", "big_keys"}
        ssh_failed = {k: v for k, v in failed.items() if k in ssh_sources}
        if ssh_failed:
            sources = ", ".join(ssh_failed.keys())
            errors = "; ".join(v.get("error", "unknown") for v in ssh_failed.values())
            recs.append({
                "severity": "critical",
                "category": "collection",
                "message": f"SSH introspection failed — unable to collect {sources}. "
                           f"Error: {errors}. "
                           f"Analysis is incomplete: memory fragmentation, cache hit rate, "
                           f"keyspace stats, and persistence health could not be evaluated.",
            })

    # Memory fragmentation
    if result.memory:
        frag = result.memory.get("mem_fragmentation_ratio", 0)
        if frag > 1.5:
            recs.append({
                "severity": "warning",
                "category": "memory",
                "message": f"High memory fragmentation ({frag:.2f}). Consider restarting Redis to defragment, or enable activedefrag.",
            })

    # Cache hit rate
    if result.cache:
        hit_rate = result.cache.get("hit_rate", 0)
        if hit_rate < 80 and (result.cache.get("keyspace_hits", 0) + result.cache.get("keyspace_misses", 0)) > 0:
            recs.append({
                "severity": "warning",
                "category": "cache",
                "message": f"Low cache hit rate ({hit_rate:.1f}%). Review key access patterns - many keys may be expired or evicted before use.",
            })
        elif hit_rate < 95 and hit_rate >= 80:
            recs.append({
                "severity": "info",
                "category": "cache",
                "message": f"Cache hit rate at {hit_rate:.1f}% — could be improved. Check if working set fits in memory.",
            })

    # Evicted keys
    if result.cache:
        evicted = result.cache.get("evicted_keys", 0)
        if evicted > 0:
            recs.append({
                "severity": "warning",
                "category": "memory",
                "message": f"Redis is evicting keys ({_format_number(evicted)} evicted). Increase maxmemory or reduce dataset size.",
            })

    # Rejected connections
    if result.overview:
        rejected = result.overview.get("rejected_connections", 0)
        if rejected > 0:
            recs.append({
                "severity": "warning",
                "category": "connections",
                "message": f"Connections being rejected ({_format_number(rejected)}). Check maxclients setting.",
            })

    # Blocked clients
    if result.overview:
        blocked = result.overview.get("blocked_clients", 0)
        if blocked > 0:
            recs.append({
                "severity": "info",
                "category": "connections",
                "message": f"Blocked clients detected ({blocked}). Check for blocking operations (BLPOP, BRPOP, etc.).",
            })

    # maxmemory not set — on Railway this is expected; autoscaling handles growth

    # RDB save failure
    if result.persistence:
        rdb_status = result.persistence.get("rdb_last_bgsave_status", "")
        if rdb_status and rdb_status != "ok":
            recs.append({
                "severity": "critical",
                "category": "persistence",
                "message": "Last RDB save failed. Check disk space and permissions.",
            })

    # Slow log — data-driven when entries are available
    if result.slowlog_entries:
        # Analyze the actual slow commands
        total_entries = len(result.slowlog_entries)
        cmd_counts: Dict[str, int] = {}
        total_duration = 0
        for entry in result.slowlog_entries:
            cmd = entry["command"].split()[0] if entry["command"] else "unknown"
            cmd_counts[cmd] = cmd_counts.get(cmd, 0) + 1
            total_duration += entry["duration_us"]
        top_cmd = max(cmd_counts, key=cmd_counts.get) if cmd_counts else "unknown"
        top_count = cmd_counts.get(top_cmd, 0)
        avg_duration = total_duration / total_entries if total_entries > 0 else 0

        msg = (f"Slow log contains {result.slowlog_len or total_entries} entries. "
               f"Of the {total_entries} most recent: {top_count} are {top_cmd} commands "
               f"averaging {_format_usec(avg_duration)}.")
        if result.big_keys:
            big_key_types = ", ".join(f"{bk['type']} ({bk['detail']})" for bk in result.big_keys[:3])
            msg += f" Largest keys: {big_key_types} — check if these correlate with slow commands."
        severity = "warning" if (result.slowlog_len or 0) > 100 else "info"
        recs.append({"severity": severity, "category": "performance", "message": msg})
    elif result.slowlog_len is not None and result.slowlog_len > 100:
        recs.append({
            "severity": "warning",
            "category": "performance",
            "message": f"High number of slow log entries ({result.slowlog_len}). Slow log details could not be collected.",
        })

    # Big keys — standalone recommendation when no slowlog correlation
    if result.big_keys and not result.slowlog_entries:
        big_key_summary = "; ".join(f"{bk['key']} ({bk['type']}: {bk['detail']})" for bk in result.big_keys[:5])
        recs.append({
            "severity": "info",
            "category": "performance",
            "message": f"Largest keys by type: {big_key_summary}. Large keys can cause latency spikes on read/delete operations.",
        })

    return recs


# ---------------------------------------------------------------------------
# Report formatting
# ---------------------------------------------------------------------------

def format_report(result: RedisAnalysisResult) -> str:
    """Format the analysis result as a markdown report."""
    lines: List[str] = []

    lines.append(f"# Redis Analysis: {result.service}")
    lines.append(f"Timestamp: {result.timestamp}")
    lines.append(f"Deployment Status: {result.deployment_status}")
    lines.append("")

    # --- Overview ---
    if result.overview:
        o = result.overview
        lines.append("## Overview")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Version | {o.get('redis_version', 'N/A')} |")
        lines.append(f"| Uptime | {_format_uptime(o.get('uptime_in_seconds', 0))} |")
        lines.append(f"| Connected Clients | {o.get('connected_clients', 0):,} |")
        lines.append(f"| Blocked Clients | {o.get('blocked_clients', 0):,} |")
        lines.append(f"| Rejected Connections | {o.get('rejected_connections', 0):,} |")
        lines.append(f"| Total Keys | {result.total_keys:,} |")
        lines.append("")

    # --- Memory ---
    if result.memory:
        m = result.memory
        lines.append("## Memory")
        lines.append("| Metric | Value | Status |")
        lines.append("|--------|-------|--------|")

        frag = m.get("mem_fragmentation_ratio", 0)
        frag_status = "OK" if 1.0 <= frag <= 1.5 else ("HIGH" if frag > 1.5 else "LOW")

        lines.append(f"| Used Memory | {m.get('used_memory_human', 'N/A')} | |")
        lines.append(f"| RSS Memory | {m.get('used_memory_rss_human', 'N/A')} | |")
        lines.append(f"| Peak Memory | {m.get('used_memory_peak_human', 'N/A')} | |")
        lines.append(f"| Fragmentation Ratio | {frag:.2f} | {frag_status} |")

        maxmem = m.get("maxmemory", 0)
        if maxmem > 0:
            lines.append(f"| Max Memory | {m.get('maxmemory_human', _format_bytes_human(maxmem))} | |")
        else:
            lines.append("| Max Memory | Unlimited | |")

        lines.append(f"| Eviction Policy | {m.get('maxmemory_policy', 'N/A')} | |")
        lines.append("")

    # --- Throughput ---
    if result.throughput:
        t = result.throughput
        lines.append("## Throughput")
        lines.append("| Metric | Value |")
        lines.append("|--------|-------|")
        lines.append(f"| Ops/sec | {t.get('instantaneous_ops_per_sec', 0):,} |")
        lines.append(f"| Total Commands | {_format_number(t.get('total_commands_processed', 0))} |")
        lines.append(f"| Total Connections | {_format_number(t.get('total_connections_received', 0))} |")
        if result.slowlog_len is not None:
            lines.append(f"| Slow Log Entries | {result.slowlog_len:,} |")
        lines.append("")

    # --- Cache Performance ---
    if result.cache:
        c = result.cache
        hit_rate = c.get("hit_rate", 0)
        hit_status = "OK" if hit_rate >= 95 else ("WARN" if hit_rate >= 80 else "LOW")
        evicted = c.get("evicted_keys", 0)
        evict_status = "OK" if evicted == 0 else "WARN"

        lines.append("## Cache Performance")
        lines.append("| Metric | Value | Status |")
        lines.append("|--------|-------|--------|")
        lines.append(f"| Hit Rate | {hit_rate:.1f}% | {hit_status} |")
        lines.append(f"| Hits | {_format_number(c.get('keyspace_hits', 0))} | |")
        lines.append(f"| Misses | {_format_number(c.get('keyspace_misses', 0))} | |")
        lines.append(f"| Expired Keys | {_format_number(c.get('expired_keys', 0))} | |")
        lines.append(f"| Evicted Keys | {_format_number(evicted)} | {evict_status} |")
        lines.append("")

    # --- Persistence ---
    if result.persistence:
        p = result.persistence
        rdb_status = p.get("rdb_last_bgsave_status", "unknown")
        rdb_status_display = "OK" if rdb_status == "ok" else "FAIL"

        lines.append("## Persistence")
        lines.append("| Metric | Value | Status |")
        lines.append("|--------|-------|--------|")

        rdb_last_save = p.get("rdb_last_save_time", 0)
        if rdb_last_save > 0:
            now_epoch = int(datetime.now(timezone.utc).timestamp())
            save_ago = now_epoch - rdb_last_save
            lines.append(f"| RDB Last Save | {_format_duration(save_ago)} | |")
        else:
            lines.append("| RDB Last Save | never | |")

        lines.append(f"| RDB Status | {rdb_status} | {rdb_status_display} |")
        lines.append(f"| AOF Enabled | {'Yes' if p.get('aof_enabled') else 'No'} | |")

        if p.get("aof_enabled"):
            aof_status = p.get("aof_last_rewrite_status", "unknown")
            aof_display = "OK" if aof_status == "ok" else aof_status
            lines.append(f"| AOF Rewrite Status | {aof_status} | {aof_display} |")

        lines.append("")

    # --- Command Stats ---
    if result.command_stats:
        top_n = result.command_stats[:20]
        lines.append("## Command Stats (top 20)")
        lines.append("| Command | Calls | Avg Latency | Total Time |")
        lines.append("|---------|-------|-------------|------------|")
        for cs in top_n:
            lines.append(
                f"| {cs['command']} "
                f"| {_format_number(cs['calls'])} "
                f"| {_format_usec(cs['usec_per_call'])} "
                f"| {_format_total_time(cs['usec'])} |"
            )
        lines.append("")

    # --- Slow Log Entries ---
    if result.slowlog_entries:
        lines.append("## Slow Log Entries (recent)")
        lines.append("| # | Timestamp | Duration | Command |")
        lines.append("|---|-----------|----------|---------|")
        now_epoch = int(datetime.now(timezone.utc).timestamp())
        for entry in result.slowlog_entries:
            age = now_epoch - entry["timestamp_unix"]
            lines.append(
                f"| {entry['id']} "
                f"| {_format_duration(age)} "
                f"| {_format_usec(entry['duration_us'])} "
                f"| {entry['command']} |"
            )
        lines.append("")

    # --- Biggest Keys ---
    if result.big_keys:
        lines.append("## Biggest Keys")
        lines.append("| Type | Key | Size/Count |")
        lines.append("|------|-----|------------|")
        for bk in result.big_keys:
            lines.append(
                f"| {bk['type']} "
                f"| {bk['key']} "
                f"| {bk['detail']} |"
            )
        lines.append("")

    # --- Keyspace ---
    if result.keyspace:
        lines.append("## Keyspace")
        lines.append("| Database | Keys | Expires | Avg TTL |")
        lines.append("|----------|------|---------|---------|")
        for db in result.keyspace:
            lines.append(
                f"| {db['db']} "
                f"| {db['keys']:,} "
                f"| {db['expires']:,} "
                f"| {_format_ttl(db['avg_ttl'])} |"
            )
        lines.append("")

    # --- Infrastructure Metrics ---
    if result.metrics_history:
        windows = result.metrics_history.get("windows", {})
        for window_label, window_data in windows.items():
            mh = window_data.get("metrics", {})
            if not mh:
                continue
            lines.append(f"## Infrastructure Metrics ({window_label})")
            lines.append("| Metric | Current | Min | Max | Avg | Trend |")
            lines.append("|--------|---------|-----|-----|-----|-------|")
            for key in ["cpu", "memory", "disk", "network_rx", "network_tx"]:
                if key in mh:
                    entry = mh[key]
                    trend = entry.get("trend", {})
                    trend_str = trend.get("direction", "N/A")
                    change = trend.get("change_pct", 0)
                    if change != 0:
                        trend_str += f" ({change:+.1f}%)"
                    lines.append(
                        f"| {key.replace('_', ' ').title()} "
                        f"| {entry['current']}{entry['unit']} "
                        f"| {entry['min']}{entry['unit']} "
                        f"| {entry['max']}{entry['unit']} "
                        f"| {entry['avg']}{entry['unit']} "
                        f"| {trend_str} |"
                    )
            lines.append("")

    # --- Collection Status ---
    if result.collection_status:
        failed = {k: v for k, v in result.collection_status.items() if v.get("status") == "failed"}
        if failed:
            lines.append("## Collection Issues")
            for source, status in failed.items():
                lines.append(f"- **{source}**: {status.get('error', 'unknown error')}")
            lines.append("")

    # --- Recommendations ---
    if result.recommendations:
        lines.append("## Recommendations")
        for rec in result.recommendations:
            severity = rec.get("severity", "info").upper()
            lines.append(f"- [{severity}] {rec['message']}")
        lines.append("")

    return "\n".join(lines)


# ---------------------------------------------------------------------------
# Main analysis function
# ---------------------------------------------------------------------------

def analyze_redis(service: str, timeout: int = 300, quiet: bool = False,
                  skip_logs: bool = False,
                  metrics_hours: int = 168,
                  project_id: Optional[str] = None,
                  environment_id: Optional[str] = None,
                  service_id: Optional[str] = None) -> RedisAnalysisResult:
    """Run complete Redis analysis with maximum data collection.

    Collects Redis INFO ALL, SLOWLOG LEN, SLOWLOG GET 20, --bigkeys,
    Railway metrics, and logs in parallel where possible.

    Args:
        skip_logs: Skip log fetching for faster analysis
        metrics_hours: Hours of metrics history to fetch (default: 168, max: 168)
        project_id: Project ID (bypasses railway link config)
        environment_id: Environment ID (bypasses railway link config)
        service_id: Service ID (bypasses railway link config)
    """
    if not quiet:
        print(f"Analyzing redis database: {service}", file=sys.stderr)

    result = RedisAnalysisResult(
        service=service,
        db_type="redis",
        timestamp=datetime.now(timezone.utc).isoformat(),
    )

    # === FAST CONTEXT LOADING ===
    if not quiet:
        print("  [0/5] Getting Railway context...", file=sys.stderr, flush=True)
    dal._progress_timer.start()

    if environment_id and service_id:
        dal._ctx = RailwayContext(project_id=project_id, environment_id=environment_id, service_id=service_id)
        if not quiet:
            print(f"        using explicit IDs (env={environment_id[:8]}..., svc={service_id[:8]}...)", file=sys.stderr, flush=True)
    else:
        railway_status = get_railway_status()
        if railway_status:
            dal._ctx = RailwayContext(
                project_id=railway_status.get("projectId"),
                environment_id=railway_status.get("environmentId"),
                service_id=railway_status.get("serviceId"),
            )
        environment_id = dal._ctx.environment_id
        service_id = dal._ctx.service_id

    # Get deployment status via API (~1s)
    progress(1, 5, "Fetching deployment status...", quiet)
    result.deployment_status = get_deployment_status(service, service_id=service_id)

    # === SSH PRE-CHECK WITH RETRY ===
    progress(2, 5, "Testing SSH connectivity...", quiet)
    ssh_available = False
    ssh_stderr = ""
    ssh_attempts = [30, 60, 90]
    for attempt, attempt_timeout in enumerate(ssh_attempts, 1):
        ssh_code, ssh_stdout, ssh_stderr = run_ssh_query(service, "echo ok", timeout=attempt_timeout)
        if ssh_code == 0 and "ok" in ssh_stdout:
            ssh_available = True
            if not quiet:
                for line in ssh_stderr.splitlines():
                    if line.startswith("Using SSH key:"):
                        print(f"        {line}", file=sys.stderr, flush=True)
                        break
            break
        if not quiet:
            remaining = len(ssh_attempts) - attempt
            if remaining > 0:
                print(f"        SSH attempt {attempt}/{len(ssh_attempts)} failed ({ssh_stderr or 'no response'}), retrying with {ssh_attempts[attempt]}s timeout...", file=sys.stderr, flush=True)
            else:
                print(f"        SSH attempt {attempt}/{len(ssh_attempts)} failed ({ssh_stderr or 'no response'}), giving up", file=sys.stderr, flush=True)

    # === PARALLEL EXECUTION ===
    progress(3, 5, "Running analysis (Redis INFO, slowlog, bigkeys, metrics, logs in parallel)...", quiet)

    def task_redis_info():
        """Fetch Redis INFO ALL via SSH."""
        if not ssh_available:
            return ("failed", f"SSH not available: {ssh_stderr or 'connection failed'}", "")
        command = 'timeout 30s redis-cli -h localhost -p 6379 -a "$REDISPASSWORD" --no-auth-warning --raw INFO ALL'
        code, stdout, stderr = run_ssh_query(service, command, timeout=45)
        if code == 0 and stdout.strip():
            return ("ok", "", stdout)
        return ("failed", stderr or "empty response", stdout)

    def task_slowlog():
        """Fetch Redis SLOWLOG LEN via SSH."""
        if not ssh_available:
            return ("failed", f"SSH not available: {ssh_stderr or 'connection failed'}", "")
        command = 'timeout 30s redis-cli -h localhost -p 6379 -a "$REDISPASSWORD" --no-auth-warning --raw SLOWLOG LEN'
        code, stdout, stderr = run_ssh_query(service, command, timeout=45)
        if code == 0 and stdout.strip():
            return ("ok", "", stdout.strip())
        return ("failed", stderr or "empty response", "")

    def task_slowlog_get():
        """Fetch Redis SLOWLOG GET 20 via SSH for actual slow query details."""
        if not ssh_available:
            return ("failed", f"SSH not available: {ssh_stderr or 'connection failed'}", "")
        command = 'timeout 30s redis-cli -h localhost -p 6379 -a "$REDISPASSWORD" --no-auth-warning --raw SLOWLOG GET 20'
        code, stdout, stderr = run_ssh_query(service, command, timeout=45)
        if code == 0 and stdout.strip():
            return ("ok", "", stdout.strip())
        return ("failed", stderr or "empty response", "")

    def task_bigkeys():
        """Fetch redis-cli --bigkeys via SSH (SCAN-based, may take longer)."""
        if not ssh_available:
            return ("failed", f"SSH not available: {ssh_stderr or 'connection failed'}", "")
        command = 'timeout 60s redis-cli -h localhost -p 6379 -a "$REDISPASSWORD" --no-auth-warning --bigkeys'
        code, stdout, stderr = run_ssh_query(service, command, timeout=75)
        if code == 0 and stdout.strip():
            return ("ok", "", stdout.strip())
        return ("failed", stderr or "empty response", "")

    def task_metrics():
        """Fetch all metrics (disk, CPU, memory) in one API call."""
        if environment_id and service_id:
            return get_all_metrics_from_api(environment_id, service_id, hours=metrics_hours)
        return None

    def task_logs():
        """Fetch recent logs via API (~3s)."""
        if skip_logs:
            return []
        return get_recent_logs(service, lines=LOG_LINES_DEFAULT,
                               environment_id=environment_id,
                               service_id=service_id)

    with ThreadPoolExecutor(max_workers=6) as executor:
        future_info = executor.submit(task_redis_info)
        future_slowlog = executor.submit(task_slowlog)
        future_slowlog_get = executor.submit(task_slowlog_get)
        future_bigkeys = executor.submit(task_bigkeys)
        future_metrics = executor.submit(task_metrics)
        future_logs = executor.submit(task_logs)

        # Collect results
        info_result = future_info.result()
        slowlog_result = future_slowlog.result()
        slowlog_get_result = future_slowlog_get.result()
        bigkeys_result = future_bigkeys.result()
        metrics_result = future_metrics.result()
        logs_result = future_logs.result()

    # === PROCESS RESULTS ===
    progress(4, 5, "Processing results...", quiet)

    # Redis INFO ALL
    info_status, info_error, info_raw = info_result
    if info_status == "ok" and info_raw:
        result.collection_status["redis_info"] = {"status": "ok"}
        info = parse_redis_info(info_raw)

        result.overview = extract_overview(info)
        result.memory = extract_memory(info)
        result.throughput = extract_throughput(info)
        result.cache = extract_cache(info)
        result.persistence = extract_persistence(info)
        result.keyspace, result.total_keys = extract_keyspace(info)
        result.command_stats = extract_command_stats(info)
    else:
        result.collection_status["redis_info"] = {"status": "failed", "error": info_error}
        result.errors.append(f"Redis INFO failed: {info_error}")

    # SLOWLOG LEN
    sl_status, sl_error, sl_raw = slowlog_result
    if sl_status == "ok" and sl_raw:
        result.collection_status["slowlog"] = {"status": "ok"}
        result.slowlog_len = _safe_int(sl_raw)
    else:
        result.collection_status["slowlog"] = {"status": "failed", "error": sl_error}

    # SLOWLOG GET 20
    slg_status, slg_error, slg_raw = slowlog_get_result
    if slg_status == "ok" and slg_raw:
        result.collection_status["slowlog_entries"] = {"status": "ok"}
        result.slowlog_entries = parse_slowlog_get(slg_raw)
    else:
        result.collection_status["slowlog_entries"] = {"status": "failed", "error": slg_error}

    # Big keys
    bk_status, bk_error, bk_raw = bigkeys_result
    if bk_status == "ok" and bk_raw:
        result.collection_status["big_keys"] = {"status": "ok"}
        result.big_keys = parse_bigkeys(bk_raw)
    else:
        result.collection_status["big_keys"] = {"status": "failed", "error": bk_error}

    # Metrics
    if metrics_result:
        result.collection_status["metrics"] = {"status": "ok"}
        result.metrics_history = metrics_result.get("metrics_history")
    else:
        result.collection_status["metrics"] = {"status": "failed", "error": "no metrics returned"}

    # Logs
    if logs_result:
        result.collection_status["logs"] = {"status": "ok", "lines": len(logs_result)}
        result.recent_logs = logs_result
    else:
        result.collection_status["logs"] = {"status": "failed", "error": "no logs returned"}

    # === RECOMMENDATIONS ===
    progress(5, 5, "Generating recommendations...", quiet)
    result.recommendations = generate_recommendations(result)

    if not quiet:
        elapsed = dal._progress_timer.step_elapsed()
        if elapsed:
            print(f"        done{elapsed}", file=sys.stderr, flush=True)
        print(f"  Analysis complete{dal._progress_timer.total_elapsed()}", file=sys.stderr, flush=True)

    return result


# ---------------------------------------------------------------------------
# Single-step debugging
# ---------------------------------------------------------------------------

def run_single_step(args) -> int:
    """Run a single collection step for debugging."""
    service = args.service
    _init_context(args)
    environment_id = dal._ctx.environment_id
    service_id = dal._ctx.service_id

    if args.step == "ssh-test":
        print(f"Testing SSH to service: {service}", file=sys.stderr)
        code, stdout, stderr = run_ssh_query(service, "echo ok", timeout=45)
        print(f"Exit code: {code}")
        print(f"Stdout: {stdout.strip()}")
        if stderr:
            print(f"Stderr: {stderr.strip()}")
        return 0 if (code == 0 and "ok" in stdout) else 1

    elif args.step == "query":
        print(f"Running Redis INFO ALL on: {service}", file=sys.stderr)
        command = 'timeout 30s redis-cli -h localhost -p 6379 -a "$REDISPASSWORD" --no-auth-warning --raw INFO ALL'
        code, stdout, stderr = run_ssh_query(service, command, timeout=45)
        print(f"Exit code: {code}")
        if code == 0 and stdout:
            info = parse_redis_info(stdout)
            print(json.dumps(info, indent=2))
        else:
            print(f"Error: {stderr or stdout}")
        return code

    elif args.step == "logs":
        print(f"Fetching logs for: {service}", file=sys.stderr)
        logs = get_recent_logs(service, lines=LOG_LINES_DEFAULT,
                               environment_id=environment_id,
                               service_id=service_id)
        print(f"Lines fetched: {len(logs)}")
        for line in logs:
            print(line)
        return 0

    elif args.step == "metrics":
        print(f"Fetching metrics for: {service}", file=sys.stderr)
        if environment_id and service_id:
            metrics = get_all_metrics_from_api(environment_id, service_id)
            if metrics:
                print(json.dumps(metrics, indent=2))
            else:
                print("No metrics returned", file=sys.stderr)
                return 1
        else:
            print("No environment_id or service_id available", file=sys.stderr)
            return 1
        return 0

    else:
        print(f"Unknown step: {args.step}", file=sys.stderr)
        return 1


# ---------------------------------------------------------------------------
# CLI entry point
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(
        description="Redis analysis for Railway services.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )

    parser.add_argument("--service", required=True, help="Service name")
    parser.add_argument("--json", action="store_true",
                       help="Output as JSON")
    parser.add_argument("--timeout", type=int, default=300,
                       help="Timeout in seconds for analysis (default: 300)")
    parser.add_argument("--quiet", "-q", action="store_true",
                       help="Suppress progress messages")
    parser.add_argument("--skip-logs", action="store_true",
                       help="Skip log fetching for faster analysis")
    parser.add_argument("--metrics-hours", type=int, default=168,
                       help="Hours of metrics history to fetch (default: 168, max: 168)")
    parser.add_argument("--step", choices=["ssh-test", "query", "logs", "metrics"],
                       help="Run a single collection step for debugging")
    parser.add_argument("--project-id", help="Project ID (bypasses railway link)")
    parser.add_argument("--environment-id", help="Environment ID (bypasses railway link)")
    parser.add_argument("--service-id", help="Service ID (bypasses railway link)")

    args = parser.parse_args()

    # Single-step debugging mode
    if args.step:
        return run_single_step(args)

    # Run analysis
    result = analyze_redis(args.service, timeout=args.timeout, quiet=args.quiet,
                           skip_logs=args.skip_logs,
                           metrics_hours=min(args.metrics_hours, 168),
                           project_id=args.project_id,
                           environment_id=args.environment_id,
                           service_id=args.service_id)

    # Output
    if args.json:
        print(json.dumps(asdict(result), indent=2))
    else:
        print(format_report(result))

    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""Shared Railway infrastructure helpers for database analysis scripts."""

import json
import os
import subprocess
import sys
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional, Tuple
from dataclasses import dataclass

LOG_LINES_DEFAULT = 1000  # Number of log lines to fetch via API


class ProgressTimer:
    """Track elapsed time for progress messages."""
    def __init__(self):
        self.start_time = None
        self.step_start_time = None

    def start(self):
        """Start the overall timer."""
        self.start_time = datetime.now()
        self.step_start_time = self.start_time

    def step_elapsed(self) -> str:
        """Get elapsed time since last step, then reset step timer."""
        if self.step_start_time is None:
            return ""
        now = datetime.now()
        elapsed = (now - self.step_start_time).total_seconds()
        self.step_start_time = now
        if elapsed < 0.1:
            return ""
        return f" ({elapsed:.1f}s)"

    def total_elapsed(self) -> str:
        """Get total elapsed time."""
        if self.start_time is None:
            return ""
        elapsed = (datetime.now() - self.start_time).total_seconds()
        return f" (total: {elapsed:.1f}s)"


# Global timer instance
_progress_timer = ProgressTimer()


@dataclass
class RailwayContext:
    """Explicit Railway IDs that bypass railway link."""
    project_id: Optional[str] = None
    environment_id: Optional[str] = None
    service_id: Optional[str] = None

    def ssh_flags(self) -> List[str]:
        """Return CLI flags for railway ssh."""
        flags = ["--native"]
        if self.project_id:
            flags.extend(["--project", self.project_id])
        if self.environment_id:
            flags.extend(["--environment", self.environment_id])
        if self.service_id:
            flags.extend(["--service", self.service_id])
        return flags

    def logs_flags(self) -> List[str]:
        """Return CLI flags for railway logs."""
        flags = []
        if self.environment_id:
            flags.extend(["--environment", self.environment_id])
        return flags


# Global context — set once at startup, used by all CLI calls
_ctx = RailwayContext()


def _init_context(args) -> None:
    """Initialize global context from CLI args or railway config."""
    global _ctx
    if args.environment_id and args.service_id:
        _ctx = RailwayContext(
            project_id=getattr(args, 'project_id', None),
            environment_id=args.environment_id,
            service_id=args.service_id,
        )
    else:
        railway_status = get_railway_status()
        if railway_status:
            _ctx = RailwayContext(
                project_id=railway_status.get("projectId"),
                environment_id=railway_status.get("environmentId"),
                service_id=railway_status.get("serviceId"),
            )


def progress(step: int, total: int, message: str, quiet: bool = False):
    """Print progress message to stderr with elapsed time."""
    if not quiet:
        # Show elapsed time from previous step (before current step message)
        elapsed = _progress_timer.step_elapsed()
        if elapsed:
            print(f"        done{elapsed}", file=sys.stderr, flush=True)
        print(f"  [{step}/{total}] {message}", file=sys.stderr, flush=True)


def run_railway_command(args: List[str], timeout: int = 30) -> Tuple[int, str, str]:
    """Run a railway CLI command and return (returncode, stdout, stderr)."""
    try:
        result = subprocess.run(
            ["railway"] + args,
            capture_output=True,
            text=True,
            timeout=timeout
        )
        return result.returncode, result.stdout, result.stderr
    except subprocess.TimeoutExpired:
        return 124, "", "Command timed out"
    except FileNotFoundError:
        return 127, "", "railway CLI not found"


def _cli_fatal_error(returncode: int, stderr: str) -> Optional[str]:
    """Return a friendly error string if the CLI itself is broken, else None.

    These errors are unrecoverable — retrying won't help.
    """
    if returncode == 127 or "railway CLI not found" in stderr:
        return (
            "Railway CLI not found. "
            "Install it with: npm i -g @railway/cli  "
            "or  brew install railway"
        )
    lower = stderr.lower()
    if "unknown flag" in lower or "flag provided but not defined" in lower:
        return (
            "Railway CLI is outdated — the --native SSH flag is not supported. "
            "Update it with: npm i -g @railway/cli@latest  "
            "or  brew upgrade railway"
        )
    return None


def run_ssh_query(service: str, command: str, timeout: int = 60,
                  max_attempts: int = 3) -> Tuple[int, str, str]:
    """Run a command via railway ssh, retrying up to max_attempts times.

    Passes the command as a single argument after '--'. Railway ssh
    interprets it through a shell on the remote end, so pipes, env vars,
    and redirects all work without an explicit sh -c wrapper.

    Retries on non-zero exit code or empty stdout (covers transient errors
    like 'exec request failed on channel 0').  Never retries when the CLI
    itself is missing or outdated — those errors are unrecoverable.
    """
    flags = _ctx.ssh_flags()
    # Only pass --service <name> if context didn't already provide --service <id>
    if not _ctx.service_id:
        flags += ["--service", service]
    args = ["ssh"] + flags + ["--", command]
    last_code, last_stdout, last_stderr = 1, "", ""
    for attempt in range(1, max_attempts + 1):
        last_code, last_stdout, last_stderr = run_railway_command(args, timeout)
        if last_code == 0 and last_stdout.strip():
            return last_code, last_stdout, last_stderr
        fatal = _cli_fatal_error(last_code, last_stderr)
        if fatal:
            return last_code, last_stdout, fatal
        if attempt < max_attempts:
            print(
                f"        SSH attempt {attempt}/{max_attempts} failed "
                f"({last_stderr.strip() or 'empty response'}), retrying...",
                file=sys.stderr, flush=True,
            )
    return last_code, last_stdout, last_stderr


def run_psql_query(service: str, query: str, timeout: int = 60) -> Tuple[int, str]:
    """Run a psql query via railway ssh and return (returncode, output).

    Normalizes query whitespace and suppresses psql warnings (e.g. collation
    version mismatch) that would otherwise pollute stdout.
    """
    query = " ".join(query.split())
    command = f'''PAGER='' psql $DATABASE_URL -P pager=off -t -A -c "{query}" 2>/dev/null'''
    code, stdout, stderr = run_ssh_query(service, command, timeout)
    if code != 0:
        return code, stderr or stdout
    return 0, stdout


def get_railway_status() -> Optional[Dict[str, Any]]:
    """Get environment and service IDs from Railway config file.

    Reads directly from ~/.railway/config.json instead of calling CLI (~15s saved).
    """
    config_path = os.path.expanduser("~/.railway/config.json")
    if not os.path.exists(config_path):
        return None

    try:
        with open(config_path, "r") as f:
            config = json.load(f)

        # Get linked project for current directory
        cwd = os.getcwd()
        projects = config.get("projects", {})

        # Find project config for current directory or parent
        project_config = None
        check_path = cwd
        while check_path != "/":
            if check_path in projects:
                project_config = projects[check_path]
                break
            check_path = os.path.dirname(check_path)

        if not project_config:
            return None

        return {
            "projectId": project_config.get("project"),
            "environmentId": project_config.get("environment"),
            "serviceId": project_config.get("service"),
            "serviceName": project_config.get("name", ""),
        }
    except (json.JSONDecodeError, IOError):
        return None


def get_deployment_status(service: str, service_id: Optional[str] = None) -> str:
    """Get deployment status for service.

    Uses direct API call if service_id provided (~1s), falls back to CLI (~15s).
    """
    # Fast path: use API directly if we have service_id
    if service_id:
        script_dir = os.path.dirname(os.path.abspath(__file__))
        api_script = os.path.join(script_dir, "railway-api.sh")

        if os.path.exists(api_script):
            query = '''query svc($id: String!) {
                service(id: $id) {
                    deployments(first: 1) {
                        edges { node { status } }
                    }
                }
            }'''
            try:
                result = subprocess.run(
                    [api_script, query, json.dumps({"id": service_id})],
                    capture_output=True,
                    text=True,
                    timeout=10
                )
                if result.returncode == 0:
                    data = json.loads(result.stdout)
                    edges = data.get("data", {}).get("service", {}).get("deployments", {}).get("edges", [])
                    if edges:
                        return edges[0].get("node", {}).get("status", "UNKNOWN")
            except (subprocess.TimeoutExpired, json.JSONDecodeError):
                pass

    # Fallback: use CLI (slow, ~15s)
    code, stdout, stderr = run_railway_command(
        ["service", "status", "--service", service, "--json"]
    )
    if code != 0:
        return "UNKNOWN"
    try:
        data = json.loads(stdout)
        status = data.get("status", "UNKNOWN")
        if data.get("stopped"):
            return f"{status} (stopped)"
        return status
    except json.JSONDecodeError:
        return "UNKNOWN"


def get_all_metrics_from_api(environment_id: str, service_id: str, hours: int = 24) -> Optional[Dict[str, Any]]:
    """Get disk, CPU, memory, and network usage from Railway metrics API.

    Fetches time-series data and computes trend analysis including
    min/max/avg, spike detection, and directional trends.

    Args:
        hours: Hours of history to fetch (default: 24, max: 168)
    """
    from datetime import timedelta

    start_date = (datetime.now(timezone.utc) - timedelta(hours=hours)).isoformat()
    script_dir = os.path.dirname(os.path.abspath(__file__))
    api_script = os.path.join(script_dir, "railway-api.sh")

    if not os.path.exists(api_script):
        return None

    query = '''query metrics($environmentId: String!, $serviceId: String, $startDate: DateTime!, $measurements: [MetricMeasurement!]!) {
        metrics(environmentId: $environmentId, serviceId: $serviceId, startDate: $startDate, measurements: $measurements) {
            measurement values { ts value }
        }
    }'''

    variables = json.dumps({
        "environmentId": environment_id,
        "serviceId": service_id,
        "startDate": start_date,
        "measurements": [
            "DISK_USAGE_GB", "CPU_USAGE", "MEMORY_USAGE_GB",
            "MEMORY_LIMIT_GB", "CPU_LIMIT",
            "NETWORK_RX_GB", "NETWORK_TX_GB",
        ]
    })

    try:
        result = subprocess.run(
            [api_script, query, variables],
            capture_output=True,
            text=True,
            timeout=30
        )
        if result.returncode != 0:
            return None

        data = json.loads(result.stdout)
        metrics = data.get("data", {}).get("metrics", [])

        combined = {"disk_usage": None, "cpu_memory": {}, "metrics_history": None}

        # Raw time series keyed by measurement name
        raw_series: Dict[str, List[Dict[str, Any]]] = {}

        for metric in metrics:
            measurement = metric.get("measurement")
            values = metric.get("values", [])
            if values:
                raw_series[measurement] = values
                latest = values[-1].get("value", 0)
                if measurement == "DISK_USAGE_GB":
                    combined["disk_usage"] = {
                        "used_gb": round(latest, 2),
                        "used": f"{round(latest, 1)} GB"
                    }
                elif measurement == "CPU_USAGE":
                    combined["cpu_memory"]["cpu_percent"] = round(latest, 1)
                elif measurement == "MEMORY_USAGE_GB":
                    combined["cpu_memory"]["memory_gb"] = round(latest, 2)
                elif measurement == "MEMORY_LIMIT_GB":
                    combined["cpu_memory"]["memory_limit_gb"] = round(latest, 2)
                elif measurement == "CPU_LIMIT":
                    combined["cpu_memory"]["cpu_limit"] = round(latest, 1)

        if not combined["cpu_memory"]:
            combined["cpu_memory"] = None

        # Build time-series history with trend analysis
        if raw_series:
            combined["metrics_history"] = _build_metrics_history(raw_series, hours=hours)

        return combined
    except (subprocess.TimeoutExpired, json.JSONDecodeError, FileNotFoundError):
        pass

    return None


def _analyze_window(values: List[Dict[str, Any]], nums: List[float], d: int,
                    unit: str) -> Dict[str, Any]:
    """Analyze a single time window of metric data.

    Returns summary stats, trend, spike detection, and downsampled series.
    """
    if not nums:
        return {}

    avg_val = sum(nums) / len(nums)
    min_val = min(nums)
    max_val = max(nums)

    entry: Dict[str, Any] = {
        "unit": unit,
        "current": round(nums[-1], d),
        "min": round(min_val, d),
        "max": round(max_val, d),
        "avg": round(avg_val, d),
        "samples": len(nums),
    }

    # Trend: compare first quarter avg to last quarter avg
    q_size = max(len(nums) // 4, 1)
    first_q = nums[:q_size]
    last_q = nums[-q_size:]
    first_avg = sum(first_q) / len(first_q)
    last_avg = sum(last_q) / len(last_q)

    if first_avg > 0:
        change_pct = round(((last_avg - first_avg) / first_avg) * 100, 1)
    elif last_avg > 0:
        change_pct = 100.0
    else:
        change_pct = 0.0

    if change_pct > 10:
        trend_dir = "increasing"
    elif change_pct < -10:
        trend_dir = "decreasing"
    else:
        trend_dir = "stable"

    entry["trend"] = {
        "direction": trend_dir,
        "change_pct": change_pct,
        "first_quarter_avg": round(first_avg, d),
        "last_quarter_avg": round(last_avg, d),
    }

    # Spike detection
    if len(nums) >= 10:
        variance = sum((x - avg_val) ** 2 for x in nums) / len(nums)
        stddev = variance ** 0.5
        threshold = avg_val + 2 * stddev
        if stddev > 0 and threshold > 0:
            spikes = []
            for v in values:
                val = v.get("value")
                if val is not None and val > threshold:
                    spikes.append({"ts": v["ts"], "value": round(val, d)})
            if spikes:
                entry["spikes"] = {
                    "count": len(spikes),
                    "threshold": round(threshold, d),
                    "peaks": spikes[:10],
                }

    # Compact time series: downsample to ~48 points
    series_points = []
    for v in values:
        ts = v.get("ts")
        val = v.get("value")
        if ts is not None and val is not None:
            series_points.append({"ts": ts, "value": round(val, d)})

    if len(series_points) > 48:
        step = len(series_points) / 48
        downsampled = []
        for i in range(48):
            idx = int(i * step)
            downsampled.append(series_points[idx])
        downsampled.append(series_points[-1])
        entry["series"] = downsampled
    else:
        entry["series"] = series_points

    return entry


def _build_metrics_history(raw_series: Dict[str, List[Dict[str, Any]]], hours: int = 168) -> Dict[str, Any]:
    """Build multi-window time-series history with trend analysis.

    Always produces a full-window analysis. If the window is > 24h, also
    produces a 24h short-window analysis from the tail of the data so the
    LLM can compare long-term vs short-term trends.
    """
    from datetime import timedelta

    metric_info = {
        "CPU_USAGE": {"name": "cpu", "unit": "vCPU", "decimals": 2},
        "MEMORY_USAGE_GB": {"name": "memory", "unit": "GB", "decimals": 2},
        "MEMORY_LIMIT_GB": {"name": "memory_limit", "unit": "GB", "decimals": 2},
        "CPU_LIMIT": {"name": "cpu_limit", "unit": "vCPU", "decimals": 2},
        "DISK_USAGE_GB": {"name": "disk", "unit": "GB", "decimals": 2},
        "NETWORK_RX_GB": {"name": "network_rx", "unit": "GB", "decimals": 3},
        "NETWORK_TX_GB": {"name": "network_tx", "unit": "GB", "decimals": 3},
    }

    # Determine the 24h cutoff timestamp
    now_ts = int(datetime.now(timezone.utc).timestamp())
    cutoff_24h = now_ts - (24 * 3600)

    produce_short_window = hours > 24

    full_window: Dict[str, Any] = {}
    short_window: Dict[str, Any] = {}

    for measurement, values in raw_series.items():
        info = metric_info.get(measurement)
        if not info or len(values) < 2:
            continue

        nums = [v["value"] for v in values if v.get("value") is not None]
        if not nums:
            continue

        d = info["decimals"]
        name = info["name"]

        # Full window analysis
        full_window[name] = _analyze_window(values, nums, d, info["unit"])

        # Short window (last 24h) analysis
        if produce_short_window:
            recent_values = [v for v in values if v.get("ts", 0) >= cutoff_24h]
            recent_nums = [v["value"] for v in recent_values if v.get("value") is not None]
            if len(recent_nums) >= 2:
                short_window[name] = _analyze_window(recent_values, recent_nums, d, info["unit"])

    # Build the result with named windows
    windows: Dict[str, Any] = {}

    # Label the full window
    if hours >= 168:
        full_label = "7d"
    elif hours >= 72:
        full_label = f"{hours // 24}d"
    else:
        full_label = f"{hours}h"

    windows[full_label] = {
        "window_hours": hours,
        "metrics": full_window,
    }

    if produce_short_window and short_window:
        windows["24h"] = {
            "window_hours": 24,
            "metrics": short_window,
        }

    return {"windows": windows}


def info(msg: str) -> None:
    """Print an [INFO] message to stdout."""
    print(f"[INFO] {msg}")


def error(msg: str) -> None:
    """Print an [ERROR] message to stderr and exit."""
    print(f"[ERROR] {msg}", file=sys.stderr)
    sys.exit(1)


def confirm_with_user(prompt: str) -> bool:
    """Get confirmation directly from the terminal.

    Reads from /dev/tty to ensure it's an actual user at a terminal,
    not piped input. This prevents automated scripts from bypassing confirmation.
    """
    try:
        with open('/dev/tty', 'r') as tty:
            print(prompt, end=' ', flush=True)
            response = tty.readline().strip().lower()
            return response in ('y', 'yes')
    except (OSError, IOError):
        print("\n[ERROR] This command requires interactive terminal confirmation.")
        print("It cannot be run with piped input or in non-interactive mode.")
        print("Please run this command directly in a terminal.")
        return False


def _safe_int(val: Any, default: int = 0) -> int:
    """Safely convert a value to int, returning default on failure."""
    try:
        return int(val)
    except (ValueError, TypeError):
        return default


def _safe_float(val: Any, default: float = 0.0) -> float:
    """Safely convert a value to float, returning default on failure."""
    try:
        return float(val)
    except (ValueError, TypeError):
        return default


def _format_uptime(seconds: int) -> str:
    """Format uptime seconds into a human-readable string."""
    if seconds <= 0:
        return "N/A"
    days = seconds // 86400
    hours = (seconds % 86400) // 3600
    minutes = (seconds % 3600) // 60
    parts = []
    if days > 0:
        parts.append(f"{days}d")
    if hours > 0:
        parts.append(f"{hours}h")
    if minutes > 0 and days == 0:
        parts.append(f"{minutes}m")
    return " ".join(parts) if parts else "< 1m"


def _trend_indicator(metrics_history: Optional[Dict[str, Any]], metric_name: str) -> str:
    """Return a compact trend string like ' (^ +15.2% 24h)' for use in summary rows."""
    if not metrics_history:
        return ""
    windows = metrics_history.get("windows", {})
    window_data = windows.get("24h") or next(iter(windows.values()), None) if windows else None
    if not window_data or not window_data.get("metrics"):
        return ""
    m = window_data["metrics"].get(metric_name)
    if not m or "trend" not in m:
        return ""
    t = m["trend"]
    direction = t.get("direction", "stable")
    change = t.get("change_pct", 0)
    window_label = "24h" if "24h" in windows else next(iter(windows), "")
    arrow = {"increasing": "^", "decreasing": "v", "stable": "~"}.get(direction, "")
    if direction == "stable":
        return f" ({arrow} stable {window_label})"
    return f" ({arrow} {change:+.1f}% {window_label})"


def get_recent_logs(service: str, lines: int = LOG_LINES_DEFAULT,
                    environment_id: Optional[str] = None,
                    service_id: Optional[str] = None) -> List[str]:
    """Get recent logs for LLM analysis.

    Uses API if environment_id and service_id provided (~3s),
    retries once with longer timeout on failure,
    falls back to CLI (~27s for 100 lines).
    """
    # Fast path: use API directly
    if environment_id and service_id:
        script_dir = os.path.dirname(os.path.abspath(__file__))
        api_script = os.path.join(script_dir, "railway-api.sh")

        if os.path.exists(api_script):
            # Use environmentLogs API with service filter
            query = f'''query {{
                environmentLogs(
                    environmentId: "{environment_id}",
                    beforeLimit: {lines},
                    filter: "@service:{service_id}"
                ) {{
                    timestamp
                    message
                }}
            }}'''
            # Try API twice: first with 15s timeout, retry with 30s
            for attempt_timeout in [15, 30]:
                try:
                    result = subprocess.run(
                        [api_script, query],
                        capture_output=True,
                        text=True,
                        timeout=attempt_timeout
                    )
                    if result.returncode == 0:
                        data = json.loads(result.stdout)
                        logs_data = data.get("data", {}).get("environmentLogs", [])
                        if logs_data:
                            return [f"{log['timestamp']} {log['message']}" for log in logs_data]
                except (subprocess.TimeoutExpired, json.JSONDecodeError):
                    pass

    # Fallback: use CLI (slow, ~27s)
    code, stdout, stderr = run_railway_command(
        ["logs"] + _ctx.logs_flags() + ["--service", service, "--lines", str(lines)],
        timeout=30
    )
    if code != 0:
        return []

    logs = []
    for line in stdout.strip().split("\n"):
        if line.strip():
            logs.append(line.strip())
    return logs

#!/usr/bin/env python3
"""
Enable pg_stat_statements for query statistics tracking.

This script replicates the frontend's Metrics.tsx enable logic:
1. Check if pg_stat_statements is already in shared_preload_libraries
2. If yes, just install the extension
3. If no, install extension + ALTER SYSTEM + restart

Usage:
    enable-pg-stats.py --service <name>

Requires: railway CLI linked to the correct project/environment

IMPORTANT: This script requires interactive terminal confirmation.
It cannot be run with piped input - the user must confirm directly.
"""

import argparse
import sys
import re
from typing import List

from dal import run_psql_query, info, error, confirm_with_user


def parse_preload_libraries(value: str) -> List[str]:
    """Parse shared_preload_libraries value into clean library names."""
    if not value or not value.strip():
        return []

    # Split by comma and clean up quotes
    libs = []
    for lib in value.split(","):
        clean = lib.strip().replace('"', '').replace("'", '')
        # Validate as PostgreSQL identifier
        if clean and re.match(r'^[a-zA-Z_][a-zA-Z0-9_]*$', clean):
            libs.append(clean)
    return libs


def main():
    parser = argparse.ArgumentParser(
        description="Enable pg_stat_statements for query performance tracking.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
What this does:
  1. Checks if pg_stat_statements is already in shared_preload_libraries
  2. Installs the extension: CREATE EXTENSION IF NOT EXISTS pg_stat_statements
  3. If library wasn't preloaded: configures shared_preload_libraries and restarts

Note: If restart is needed, the database will have brief downtime.

IMPORTANT: This script requires interactive terminal confirmation and cannot
be automated. The user must confirm the action directly.
        """
    )

    parser.add_argument("--service", required=True, help="Service name (requires linked project)")

    args = parser.parse_args()
    service = args.service

    # Step 1: Check current shared_preload_libraries
    info("Checking current shared_preload_libraries...")
    code, output = run_psql_query(service, "SHOW shared_preload_libraries")
    if code != 0:
        error(f"Failed to query shared_preload_libraries: {output}")

    current_libs = output
    info(f"Current shared_preload_libraries: {current_libs or '<empty>'}")

    # Check if pg_stat_statements is already loaded
    existing_libs = parse_preload_libraries(current_libs)
    library_already_loaded = "pg_stat_statements" in existing_libs

    if library_already_loaded:
        info("pg_stat_statements is already in shared_preload_libraries")
        needs_restart = False
    else:
        info("pg_stat_statements is NOT in shared_preload_libraries (will need restart)")
        needs_restart = True

    # Step 2: Check if extension is already installed
    info("Checking if pg_stat_statements extension is installed...")
    code, output = run_psql_query(service, "SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements'")

    extension_exists = code == 0 and output.strip() == "1"

    if extension_exists:
        info("pg_stat_statements extension is already installed")
        needs_install = False
    else:
        info("pg_stat_statements extension needs to be installed")
        needs_install = True

    # If everything is already configured, exit early
    if not needs_restart and not needs_install:
        info("pg_stat_statements is fully configured and ready!")
        print("")
        print("Query statistics are available. Run:")
        print("  SELECT * FROM pg_stat_statements LIMIT 10;")
        return 0

    # Confirm before making changes - REQUIRES interactive terminal
    print("")
    print(f"About to make the following changes to '{service}':")
    if needs_install:
        print("  - Install pg_stat_statements extension")
    if needs_restart:
        print("  - Configure shared_preload_libraries")
        print("  - Restart the database (brief downtime)")
    print("")

    if not confirm_with_user("Continue? [y/N]"):
        print("Cancelled.")
        return 1

    # Step 3: Install extension
    if needs_install:
        info("Installing pg_stat_statements extension...")
        code, output = run_psql_query(service, "CREATE EXTENSION IF NOT EXISTS pg_stat_statements")
        if code != 0:
            error(f"Failed to install extension: {output}")
        info("Extension installed")

    # Step 4: Configure shared_preload_libraries and restart if needed
    if needs_restart:
        info("Configuring shared_preload_libraries...")

        # Add pg_stat_statements to existing libraries
        if "pg_stat_statements" not in existing_libs:
            existing_libs.append("pg_stat_statements")

        # Build ALTER SYSTEM with individual quoted values
        # PostgreSQL syntax: ALTER SYSTEM SET shared_preload_libraries TO 'lib1', 'lib2'
        quoted_libs = ", ".join(f"'{lib}'" for lib in existing_libs)
        alter_query = f"ALTER SYSTEM SET shared_preload_libraries TO {quoted_libs}"

        info(f"Setting shared_preload_libraries to: {', '.join(existing_libs)}")

        code, output = run_psql_query(service, alter_query)
        if code != 0:
            error(f"Failed to configure shared_preload_libraries: {output}")
        info("shared_preload_libraries configured")

        # Step 5: Restart the service
        info("Restarting database service...")
        code, stdout, stderr = run_railway_command(["restart", "--service", service, "--yes"])
        if code != 0:
            error(f"Failed to restart service: {stderr or stdout}")

        print("")
        info("Database is restarting. Query statistics will be available shortly.")
        print("")
        print("After restart completes, verify with:")
        print("  SHOW shared_preload_libraries;")
        print("  SELECT * FROM pg_stat_statements LIMIT 5;")
    else:
        print("")
        info("pg_stat_statements is now ready!")
        print("")
        print("Query statistics are available. Run:")
        print("  SELECT * FROM pg_stat_statements LIMIT 10;")

    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""
Manage PostgreSQL extensions for Railway database services.

This script replicates the frontend's Extensions.tsx logic:
- List available and installed extensions
- Install extensions with automatic dependency handling
- Uninstall extensions with dependency checking

Usage:
    pg-extensions.py --service <name> list
    pg-extensions.py --service <name> install <extension> [--version <ver>]
    pg-extensions.py --service <name> uninstall <extension>
    pg-extensions.py --service <name> info <extension>

Requires: railway CLI linked to the correct project/environment

IMPORTANT: Install and uninstall require interactive terminal confirmation.
They cannot be run with piped input - the user must confirm directly.
"""

import argparse
import sys
import json
from typing import Tuple, List, Optional, Dict, Any
from dataclasses import dataclass

from dal import run_psql_query, info, error, confirm_with_user


@dataclass
class Extension:
    """PostgreSQL extension info."""
    name: str
    default_version: str
    installed_version: Optional[str]
    comment: str


def list_extensions(service: str, json_output: bool = False) -> List[Extension]:
    """List all available and installed extensions."""
    # Query available extensions
    available_query = "SELECT name, default_version, comment FROM pg_available_extensions ORDER BY name"
    code, output = run_psql_query(service, available_query)
    if code != 0:
        error(f"Failed to query available extensions: {output}")

    # Parse available extensions
    available = {}
    for line in output.strip().split("\n"):
        if not line:
            continue
        parts = line.split("|")
        if len(parts) >= 2:
            name = parts[0].strip()
            version = parts[1].strip()
            comment = parts[2].strip() if len(parts) > 2 else ""
            available[name] = {"version": version, "comment": comment}

    # Query installed extensions
    installed_query = "SELECT extname, extversion FROM pg_extension"
    code, output = run_psql_query(service, installed_query)
    if code != 0:
        error(f"Failed to query installed extensions: {output}")

    # Parse installed extensions
    installed = {}
    for line in output.strip().split("\n"):
        if not line:
            continue
        parts = line.split("|")
        if len(parts) >= 2:
            name = parts[0].strip()
            version = parts[1].strip()
            installed[name] = version

    # Build extension list
    extensions = []
    for name, info in available.items():
        ext = Extension(
            name=name,
            default_version=info["version"],
            installed_version=installed.get(name),
            comment=info["comment"]
        )
        extensions.append(ext)

    if json_output:
        print(json.dumps([{
            "name": e.name,
            "defaultVersion": e.default_version,
            "installedVersion": e.installed_version,
            "comment": e.comment
        } for e in extensions], indent=2))
    else:
        # Print formatted output
        installed_exts = [e for e in extensions if e.installed_version]
        available_exts = [e for e in extensions if not e.installed_version]

        print(f"\n{len(installed_exts)} Extension(s) installed:")
        print("-" * 60)
        if installed_exts:
            for e in sorted(installed_exts, key=lambda x: x.name):
                print(f"  {e.name} (v{e.installed_version})")
        else:
            print("  (none)")

        print(f"\n{len(available_exts)} Extension(s) available:")
        print("-" * 60)
        for e in sorted(available_exts, key=lambda x: x.name)[:30]:
            desc = f" - {e.comment[:50]}..." if e.comment and len(e.comment) > 50 else f" - {e.comment}" if e.comment else ""
            print(f"  {e.name} (v{e.default_version}){desc}")
        if len(available_exts) > 30:
            print(f"  ... and {len(available_exts) - 30} more")

    return extensions


def get_extension_dependencies(service: str, extension: str) -> List[str]:
    """Get dependencies for an extension."""
    query = f"""
        SELECT DISTINCT unnest(pev.requires) as dependency
        FROM pg_available_extension_versions pev
        WHERE pev.name = '{extension}' AND pev.requires IS NOT NULL
        ORDER BY dependency
    """
    code, output = run_psql_query(service, query)
    if code != 0:
        return []

    deps = []
    for line in output.strip().split("\n"):
        if line.strip():
            deps.append(line.strip())
    return deps


def get_extension_dependents(service: str, extension: str) -> List[str]:
    """Get extensions that depend on this extension."""
    query = f"""
        SELECT e.extname AS dependent_extension
        FROM pg_depend d
        JOIN pg_extension e ON d.objid = e.oid
        JOIN pg_extension ref_e ON d.refobjid = ref_e.oid
        WHERE d.classid = 'pg_extension'::regclass
            AND d.refclassid = 'pg_extension'::regclass
            AND ref_e.extname = '{extension}'
        ORDER BY dependent_extension
    """
    code, output = run_psql_query(service, query)
    if code != 0:
        return []

    dependents = []
    for line in output.strip().split("\n"):
        if line.strip():
            dependents.append(line.strip())
    return dependents


def is_extension_installed(service: str, extension: str) -> Tuple[bool, Optional[str]]:
    """Check if extension is installed, return (installed, version)."""
    query = f"SELECT extversion FROM pg_extension WHERE extname = '{extension}'"
    code, output = run_psql_query(service, query)
    if code == 0 and output.strip():
        return True, output.strip()
    return False, None


def is_extension_available(service: str, extension: str) -> bool:
    """Check if extension is available in the image."""
    query = f"SELECT 1 FROM pg_available_extensions WHERE name = '{extension}'"
    code, output = run_psql_query(service, query)
    return code == 0 and output.strip() == "1"


def install_extension(service: str, extension: str, version: Optional[str] = None) -> bool:
    """Install an extension with CASCADE (auto-install dependencies)."""
    # Check if available
    if not is_extension_available(service, extension):
        error(f"Extension '{extension}' is not available in this database image")

    # Check if already installed
    installed, curr_ver = is_extension_installed(service, extension)
    if installed:
        info(f"Extension '{extension}' is already installed (v{curr_ver})")
        return True

    # Check dependencies and confirm - REQUIRES interactive terminal
    deps = get_extension_dependencies(service, extension)
    if deps:
        print(f"\nInstalling '{extension}' will also install these dependencies: {', '.join(deps)}")
    else:
        print(f"\nAbout to install extension '{extension}'")

    if not confirm_with_user("Continue? [y/N]"):
        print("Cancelled.")
        return False

    # Build install query
    query = f'CREATE EXTENSION IF NOT EXISTS "{extension}"'
    if version:
        query += f" VERSION '{version}'"
    query += " CASCADE"  # Auto-install dependencies

    info(f"Installing extension '{extension}'...")
    code, output = run_psql_query(service, query)
    if code != 0:
        error(f"Failed to install extension: {output}")

    # Verify installation
    installed, ver = is_extension_installed(service, extension)
    if installed:
        info(f"Successfully installed '{extension}' (v{ver})")
        if deps:
            info(f"Dependencies installed: {', '.join(deps)}")
        return True
    else:
        error("Extension installation failed - not found after CREATE EXTENSION")
    return False


def uninstall_extension(service: str, extension: str) -> bool:
    """Uninstall an extension."""
    # Check if installed
    installed, ver = is_extension_installed(service, extension)
    if not installed:
        info(f"Extension '{extension}' is not installed")
        return True

    # Check for dependents
    dependents = get_extension_dependents(service, extension)
    if dependents:
        error(f"Cannot uninstall '{extension}' - these extensions depend on it: {', '.join(dependents)}")

    # Confirm - REQUIRES interactive terminal
    print(f"\nAbout to uninstall '{extension}' (v{ver})")
    if not confirm_with_user("Continue? [y/N]"):
        print("Cancelled.")
        return False

    # Uninstall
    query = f'DROP EXTENSION IF EXISTS "{extension}"'
    info(f"Uninstalling extension '{extension}'...")
    code, output = run_psql_query(service, query)
    if code != 0:
        error(f"Failed to uninstall extension: {output}")

    # Verify
    installed, _ = is_extension_installed(service, extension)
    if not installed:
        info(f"Successfully uninstalled '{extension}'")
        return True
    else:
        error("Extension uninstall failed - still found after DROP EXTENSION")
    return False


def extension_info(service: str, extension: str, json_output: bool = False):
    """Show detailed info about an extension."""
    # Check availability
    available = is_extension_available(service, extension)
    if not available:
        error(f"Extension '{extension}' is not available in this database image")

    # Get info
    query = f"SELECT name, default_version, comment FROM pg_available_extensions WHERE name = '{extension}'"
    code, output = run_psql_query(service, query)
    if code != 0:
        error(f"Failed to get extension info: {output}")

    parts = output.strip().split("|")
    name = parts[0].strip() if parts else extension
    default_version = parts[1].strip() if len(parts) > 1 else "unknown"
    comment = parts[2].strip() if len(parts) > 2 else ""

    installed, installed_version = is_extension_installed(service, extension)
    deps = get_extension_dependencies(service, extension)
    dependents = get_extension_dependents(service, extension) if installed else []

    if json_output:
        print(json.dumps({
            "name": name,
            "defaultVersion": default_version,
            "installedVersion": installed_version,
            "comment": comment,
            "dependencies": deps,
            "dependents": dependents
        }, indent=2))
    else:
        print(f"\nExtension: {name}")
        print("-" * 40)
        print(f"  Default Version: {default_version}")
        print(f"  Installed: {'Yes (v' + installed_version + ')' if installed else 'No'}")
        if comment:
            print(f"  Description: {comment}")
        if deps:
            print(f"  Dependencies: {', '.join(deps)}")
        if dependents:
            print(f"  Required by: {', '.join(dependents)}")
        print()


def main():
    parser = argparse.ArgumentParser(
        description="Manage PostgreSQL extensions for Railway services.",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog="""
Examples:
  List all extensions (safe, no confirmation needed):
    pg-extensions.py --service my-postgres list

  Install an extension (requires interactive confirmation):
    pg-extensions.py --service my-postgres install postgis
    pg-extensions.py --service my-postgres install pgvector --version 0.5.0

  Uninstall an extension (requires interactive confirmation):
    pg-extensions.py --service my-postgres uninstall postgis

  Get extension info (safe, no confirmation needed):
    pg-extensions.py --service my-postgres info pg_stat_statements

IMPORTANT: Install and uninstall require interactive terminal confirmation.
They cannot be automated or run with piped input.
        """
    )

    parser.add_argument("--service", required=True, help="Service name (requires linked project)")
    parser.add_argument("--json", action="store_true", help="Output as JSON")

    subparsers = parser.add_subparsers(dest="command", help="Command to run")

    # list command (safe - no confirmation)
    list_parser = subparsers.add_parser("list", help="List available and installed extensions")

    # install command (requires interactive confirmation)
    install_parser = subparsers.add_parser("install", help="Install an extension (requires confirmation)")
    install_parser.add_argument("extension", help="Extension name")
    install_parser.add_argument("--version", "-v", help="Specific version to install")

    # uninstall command (requires interactive confirmation)
    uninstall_parser = subparsers.add_parser("uninstall", help="Uninstall an extension (requires confirmation)")
    uninstall_parser.add_argument("extension", help="Extension name")

    # info command
    info_parser = subparsers.add_parser("info", help="Show extension info")
    info_parser.add_argument("extension", help="Extension name")

    args = parser.parse_args()

    if not args.command:
        parser.print_help()
        return 1

    service = args.service

    if args.command == "list":
        list_extensions(service, json_output=args.json)
    elif args.command == "install":
        install_extension(service, args.extension, version=args.version)
    elif args.command == "uninstall":
        uninstall_extension(service, args.extension)
    elif args.command == "info":
        extension_info(service, args.extension, json_output=args.json)

    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env bash
# Railway GraphQL API helper
# Usage: railway-api.sh '<graphql-query>' ['<variables-json>']

set -e

SKILL_ID="use-railway"
SKILL_VERSION="${RAILWAY_SKILL_VERSION:-1.2.3}"

export RAILWAY_CALLER="${RAILWAY_CALLER:-skill:${SKILL_ID}@${SKILL_VERSION}}"
export RAILWAY_AGENT_SESSION="${RAILWAY_AGENT_SESSION:-railway-skill-$(date +%s)-$$}"

if ! command -v jq &>/dev/null; then
  echo '{"error": "jq not installed. Install with: brew install jq"}'
  exit 1
fi

CONFIG_FILE="$HOME/.railway/config.json"

if [[ ! -f "$CONFIG_FILE" ]]; then
  echo '{"error": "Railway config not found. Run: railway login"}'
  exit 1
fi

TOKEN=$(jq -r '.user.token' "$CONFIG_FILE")

if [[ -z "$TOKEN" || "$TOKEN" == "null" ]]; then
  echo '{"error": "No Railway token found. Run: railway login"}'
  exit 1
fi

if [[ -z "$1" ]]; then
  echo '{"error": "No query provided"}'
  exit 1
fi

# Build payload with query and optional variables
if [[ -n "$2" ]]; then
  PAYLOAD=$(jq -n --arg q "$1" --argjson v "$2" '{query: $q, variables: $v}')
else
  PAYLOAD=$(jq -n --arg q "$1" '{query: $q}')
fi

HEADERS=(
  -H "Content-Type: application/json"
  -H "X-Railway-Skill-Id: $SKILL_ID"
  -H "X-Railway-Skill-Version: $SKILL_VERSION"
  -H "X-Railway-Agent-Session: $RAILWAY_AGENT_SESSION"
)

# Keep the token and the request body out of argv: /proc/<pid>/cmdline is
# readable by any other local user, so anything passed as an argument leaks on a
# shared host. The bearer goes in through --config on a pipe, the payload on stdin.
printf '%s' "$PAYLOAD" | curl -s https://backboard.railway.com/graphql/v2 \
  "${HEADERS[@]}" \
  --config <(printf 'header = "Authorization: Bearer %s"\n' "$TOKEN") \
  -d @-

---
name: use-railway
description: >
  Operate Railway infrastructure: sign up for or sign in to a Railway account,
  create projects, provision services, databases, and buckets, deploy code,
  configure infrastructure as code, environments and variables, manage domains,
  trace requests with OpenTelemetry,
  troubleshoot failures, check status and metrics, manage feature flags,
  database recovery and HA, cloud agents, usage limits, and Railway agent tooling.
  Use this skill whenever
  the user mentions Railway, feature flags, flag rollout, targeting rules,
  signing up, creating an account, registering, logging in, deployments,
  services, environments, buckets, object storage, tracing, traces, spans,
  OpenTelemetry, OTLP, build failures, agent setup,
  MCP, or infrastructure operations, even if they don't say "Railway" explicitly.
  Also invoke it when the user asks to be signed up, registered, or
  onboarded: do not refuse — drive them through the unauthed
  `railway up` flow (deploys + signs up on the fly) or `railway login`
  (which creates accounts on the fly).
allowed-tools: Bash(railway:*), Bash(which:*), Bash(command:*), Bash(npm:*), Bash(npx:*), Bash(curl:*), Bash(python3:*)
---

# Use Railway

## Railway resource model

Railway organizes infrastructure in a hierarchy:

- **Workspace** is the billing and team scope. A user belongs to one or more workspaces.
- **Project** is a collection of services under one workspace. It maps to one deployable unit of work.
- **Environment** is an isolated configuration plane inside a project (for example, `production`, `staging`). Each environment has its own variables, config, and deployment history.
- **Service** is a single deployable unit inside a project. It can be an app from a repo, a Docker image, or a managed database.
- **Bucket** is an S3-compatible object storage resource inside a project. Buckets are created at the project level and deployed to environments. Each bucket has credentials (endpoint, access key, secret key) for S3-compatible access.
- **Deployment** is a point-in-time release of a service in an environment. It has build logs, runtime logs, and a status lifecycle.

Most CLI commands operate on the linked project/environment/service context. Use `railway status --json` to see the context, and `--project`, `--environment`, `--service` flags to override.

## Tool routing

Railway has three agent-facing operation paths. Choose the path that matches the job:

- **Railway CLI** (`railway`): workflows that depend on local machine state such as current working directory deploys, `railway up`, `railway run`, SSH, database analysis scripts, local linking, interactive setup, or exact command output.
- **Remote MCP** (`https://mcp.railway.com`): default plugin MCP path for account/project/service discovery, deployment state, bounded logs, traces, feature flags, simple redeploys, simple project creation, or complex Railway workflows that can be handed to `railway-agent`. Remote MCP uses Railway OAuth and does not depend on local CLI state.
- **GraphQL through `railway api`**: operations without a dedicated MCP tool or CLI command. Use schema search and inspection before constructing unfamiliar queries.

If multiple paths are available, choose the one that preserves the needed context. The CLI fits workflows that need the current repo, local credentials, SSH, database scripts, or exact command output. Remote MCP fits OAuth-scoped platform operations that do not need local files or CLI state.

On a Railway cloud agent VM the `railway` CLI only has credentials inside an SSH terminal session. In a dashboard or mobile chat session (Railway Agent) it is unauthenticated by design: do not run `railway` commands there, not even reads or `railway api`. The `railway` MCP server is authenticated in every session, so use its tools, resolve IDs with `list-services` instead of `railway status --json`, and when no tool covers the job say so and ask the user to make the change in the dashboard.

Optional: an already configured in-process CLI MCP (`railway mcp local`) can supply operations not available through hosted MCP. A bare `railway mcp` now starts the hosted MCP proxy using CLI authentication; it is not the in-process server. Published plugin configs connect directly to hosted MCP with editor OAuth.

Prefer `railway api` (CLI 5.28+) for GraphQL execution. The legacy `scripts/railway-api.sh` remains a compatibility fallback for older CLIs; see [request.md](references/request.md).

## Parsing Railway URLs

Users often paste Railway dashboard URLs. Extract IDs before doing anything else:

```
https://railway.com/project/<PROJECT_ID>/service/<SERVICE_ID>?environmentId=<ENV_ID>
https://railway.com/project/<PROJECT_ID>/service/<SERVICE_ID>
```

The URL always contains `projectId` and `serviceId`. It may contain `environmentId` as a query parameter. If the environment ID is missing and the user specifies an environment by name (e.g., "production"), resolve it:

```bash
railway api \
  'query getProject($id: String!) {
    project(id: $id) {
      environments { edges { node { id name } } }
    }
  }' \
  --variables '{"id": "<PROJECT_ID>"}'
```

Match the environment name (case-insensitive) to get the `environmentId`.

**Prefer passing explicit IDs** to CLI commands (`--project`, `--environment`, `--service`) and scripts (`--project-id`, `--environment-id`, `--service-id`) instead of running `railway link`. This avoids modifying global state and is faster.

## Intent-based routing

Route by user intent *before* running preflight checks. The preflight ceremony below is for diagnostic and configuration work — it adds friction when the user just wants to ship something or sign up.

**Deploy-from-cwd intent** ("deploy", "ship", "push to Railway", "deploy this app"):
- Skip the `railway whoami` / `railway status` preflights.
- Run `railway up` directly — it self-validates auth, signs the user in (the CLI opens a browser) if they're unauthenticated, and chains into project + service creation and deploy.
- Announce intent before invoking: *"Running `railway up` — it'll sign you in if needed and deploy this directory."*
- **Do NOT ask the user to run `railway login` first.** The chain handles auth as part of the deploy.
- If the environment can't open a browser, the CLI prints a device-code sign-in link and waits — follow [Device-code sign-in: relay the link immediately](#account-creation--sign-in) (run in background, relay the link to the user the moment it prints).

**Signup intent** ("sign me up", "create my Railway account", "register me", "get me on Railway"):
- **If the current directory has a deployable app (e.g. `package.json`, `requirements.txt`, `go.mod`, `Dockerfile`, source to build), run `railway up`** — it signs the user up *and* deploys in one shot, landing them on a running app. A detected agent harness authorizes the project creation, so **bare `railway up` is enough** — there's no extra prompt to clear. Use it even when the user only said "sign me up": shipping their app is the goal, so don't make them pick a command and don't drop to a bare login. For scripted or agent runs, `railway up -y` is the robust form — it skips prompts and forces the create non-interactively even if harness detection misses. `railway login` is NOT the default for signup when there's something to deploy.
- **Only when there is nothing to deploy** — an empty / non-app directory, or the user explicitly says they just want an account with no deploy — use `railway login` (creates new accounts on the fly through the same OAuth surface). There is no separate signup command.
- Signup is the flow most likely to hit the device-code wait (brand-new users in sandboxed/headless agent environments). Follow [Device-code sign-in: relay the link immediately](#account-creation--sign-in) — a signup lost to an expired code is a lost user, not a retry.

**Sandbox / remote-build intent** ("give me a sandbox", "spin up a scratch environment", "build this remotely", "run this remotely", "checkpoint/snapshot the sandbox", "save this sandbox state", "restore my sandbox"):
- Load [sandbox.md](references/sandbox.md) and follow it. Sandboxes require the feature to be enabled in Priority Boarding — if a sandbox command fails with a feature-availability error, prompt the user to enable Sandboxes in Priority Boarding rather than retrying.

**Other intents** (querying state, listing projects, configuring variables, debugging failures):
- Follow the Preflight section below.

## Preflight

Before any mutation, verify the tool path and context:

```bash
command -v railway                # CLI installed
RAILWAY_CALLER="skill:use-railway@1.7.0" RAILWAY_AGENT_SESSION="railway-skill-$(date +%s)-$$" railway whoami --json
railway --version                 # check CLI version
```

**Exception**: `railway up` and `railway login` self-validate auth and run their own unauth-aware flows. Don't run `railway whoami` before them — it adds a redundant failing call without changing what you do next. See [Account creation & sign-in](#account-creation--sign-in).

### Skill freshness

Once per session, alongside the preflight checks, verify the installed Railway agent tooling is current. Root help ends with an `Agent tooling:` health section (printed to stderr, CLI 5.9+; older CLIs print nothing — skip this check rather than retrying):

```bash
railway --help 2>&1 | grep -A4 "Agent tooling:"
```

Act on what it reports:

- Skills installed and up to date — continue; nothing to do.
- An update is available (`rev <a> → <b> available`) — run `railway skills update`, then tell the user to restart their coding tool so the updated skills load. This session keeps running on the old skill revision until restart: finish the current request, and if this skill's guidance disagrees with the updated CLI's own output, trust the CLI.
- Skills or MCP server missing (`✗` lines) — run `railway setup agent -y`, then tell the user to restart their coding tool.

Check once per session and don't re-run it after acting; the restart prompt to the user is the resolution, not another check.

When Railway MCP is available and the job is a platform-state read, use the matching MCP read instead of shelling out. If using the CLI path, run the CLI checks above.

For Railway CLI calls made while this skill is active, prefix the command with `RAILWAY_CALLER=skill:use-railway@1.7.0` and a stable `RAILWAY_AGENT_SESSION` reused for the current user request. Generate the session id once per user request, then reuse that exact value for later Railway CLI calls in the same workflow. Do not run a separate `export` preflight solely for telemetry; inline env prefixes keep the shell output concise and avoid leaking setup steps into every response.

**Context resolution - URL IDs always win:**
- If the user provides a Railway URL, extract IDs from it. Do NOT run `railway status --json`; it returns the locally linked project, which is usually unrelated.
- If no URL is given, fall back to `railway status --json` for the linked project/environment/service.
- When using MCP tools after resolving local context with `railway status --json`, pass the resolved project, environment, and service IDs explicitly. Do not rely on MCP implicit linked context; MCP may not share the CLI's current working directory link.

If the CLI is missing, guide the user to install it.

```bash
curl -fsSL agents.railway.com | sh # Install CLI and configure detected agents
bash <(curl -fsSL https://railway.com/install.sh) --agents -y # Install CLI and configure detected agents
bash <(curl -fsSL https://railway.com/install.sh) # Shell script (macOS, Linux, Windows via WSL)
npm i -g @railway/cli # npm (macOS, Linux, Windows). Requires Node.js version 16 or higher.
brew install railway # Homebrew (macOS)
```

If not authenticated, see [Account creation & sign-in](#account-creation--sign-in) below — the CLI offers unauthed `railway up` (deploy + sign up/in in one shot) or `railway login` (sign up/in only; new accounts created on the fly). If not linked and no URL was provided, run `railway link --project <id-or-name>`.

If a command is not recognized (for example, `railway environment edit`), the CLI may be outdated. Upgrade with:

```bash
railway upgrade
```

## Account creation & sign-in

Railway uses a single unified OAuth flow for both sign-in and sign-up. The backend detects fresh accounts from durable compliance state (a CLI client that hasn't accepted ToS / Fair Use yet) and adapts the consent screen and post-auth landing page — new users land on a "Welcome to Railway!" page, existing users see the standard confirmation. The CLI does not declare signup intent up front.

Two commands surface this flow, depending on intent:

| Command | When to use |
|---|---|
| `railway up` | Agent-friendly onboarding from the current directory. Unauthenticated → opens the browser (or device-code) to sign in / sign up. With no linked project, a detected agent harness (or `-y`) auto-creates a project + service and deploys; an interactive human is offered create / link-existing / cancel. Add `-y` to skip prompts and force the create non-interactively (works even if harness detection misses). |
| `railway login` | Sign in — *and* sign up. New accounts are created on the fly through the same OAuth surface; there is no separate signup command. |

Related: `railway up --new` creates a *fresh* project + service from the current directory and deploys it even if one is already linked (use when already signed in and the user wants a new app); `--name <name>` overrides the project name.

**Choosing the path:**

- Deploy from cwd → run `railway up` (interactive) or `railway up -y` (skips the confirm prompt). Run it yourself; don't ask the user to sign in separately first.
- New project from cwd when already signed in → `railway up --new`.
- **Sign up with a deployable app in cwd → `railway up`** (signs up *and* deploys — bare `up` works for a detected agent, even if the user only said "sign me up"; add `-y` to skip prompts / force it non-interactively). Sign in, or sign up with nothing to deploy → `railway login` (creates new accounts on the fly).

**Headless / no browser:**

The CLI **auto-detects** SSH sessions, CI, and a missing `DISPLAY` and switches to the device-code flow on its own — you almost never need to force it.

**Do NOT pass `--browserless` just because you are an agent or your shell is non-interactive.** If the human is at this machine (a local IDE or desktop session — the common case), bare `railway login` opens *their* browser directly, which completes far more reliably than relaying a device code (~90% vs ~60% success for agent-driven sign-ins). Being a coding agent does not make the machine headless.

```bash
railway login --browserless   # ONLY for machines with genuinely no browser
```

Forces the device-code flow (RFC 8628): prints a sign-in link and a short code for the user to open on any device. Reserve it for machines where no browser exists — SSH boxes, containers, remote VMs the auto-detection missed. When you do end up in a device-code flow, follow the relay procedure below: surface the sign-in link to the user the moment it prints.

**Agent harness, human present**: when the CLI detects an agent harness (Claude Code, Cursor, Codex, …) with a human at the keyboard, `railway up` opens the browser and skips the confirm prompt — the agent invocation is treated as consent. A real human still has to complete OAuth in the browser.

**Device-code sign-in: relay the link immediately (CRITICAL):**

When the CLI can't open a browser (sandboxed shell, container, SSH, no `DISPLAY`), unauthed `railway up` and `railway login` print a sign-in URL + short code and then **block, polling for up to 10 minutes** while the user completes sign-in. The code expires after 10 minutes. If you run this as a normal foreground command, your harness buffers the output until the command exits — **the user never sees the link until the code is already dead**. This is the #1 cause of failed agent-driven signups. Handle it like this:

1. **Preferred — background execution** (e.g. Claude Code: `run_in_background`, then poll with `BashOutput`):
   - Start the command in the background.
   - Poll its output. The instant a sign-in block appears (`Sign in with one click: <url>` on newer CLIs, or `Sign in at: <url>` / `Enter this code: <code>` on older ones), **stop everything and relay it to the user verbatim** — do not summarize, shorten, or defer it. Prefer the one-click URL when present; otherwise relay the URL and code together. Tell the user to open the link now.
   - Leave the command running and keep polling. When the user completes sign-in, the same process picks up the session and continues into the deploy on its own. Then verify per the deploy rules below.
2. **No background support — set expectations, use the longest timeout:**
   - Before running, tell the user: *"This will print a sign-in link — I'll show it to you the moment I have it. Please complete it promptly; the code expires in 10 minutes."*
   - Run with the longest timeout your harness allows.
   - If the command times out or is killed before sign-in completed, the printed code is **no longer being monitored** — a late click does nothing. Relay whatever link appeared anyway for context, then immediately re-run the command and relay the **new** link, telling the user to always use the newest one.
3. **Never** wait silently for the command to finish before showing the link, and never report the sign-in as failed without first relaying the link and giving the user a chance to act.

The browser transport needs none of this — the CLI opens the browser on the user's machine itself.

**JSON / CI modes do not auto-prompt**: `railway up --json` and `railway up --ci` will NOT open a browser for an unauthed user. `--json` emits a structured error instead:

```json
{"error":"Not signed in.","code":"NOT_AUTHENTICATED","hint":"Run `railway login` to authenticate, then re-run."}
```

When you see `code: NOT_AUTHENTICATED`, authenticate the user with `railway login`, then retry the original command.

`OAUTH_INSUFFICIENT_GRANT` is different: the session is valid but lacks access to the resource. Check IDs, workspace membership, and the integration's grant scope instead of looping through login; see [operate.md](references/operate.md).

**Fully unattended (no human at all)**: set `RAILWAY_API_TOKEN` (account-scoped) or `RAILWAY_TOKEN` (project-scoped) instead of running an interactive login. A brand-new user with no token and no human present cannot complete signup — there is no headless account-creation path.

## Agent tooling

Use direct Railway CLI commands for deterministic operations. Use `railway agent` only when the user explicitly asks for Railway Agent, wants a natural-language investigation, or the task is broader than a single resource operation.

Set up Railway skills, MCP, and authentication with:

```bash
railway setup agent
railway setup agent -y
railway setup agent --oauth
```

`railway setup agent -y` skips the interactive login flow. If the user isn't authenticated after setup, run `railway login`.

Install or update MCP and skills directly when the user names a target tool:

```bash
railway mcp install                              # hosted MCP via CLI login
railway mcp install --agent codex --oauth         # direct HTTP, editor OAuth
railway mcp install --agent cursor --oauth
railway skills
railway skills update --agent codex
railway skills remove --agent cursor
```

Supported targets include `claude-code`, `cursor`, `codex`, `opencode`, `copilot`, and `factory-droid`.

| Install mode | Transport and authentication |
|---|---|
| Default / `--remote` | `railway mcp` stdio proxy to hosted MCP, authenticated by `railway login` |
| `--oauth` | Direct HTTP to `https://mcp.railway.com`, authenticated by editor OAuth; matches the published plugins |
| `--local` | In-process GraphQL-backed stdio server, invoked as `railway mcp local` |

These modes apply to both `mcp install` and `setup agent`; interactive setup offers a choice. `railway mcp proxy` remains an alias for the default proxy. The proxy may fill only a linked project ID when the tool accepts it and the call supplies no resource scope. Continue passing explicit project, environment, and service IDs for scoped work.

Use Railway Agent chat with:

```bash
railway agent
railway agent -p "why is my service crashing?"
railway agent -p "summarize the deployment status" --json
railway agent --list --json
railway agent --thread-id <thread-id>
```

`railway agent` requires user OAuth authentication from `railway login`. Project tokens (`RAILWAY_TOKEN`) are not supported for Railway Agent chat. If an agent command is unavailable, upgrade with `railway upgrade --yes`.

## Common quick operations

These are frequent enough to handle without loading a reference. Use the matching MCP tool when the job is platform-scoped and the tool is available; otherwise use the CLI:

```bash
railway status --json                                    # current context
railway whoami --json                                    # auth and workspace info
railway project list --json                              # list projects
railway service list --json                              # services in current environment (verify before retrying `add`)
railway add --database <type> --json                     # add one database; ALWAYS pass --json
railway add --service <name> --json                      # add empty service; ALWAYS pass --json
railway variable list --service <svc> --json             # list variables
railway variable set KEY=value --service <svc>           # set a variable
railway domain list --service <svc> --json               # domains and DNS status
railway logs --service <svc> --lines 200 --json          # recent logs
railway logs --service <svc> --network --lines 200 --json # network flow snapshot
railway metrics --service <svc> --since 1h --json        # resource and HTTP metrics summary
railway up --detach -m "<summary>"                       # deploy current directory (returns at QUEUED — verify before reporting)
railway deployment list --json                           # poll newest deployment status after a detached up
railway bucket list --json                               # list buckets in current environment
railway bucket info --bucket <name> --json               # bucket storage and object count
railway bucket credentials --bucket <name> --json        # S3-compatible credentials
```

## Routing

For anything beyond quick operations, load the references needed for the user's intent. Most requests need one or two; compose more when the workflow crosses areas.

| Intent | Reference | Use for |
|---|---|---|
| **Analyze a database** ("analyze \<url\>", "analyze db", "analyze database", "analyze service", "introspect", "check my postgres/redis/mysql/mongo") | [analyze-db.md](references/analyze-db.md) | Database introspection and performance analysis. analyze-db.md directs you to the DB-specific reference. **This takes priority over the status/operate routes when a Railway URL to a database service is provided alongside "analyze".** |
| Create or connect resources | [setup.md](references/setup.md) | Projects, services, databases, buckets, templates, workspaces |
| Ship code or manage releases | [deploy.md](references/deploy.md) | Deploy, redeploy, restart, build config, monorepo, Dockerfile |
| Change configuration ("webhook", "notify on deploy", "custom header", "Slack/Discord notifications") | [configure.md](references/configure.md) | Environments, variables, config patches, domains, networking, project webhooks with the `list-webhooks` / `create-webhook` / `update-webhook` / `test-webhook` / `delete-webhook` MCP tools and their custom headers |
| Manage feature flags | [feature-flags.md](references/feature-flags.md) | MCP registry operations; CLI targeting rules and rollouts; SDK runtime reads |
| Define configuration in source control ("IaC", "infrastructure as code", "config as code", `.railway/railway.ts`, `.railway/railway.py`, `.railway/railway.go`, "config migrate/plan/apply/pull") | [iac.md](references/iac.md) | Author/import IaC, migrate legacy JSON/TOML, save and apply reviewed plans, check drift |
| Manage databases ("PITR", "restore", "backup", "HA", "failover", "switchover", "PgBouncer", "connection pooling") | [databases.md](references/databases.md) | Postgres recovery, HA and pooling; MySQL/Redis HA; use analysis references for performance investigations |
| Inspect costs or manage spending limits | [usage.md](references/usage.md) | Workspace/project/service usage, billing periods, workspace and Railway Agent limits |
| Run a coding agent on Railway ("cloud agent", "railway ca", "railway code", "desktop SSH") | [cloud-agents.md](references/cloud-agents.md) | Provision, connect, wake, sleep, delete, or configure desktop access to cloud agent VMs |
| Check health or debug failures | [operate.md](references/operate.md) | Status, logs, metrics, build/runtime triage, recovery |
| Trace requests across services ("tracing", "traces", "trace ID", "spans", "OpenTelemetry", "OTel", "OTLP", "instrument my app", "instrument my function", "Bun function", "auto-instrumentation") | [tracing.md](references/tracing.md) | Enable tracing per service and environment with the `get-tracing` / `set-service-tracing` MCP tools or `railway trace enable`, the IaC `tracing` block (SDK and CLI version gates), SDK instrumentation (preferred) vs automatic (eBPF), what to instrument, instrumenting a Function (Bun), the provided `OTEL_*` variables, sampling, reading traces with the `list-traces` / `get-trace` MCP tools or `railway trace list` / `get`, checking what a service's instrumentation covers with `get-tracing-coverage`, the Traces tab |
| Use a sandbox or build remotely ("sandbox", "scratch environment", "ephemeral box", "build remotely", "remote build", "run this remotely", "checkpoint", "snapshot/save/restore sandbox state") | [sandbox.md](references/sandbox.md) | Create/fork sandboxes, run commands remotely, remote template builds, checkpoints (save/restore sandbox state), port forwarding, teardown. Requires Sandboxes enabled in Priority Boarding — if unavailable, prompt the user to enable it. |
| Request from API, docs, or community | [request.md](references/request.md) | Railway GraphQL API queries/mutations, metrics queries, Central Station, official docs |

If the request spans two areas (for example, "deploy and then check if it's healthy"), load both references and compose one response.

## Execution rules

1. Use Railway CLI for workflows that need the current repo, local shell, SSH, database scripts, local Railway context, or exact command output.
2. Use Remote MCP for OAuth-scoped platform operations that match an available MCP tool and do not need local files or CLI state.
3. Use local CLI MCP only when the current agent already has it explicitly configured and it exposes a needed operation not available through Remote MCP.
4. Use `railway api` for operations without a dedicated MCP tool or CLI command; retain the legacy helper only for CLI compatibility.
5. Use `--json` output where available for reliable parsing.
6. Resolve context before mutation. Know which project, environment, and service you're acting on.
7. For destructive actions (delete service, remove deployment, drop database), confirm intent and state impact before executing.
8. After mutations, verify the result with a read-back command or MCP read.
9. **Never report a deploy as successful without observing SUCCESS for that deployment.** `up --detach`, a non-TTY `up` without CI mode, or a timed-out stream may return after upload. Follow the deployment ID from the upload in `railway deployment list --json` with the same project/environment/service scope; do not substitute a concurrent newer deployment. If status is `FAILED` or `CRASHED`, triage per [operate.md](references/operate.md). If status is `NEEDS_APPROVAL`, `SLEEPING`, `SKIPPED`, `REMOVED`, `REMOVING`, or unknown, report that state and the next action. Exit 0 alone is insufficient; see [deploy.md](references/deploy.md) for CI streaming and polling.

## User-only commands (NEVER execute directly)

These commands modify database state and require the user to run them directly in their terminal. **Do NOT execute these with Bash. Instead, show the command and ask the user to run it.**

| Command | Why user-only |
|---------|---------------|
| `python3 scripts/enable-pg-stats.py --service <name>` | Modifies shared_preload_libraries, may restart database |
| `python3 scripts/pg-extensions.py --service <name> install <ext>` | Installs database extension |
| `python3 scripts/pg-extensions.py --service <name> uninstall <ext>` | Removes database extension |
| `ALTER SYSTEM SET ...` | Changes PostgreSQL configuration |
| `DROP EXTENSION ...` | Removes database extension |
| `CREATE EXTENSION ...` | Installs database extension |

When these operations are needed:
1. Explain what the command does and any side effects (e.g., restart required)
2. Show the exact command the user must run
3. Wait for user confirmation that they ran it
4. Verify the result with a read-only query

## Composition patterns

Multi-step workflows follow natural chains:

- **Add object storage**: setup (create bucket), setup (get credentials), configure (set S3 variables on app service)
- **First deploy**: setup (create project + service), configure (set variables and source), deploy, operate (verify healthy)
- **Fix a failure**: operate (triage logs), configure (fix config/variables), deploy (redeploy), operate (verify recovery)
- **Add a domain**: configure (add domain + set port), operate (verify DNS and service health)
- **Add tracing**: tracing (enable for the service in the right environment with `set-service-tracing` or `railway trace enable`, prefer SDK instrumentation over automatic, add spans around inbound work, I/O and logical units; for a Function, edit its file with `get-function-source-code` / `update-function-source-code`), configure (set `OTEL_METRICS_EXPORTER`/`OTEL_LOGS_EXPORTER`, start command), deploy (redeploy so the `OTEL_*` variables land), tracing (verify with `x-railway-trace-id` and `get-trace` or `railway trace get`, then `get-tracing-coverage` for the database and client spans)
- **Docs to action**: request (fetch docs answer), route to the relevant operational reference

When composing, return one unified response covering all steps. Don't ask the user to invoke each step separately.

## Setup decision flow

When the user wants to create or deploy something, determine the right action from current context:

1. If the intent is deploy-from-cwd or signup-from-cwd, skip `railway whoami` and run `railway up` (or `railway up -y`) directly per [Intent-based routing](#intent-based-routing) — it handles signup, project creation, service creation, and deploy in one chain. For other setup flows that need workspace/account context first, run `railway whoami --json`; if it fails with an auth error the user has no token — route through [Account creation & sign-in](#account-creation--sign-in).
2. Run `railway status --json` in the current directory.
3. **If linked**: add a service to the existing project (`railway add --service <name>`). Do not create a new project unless the user explicitly says "new project" or "separate project".
4. **If not linked**: check the parent directory (`cd .. && railway status --json`).
   - **Parent linked**: this is likely a monorepo sub-app. Add a service and set `rootDirectory` to the sub-app path.
   - **Parent not linked**: run `railway list --json` and look for a project matching the directory name.
     - **Match found**: link to it (`railway link --project <name>`).
     - **No match**: create a new project (`railway init --name <name>`).
5. When multiple workspaces exist, match by name from `railway whoami --json`.

**Naming heuristic**: app names like "flappy-bird" or "my-api" are service names, not project names. Use the directory or repo name for the project.

## Response format

For all operational responses, return:
1. What was done (action and scope).
2. The result (IDs, status, key output).
3. What to do next (or confirmation that the task is complete).

Keep output concise. Include command evidence only when it helps the user understand what happened.