references/multi-env-isolation.md
# Multi-Environment Isolation Checklist
When creating a second Terraform environment (`staging`, `lab`, etc.) in the same cloud account alongside production, every item below must be verified. Skip one and you get silent name collisions or cross-contamination.
## Configuration contract parity
Keep one machine-readable list of runtime-required keys. Every environment must contain each key
exactly once with a non-empty value. Only values differ by environment; requiredness does not.
Do not use these as environment isolation mechanisms:
- A production-only default that lets a missing key render as empty or as a guessed hostname
- A staging-only assertion that production never runs
- A hand-maintained list of variables inside one deploy writer
- A validator that reads the operator shell instead of the candidate environment artifact
Render each environment through the same production parser and compare the resulting key set before
planning. For Compose, remember that shell variables outrank `--env-file` and project `.env`; sanitize
ambient overrides or print `docker compose config --environment` as evidence.
## Terraform state isolation
Two environments MUST use different state paths. Same OSS/S3 bucket is fine — different prefix isolates completely:
```hcl
# production
backend "oss" {
bucket = "myproject-terraform-state"
prefix = "environments/production"
}
# staging
backend "oss" {
bucket = "myproject-terraform-state" # same bucket OK
prefix = "environments/staging" # different prefix = isolated state
}
```
**Verification**: `terraform state list` in one environment must show ZERO resources from the other.
## Resource naming collision matrix
Grep every `.tf` file for hardcoded names. Every globally-unique resource will collide.
### Must rename (apply will fail)
| Resource | Uniqueness scope | Fix pattern |
|---|---|---|
| SSH key pair (`key_pair_name`) | Region | `"${env}-deploy"` |
| SLS log project (`project_name`) | Account | `"${env}-logs"` |
| CloudMonitor contact (`alarm_contact_name`) | Account | `"${env}-ops"` |
| CloudMonitor contact group | Account | `"${env}-ops"` |
### Should rename (won't fail but causes confusion)
| Resource | Issue if same name |
|---|---|
| Security group name | Two SGs with same name in same VPC, can't tell apart in console |
| ECS instance name/hostname | Two instances named `myapp-spot` in console |
| Data disk name | Same in disk list |
| Auto snapshot policy name | Same in policy list |
| SLS machine group name | Logs from both instances land in same group |
### Pattern: Use a module name variable
```hcl
# production main.tf
module "app" {
source = "../../modules/spot-with-data-disk"
name = "production-spot" # flows to instance_name, disk_name, snapshot_policy_name
}
# staging main.tf
module "app" {
source = "../../modules/spot-with-data-disk"
name = "staging-spot" # all child resource names auto-isolated
}
```
## DNS record isolation
### The duplication trap
Two Terraform environments creating A records for `@` (root) in the same Cloudflare zone:
- Each gets its own Cloudflare record ID (independent)
- Cloudflare now has TWO A records for the same domain
- DNS round-robins between the two IPs
- ~50% of traffic goes to the wrong instance
### Correct patterns
**Pattern A: Subdomain isolation** (recommended for staging/lab):
```hcl
# Production: root domain records
resource "cloudflare_dns_record" "prod" {
name = "@" # example.com
}
# Staging: subdomain records only
resource "cloudflare_dns_record" "staging" {
name = "staging" # staging.example.com
}
```
**Pattern B: Separate zones** (for fully independent deployments):
Each environment gets its own domain/zone. No shared Cloudflare zone IDs.
**Pattern C: One environment owns DNS** (production):
Only production has DNS resources. Other environments access via IP only.
### Destroy safety
When one environment is destroyed:
- Its DNS records are deleted (by their specific Cloudflare record IDs)
- Other environments' DNS records are NOT affected
- **Verify before destroy**: Compare DNS record IDs between environments:
```bash
terraform state show 'cloudflare_dns_record.app["root"]' | grep "^id"
```
IDs must be different.
## Shared resources (safe to share)
These are referenced but NOT managed by the second environment:
| Resource | Why safe |
|---|---|
| VPC / VSwitch | Referenced by ID, not created |
| Cloudflare zone ID | Referenced, records are independent |
| OSS state bucket | Different prefix = different state |
| SSH public key content | Same key, different key pair resource |
| Cloud provider credentials | Same account, different resources |
## Makefile pattern for multi-environment
```makefile
ENV ?= production
ENV_DIR := environments/$(ENV)
init: ; cd $(ENV_DIR) && terraform init
plan: pre-deploy ; cd $(ENV_DIR) && terraform plan -out=tfplan
apply: ; cd $(ENV_DIR) && terraform apply tfplan
drift: ; cd $(ENV_DIR) && terraform plan -detailed-exitcode
```
Usage: `make plan ENV=staging`
In a real repository, keep Terraform behind its existing canonical Make/CI wrapper rather than
copying this minimal example over stronger saved-plan, digest, authorization, or provenance gates.
references/pre-deploy-validation.md
# Pre-Deploy Validation Ladder
Run validation at the earliest phase that can still prevent the failure. Do not collapse syntax,
candidate behavior, plan scope, deployed state, and user-visible outcome into one green check.
## 1. Static configuration
Run the repository's canonical formatter and wrapper. If no wrapper exists:
```bash
terraform fmt -check -recursive
terraform init -backend=false -input=false
terraform validate
```
`terraform validate` checks syntax and internal consistency. It does not consult remote state or
provider APIs and cannot prove environment values, runtime modules, network access, or user behavior.
Use variable validation and resource preconditions for values Terraform knows before mutation.
Postconditions run after a resource operation and do not roll back earlier effects. Terraform `check`
blocks report warnings and continue, so they are observability rather than a destructive-change gate.
## 2. One environment schema
Define runtime-required keys once in a machine-readable manifest or derive them from the canonical
configuration. Apply the same rules to staging and production:
- Every required key appears exactly once.
- Missing and empty are both failures.
- Any unresolved `${IDENTIFIER}` token in a required effective value is a failure, including a
placeholder that references a different missing key.
- Optional keys are explicitly classified; they do not share the required list.
- Environment files may change values, never requiredness.
- Caddy/Compose defaults are product defaults only, not a way to hide incomplete deployment input.
Calibrate the checker with one known-good environment and known-bad fixtures for missing, duplicate,
explicit-empty, self-placeholder, and foreign-placeholder values. A validator without a dangerous-input
test has not proved it can fail.
## 3. Render the effective runtime model
Docker Compose interpolation precedence is shell, then `--env-file`, then project `.env`. Prevent an
operator's ambient export from changing a reviewed release. Render the candidate with explicit project,
Compose, and env paths, then inspect the canonical model:
```bash
docker compose \
--project-directory "$CANDIDATE_ROOT" \
--env-file "$CANDIDATE_ROOT/environment.env" \
-f "$CANDIDATE_ROOT/compose.yaml" \
config --format json > "$CANDIDATE_ROOT/compose.rendered.json"
```
Run this command from a sanitized environment or explicitly reject ambient keys that can override the
candidate. Confirm the rendered service contains the exact immutable image and the complete environment
map. Do not rebuild a hand-selected env list in each deploy writer.
## 4. Run the production validator before live mutation
Use the same parser/module set that production will run. For Caddy, run the exact image digest with the
candidate directory and its full Compose-derived environment:
```bash
set -euo pipefail
GATEWAY_SERVICE=claude4dev-gateway
: "${CANDIDATE_ROOT:?candidate root is required}"
: "${EXPECTED_CADDY_IMAGE_DIGEST:?reviewed Caddy image digest is required}"
GATEWAY_ENV_FILE="$CANDIDATE_ROOT/gateway.effective.env"
REQUIRED_ENV_FILE="$CANDIDATE_ROOT/gateway/required-env.keys"
[ -s "$REQUIRED_ENV_FILE" ] \
|| { echo "FATAL: required-key manifest is missing or empty" >&2; exit 1; }
RENDERED_GATEWAY_IMAGE="$(jq -er --arg service "$GATEWAY_SERVICE" '
.services[$service].image
| select(type == "string" and length > 0)
' "$CANDIDATE_ROOT/compose.rendered.json")"
printf '%s\n' "$EXPECTED_CADDY_IMAGE_DIGEST" \
| grep -Eq '@sha256:[0-9a-f]{64}$' \
|| { echo "FATAL: expected Caddy image is not an immutable digest" >&2; exit 1; }
[ "$RENDERED_GATEWAY_IMAGE" = "$EXPECTED_CADDY_IMAGE_DIGEST" ] \
|| { echo "FATAL: rendered gateway image differs from the reviewed digest" >&2; exit 1; }
jq -e --arg service "$GATEWAY_SERVICE" '
.services[$service].environment
| type == "object"
and all(to_entries[];
(.key | test("^[A-Za-z_][A-Za-z0-9_]*$"))
and (.value | type == "string")
and ((.value | contains("\n")) | not)
and ((.value | test("\\$\\{[A-Za-z_][A-Za-z0-9_]*\\}")) | not)
)
' "$CANDIDATE_ROOT/compose.rendered.json" >/dev/null \
|| { echo "FATAL: rendered gateway environment contains an invalid key, value, or unresolved placeholder" >&2; exit 1; }
required_count=0
seen_required=' '
while IFS= read -r required_key || [ -n "$required_key" ]; do
case "$required_key" in
''|'#'*) continue ;;
[A-Za-z_]*)
case "$required_key" in
*[!A-Za-z0-9_]*) echo "FATAL: invalid required key: $required_key" >&2; exit 1 ;;
esac
;;
*) echo "FATAL: invalid required key: $required_key" >&2; exit 1 ;;
esac
case "$seen_required" in
*" $required_key "*) echo "FATAL: duplicate required key: $required_key" >&2; exit 1 ;;
esac
seen_required="$seen_required$required_key "
required_count=$((required_count + 1))
if ! required_value="$(jq -er --arg service "$GATEWAY_SERVICE" --arg key "$required_key" '
.services[$service].environment[$key] | select(type == "string")
' "$CANDIDATE_ROOT/compose.rendered.json")"; then
echo "FATAL: required gateway environment key is missing: $required_key" >&2
exit 1
fi
[ -n "$required_value" ] \
|| { echo "FATAL: required gateway environment key is empty: $required_key" >&2; exit 1; }
if printf '%s\n' "$required_value" | grep -Eq '\$\{[A-Za-z_][A-Za-z0-9_]*\}'; then
echo "FATAL: required gateway environment key is unresolved: $required_key" >&2
exit 1
fi
done < "$REQUIRED_ENV_FILE"
[ "$required_count" -gt 0 ] \
|| { echo "FATAL: required-key manifest contains no keys" >&2; exit 1; }
jq -r --arg service "$GATEWAY_SERVICE" '
.services[$service].environment
| to_entries | sort_by(.key)[] | "\(.key)=\(.value)"
' "$CANDIDATE_ROOT/compose.rendered.json" > "$GATEWAY_ENV_FILE"
chmod 600 "$GATEWAY_ENV_FILE"
docker run --rm --pull=never --network none \
--env-file "$GATEWAY_ENV_FILE" \
-v "$CANDIDATE_ROOT/gateway:/etc/caddy:ro" \
"$EXPECTED_CADDY_IMAGE_DIGEST" \
caddy adapt --config /etc/caddy/Caddyfile --validate
```
`caddy adapt` alone is weaker: `--validate` also loads and provisions the adapted configuration. Keep
network disabled unless validation genuinely requires network access, and document that exception.
Validate before writing the live env/config tree or restarting the service. Promote the same bytes that
passed; avoid overlay extraction that leaves deleted stale files behind. Enumerate every normal and
recovery writer and route all of them through the shared validator.
## 5. Review one executable plan
Generate a saved plan, inspect it with `terraform show`, and bind it to:
- the exact environment and backend/workspace;
- source and release-artifact identities;
- resource addresses and action scope;
- immutable helper/validator bytes;
- a digest shown to the human reviewer.
Apply that exact plan file. In saved-plan mode, Terraform treats passing the plan as approval and does
not prompt; if production requires a fresh explicit decision, implement it in the wrapper at the last
reversible point. Do not use target names or environment variables as proof of authorization.
## 6. Issue promotion evidence only after live verification
A zero exit from `terraform apply` proves only that Terraform completed its operation. Run required
environment-specific live verifiers next. Record the staging/promotion receipt only after they all pass,
and bind it to the saved-plan digest, exact source/artifact identities, verifier set, and timestamp.
Before production, freshly read the authoritative remote branch and prove the candidate commit belongs
to it; cached tracking refs are not provenance. After apply, independently read back deployed identity,
service health, and the real user path. Post-deploy checks remain necessary, but they are detection and
recovery evidence, not a substitute for the pre-mutation gate.
## 7. Validate external dependencies at their authority
Keep these checks in the project's canonical pre-deploy wrapper instead of trusting string shape:
- Resolve every hostname the candidate gateway serves and compare it with the environment's intended
target. When DNS is Terraform-managed, inspect the planned/current provider record IDs too.
- Verify OAuth/OIDC issuer and callback identities against the authoritative application configuration,
including already-initialized databases that no longer replay first-boot seed files.
- Verify the selected SSH key exists, has suitable permissions, and reaches the attested host identity;
a path existing locally does not prove it is the key for that host.
- Verify credentials through the provider's read-only identity/status endpoint, then exercise the exact
permission needed by the plan when that can be done without mutation.
Fail with the concrete mismatch and next action. Do not continue to apply merely to collect a more
expensive version of the same error.
## Primary contracts
- HashiCorp: provisioners are a last resort because their behavior is not predictably modeled:
<https://developer.hashicorp.com/terraform/language/provisioners>
- HashiCorp: saved-plan apply executes the reviewed plan without another approval prompt:
<https://developer.hashicorp.com/terraform/cli/commands/apply>
- HashiCorp: validation, preconditions, postconditions, and non-blocking checks run at different phases:
<https://developer.hashicorp.com/terraform/language/validate>
- Docker: Compose interpolation precedence and effective environment:
<https://docs.docker.com/compose/how-tos/environment-variables/variable-interpolation/>
- Docker: `docker compose config` renders the actual engine model:
<https://docs.docker.com/reference/cli/docker/compose/config/>
- Caddy: `caddy adapt --validate` is stronger than adaptation alone:
<https://caddyserver.com/docs/command-line#caddy-adapt>
references/release-safety-and-environment-parity.md
# Release Safety and Environment Parity
Use this reference when Terraform or a recovery tool can change a running service, especially a shared
gateway. The invariant is simple: no candidate reaches a live path until the exact bytes, environment,
and runtime have been validated together.
## Freeze the real outcome and mutation surface
Write down:
- User-visible outcome and the real path that proves it
- Exact environment and service
- Every writer that can change that runtime, including broad deploy modules and break-glass recovery
- Last reversible point
- Authorized action and non-goals
- Recovery path if the mutation fails after it starts
A focused resource name does not imply a focused blast radius. Inspect provisioner bodies and dependency
edges: a "service deploy" resource may also extract a gateway directory, replace the runtime env, and
recreate a shared front door.
## Use one required-key contract
Keep required keys in one manifest or derive them from one canonical schema. Run the same validator for
every environment and every writer.
| Allowed difference | Forbidden difference |
|---|---|
| Domain, credential value, capacity, feature value | Required in staging but optional/defaulted in production |
| Environment-specific resource identity | Empty accepted by one writer but rejected by another |
| Explicit optional feature disabled in an environment | A hand-picked env list that omits a newly used key |
Treat unset and explicit-empty separately in fixtures, but reject both for required keys. Caddy replaces
`{$VAR}` before parsing and can expand it to an empty token. Compose `${VAR:?message}` rejects unset and
empty, while `${VAR?message}` rejects only unset. Pick the form from the schema, not by habit.
Do not "fix" a duplicated tracked secret by replacing it with a marker until the complete injection chain
is proven: one named secret SSOT, CI/Make export, renderer replacement, non-empty candidate readback, and a
negative test for missing input. Removing the only usable value is an outage, not secret management.
## Validate the exact bundle
Build a candidate directory away from live paths. It must contain the exact reviewed source archive,
selected environment artifact, Compose file, gateway files, generated files owned by adjacent modules,
and immutable image digest.
Then:
1. Render Compose's canonical JSON from explicit candidate paths and a controlled interpolation environment.
2. Extract the gateway service's full environment map and exact image from that rendered model.
3. Reject any missing/empty required key, any unresolved `${IDENTIFIER}` token, and any image mismatch.
4. Run the exact image with the complete environment and candidate config using the runtime's strongest
non-starting validation mode.
5. Promote only the candidate bytes that passed, under the same deployment lock.
6. Use exact sync semantics for managed directories so a deleted config cannot survive as stale live input.
Do not validate with a host-installed binary, a different image tag, a manually reconstructed env subset,
or the current live config. Those validate a different system.
## Bind staging, source provenance, and production authorization
- Let staging test a clean local candidate before it is merged.
- Record a staging receipt only after apply and every required live verifier passes. A recorder callable
by itself must rerun or cryptographically consume that evidence; it cannot mint a receipt from plan text.
- Before production, freshly fetch/read the authoritative remote branch and prove the candidate commit is
in its history. Reject fetch failure; never fall back to a cached remote-tracking ref.
- Ask for production authorization at the last reversible point, using a channel the applying process
cannot forge. Plan digests and blast-radius acknowledgements prove review, not permission.
- Keep the invalid-config recovery path separate and narrow: exact target, approved known-good bytes,
prevalidation, compare-and-swap live identity, recovery ledger, and public readback.
## Keep pre- and post-mutation checks distinct
Pre-mutation validation prevents a known bad candidate from touching live state. Post-deploy acceptance
detects runtime, dependency, routing, and user-journey failures that static validation cannot know. Run
both; never describe a post-deploy failure as evidence that the pre-deploy gate worked.
## Minimum fixtures
Healthy fixtures:
- Staging and production both provide the same required key set with different valid values.
- Exact image accepts the rendered candidate.
- Saved plan, source/artifact provenance, live verifiers, and plan-bound production authorization/audit
all match; the apply runner remains non-interactive and headless-compatible.
Dangerous fixtures:
- Required key absent.
- Required key present but empty.
- Required key remains an unresolved self- or foreign-key placeholder.
- Ambient shell variable overrides the candidate env file.
- Config uses a module absent from the exact production image.
- A broad deploy writer bypasses the focused gateway validator.
- Live directory retains a config deleted from the candidate.
- A standalone recorder tries to issue a staging receipt after a failed verifier.
- Candidate commit exists only locally or fetch of authoritative main fails.
- Production orchestration cannot prove that its recorded authorization/audit is bound to the exact
plan digest, environment, and source/artifact identities.
For every pre-mutation failure, assert the live manifest/env and restart count are unchanged. Calibrate the
gate on known-good inputs before enabling it fail-closed; a false positive trains operators to bypass it.
references/zero-to-deploy-checklist.md
# Zero-to-Deployment Checklist
A fresh instance with an empty data disk exposes every implicit dependency that production silently relies on. This checklist covers everything that must be explicitly created before services will start.
## Pre-flight: cloud-init must handle
These run at OS boot, before Terraform provisioners:
- [ ] **Mount data disk**: Format if new (`blkid` check), mount to `/data`, add to fstab
- [ ] **Create service directories**: `mkdir -p /data/{service1,service2,...}` — file provisioners fail if target dir doesn't exist
- [ ] **Install Docker + Compose**: Curl installer, enable systemd service
- [ ] **Configure swap**: `fallocate` on data disk (NOT system disk)
- [ ] **SSH hardening**: key-only auth, no password root login
- [ ] **Firewall**: UFW + DOCKER-USER iptables chain
- [ ] **Debconf preseed**: For any package with interactive prompts (iptables-persistent, etc.)
- [ ] **Signal readiness**: Write timestamp to `/data/cloud-init.log`
## Provisioner ordering
Terraform provisioners execute in declaration order within a resource, but resources execute in parallel unless `depends_on` is set.
```
gateway_backend_deploy ──────────→ channel_sync (depends_on gateway backend)
→ identity_sync (depends_on gateway backend)
→ object_store_sync (depends_on gateway backend)
app_deploy (depends_on gateway_backend_deploy)
├─ wait for cloud-init
├─ upload source (tarball via file provisioner)
├─ upload .env (staging variant)
├─ start stateful (postgres, redis) --no-recreate
├─ run DB migrations
├─ build stateless images
├─ fix volume permissions
├─ start stateless (relay, api, frontend, gateway)
└─ verify health
```
## Database bootstrap
### PostgreSQL databases
PostgreSQL `docker-entrypoint-initdb.d` scripts only run when the data directory is empty (first-ever start). On subsequent starts — even if a database doesn't exist — init scripts are skipped.
**Fix**: Explicitly create databases in provisioner:
```bash
# Wait for postgres healthy
sleep 10
# Create database if missing (idempotent)
docker exec my-postgres psql -U postgres -tc \
"SELECT 1 FROM pg_database WHERE datname='mydb'" | grep -q 1 \
|| docker exec my-postgres psql -U postgres -c "CREATE DATABASE mydb;"
```
### Schema migrations
Migrations must be idempotent. Track applied versions:
```bash
PSQL='docker compose exec -T postgres psql -v ON_ERROR_STOP=1 -U myuser -d mydb'
# Create tracking table
$PSQL -tAc "CREATE TABLE IF NOT EXISTS schema_migrations (
version TEXT PRIMARY KEY,
applied_at TIMESTAMPTZ DEFAULT now()
)"
# Apply each migration file in order
for f in migrations/*.sql; do
VER=$(basename $f)
APPLIED=$($PSQL -tAc "SELECT 1 FROM schema_migrations WHERE version='$VER'" | tr -d ' ')
if [ "$APPLIED" = "1" ]; then
echo "Skip: $VER"
else
echo "Apply: $VER"
{ echo 'BEGIN;'; cat $f; echo 'COMMIT;'; } | $PSQL
$PSQL -tAc "INSERT INTO schema_migrations(version) VALUES ('$VER') ON CONFLICT DO NOTHING"
fi
done
```
## Docker build on remote
### Proxy mode
Docker Compose interpolation gives the invoking shell higher precedence than `--env-file` or the
project `.env`. An explicit one-command assignment therefore overrides the reviewed env file; an old
ambient export can do the same accidentally.
```bash
# Explicit one-off override: this value wins over .env for interpolation.
DOCKER_WITH_PROXY_MODE=disabled docker compose build myapp
# Audited release: make the selected env file authoritative and inspect the effective inputs first.
env -u DOCKER_WITH_PROXY_MODE \
docker compose --env-file .env config --environment
env -u DOCKER_WITH_PROXY_MODE \
docker compose --env-file .env build myapp
```
Do not edit a tracked or shared `.env` merely to perform a one-off build. For a release, freeze the
intended environment artifact, remove ambient overrides for its keys, and review the rendered Compose
model before the build or deployment mutates live state.
### Memory management
Building Docker images while 10+ containers run can OOM on small instances (8GB). Strategy:
```bash
# Stop non-critical containers to free RAM
cd /data/other-project && docker compose stop search-engine analytics-db || true
# Build (memory-intensive)
cd /data/myproject && docker compose build myapp
# Restart stopped containers
cd /data/other-project && docker compose up -d search-engine analytics-db || true
```
## Volume permissions
Containers running as non-root need writable volume directories:
```bash
# Before docker compose up:
mkdir -p data-dir logs-dir
chown -R 1001:1001 data-dir logs-dir # match container UID
```
Find the UID from the Dockerfile:
```dockerfile
RUN adduser -S myuser -u 1001 -G mygroup
USER myuser # runs as uid 1001
```
## Environment-specific .env files
Production and staging need different values under one schema. A key required by the runtime must be
required, present exactly once, and non-empty in both. Do not make production rely on an implicit
fallback merely because staging has an explicit value.
| Variable | Production | Staging |
|---|---|---|
| `FRONTEND_URL` | `https://myapp.com` | `https://staging.myapp.com` |
| `CORS_ORIGIN` | `https://myapp.com` | `https://staging.myapp.com` |
| `NEW_API_URL` | `http://api-container:3000` | Same (internal Docker network) |
| `DOCKER_WITH_PROXY_MODE` | `required` (if behind proxy) | `disabled` (direct internet) |
**Pattern**: Create environment-specific source files from the same schema. Freeze the selected file
into the release artifact or saved plan before apply:
```hcl
locals {
env_src = "${local.repo}/.env.staging" # staging-specific
}
provisioner "file" {
source = local.env_src
destination = "${local.deploy_dir}/.env"
}
```
Generic source sync must exclude repository env templates so it cannot overwrite the explicitly
selected runtime environment:
```
--exclude=.env --exclude='.env.*'
```
Before promoting it, render the candidate's complete Compose service environment and validate the
candidate configuration with the exact immutable image that will run it. Never select six remembered
keys from a seven-key service; derive the set from the one required-key manifest.
## Verification template
After all services start, verify in the provisioner (not ad-hoc SSH):
```bash
sleep 20
echo '=== Service logs ==='
docker logs my-critical-service --tail 20 2>&1 || true
echo '=== All containers ==='
docker ps --format 'table {{.Names}}\t{{.Status}}' 2>&1 || true
# Final gate (only line that can fail)
docker ps --filter name=my-critical-service --format '{{.Status}}' | grep -q healthy \
|| { echo 'FATAL: service unhealthy'; exit 1; }
```
SKILL.md
---
name: terraform-skill
description: >-
Diagnoses and designs safe Terraform releases, provisioners, multi-environment
isolation, and fresh-host bootstrap. Use when writing or reviewing plan/apply
wrappers, null_resource, remote-exec, local-exec, file provisioners, cloud-init,
Docker Compose or Caddy deployment; when staging and production configuration may
differ; when an IaC rollout can mutate a shared gateway; or when debugging drift,
saved-plan, provenance, TLS, Restarting/unhealthy containers, DNS duplication,
snapshot contamination, and post-apply failures. It emphasizes exact reviewed
artifacts, pre-mutation validation, explicit production authorization, and
independent live readback.
---
# Terraform Release and Provisioner Safety
Prevent a valid-looking Terraform workflow from publishing unvalidated bytes or widening a change's
blast radius. Keep the user's business outcome and the actual mutation surface ahead of plan counts,
green wrappers, or process completeness.
## Route the task
- Release, shared gateway, saved plan, staging receipt, or production promotion: read
[release-safety-and-environment-parity.md](references/release-safety-and-environment-parity.md).
- A second environment, DNS ownership, state, or snapshots: read
[multi-env-isolation.md](references/multi-env-isolation.md).
- Pre-deploy checks or a validator: read
[pre-deploy-validation.md](references/pre-deploy-validation.md).
- Fresh instance or empty data disk: read
[zero-to-deploy-checklist.md](references/zero-to-deploy-checklist.md).
- One known provisioner symptom: use the matching pattern below.
## Operating contract
1. Use the repository's canonical wrapper when it has one. Do not bypass it with a raw Terraform,
SSH, SCP, helper-script, or console path because a gate rejects the planned release.
2. Prefer provider resources, image baking, cloud-init, or configuration management. HashiCorp
recommends exhausting purpose-built alternatives because Terraform cannot model provisioner side
effects predictably. When a provisioner remains necessary, make its artifact, target, lock,
validation, and readback explicit.
3. Inventory every resource or recovery tool that can write the target runtime. A validator attached
to only one writer does not protect another writer of the same shared service.
4. Give staging and production one required-key schema. Let values differ; never let a key be required
in one environment and optional, defaulted, absent, or allowed-empty in another.
5. Validate the exact candidate bytes, Compose-rendered environment, and immutable runtime image
before the first live write or restart. Keep post-deploy checks too: they detect damage but cannot
prevent the first bad mutation.
6. Treat a saved plan as an executable artifact. Bind it to reviewed source/artifact identity and apply
that exact file. A successful apply is not a staging receipt; record promotion evidence only after
every required live verifier succeeds.
7. Require an explicit production decision at the last reversible point. A deadline, `PLAN_DIGEST`,
`CONFIRM_*`, or agent inference is not production authorization.
8. Stop after the requested result is verified. Do not turn a single-service fix into full-stack drift
reconciliation, recovery redesign, or unrelated hardening.
## Provisioner traps (symptom → fix)
Use these incident-derived symptom patterns to choose the next falsifying check. Confirm the current
source and runtime before promoting a historical cause into the present diagnosis.
### `docker: not found` in remote-exec
cloud-init still installing Docker when provisioner SSHs in.
```hcl
provisioner "remote-exec" {
inline = [
"cloud-init status --wait",
"command -v docker >/dev/null || { echo 'FATAL: Docker not ready'; exit 1; }",
]
}
```
### `rsync: connection unexpectedly closed` in local-exec
Do not infer a universal Terraform limitation from this symptom. A second SSH client can lose to the
target's connection budget, SSH policy, or a competing deploy. Keep `local-exec` local: package an
immutable artifact there, then use a Terraform-managed upload or a purpose-built deploy system. Give
every apply a unique remote staging path; never share `/tmp/src.tar.gz` across concurrent applies.
```hcl
provisioner "local-exec" {
command = "tar czf /tmp/src-${self.id}.tar.gz --exclude=node_modules --exclude=.git -C ${path.module}/../../.. myproject"
}
provisioner "file" {
source = "/tmp/src-${self.id}.tar.gz"
destination = "/tmp/src-${self.id}.tar.gz"
}
provisioner "remote-exec" {
inline = ["tar xzf /tmp/src-${self.id}.tar.gz -C /data/ && rm -f /tmp/src-${self.id}.tar.gz"]
}
```
macOS BSD tar: `--exclude` must come BEFORE the source argument.
### `cloud-init status` shows "running" forever
`apt-get -y` does not suppress debconf dialogs. Packages like `iptables-persistent` block on TTY prompts.
```yaml
- |
echo iptables-persistent iptables-persistent/autosave_v4 boolean true | debconf-set-selections
echo iptables-persistent iptables-persistent/autosave_v6 boolean true | debconf-set-selections
DEBIAN_FRONTEND=noninteractive apt-get install -y iptables-persistent
```
Known offenders: `iptables-persistent`, `postfix`, `mysql-server`, `wireshark-common`.
### `EACCES: permission denied` in container logs, container Restarting
Host volume dirs are root-owned; container runs as non-root (uid 1001). Fix before `docker compose up`:
```bash
mkdir -p /data/myapp/data /data/myapp/logs
chown -R 1001:1001 /data/myapp/data /data/myapp/logs
```
Find UID: grep `adduser.*-u` or `USER` in Dockerfile.
### Provisioner fails but no diagnostic output
Keep fail-fast behavior; attach diagnostics to failure instead of disabling `set -e`. Otherwise an
early failed command can be overwritten by a later green health check.
```hcl
provisioner "remote-exec" {
inline = [
"set -eu",
"trap 'rc=$?; if [ $rc -ne 0 ]; then docker logs myapp --tail 20 2>&1 || true; docker ps --format \\\"table {{.Names}}\\\\t{{.Status}}\\\" || true; fi; exit $rc' EXIT",
"docker compose up -d",
"sleep 15",
"docker ps --filter name=myapp --format '{{.Status}}' | grep -q healthy || exit 1",
]
}
```
### Container `Restarting` — database tables missing
DB migrations not in provisioner. PostgreSQL `docker-entrypoint-initdb.d` only runs on empty data dir. Explicitly create DB + run migrations:
```bash
# After postgres healthy:
docker exec pg psql -U postgres -tc "SELECT 1 FROM pg_database WHERE datname='mydb'" | grep -q 1 \
|| docker exec pg psql -U postgres -c "CREATE DATABASE mydb;"
# Idempotent migrations:
for f in migrations/*.sql; do
VER=$(basename $f)
APPLIED=$($PSQL -tAc "SELECT 1 FROM schema_migrations WHERE version='$VER'" | tr -d ' ')
[ "$APPLIED" = "1" ] && continue
{ echo 'BEGIN;'; cat $f; echo 'COMMIT;'; } | $PSQL
$PSQL -tAc "INSERT INTO schema_migrations(version) VALUES ('$VER') ON CONFLICT DO NOTHING"
done
```
### Compose uses an unexpected value despite `.env`
Compose interpolation gives the invoking shell higher precedence than `--env-file` or project `.env`.
An old exported value can therefore override the reviewed environment silently. Inspect what Compose
actually used; unset ambient overrides when the env file is meant to be authoritative.
```bash
# Inspect interpolation inputs and the rendered model.
docker compose --env-file .env config --environment
docker compose --env-file .env config --format json > compose.rendered.json
# Make the reviewed env file authoritative for this key.
env -u DOCKER_WITH_PROXY_MODE docker compose --env-file .env build
```
### TLS handshake fails: `Invalid format for Authorization header`
Caddy's Cloudflare DNS module expects a scoped API Token through Bearer authentication. Do not infer
credential type, validity, or permissions from length/prefix alone. Verify the token with Cloudflare's
official endpoint, then exercise the exact zone operation or provider path required by the release.
```bash
curl -fsS https://api.cloudflare.com/client/v4/user/tokens/verify \
-H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
| jq -e '.success == true and .result.status == "active"' >/dev/null
```
If the credential is absent or wrong, create a least-privilege API Token through Cloudflare's current
dashboard/API flow and grant only the zones/operations the provider needs. Follow the official creation
contract rather than copying permission-group IDs that may drift:
<https://developers.cloudflare.com/fundamentals/api/get-started/create-token/>.
### TLS fails on staging but works on production — hardcoded domains
Caddyfile or compose has literal domain names. Staging Caddy loads production config, tries to get certs for domains it doesn't own → ACME fails.
**Caddyfile**: Use `{$VAR}` — Caddy evaluates env vars at startup.
```caddy
# WRONG
example.com { tls { dns cloudflare {env.CLOUDFLARE_API_TOKEN} } }
# RIGHT
{$LOBEHUB_DOMAIN} { tls { dns cloudflare {env.CLOUDFLARE_API_TOKEN} } }
```
**Compose**: Use `${VAR:?required}` — fail-fast if unset or empty.
```yaml
# WRONG
- APP_URL=https://example.com
# RIGHT
- APP_URL=${APP_URL:?APP_URL is required}
```
Pass the env var to the gateway container so Caddy can read it:
```yaml
environment:
- LOBEHUB_DOMAIN=${LOBEHUB_DOMAIN:?LOBEHUB_DOMAIN is required}
- CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN:?required for DNS-01 TLS}
```
Do not stop at this local assertion. Put all runtime-required keys in one schema, require the same set
from every environment file, render the exact Compose service environment, and run the exact deployed
Caddy image with that full environment before mutating live files. Caddy `{$VAR}` expansion can become
an empty token before parsing; a Caddyfile default is not an environment-completeness check.
### OAuth login fails: `Social sign in failed`
Casdoor `init_data.json` contains hardcoded redirect URIs. `--createDatabase=true` only applies init_data on first-ever DB creation — not on restarts. Fix via SQL in provisioner:
```bash
# Replace production domain with staging in existing Casdoor DB
$PSQL -c "UPDATE application SET redirect_uris = REPLACE(redirect_uris,
'example.com', 'staging.example.com')
WHERE name='lobechat'
AND redirect_uris LIKE '%example.com%'
AND redirect_uris NOT LIKE '%staging.example.com%';"
```
Also check `AUTH_CASDOOR_ISSUER` — it must match the Casdoor subdomain (`auth.staging.example.com`), not the app root domain.
## Multi-environment isolation
Before creating a second environment, grep `.tf` files for hardcoded names. See [references/multi-env-isolation.md](references/multi-env-isolation.md) for the complete matrix.
Environment isolation does not mean configuration-contract drift. Keep one required-key manifest and
the same validation path for every environment. Staging and production may use different domains,
credentials, instance sizes, and feature values; they must not disagree about whether a runtime key is
required, optional, allowed-empty, or silently defaulted.
**Will fail on apply** (globally unique):
| Resource | Scope | Fix |
|---|---|---|
| SSH key pair | Region | `"${env}-deploy"` |
| SLS log project | Account | `"${env}-logs"` |
| CloudMonitor contact | Account | `"${env}-ops"` |
**DNS duplication trap**: Two environments creating A records for the same name in the same Cloudflare zone → two independent record IDs → DNS round-robin → ~50% traffic to wrong instance. Fix: use subdomain isolation (`staging.example.com`) or separate zones. Remember to create DNS records for ALL subdomains Caddy serves (e.g., `auth.staging`, `minio.staging`).
**Snapshot cross-contamination**: Unfiltered `data "alicloud_ecs_snapshots"` returns ALL account snapshots. New env inherits old 100GB snapshot, fails creating 40GB disk. Gate with variable:
```hcl
locals {
latest_snapshot_id = var.enable_snapshot_recovery && length(local.available_snapshots) > 0
? local.available_snapshots[0].snapshot_id : null
}
```
Do NOT add `count` to the data source — changes its state address, causes drift.
## Pre-deploy validation
Run the cheapest checks first, but do not let a syntax check certify runtime behavior. HashiCorp's
`terraform validate` checks syntax and internal consistency without remote state or provider APIs.
Preconditions can block before their resource action; postconditions run after change and do not undo
what already happened; `check` assertions warn and continue. Choose the mechanism by when damage must
be prevented.
Key checks (see [references/pre-deploy-validation.md](references/pre-deploy-validation.md)):
1. Format, initialize without backend where appropriate, and run `terraform validate`.
2. Compare every environment against one required-key schema; reject missing, duplicate, and empty values.
3. Render the exact Compose model from the candidate files and controlled interpolation environment.
4. Run the production validator from the exact immutable image/module set against dangerous and healthy fixtures.
5. Generate a saved plan; review resource addresses, actions, target scope, artifact identity, and digest.
6. Apply that exact plan only after the required environment-specific authorization.
7. Verify deployed identity and the real user path independently; only then issue a staging/promotion receipt.
## Zero-to-deployment
Fresh disks expose every implicit dependency. See [references/zero-to-deploy-checklist.md](references/zero-to-deploy-checklist.md).
Key items that break provisioners on fresh instances:
1. **Directories**: `mkdir -p /data/{svc1,svc2}` in cloud-init — `file` provisioner fails if target dir missing
2. **Databases**: Explicit `CREATE DATABASE` — PG init scripts only run on empty data dir
3. **Migrations**: Tracked in `schema_migrations` table, applied idempotently
4. **Provisioner ordering**: `depends_on` between resources sharing Docker networks
5. **Memory**: Stop non-critical containers during Docker build on small instances (≤8GB)
6. **Environment contract**: Every runtime-required value uses the same required-key schema in every environment
7. **Credential capability**: Verify current status and the exact required operation; do not trust shape alone
tests/check-compose-precedence.sh
#!/usr/bin/env bash
set -euo pipefail
skill_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
checklist="$skill_root/references/zero-to-deploy-checklist.md"
release_contract="$skill_root/references/release-safety-and-environment-parity.md"
predeploy="$skill_root/references/pre-deploy-validation.md"
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
stale_claim='Command-line env vars do NOT override `.env` values for compose interpolation.'
if grep -RFn --include='*.md' -- "$stale_claim" "$skill_root"; then
echo "ERROR: stale Docker Compose interpolation precedence claim remains" >&2
exit 1
fi
grep -Fq 'Docker Compose interpolation gives the invoking shell higher precedence' "$checklist"
grep -Fq 'DOCKER_WITH_PROXY_MODE=disabled docker compose build myapp' "$checklist"
grep -Fq 'env -u DOCKER_WITH_PROXY_MODE' "$checklist"
if grep -Eq 'Production apply .*headless|GUI|interactive (apply|approval)|TTY.*approval' "$release_contract"; then
echo "ERROR: release contract requires a host-specific interactive apply gate" >&2
exit 1
fi
grep -Fq 'plan-bound production authorization/audit' "$release_contract"
grep -Fq 'the apply runner remains non-interactive and headless-compatible.' "$release_contract"
grep -Fq 'compose.rendered.json" > "$GATEWAY_ENV_FILE"' "$predeploy"
grep -Fq -- '--env-file "$GATEWAY_ENV_FILE"' "$predeploy"
grep -Fq '(.value | type == "string")' "$predeploy"
grep -Fq '.services[$service].image' "$predeploy"
grep -Fq '[ "$RENDERED_GATEWAY_IMAGE" = "$EXPECTED_CADDY_IMAGE_DIGEST" ]' "$predeploy"
grep -Fq 'test("\\$\\{[A-Za-z_][A-Za-z0-9_]*\\}")' "$predeploy"
grep -Fq 'REQUIRED_ENV_FILE="$CANDIDATE_ROOT/gateway/required-env.keys"' "$predeploy"
grep -Fq '[ -n "$required_value" ]' "$predeploy"
grep -Fq 'placeholder that references a different missing key' "$predeploy"
grep -Fq 'unresolved self- or foreign-key placeholder' "$release_contract"
# Execute the exact documented validator block. Static prose checks previously
# stayed green while the block omitted image parity and placeholder rejection.
awk '
/^```bash$/ { in_bash = 1; next }
in_bash && /^set -euo pipefail$/ { capture = 1 }
capture && /^```$/ { exit }
capture { print }
' "$predeploy" > "$tmp/documented-validator.sh"
test -s "$tmp/documented-validator.sh"
mkdir -p "$tmp/bin"
cat > "$tmp/bin/docker" <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
printf '%s\n' "$@" > "${DOCKER_CALL_LOG:?}"
EOF
chmod +x "$tmp/bin/docker"
image_a="ghcr.io/example/gateway@sha256:$(printf 'a%.0s' {1..64})"
image_b="ghcr.io/example/gateway@sha256:$(printf 'b%.0s' {1..64})"
write_fixture() {
local root=$1 value=$2 image=$3
mkdir -p "$root/gateway"
printf '%s\n' MB_SITE_ADDRESS MB_COOKIE_TOKEN > "$root/gateway/required-env.keys"
jq -n --arg value "$value" --arg image "$image" '{
services: {
"claude4dev-gateway": {
image: $image,
environment: {
MB_SITE_ADDRESS: "example.invalid",
MB_COOKIE_TOKEN: $value,
OPTIONAL_NOTE: ""
}
}
}
}' > "$root/compose.rendered.json"
}
run_documented_validator() {
local root=$1 expected_image=$2 log=$3
CANDIDATE_ROOT="$root" \
EXPECTED_CADDY_IMAGE_DIGEST="$expected_image" \
DOCKER_CALL_LOG="$log" \
PATH="$tmp/bin:$PATH" \
bash "$tmp/documented-validator.sh"
}
write_fixture "$tmp/healthy" token-value "$image_a"
run_documented_validator "$tmp/healthy" "$image_a" "$tmp/healthy.docker"
grep -Fq -- '--env-file' "$tmp/healthy.docker"
grep -Fq -- "$image_a" "$tmp/healthy.docker"
write_fixture "$tmp/image-drift" token-value "$image_a"
if run_documented_validator "$tmp/image-drift" "$image_b" "$tmp/image-drift.docker" 2>/dev/null; then
echo "ERROR: documented validator accepted a rendered image mismatch" >&2
exit 1
fi
test ! -e "$tmp/image-drift.docker"
write_fixture "$tmp/foreign-placeholder" '${MISSING_OTHER}' "$image_a"
if run_documented_validator "$tmp/foreign-placeholder" "$image_a" "$tmp/foreign-placeholder.docker" 2>/dev/null; then
echo "ERROR: documented validator accepted a foreign unresolved placeholder" >&2
exit 1
fi
test ! -e "$tmp/foreign-placeholder.docker"
write_fixture "$tmp/required-empty" '' "$image_a"
if run_documented_validator "$tmp/required-empty" "$image_a" "$tmp/required-empty.docker" 2>/dev/null; then
echo "ERROR: documented validator accepted an explicit-empty required value" >&2
exit 1
fi
test ! -e "$tmp/required-empty.docker"
write_fixture "$tmp/required-missing" token-value "$image_a"
jq 'del(.services["claude4dev-gateway"].environment.MB_COOKIE_TOKEN)' \
"$tmp/required-missing/compose.rendered.json" > "$tmp/required-missing/compose.rendered.next.json"
mv "$tmp/required-missing/compose.rendered.next.json" "$tmp/required-missing/compose.rendered.json"
if run_documented_validator "$tmp/required-missing" "$image_a" "$tmp/required-missing.docker" 2>/dev/null; then
echo "ERROR: documented validator accepted a missing required value" >&2
exit 1
fi
test ! -e "$tmp/required-missing.docker"
write_fixture "$tmp/required-duplicate" token-value "$image_a"
printf '%s\n' MB_COOKIE_TOKEN >> "$tmp/required-duplicate/gateway/required-env.keys"
if run_documented_validator "$tmp/required-duplicate" "$image_a" "$tmp/required-duplicate.docker" 2>/dev/null; then
echo "ERROR: documented validator accepted a duplicate required key" >&2
exit 1
fi
test ! -e "$tmp/required-duplicate.docker"
echo "Compose precedence and headless release documentation are internally consistent."