references/advanced-strategies.md
# Advanced Deployment Strategies Reference
Extended configurations, platform-specific patterns, and advanced rollback strategies.
Core patterns and decision tables live in [`../SKILL.md`](../SKILL.md).
---
## Platform-Specific Pipeline Configurations
### GitHub Actions — Full Production Pipeline with Reusable Workflows
```yaml
# .github/workflows/production.yml
name: Production Pipeline
on:
push:
branches: [main]
workflow_dispatch:
inputs:
skip_tests:
type: boolean
default: false
permissions:
contents: read
id-token: write # for OIDC auth to cloud providers
jobs:
build:
runs-on: ubuntu-latest
outputs:
image_tag: ${{ steps.meta.outputs.version }}
steps:
- uses: actions/checkout@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log in to registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ghcr.io/${{ github.repository }}
tags: |
type=sha,prefix=,format=short
- name: Build and push (with layer cache)
uses: docker/build-push-action@v5
with:
push: true
tags: ${{ steps.meta.outputs.tags }}
cache-from: type=gha
cache-to: type=gha,mode=max
security-scan:
needs: build
runs-on: ubuntu-latest
steps:
- name: Run Trivy vulnerability scan
uses: aquasecurity/trivy-action@0.28.0
with:
image-ref: ghcr.io/${{ github.repository }}:${{ needs.build.outputs.image_tag }}
exit-code: 1
severity: CRITICAL,HIGH
- name: SAST with Semgrep
uses: semgrep/semgrep-action@v1
with:
config: auto
test:
needs: build
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: test
options: >-
--health-cmd pg_isready
--health-interval 10s
steps:
- uses: actions/checkout@v4
- name: Run test suite
run: make test-ci
env:
DATABASE_URL: postgres://postgres:test@localhost/test
deploy-staging:
needs: [test, security-scan]
environment:
name: staging
url: https://staging.example.com
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/deploy-staging
aws-region: us-east-1
- name: Deploy to EKS staging
run: |
aws eks update-kubeconfig --name my-cluster-staging
kubectl set image deployment/my-app \
app=ghcr.io/${{ github.repository }}:${{ needs.build.outputs.image_tag }}
kubectl rollout status deployment/my-app --timeout=5m
e2e-tests:
needs: deploy-staging
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Playwright E2E suite
run: npx playwright test --reporter=github
env:
BASE_URL: https://staging.example.com
deploy-production:
needs: e2e-tests
environment:
name: production
url: https://app.example.com
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials (OIDC)
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/deploy-production
aws-region: us-east-1
- name: Deploy canary to production
run: |
aws eks update-kubeconfig --name my-cluster-prod
kubectl argo rollouts set image my-app \
app=ghcr.io/${{ github.repository }}:${{ needs.build.outputs.image_tag }}
- name: Monitor canary promotion
run: |
kubectl argo rollouts status my-app --timeout=30m
- name: Rollback on failure
if: failure()
run: kubectl argo rollouts abort my-app
```
---
### GitLab CI — Multi-Environment Pipeline with Dynamic Environments
```yaml
# .gitlab-ci.yml
stages:
- build
- test
- staging
- production
variables:
DOCKER_DRIVER: overlay2
IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
build:
stage: build
image: docker:24
services:
- docker:24-dind
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build --cache-from $CI_REGISTRY_IMAGE:latest -t $IMAGE .
- docker push $IMAGE
- docker tag $IMAGE $CI_REGISTRY_IMAGE:latest
- docker push $CI_REGISTRY_IMAGE:latest
test:unit:
stage: test
image: $IMAGE
script:
- make test
coverage: '/coverage: \d+\.\d+%/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
test:security:
stage: test
image:
name: aquasec/trivy:latest
entrypoint: [""]
script:
- trivy image --exit-code 1 --severity CRITICAL,HIGH $IMAGE
deploy:staging:
stage: staging
environment:
name: staging
url: https://staging.example.com
on_stop: stop:staging
script:
- kubectl apply -f k8s/staging/
- kubectl set image deployment/my-app app=$IMAGE -n staging
- kubectl rollout status deployment/my-app -n staging
only:
- main
stop:staging:
stage: staging
environment:
name: staging
action: stop
script:
- kubectl delete namespace staging --ignore-not-found
when: manual
only:
- main
deploy:production:
stage: production
environment:
name: production
url: https://app.example.com
script:
- kubectl set image deployment/my-app app=$IMAGE -n production
- kubectl rollout status deployment/my-app -n production --timeout=10m
when: manual
only:
- main
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
when: manual
allow_failure: false
```
---
### Azure Pipelines — Multi-Stage with Approvals and Environments
```yaml
# azure-pipelines.yml
trigger:
branches:
include:
- main
variables:
imageRepository: 'myapp'
containerRegistry: 'myregistry.azurecr.io'
tag: '$(Build.BuildId)'
stages:
- stage: Build
displayName: 'Build & Test'
jobs:
- job: BuildAndTest
pool:
vmImage: ubuntu-latest
steps:
- task: Docker@2
displayName: Build image
inputs:
command: build
repository: $(imageRepository)
dockerfile: Dockerfile
tags: $(tag)
- task: Docker@2
displayName: Push image
inputs:
command: push
repository: $(imageRepository)
tags: $(tag)
- script: make test
displayName: Run tests
- stage: Staging
displayName: 'Deploy to Staging'
dependsOn: Build
jobs:
- deployment: DeployStaging
environment: staging
pool:
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- task: KubernetesManifest@0
inputs:
action: deploy
manifests: k8s/staging/*.yaml
containers: $(containerRegistry)/$(imageRepository):$(tag)
- stage: Production
displayName: 'Deploy to Production'
dependsOn: Staging
jobs:
- deployment: DeployProduction
environment:
name: production
resourceType: Kubernetes
pool:
vmImage: ubuntu-latest
strategy:
canary:
increments: [10, 25, 50]
preDeploy:
steps:
- task: ManualValidation@0
inputs:
notifyUsers: 'release-managers@example.com'
instructions: 'Verify staging metrics. Approve to start canary.'
onTimeout: reject
deploy:
steps:
- task: KubernetesManifest@0
inputs:
action: deploy
manifests: k8s/production/*.yaml
containers: $(containerRegistry)/$(imageRepository):$(tag)
postRouteTraffic:
steps:
- script: ./scripts/verify-deployment.sh https://app.example.com
on:
failure:
steps:
- task: KubernetesManifest@0
inputs:
action: reject
```
---
## Multi-Region Canary Promotion
Deploy to a pilot region first, validate, then promote to remaining regions in parallel:
```yaml
# deploy-multiregion.yml (GitHub Actions)
jobs:
deploy-pilot:
environment: production-us-east-1
runs-on: ubuntu-latest
steps:
- name: Deploy canary to pilot region
run: |
aws eks update-kubeconfig --name cluster-us-east-1 --region us-east-1
kubectl argo rollouts set image my-app app=$IMAGE
kubectl argo rollouts status my-app --timeout=20m
deploy-secondary:
needs: deploy-pilot
strategy:
matrix:
region: [us-west-2, eu-west-1, ap-southeast-1]
environment: production-${{ matrix.region }}
runs-on: ubuntu-latest
steps:
- name: Deploy to ${{ matrix.region }}
run: |
aws eks update-kubeconfig --name cluster-${{ matrix.region }} \
--region ${{ matrix.region }}
kubectl argo rollouts set image my-app app=$IMAGE
kubectl argo rollouts status my-app --timeout=20m
```
---
## Advanced Argo Rollouts Patterns
### Experiment-Based Canary (A/B Analysis)
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: my-app
spec:
replicas: 20
strategy:
canary:
canaryService: my-app-canary
stableService: my-app-stable
trafficRouting:
istio:
virtualService:
name: my-app-vsvc
analysis:
templates:
- templateName: success-rate
- templateName: latency-p99
startingStep: 1
args:
- name: service-name
value: my-app-canary
steps:
- setWeight: 5
- experiment:
templates:
- name: baseline
specRef: stable
- name: canary
specRef: canary
analyses:
- templateName: ab-test
requiredForCompletion: true
- setWeight: 20
- pause: { duration: 10m }
- setWeight: 50
- pause: { duration: 15m }
- setWeight: 100
```
### Header-Based Canary (Internal Testing Before Traffic Split)
```yaml
# Route users with X-Canary: true header to canary pods
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-app-vsvc
spec:
http:
- match:
- headers:
x-canary:
exact: "true"
route:
- destination:
host: my-app-canary
- route:
- destination:
host: my-app-stable
weight: 90
- destination:
host: my-app-canary
weight: 10
```
---
## Database Migration Rollback Strategies
### Expand/Contract Pattern (Safe Schema Changes)
Never break backward compatibility within a single deployment cycle. Use the expand/contract pattern across three releases:
```
Release N: Add nullable column (expand)
Release N+1: Backfill data, deploy new code that reads new column
Release N+2: Drop old column (contract) — safe because no code references it
```
```bash
# Release N migration — backward compatible
cat > migrations/V20240315__expand_add_email_v2.sql <<'EOF'
ALTER TABLE users ADD COLUMN email_v2 VARCHAR(255);
CREATE INDEX CONCURRENTLY idx_users_email_v2 ON users(email_v2);
EOF
# Release N+2 migration — contract (only after old code retired)
cat > migrations/V20240415__contract_drop_email.sql <<'EOF'
ALTER TABLE users DROP COLUMN email;
ALTER TABLE users RENAME COLUMN email_v2 TO email;
EOF
```
### Flyway with Undo Scripts
```bash
# Forward migration
flyway migrate
# Undo the most recent migration (requires Flyway Teams)
flyway undo
# Undo a specific version
flyway undo -target=20240315
```
Undo script naming convention: `U20240315__expand_add_email_v2.sql` (prefix `U` instead of `V`).
### Zero-Downtime Index Creation (PostgreSQL)
```sql
-- Blocking (avoid in production):
CREATE INDEX idx_users_email ON users(email);
-- Non-blocking (safe for live systems):
CREATE INDEX CONCURRENTLY idx_users_email ON users(email);
```
Use `CREATE INDEX CONCURRENTLY` in all production migrations. Add it to a pre-deploy migration step that runs before the application update.
---
## Blue-Green with Database (Full Example)
```yaml
# blue-green-deploy.yml
steps:
- name: Deploy green environment
run: |
kubectl apply -f k8s/green/
- name: Run database migrations (forward, backward-compatible)
run: |
kubectl exec -n production deploy/migration-job -- flyway migrate
- name: Smoke test green
run: |
kubectl port-forward -n production svc/my-app-green 8080:80 &
sleep 3
curl -sf http://localhost:8080/health/ready
- name: Switch traffic to green
run: |
kubectl patch service my-app \
-p '{"spec":{"selector":{"slot":"green"}}}'
- name: Verify green is live
run: ./scripts/verify-deployment.sh https://app.example.com
- name: Scale down blue
run: |
kubectl scale deployment my-app-blue --replicas=0
- name: Rollback to blue on failure
if: failure()
run: |
kubectl patch service my-app \
-p '{"spec":{"selector":{"slot":"blue"}}}'
kubectl scale deployment my-app-blue --replicas=5
```
---
## Deployment Freeze Automation
Automatically block production deployments during freeze windows:
```python
#!/usr/bin/env python3
# scripts/check-freeze-window.py
import sys
from datetime import datetime, timezone
FREEZE_WINDOWS = [
# (month, day_start, day_end, description)
(11, 25, 30, "US Thanksgiving"),
(12, 20, 31, "Year-end freeze"),
(1, 1, 2, "New Year"),
]
now = datetime.now(timezone.utc)
month, day = now.month, now.day
for fm, d_start, d_end, label in FREEZE_WINDOWS:
if fm == month and d_start <= day <= d_end:
print(f"BLOCKED: Deployment freeze active — {label}")
print("Override with FORCE_DEPLOY=true environment variable if critical.")
if not os.environ.get("FORCE_DEPLOY"):
sys.exit(1)
print("No active freeze window — deployment allowed.")
```
Use in pipeline:
```yaml
- name: Check deployment freeze window
run: python scripts/check-freeze-window.py
env:
FORCE_DEPLOY: ${{ vars.FORCE_DEPLOY }}
```
---
## Notification Templates
### Slack Deployment Notification
```bash
#!/usr/bin/env bash
# scripts/notify-slack.sh
STATUS="${1:?pass 'success' or 'failure'}"
WEBHOOK="${SLACK_WEBHOOK:?SLACK_WEBHOOK not set}"
REPO="${GITHUB_REPOSITORY:-unknown}"
SHA="${GITHUB_SHA:-unknown}"
ACTOR="${GITHUB_ACTOR:-unknown}"
if [ "$STATUS" = "success" ]; then
COLOR="good"
EMOJI=":white_check_mark:"
TEXT="Production deploy succeeded"
else
COLOR="danger"
EMOJI=":red_circle:"
TEXT="Production deploy FAILED — rollback triggered"
fi
curl -sf -X POST "$WEBHOOK" \
-H "Content-Type: application/json" \
-d "{
\"attachments\": [{
\"color\": \"$COLOR\",
\"text\": \"$EMOJI $TEXT\",
\"fields\": [
{\"title\": \"Repo\", \"value\": \"$REPO\", \"short\": true},
{\"title\": \"SHA\", \"value\": \"${SHA:0:7}\", \"short\": true},
{\"title\": \"Deployed by\", \"value\": \"$ACTOR\", \"short\": true}
]
}]
}"
```
references/details.md
# deployment-pipeline-design — detailed patterns and worked examples
## Pipeline Stages
### Standard Pipeline Flow
```
┌─────────┐ ┌──────┐ ┌─────────┐ ┌────────┐ ┌──────────┐
│ Build │ → │ Test │ → │ Staging │ → │ Approve│ → │Production│
└─────────┘ └──────┘ └─────────┘ └────────┘ └──────────┘
```
### Detailed Stage Breakdown
1. **Source** - Code checkout, dependency graph resolution
2. **Build** - Compile, package, containerize, sign artifacts
3. **Test** - Unit, integration, SAST/SCA security scans
4. **Staging Deploy** - Deploy to staging environment with smoke tests
5. **Integration Tests** - E2E, contract tests, performance baselines
6. **Approval Gate** - Manual or automated metric-based gate
7. **Production Deploy** - Canary, blue-green, or rolling strategy
8. **Verification** - Deep health checks, synthetic monitoring
9. **Rollback** - Automated rollback on failure signals
## Approval Gate Patterns
### Pattern 1: Manual Approval (GitHub Actions)
```yaml
production-deploy:
needs: staging-deploy
environment:
name: production
url: https://app.example.com
runs-on: ubuntu-latest
steps:
- name: Deploy to production
run: kubectl apply -f k8s/production/
```
Environment protection rules in GitHub enforce required reviewers before this job starts. Configure reviewers at **Settings → Environments → production → Required reviewers**.
### Pattern 2: Time-Based Approval (GitLab CI)
```yaml
deploy:production:
stage: deploy
script:
- deploy.sh production
environment:
name: production
when: delayed
start_in: 30 minutes
only:
- main
```
### Pattern 3: Multi-Approver (Azure Pipelines)
```yaml
stages:
- stage: Production
dependsOn: Staging
jobs:
- deployment: Deploy
environment:
name: production
resourceType: Kubernetes
strategy:
runOnce:
preDeploy:
steps:
- task: ManualValidation@0
inputs:
notifyUsers: "team-leads@example.com"
instructions: "Review staging metrics before approving"
```
### Pattern 4: Automated Metric Gate
Use an AnalysisTemplate (Argo Rollouts) or a custom gate script to block promotion when error rates exceed a threshold:
```yaml
# Argo Rollouts AnalysisTemplate — blocks canary promotion automatically
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate
spec:
metrics:
- name: success-rate
interval: 60s
successCondition: "result[0] >= 0.95"
failureCondition: "result[0] < 0.90"
inconclusiveLimit: 3
provider:
prometheus:
address: http://prometheus:9090
query: |
sum(rate(http_requests_total{status!~"5..",job="my-app"}[2m]))
/ sum(rate(http_requests_total{job="my-app"}[2m]))
```
## Deployment Strategies
### Decision Table
| Strategy | Downtime | Rollback Speed | Cost Impact | Best For |
|-------------|----------|----------------|-----------------|----------------------------------|
| Rolling | None | ~minutes | None | Most stateless services |
| Blue-Green | None | Instant | 2x infra (temp) | High-risk or database migrations |
| Canary | None | Instant | Minimal | High-traffic, metric-driven |
| Recreate | Yes | Fast | None | Dev/test, batch jobs |
| Feature Flag | None | Instant | None | Gradual feature exposure |
### 1. Rolling Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 10
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 2 # at most 12 pods during rollout
maxUnavailable: 1 # at least 9 pods always serving
```
Characteristics: gradual rollout, zero downtime, easy rollback, best for most applications.
### 2. Blue-Green Deployment
```bash
# Switch traffic from blue to green
kubectl apply -f k8s/green-deployment.yaml
kubectl rollout status deployment/my-app-green
# Flip the service selector
kubectl patch service my-app -p '{"spec":{"selector":{"version":"green"}}}'
# Rollback instantly if needed
kubectl patch service my-app -p '{"spec":{"selector":{"version":"blue"}}}'
```
Characteristics: instant switchover, easy rollback, doubles infrastructure cost temporarily, good for high-risk deployments with long warm-up times.
### 3. Canary Deployment (Argo Rollouts)
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: my-app
spec:
replicas: 10
strategy:
canary:
analysis:
templates:
- templateName: success-rate
startingStep: 2
steps:
- setWeight: 10
- pause: { duration: 5m }
- setWeight: 25
- pause: { duration: 5m }
- setWeight: 50
- pause: { duration: 10m }
- setWeight: 100
```
Characteristics: gradual traffic shift, real-user metric validation, automated promotion or rollback, requires Argo Rollouts or a service mesh.
### 4. Feature Flags
```python
from flagsmith import Flagsmith
flagsmith = Flagsmith(environment_key="API_KEY")
if flagsmith.has_feature("new_checkout_flow"):
process_checkout_v2()
else:
process_checkout_v1()
```
Characteristics: deploy without releasing, A/B testing, instant rollback per user segment, granular control independent of deployment.
## Pipeline Orchestration
### Multi-Stage Pipeline Example (GitHub Actions)
```yaml
name: Production Pipeline
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
outputs:
image: ${{ steps.build.outputs.image }}
steps:
- uses: actions/checkout@v4
- name: Build and push Docker image
id: build
run: |
IMAGE=myapp:${{ github.sha }}
docker build -t $IMAGE .
docker push $IMAGE
echo "image=$IMAGE" >> $GITHUB_OUTPUT
test:
needs: build
runs-on: ubuntu-latest
steps:
- name: Unit tests
run: make test
- name: Security scan
run: trivy image ${{ needs.build.outputs.image }}
deploy-staging:
needs: test
environment:
name: staging
runs-on: ubuntu-latest
steps:
- name: Deploy to staging
run: kubectl apply -f k8s/staging/
integration-test:
needs: deploy-staging
runs-on: ubuntu-latest
steps:
- name: Run E2E tests
run: npm run test:e2e
deploy-production:
needs: integration-test
environment:
name: production # blocks here until required reviewers approve
runs-on: ubuntu-latest
steps:
- name: Canary deployment
run: |
kubectl apply -f k8s/production/
kubectl argo rollouts promote my-app
verify:
needs: deploy-production
runs-on: ubuntu-latest
steps:
- name: Deep health check
run: |
for i in {1..12}; do
STATUS=$(curl -sf https://app.example.com/health/ready | jq -r '.status')
[ "$STATUS" = "ok" ] && exit 0
sleep 10
done
exit 1
- name: Notify on success
run: |
curl -X POST ${{ secrets.SLACK_WEBHOOK }} \
-d '{"text":"Production deployment successful: ${{ github.sha }}"}'
```
## Health Checks
### Shallow vs Deep Health Endpoints
A shallow `/ping` returns 200 even when downstream dependencies are broken. Use a deep readiness endpoint that verifies actual dependencies before promoting traffic.
```python
# /health/ready — checks real dependencies, used by pipeline gate
@app.get("/health/ready")
async def readiness():
checks = {
"database": await check_db_connection(),
"cache": await check_redis_connection(),
"queue": await check_queue_connection(),
}
status = "ok" if all(checks.values()) else "degraded"
code = 200 if status == "ok" else 503
return JSONResponse({"status": status, "checks": checks}, status_code=code)
```
### Post-Deployment Verification Script
```bash
#!/usr/bin/env bash
# verify-deployment.sh — run after every production deploy
set -euo pipefail
ENDPOINT="${1:?usage: verify-deployment.sh <base-url>}"
MAX_ATTEMPTS=12
SLEEP_SECONDS=10
for i in $(seq 1 $MAX_ATTEMPTS); do
STATUS=$(curl -sf "$ENDPOINT/health/ready" | jq -r '.status' 2>/dev/null || echo "unreachable")
if [ "$STATUS" = "ok" ]; then
echo "Health check passed after $((i * SLEEP_SECONDS))s"
exit 0
fi
echo "Attempt $i/$MAX_ATTEMPTS: status=$STATUS — retrying in ${SLEEP_SECONDS}s"
sleep "$SLEEP_SECONDS"
done
echo "Health check failed after $((MAX_ATTEMPTS * SLEEP_SECONDS))s"
exit 1
```
## Rollback Strategies
### Automated Rollback in Pipeline
```yaml
deploy-and-verify:
steps:
- name: Deploy new version
run: kubectl apply -f k8s/
- name: Wait for rollout
run: kubectl rollout status deployment/my-app --timeout=5m
- name: Post-deployment health check
id: health
run: ./scripts/verify-deployment.sh https://app.example.com
- name: Rollback on failure
if: failure()
run: |
kubectl rollout undo deployment/my-app
echo "Rolled back to previous revision"
```
### Manual Rollback Commands
```bash
# List revision history with change-cause annotations
kubectl rollout history deployment/my-app
# Rollback to previous version
kubectl rollout undo deployment/my-app
# Rollback to a specific revision
kubectl rollout undo deployment/my-app --to-revision=3
# Verify rollback completed
kubectl rollout status deployment/my-app
```
For advanced rollback strategies including database migration rollbacks and Argo Rollouts abort flows, see [`references/advanced-strategies.md`](references/advanced-strategies.md).
## Monitoring and Metrics
### Key DORA Metrics to Track
| Metric | Target (Elite) | How to Measure |
|--------------------------|----------------|------------------------------------------|
| Deployment Frequency | Multiple/day | Pipeline run count per day |
| Lead Time for Changes | < 1 hour | Commit timestamp → production deploy |
| Change Failure Rate | < 5% | Failed deploys / total deploys |
| Mean Time to Recovery | < 1 hour | Incident open → service restored |
### Post-Deployment Metric Verification
```yaml
- name: Verify error rate post-deployment
run: |
sleep 60 # allow metrics to accumulate
ERROR_RATE=$(curl -sf "$PROMETHEUS_URL/api/v1/query" \
--data-urlencode 'query=sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))' \
| jq '.data.result[0].value[1]')
echo "Current error rate: $ERROR_RATE"
if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
echo "Error rate $ERROR_RATE exceeds 1% threshold — triggering rollback"
exit 1
fi
```
## Pipeline Best Practices
1. **Fail fast** — Run quick checks (lint, unit tests) before slow ones (E2E, security scans)
2. **Parallel execution** — Run independent jobs concurrently to minimize total pipeline time
3. **Caching** — Cache dependency layers and build artifacts between runs
4. **Artifact promotion** — Build once, promote the same artifact through all environments
5. **Environment parity** — Keep staging infrastructure as close to production as possible
6. **Secrets management** — Use secret stores (Vault, AWS Secrets Manager, GitHub encrypted secrets) — never hardcode
7. **Deployment windows** — Prefer low-traffic windows; enforce change freeze periods via gate policies
8. **Idempotent deploys** — Ensure re-running a deploy produces the same result
9. **Rollback automation** — Trigger rollback automatically on health check or metric threshold failure
10. **Annotate deployments** — Send deployment markers to monitoring tools (Datadog, Grafana) for correlation
SKILL.md
---
name: deployment-pipeline-design
description: Design multi-stage CI/CD pipelines with approval gates, security checks, and deployment orchestration. Use this skill when designing zero-downtime deployment pipelines, implementing canary rollout strategies, setting up multi-environment promotion workflows, or debugging failed deployment gates in CI/CD.
---
# Deployment Pipeline Design
Architecture patterns for multi-stage CI/CD pipelines with approval gates, deployment strategies, and environment promotion workflows.
## Purpose
Design robust, secure deployment pipelines that balance speed with safety through proper stage organization, automated quality gates, and progressive delivery strategies. This skill covers both the structural design of pipeline architecture and the operational patterns for reliable production deployments.
## Input / Output
### What You Provide
- **Application type**: Language/runtime, containerized or bare-metal, monolith or microservices
- **Deployment target**: Kubernetes, ECS, VMs, serverless, or platform-as-a-service
- **Environment topology**: Number of environments (dev/staging/prod), region layout, air-gap requirements
- **Rollout requirements**: Acceptable downtime, rollback SLA, traffic splitting needs, canary vs blue-green preference
- **Gate constraints**: Approval teams, required test coverage thresholds, compliance scans (SAST, DAST, SCA)
- **Monitoring stack**: Prometheus, Datadog, CloudWatch, or other metrics sources used for automated promotion decisions
### What This Skill Produces
- **Pipeline configuration**: Stage definitions, job dependencies, parallelism, and caching strategy
- **Deployment strategy**: Chosen rollout pattern with annotated configuration (canary weights, blue-green switchover, rolling parameters)
- **Health check setup**: Shallow vs deep readiness probes, post-deployment smoke test scripts
- **Gate definitions**: Automated metric thresholds and manual approval workflows
- **Rollback plan**: Automated rollback triggers and manual runbook steps
## When to Use
- Design CI/CD architecture for a new service or platform migration
- Implement deployment gates between environments
- Configure multi-environment pipelines with mandatory security scanning
- Establish progressive delivery with canary or blue-green strategies
- Debug pipelines where stages succeed but production behavior is wrong
- Reduce mean time to recovery by automating rollback on metric degradation
## Detailed patterns and worked examples
Detailed pattern documentation lives in `references/details.md`. Read that file when the navigation tier above is insufficient.
## Troubleshooting
### Health check passes in pipeline but service is unhealthy in production
The pipeline health check is hitting a shallow `/ping` endpoint that returns 200 even when the database is unreachable. Use a deep readiness check that verifies actual dependencies (see Health Checks section above).
### Canary deployment never promotes to 100%
Argo Rollouts requires a valid `AnalysisTemplate` to auto-promote. If the Prometheus query returns no data (e.g., metric name changed), the analysis stays inconclusive and promotion stalls. Add `inconclusiveLimit` so the rollout fails fast rather than hanging:
```yaml
spec:
metrics:
- name: error-rate
failureCondition: "result[0] > 0.05"
inconclusiveLimit: 2 # fail after 2 inconclusive results, not hang indefinitely
provider:
prometheus:
query: |
sum(rate(http_requests_total{status=~"5.."}[2m]))
/ sum(rate(http_requests_total[2m]))
```
### Staging deploy succeeds but production job never starts
Check that production environment protection rules are configured — a missing reviewer assignment means the approval gate waits indefinitely with no notification. In GitHub Actions, ensure `Required reviewers` is set to an existing user or team in **Settings → Environments → production**.
### Docker layer cache busted on every run causing slow builds
If `COPY . .` appears before dependency installation, any source file change invalidates the dependency layer. Reorder to copy dependency manifests first:
```dockerfile
# Good: dependencies cached separately from source code
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
```
### Rollback leaves database migrations applied to old code
A service rollback without a migration rollback causes schema/code mismatch errors. Always make migrations backward-compatible (additive only) for at least one release cycle, and keep undo scripts versioned alongside the migration:
```bash
# migrations/V20240315__add_nullable_column.sql (forward)
# migrations/V20240315__add_nullable_column.undo.sql (backward)
```
Never run destructive migrations (DROP COLUMN, ALTER NOT NULL) until the old code version is fully retired from all environments.
## Advanced Topics
For platform-specific pipeline configurations, multi-region promotion workflows, and advanced Argo Rollouts patterns, see:
- [`references/advanced-strategies.md`](references/advanced-strategies.md) — Extended YAML examples, platform-specific configs (GitHub Actions, GitLab CI, Azure Pipelines), multi-region canary patterns, and database migration rollback strategies
## Related Skills
- `github-actions-templates` - For GitHub Actions implementation patterns and reusable workflows
- `gitlab-ci-patterns` - For GitLab CI/CD pipeline implementation
- `secrets-management` - For secrets handling in CI/CD pipelines