assets/deployment-strategy-matrix.md
# Deployment Strategy Matrix
> Choose a strategy per release based on risk, reversibility, and audience. Strategies combine: feature flags gate a feature's visibility while a canary gates the binary. Rollback time assumes an immutable, previously healthy artifact is available.
## Comparison
| Strategy | Speed to full rollout | Safety (defect exposure) | Rollback time | Complexity | When to use |
|----------|----------------------|--------------------------|---------------|------------|-------------|
| Rolling | Fast — minutes | Moderate — all instances briefly run the new version as it spreads | Minutes (redeploy previous artifact) | Low | Default for stateless services with good monitoring |
| Blue-green | Fast cutover; swap in seconds | High — the full old environment stays live | Seconds (router change) | Medium; 2× resources | Risky changes needing instant rollback; infrastructure you can afford to double |
| Canary | Medium — staged % over minutes–hours | High — defects hit a small slice first | Minutes (redeploy / auto-rollback of the canary) | Medium | Risky changes; comparing against a control population |
| Ring | Slow by design — hours–days across cohorts | Very high — exposure grows by ring | Slow (rings are users/devices, hard to recall) | High | Large user bases, clients/devices, compliance-driven exposure limits |
| Feature flag | Instant on/off — no redeploy | Depends on rollout %; off = zero exposure | Seconds (toggle off) | Medium (flag debt) | Decoupling deploy from release; dark launch; kill switch |
| Shadow | Immediate (traffic mirrored) | No user impact (nothing live is served) | N/A (nothing served) | High | Validating performance/behavior without exposing users |
## Decision Guidance
| Situation | Pick |
|-----------|------|
| Stateless service, good monitoring | Rolling or canary |
| Change can break irreversibly (schema, data) | Canary + expand/contract migration; never rely on rollback past finalization |
| Instant rollback is a hard requirement | Blue-green (or canary with auto-rollback) |
| Mobile / desktop / IoT users | Ring (phased release) + feature flags / kill switch — true rollback is impossible |
| Defect can be gated behind a flag | Flag off is the fastest lever — reach for it before any redeploy |
| Need to compare behavior vs. control | Canary with canary-vs-control metrics |
| Regulatory exposure limits | Ring with approved exposure caps per cohort |
## Gotchas
- **Teeing / shadowing limits:** synthetic or mirrored traffic does not validate stateful systems (e.g., billing) — a shadow run can look perfect and still fail on real writes.
- **Before/after comparisons lie:** compare canary vs. control populations, not the same metric over time.
- **Flags are not free:** every flag left behind is rollback speed today and operational debt tomorrow — plan removal in the flag's lifecycle.
- **Canary size × error budget:** a 5% canary at 20% error costs ~1% of the overall budget — size the canary so a bad release stays inside the budget.
## Sources and Further Reading
- Google SRE Workbook ch. 16 — Canarying Releases: https://sre.google/workbook/canarying-releases/
- Google SRE Book ch. 8 — Release Engineering: https://sre.google/sre-book/release-engineering/
- Google Cloud — Reliable releases and rollbacks: https://cloud.google.com/blog/products/gcp/reliable-releases-and-rollbacks-cre-life-lessons
- Martin Fowler — Parallel Change: https://martinfowler.com/bliki/ParallelChange.html
- Runway — Rollbacks on mobile: https://www.runway.team/blog/rollbacks-on-mobile-yes-they-are-possible-and-this-is-why-you-need-them
assets/dora-metrics-reference.md
# DORA Metrics Quick Reference
> The five research-backed software delivery metrics, how to compute them, and the classic thresholds. Use for measuring, dashboarding, and reporting delivery performance. The metrics measure system outcomes, not individual performance — do not use them for personal evaluation.
## The Five Metrics
| Metric | What it measures | Formula / unit | Data source |
|--------|------------------|----------------|-------------|
| Deployment frequency (DF) | How often code reaches a production environment | Successful production deployments per day (or per week) | Deploy logs / CI-CD platform / GitOps sync records |
| Change lead time (CLT) | Time from commit to running in production | Median of (deploy finished_at − commit created_at) over deployed commits; unit: hours/days | Version-control commit timestamps + deploy records |
| Change failure rate (CFR) | Share of deployments that cause degraded service | Failed or remediated deploys ÷ total deploys × 100 (%) | Deploy ↔ incident correlation (rollbacks, hotfixes, incident tickets tied to a deploy) |
| Failed deployment recovery time | Time to restore service after a failed deploy | Median time from failed deploy start to next successful deploy; unit: minutes/hours | Incident + deploy timeline |
| Deployment rework rate | Share of deployments needing rework (rollback, hotfix, forward fix) | Unplanned rework deployments ÷ total deployments × 100 (%) | Deploy records flagged as unplanned |
## Classic 2024 Thresholds (the last four-tier table)
| Tier | Deployment frequency | Change lead time | Change failure rate | Failed deployment recovery time |
|------|----------------------|------------------|---------------------|-------------------------------|
| Elite | On-demand (multiple deploys/day) | Less than one day | 5% | Less than one hour |
| High | Daily to weekly | One day to one week | 20% | Less than one day |
| Medium | Weekly to monthly | One week to one month | 10% | Less than one day |
| Low | Monthly to biannual | One to six months | 40% | One week to one month |
Note the 2024 **inversion**: High shows a higher change failure rate (20%) than Medium (10%) — clusters are descriptive groupings, not a monotonic scorecard.
> **2025 change caveat —** the DORA team **retired the Elite/High/Medium/Low tiers entirely** in the 2025 report (renamed "State of AI-assisted Software Development"), replacing them with seven qualitative archetypes built on eight measures. 2025 publishes metric *distributions*, not tiers. **Deployment rework rate** was added in 2024 as the fifth metric, not in 2025. Do not hard-code the 2024 threshold table into dashboards; treat it as a historical reference point anchored to the 2024 report.
## Top Pitfalls
- **PRs ≠ deploys.** Count deployments of code to production, not merged pull requests or commits.
- **Mean vs. median.** Use the median for lead time and recovery time — the mean is skewed by rare long outliers.
- **Repo vs. service.** Measure per deployable service, not per repository (a monorepo may contain many services).
- **Ignoring rollbacks.** A rolled-back deploy is a failure — excluding it inflates both DF and CFR.
- **Time-source mismatch.** Commit and deploy timestamps must be comparable (UTC, NTP-synced) or lead time is meaningless.
- **Manual counting.** Spreadsheets drift; derive the metrics from pipelines and GitOps records automatically.
- **Gaming the metric.** Raising DF without improving CFR or lead time just amplifies bad change.
## Sources and Further Reading
- DORA — research and metric definitions: https://dora.dev/
- Accelerate (Forsgren, Humble, Kim, 2018): https://itrevolution.com/product/accelerate/
- DORA metrics measurement guidance: https://dora.dev/research/measurement/
- Google Cloud DORA blog (2025 tier retirement): https://cloud.google.com/blog/products/devops-sre
assets/release-toolchain-cheatsheet.md
# Release Toolchain Cheatsheet
> One-liner per tool with a "pick when" note. Status reflects the 2025–2026 landscape; verify current vendor support before committing. This is a selection map, not an endorsement.
## CI/CD
| Tool | One-liner | Pick when |
|------|-----------|-----------|
| GitHub Actions | Cloud or self-hosted workflows with a huge ecosystem and native artifact attestation | You live on GitHub and want tight repo→CI coupling plus SLSA provenance out of the box |
| GitLab CI | Single-application CI/CD with built-in environments, review apps, and protected environments | You use GitLab and want CI + CD + security in one platform |
| Jenkins | Mature, plugin-driven automation server you self-host | Legacy estates, on-prem requirements, maximum plugin flexibility |
| CircleCI | Fast cloud CI with strong caching and parallelism | Small-to-mid teams wanting speed with minimal ops |
| Buildkite | Hybrid model: your infrastructure, their orchestration | You need your own runners for compliance/performance but want managed orchestration |
| Tekton | Kubernetes-native CI/CD building blocks | You are Kubernetes-native and want pipelines as CRDs |
| Dagger | CI/CD as code in your own language, runs anywhere | You want portable pipelines free of a single vendor |
## Release Automation
| Tool | One-liner | Pick when |
|------|-----------|-----------|
| semantic-release | Auto-version + auto-changelog from conventional commits | Single-package repos on GitHub; zero-touch semantic releases |
| release-please | Google-style release PRs; manifest mode for monorepos | Monorepos needing per-package versioning with a release train |
| changesets | Intentional, PR-driven version bumps for monorepos | You want humans to decide release content, not automagic |
| release-drafter | Drafts release notes from merged PR labels | You want fast GitHub release notes driven by PR labels |
| git-cliff | Changelog generator from conventional commits, highly configurable | You want a changelog generated locally or in any CI |
## Artifact Repositories
| Tool | One-liner | Pick when |
|------|-----------|-----------|
| Artifactory | Universal artifact manager with proxy, security, and promotion | Multi-format artifacts (containers, Maven, npm, PyPI) in one place |
| Nexus | Open-source artifact repository (Maven/npm/PyPI) | You want a self-hosted, budget-friendly repo, especially for JVM ecosystems |
| GHCR / ECR / GAR | Cloud-native container registries with IAM and signing | You are all-in on one cloud and want zero extra infrastructure |
## GitOps / Deployment
| Tool | One-liner | Pick when |
|------|-----------|-----------|
| Argo CD | Declarative GitOps for Kubernetes with sync, rollback, and SSO | You want Git as the single source of truth for cluster state |
| Flux | CNCF GitOps toolkit with automation and OCI support | You want GitOps + progressive delivery with strong multi-tenancy |
| Argo Rollouts | Advanced rollout strategies (canary/blue-green) for Kubernetes | You need canary with automated analysis on Kubernetes |
| Flagger | Progressive delivery operator that automates canary releases | You want metric-driven automated canary promotion |
| Harness | Enterprise CD / feature-flag / verification platform | You want a managed, approval-heavy enterprise platform |
| Spinnaker | Cloud-native CD with complex pipelines; declining maintenance | You inherited it; new projects should look elsewhere |
## Feature Flags
| Tool | One-liner | Pick when |
|------|-----------|-----------|
| LaunchDarkly | Enterprise feature management with targeting + experimentation | You need scale, audit, and kill-switch-grade reliability |
| Flagsmith | Open-core feature flags with remote config | You want a self-hostable, OSS-friendly option |
| Unleash | Open-source feature toggles, simple and fast | You want a lean OSS toggle server with good SDKs |
| OpenFeature | Vendor-neutral open standard for flag evaluation | You want to avoid lock-in and swap providers later |
## Security / Supply Chain
| Tool | One-liner | Pick when |
|------|-----------|-----------|
| Syft | Generates SBOMs from images/filesystems (SPDX, CycloneDX) | You need SBOM generation in every build |
| Trivy | Fast vulnerability scanner for images, repos, and SBOMs | You want one scanner for CI + registry + cluster |
| Cosign / sigstore | Keyless container signing + attestation | You want verifiable signatures without key management |
| slsa-github-generator | GitHub Actions producing SLSA L3 provenance | You want build provenance auditors accept |
| Dependabot / Renovate | Automated dependency-update PRs | You want dependency drift under control continuously |
## Observability
| Tool | One-liner | Pick when |
|------|-----------|-----------|
| Prometheus + Grafana | Open-source metrics + dashboards/alerting | You want the standard OSS monitoring stack |
| Datadog | SaaS APM, logs, metrics, and SLOs in one | You want a managed all-in-one with SLO/error-budget features |
| Sentry | Error tracking with release association | You want crash/exception tracking tied to releases |
| OpenTelemetry | Vendor-neutral telemetry standard (traces/metrics/logs) | You want to future-proof instrumentation |
## Sources and Further Reading
- Continuous Delivery Foundation: https://cd.foundation/
- GitHub Actions artifact attestations: https://docs.github.com/en/actions/security-for-github-actions
- Argo CD documentation: https://argo-cd.readthedocs.io/
- OpenFeature: https://openfeature.dev/
- SLSA framework: https://slsa.dev/
assets/versioning-decision-table.md
# Versioning Decision Table
> Choose a versioning scheme, then encode the rules in your release tooling. Canonical definitions: SemVer 2.0.0, CalVer, and Conventional Commits.
## Scheme Selection
| Question | If yes → |
|----------|----------|
| Do you have a public API or installable library where consumers depend on compatibility guarantees? | **SemVer** — each version carries semantic meaning |
| Is the product time-bound (releases must be date-identifiable) or driven by external events (regulatory, compliance)? | **CalVer** — date-based |
| Do multiple components ship on the same train with a shared promise (product suite, mobile app)? | **Fixed / one-version** — a single version for the product |
| Do components evolve independently and integrate via registries? | **Independent versioning** — per-component SemVer |
| Scheme | Format | Best for | Example |
|--------|--------|----------|---------|
| SemVer | MAJOR.MINOR.PATCH[-prerelease][+build] | Libraries, APIs, anything with consumers | 2.4.0, 2.4.0-rc.1 |
| CalVer | Date segments + optional modifier | Products with time-based releases | Ubuntu 24.04, pip 24.3 |
| Fixed / one-version | One version across all components | Release trains, product suites, mobile apps | 2026.08.1 |
| Independent | Per-component versions | Microservices, monorepo packages | api 3.2.1, web 1.9.0 |
## SemVer Rules
| Component | Rule |
|-----------|------|
| MAJOR | Incompatible API change |
| MINOR | Backward-compatible new functionality |
| PATCH | Backward-compatible bug fix |
| 0.y.z | Initial development: anything may change; consumers should pin |
| Prerelease | `-alpha.1`, `-beta.2`, `-rc.1` — lower precedence than the final release |
| Build metadata | `+build.123`, `+exp.sha.5114f85` — ignored in precedence, useful for provenance |
## Bump Rules from Conventional Commits
| Commit type | Bump | Example |
|-------------|------|---------|
| `BREAKING CHANGE:` footer, or `feat!` / `fix!` | MAJOR | `feat!: drop v1 API` |
| `feat` | MINOR | `feat(auth): add refresh tokens` |
| `fix` | PATCH | `fix(api): retry on 429` |
| `perf`, `refactor`, `docs`, `test`, `chore`, `ci`, `build`, `style` | PATCH in this skill | `docs: update readme` |
For `0.y.z`, this skill's Release Please-compatible policy maps both
`feat` and breaking changes to MINOR (`0.5.0` -> `0.6.0`), while fixes and
other changes remain PATCH. At `1.0.0` and later, normal SemVer priority
applies.
> **Gotcha —** the bump is decided by the highest-priority type in the release range: one `BREAKING CHANGE` forces a MAJOR at 1.0.0+, or a MINOR bump in 0.x, even if the rest are fixes. Automate with the `version_bump.py` script.
## Prerelease and Build Metadata Rules
- Prerelease identifiers: dot-separated alphanumerics + hyphens; numeric identifiers have no leading zeros (`rc.1`, not `rc.01`).
- Prerelease precedence: `1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta < 1.0.0-rc.1 < 1.0.0`.
- Build metadata (`+...`) does not affect precedence: `1.0.0+001 == 1.0.0+002`.
- Promotion pattern: build once, record the digest, tag with SemVer, promote the same immutable artifact through environments — never rebuild for promotion.
## Sources and Further Reading
- Semantic Versioning 2.0.0: https://semver.org/
- Conventional Commits 1.0.0: https://www.conventionalcommits.org/en/v1.0.0/
- CalVer: https://calver.org/
- Keep a Changelog: https://keepachangelog.com/en/1.1.0/
- Changesets (independent versioning in monorepos): https://github.com/changesets/changesets
evals/evals.json
{"schema_version": 1, "skill_name": "release-engineering", "evals": [{"id": "release-plan-design", "prompt": "We are shipping v2.4.0 of our SaaS product next month and I need a release plan covering the timeline, who does what, and what could go wrong. Can you put one together?", "expected_output": "A complete release plan naming the version and target date, a timeline with milestones and branch cut, owners assigned via RACI, scope items derived from the changelog/commits, risks with mitigations, a rollout plan, a rollback contingency, a communications plan, and recorded sign-offs. The plan follows the skill's release-plan template structure.", "assertions": ["The response names the release version and target date with a milestone timeline including branch cut", "The response assigns named owners for each task using a RACI-style responsibility matrix", "The response derives scope items from changelog entries or commit history rather than guessing", "The response lists risks with mitigations and a rollback contingency for the rollout", "The response includes a communications plan and explicit sign-offs before go-live"]}, {"id": "progressive-delivery-selection", "prompt": "We deploy to production every day and I want to start rolling out changes more gradually instead of flipping the switch all at once. Which progressive delivery strategy should we use and how do we gate it?", "expected_output": "A strategy recommendation comparing canary, blue/green, rings, percentage rollouts, and feature flags on speed, safety, and rollback time, built on the deploy-vs-release distinction. The rollout is gated on SLI-derived metrics and error budget, compares canary vs control concurrently rather than before/after, ramps exposure exponentially with verification between steps, and wires guardrail divergence to auto-rollback.", "assertions": ["The response explicitly separates deploy from release and recommends a concrete strategy from canary, blue/green, rings, percentage, or feature flags with trade-offs", "The response gates rollout steps on SLI-derived metrics and the service's error budget", "The response requires comparing canary/experimental population against a control concurrently, never before/after", "The response ramps exposure exponentially (e.g., 1% to 10% to 50% to 100%) with verification between steps", "The response wires guardrail divergence to pause, auto-rollback, or page rather than continuing the ramp"]}, {"id": "version-bump-determination", "prompt": "I have a changelog with breaking change, feat, and fix commits since v1.9.0 and need to know what the next version should be and why.", "expected_output": "A version bump computed from Conventional Commits semantics: a BREAKING CHANGE footer or feat!/fix! maps to a MAJOR bump, feat to MINOR, fix to PATCH, and no-bump types (chore, docs, refactor, ci) do not advance the version. For a breaking change after v1.9.0 the computed version is 2.0.0, validated against strict SemVer with the bump rule justified per commit type.", "assertions": ["The response maps a BREAKING CHANGE footer or feat!/fix! to a MAJOR bump", "The response maps feat commits to MINOR and fix commits to PATCH bumps", "The response ignores no-bump types such as chore, docs, refactor, or ci", "The response computes 2.0.0 as the next version after 1.9.0 when a breaking change is present", "The response validates the computed version against strict SemVer and justifies each bump rule"]}, {"id": "rollback-plan-stateful-service", "prompt": "Our service shares a Postgres database and the last release included a schema migration that already ran in production. I need a rollback plan for when this release goes wrong.", "expected_output": "A rollback plan built on expand/contract (parallel change) with forward-only migrations: the schema must support the previous release so the binary can be rolled back safely, rollback is safe only before finalization, and after finalization the only path is a new forward migration. The plan rejects git revert as a rollback for schema/data changes, adds manual checkpoints for data-touching operations, and treats backup/restore with RPO/RTO as a last resort.", "assertions": ["The response applies expand/contract (parallel change) so old and new schema coexist across releases", "The response states rollback is safe only before finalization and requires a forward migration afterwards", "The response rejects git revert as a rollback mechanism for schema or data changes", "The response requires forward-only, append-only migrations and manual checkpoints before data-touching operations", "The response treats backup/restore (with RPO/RTO) as a last resort rather than a primary rollback path"]}, {"id": "dora-metrics-computation", "prompt": "I exported our deployment and commit events for the last 30 days and want to compute our DORA metrics from the raw data, not from whatever the vendor dashboard shows.", "expected_output": "A computation of all five DORA metrics from raw events: deployment frequency as successful production deploys per day, change lead time aggregated as the median across changes, change failure rate as failed over total deployments, failed deployment recovery time as median time to recover from change-caused failures, and deployment rework rate as the share of unplanned bug-fix deploys. Metrics are scoped to production only, use medians not means, and flag measurement pitfalls such as PRs vs deploys and ignoring rollbacks.", "assertions": ["The response computes deployment frequency from successful production deployments per day", "The response aggregates change lead time as the median across changes, not the mean", "The response computes change failure rate as failed deployments divided by total deployments", "The response scopes every metric to production or release-to-users and includes failed deployment recovery time and deployment rework rate", "The response flags measurement pitfalls such as counting PR merges as deploys, ignoring rollbacks, or using mean instead of median"]}, {"id": "release-readiness-checklist", "prompt": "Our release candidate is ready and I need a readiness review structure so we can decide go or no-go with actual evidence instead of vibes.", "expected_output": "A readiness checklist organized into the four dimensions — functional, non-functional, operational, and governance — where every item has a named owner and an evidence link, plus a go/no-go decision block. Operational items cover monitoring live before go-live, runbooks, and a rehearsed rollback; governance items cover approval and the ticket-to-verification audit chain.", "assertions": ["The response organizes readiness into the four dimensions: functional, non-functional, operational, and governance", "The response assigns a named owner (a person, not a team) and an evidence link to every checklist item", "The response includes an explicit go/no-go decision block with recorded verdict", "The response requires monitoring and alerting to be live before go-live and a rehearsed rollback path", "The response ties governance items to the audit chain of ticket to PR review to CI to approval to deploy log to verification"]}, {"id": "feature-flag-cleanup-plan", "prompt": "We have dozens of feature flags that have been on for a year and nobody remembers what they do. How do we clean them up safely?", "expected_output": "A flag cleanup plan following the remove-then-archive order: remove all code references first (verified with code-reference scanning), then archive the flag key in the platform — never delete or reuse keys. The plan assigns owners and expiry at creation, uses time-bomb checks to enforce removal, targets a 90-120 day archive cadence, tests both ON and OFF states, and warns about the Knight Capital flag-reuse failure.", "assertions": ["The response requires removing all code references before archiving the flag", "The response says to archive flag keys rather than delete them, and never reuse a key", "The response recommends code-reference scanning and time-bomb expiry checks to enforce removal", "The response targets archiving temporary flags within roughly 90-120 days", "The response warns about the Knight Capital failure mode of reactivating a stale flag and requires testing both ON and OFF states"]}, {"id": "anti-trigger-incident-debugging", "prompt": "Our checkout service is throwing intermittent 500 errors in production and I need help doing root-cause analysis to find the underlying fault.", "expected_output": "The agent declines to apply release-engineering to this request, recognizing that production incident root-cause debugging and fault localization fall outside its scope. It routes the user to the systematic-debugging skill for root-cause analysis and fault localization, and to site-reliability-engineering for on-call and incident response operations.", "assertions": ["The response declines release-engineering as the appropriate skill for production incident root-cause debugging", "The response routes the user to systematic-debugging for fault localization and root-cause analysis", "The response names site-reliability-engineering for on-call and incident-response operations", "The response does not attempt to apply release planning, rollback, or DORA methodology to the debugging task", "The response explains that incident root-cause debugging sits outside the release-engineering negative boundary"]}]}
pytest.ini
[pytest]
# Override root pyproject.toml coverage settings.
# release-engineering tests are subprocess-based and do not measure
# the root scripts/ or eval_runner/ packages.
addopts = -ra --strict-markers --tb=short
README.md
# Release Engineering
Senior-to-principal release engineering methodology: pipelines, process, artifacts, gates, compliance, and metrics that move software from commit to customer safely.
## Why Install This Skill
Releasing software is where engineering risk becomes customer impact. This skill gives your agent the working methodology of a senior release engineer: designing CD pipelines that promote one immutable artifact through every environment, planning rollbacks before you need them, choosing between canary, blue-green, and feature-flag rollouts, and computing DORA metrics from real events instead of guesses.
It also covers the parts of release work that quietly break teams: versioning and changelog discipline, audit-ready change records for SOC 2 / SOX / PCI, supply-chain integrity (SBOM, signing, provenance), and the ceremony of multi-team release trains. Install once and your agent can draft a release plan, compute the next SemVer from commit history, validate a changelog, build a rollback runbook, and report the five DORA metrics — without you hand-writing a single template.
## What You Get
| Directory | Contents |
|-----------|----------|
| `references/` | 15 dense topic files: role-and-career, skills-competency-model, release-process-models, cd-and-pipeline-stages, progressive-delivery, change-governance-and-compliance, readiness-and-quality-gates, rollback-and-recovery, versioning-and-artifacts, feature-flag-lifecycle, monorepo-polyrepo-release, toolchain-landscape, supply-chain-security, metrics-and-dora, release-operations-and-triage |
| `templates/` | 6 fillable templates: release-plan, release-readiness-checklist, rollback-runbook, release-notes, change-governance-record, hotfix-emergency-release-plan |
| `assets/` | 4 quick-reference files: dora-metrics-reference, versioning-decision-table, deployment-strategy-matrix, release-toolchain-cheatsheet |
| `scripts/` | 5 Python CLIs: version_bump (next-SemVer from conventional commits, including the documented 0.x policy), semver_check (validate/compare/sort), changelog_check (Keep a Changelog and Release Please validator), dora_metrics (five-metric computation), release_plan_scaffold (plan generator) |
| `evals/` | Schema-v1 output-quality eval manifest (8 cases) |
## Quick Start
Compute the next version from the commits since the last tag:
```bash
python3 release-engineering/scripts/version_bump.py --current-version 1.4.0 --git-range v1.4.0..HEAD
```
Validate a changelog before it ships:
```bash
python3 release-engineering/scripts/changelog_check.py CHANGELOG.md
# Or select Release Please's linked-header format explicitly:
python3 release-engineering/scripts/changelog_check.py CHANGELOG.md --format release-please
```
Compute the five DORA metrics from deployment and commit event data:
```bash
python3 release-engineering/scripts/dora_metrics.py --events deploy-events.json --environment prod
```
## Triggers
- Release planning, timelines, and rollout strategy
- CD pipeline design and promotion-stage reviews
- Version bumps, SemVer validation, and conventional-commit classification
- Changelog authoring and validation
- Go/no-go readiness reviews and release candidates
- Rollback runbook writing and rehearsal
- Canary, blue-green, ring, and feature-flag rollouts
- Feature flag lifecycle and cleanup
- Release trains, branch cuts, and stabilization windows
- DORA metric definitions and computation
- Change-control and audit evidence (SOC 2, SOX, PCI)
- SBOM, signing, provenance, and registry hygiene
- Hotfixes, break-glass changes, and emergency releases
- Multi-team release coordination
## Requirements
- Python 3.8+ for scripts (standard library only, no third-party packages)
- No specific CI platform, deployment tool, or version control mandate
- Works with any stack; examples reference GitHub Actions, GitLab, Argo CD, LaunchDarkly, and others as illustrations
references/cd-and-pipeline-stages.md
# Continuous Delivery and Pipeline Stages
The pipeline is where release engineering becomes machinery: it decides what can ship, how fast it can ship, and — if designed badly — how slowly and how riskily. The canonical pattern is deceptively simple: build an artifact **once**, then **promote** that same immutable artifact through progressively more production-like environments. Most pipeline failures come from violating one of the words in that sentence: the artifact gets rebuilt, or the environments diverge, or the gates stop meaning anything.
## Continuous Delivery vs Continuous Deployment
The two terms are routinely conflated; the distinction is one human decision.
- **Continuous Delivery (CD):** software is *always releasable*. The current version of every service could be deployed to production at a moment's notice, but a human or business process chooses when and how often to actually release. The Thoughtworks CD working group's indicators: the software is deployable throughout its lifecycle; keeping it deployable is prioritized over new features; fast, automated feedback on production-readiness is available; and any version can be pushed to any environment at the push of a button.
- **Continuous Deployment:** every change that survives the pipeline is *automatically* put into production, often many times per day. Continuous deployment is CD plus the removal of the human release decision.
| | Continuous Delivery | Continuous Deployment |
|---|---|---|
| Deployable at any time | Yes | Yes |
| Who triggers production deploy | Human/business decision (push-button) | The pipeline, automatically |
| Prerequisite | Automated gates + release candidate discipline | CD + high-confidence automated gates + fast rollback |
| Typical cadence | On-demand, business-chosen | Many per day |
> **Gotcha — "we do CD" meaning "we have CI":** Continuous integration (frequent merges to trunk with automated builds) is a prerequisite, not the deliverable. If a human still edits environment config by hand or rebuilds per environment, you have CI, not CD.
## The Canonical Pipeline: Build Once, Promote Many
The deployment pipeline (Humble & Farley's *Continuous Delivery* is the canonical source) has a fixed spine:
```
Commit → Build → Unit tests → Integration/system tests → Package/artifact
→ Promote dev → Promote staging/pre-prod → Deploy production → Verify
```
The two load-bearing rules:
1. **Build once.** The exact artifact that ran tests is the exact artifact that reaches production. Rebuilding per environment means production runs code that was never tested — the pipeline's core guarantee is void. "Only build binaries once" is the founding rule of the field, and it is still violated more often than it should be.
2. **Promote, don't re-package.** Promotion moves an immutable artifact between environments, ideally by moving a pointer or label rather than copying bits. Google's MPM packages are content-hashed, versioned, and signed, with movable labels (`dev`, `canary`, `production`) pointing at immutable versions. Content-addressed storage and digest-pinned references make promotion verifiable: the promoted digest equals the tested digest.
### Stage Inventory
| Stage | Responsibility | Typical gates | Promotion evidence |
|-------|----------------|---------------|--------------------|
| Source | Trigger on commit/PR; capture commit metadata | Branch protection, review | Commit SHA, author, PR number |
| Build | Produce the artifact hermetically | Hermetic/reproducible build succeeds | Artifact digest (SHA-256) |
| Unit + static analysis | Fast correctness and lint | Test suite, SAST, dependency scan | Test reports, scan results |
| Integration/system | Cross-component behavior | Contract tests, E2E, integration suite | Test reports pinned to digest |
| Package/publish | Store immutable artifact + metadata | Signing, SBOM generation, provenance attestation | Signed digest, SBOM, attestation |
| Promotion (dev → staging) | Move artifact through pre-prod | Deploy + smoke tests + config validation | Deploy log per environment |
| Promotion (staging → prod) | Deploy to production | Canary/metric gates, error-budget check, approval if any | Deploy log, verification window |
Each stage gates the next; the artifact only promotes when the previous stage passes. The evidence each stage emits is what makes the pipeline auditable (see [change-governance-and-compliance.md](./change-governance-and-compliance.md)) — a pipeline that emits no structured records produces no audit trail even though it "does CI."
### Push-on-Green vs Select-a-Build
**Push-on-green** is the Google-flavored extreme: deploy every build that passes all tests, automatically. Google describes it as one end of a spectrum — some Google teams deploy every green build; others **build hourly and select** a build to promote based on test results and feature content. Both are legitimate; the selection model adds human judgment about *what* to ship, while push-on-green optimizes for velocity. Start with selection, graduate to push-on-green as metrics and rollback mature. Push-on-green without fast automated rollback (see [rollback-and-recovery.md](./rollback-and-recovery.md)) is a gamble, not a practice.
## Stage Gates and Their Failure Modes
| Gate type | Example | Failure mode |
|-----------|---------|--------------|
| Compile/build | Hermetic build succeeds | Build machine state leaks in (non-hermetic) |
| Unit/integration tests | Suites run on the artifact | Flaky tests → gate ignored or rerun into green |
| Security scans | SAST, dependency/CVE scan, SBOM check | Scans run on a *different* artifact than the one shipped |
| Performance check | Peak-load + margin benchmark on staging | Staging capacity ≠ production capacity → false pass |
| Approval | Human sign-off to promote | Approval without evidence; CAB delay (see below) |
| Post-deploy verification | Canary metrics, smoke tests | Verification watches the wrong SLIs or no SLIs |
The meta-failure mode is **gate erosion**: when a gate blocks frequently for reasons engineers believe are spurious, teams start bypassing it (hotfix overrides, force-promote), and the gate becomes theater. Fix the gate's false positives rather than adding more gates on top. A gate you cannot trust should be deleted; an untrusted gate that stays is worse than none, because it trains the org to ignore gates.
## The Evidence on Approval Gates (CABs)
The empirical case against heavyweight human approval is one of the strongest findings in the delivery literature (Accelerate / 2019 State of DevOps, via Forsgren et al.):
- External approval (Change Advisory Board or manager sign-off) is **negatively correlated** with lead time, deployment frequency, and restore time.
- It has **no correlation** with change-failure rate — approval does not make releases safer.
- The research describes heavyweight approval as "worse than no approval process": it slows delivery without improving stability.
The recommended replacement is **lightweight peer review** (pair programming or intra-team code review) **combined with a deployment pipeline that detects and rejects bad changes**. The pipeline is the enforcement mechanism; the review is the quality input. For teams under audit pressure, the pipeline *also* produces better evidence than a CAB minute ever did — immutable, timestamped, and linked (see [change-governance-and-compliance.md](./change-governance-and-compliance.md)).
> **Gotcha — "one more sign-off" as a risk control:** Adding an approver does not reduce change-failure rate — the Accelerate data says so directly. If a release keeps failing, add automated gates and smaller batches, not another approval step.
## Pipeline-as-Code
The pipeline definition itself must be versioned, reviewed, and tested like production code:
- **Versioned:** pipeline YAML/Starlark lives in the source repo, so a release's pipeline is reproducible from history — and a changed pipeline is diffable in review.
- **Reviewed:** pipeline changes go through the same PR review as application code — a pipeline edit is a production change, because the pipeline is what reaches production.
- **Tested:** exercise the pipeline on a branch/PR environment before it runs on `main`; validate inputs (secrets, parameters) rather than assuming them.
- **Auditable:** pipeline runs emit structured events (start/end timestamps per stage, commit SHA, artifact digest, stage result) that feed dashboards and postmortems. Without these events, "what did we deploy when and why" is tribal knowledge.
## Hermetic Builds
A **hermetic build** produces a byte-for-byte reproducible artifact whose output depends only on declared inputs — not on the build machine, the time of day, or what happens to be in a package registry. Hermeticity is what makes build-once meaningful:
- **Determinism:** the same commit always yields the same artifact; no "works on my machine" drift.
- **Trust:** no surprise dependency fetched at build time (see [supply-chain-security.md](./supply-chain-security.md)).
- **Rollback safety:** a prior artifact can be reproduced exactly if needed (see [rollback-and-recovery.md](./rollback-and-recovery.md)).
- **Auditability:** you know exactly what went into an artifact because the build environment is pinned.
Practical steps: pin toolchains and base images (digest references, not tags); vendor or mirror dependencies behind an internal registry with lockfiles and hashes; disable network access during builds (sandbox or `--network none`); use content-addressable build tools (Bazel, Nix) where feasible. The tradeoff is real — dependency mirroring and build infrastructure are overhead — but it buys determinism, cacheability, and supply-chain guarantees that no test suite can provide.
## The Self-Service Model
Google's release-engineering philosophy (SRE Book ch. 8) names four principles that the pipeline should embody:
1. **Self-service** — teams control their own release cadence through the pipeline; no ticket to a central release team.
2. **High velocity** — frequent releases with fewer changes per version (small batches).
3. **Hermetic builds** — reproducible artifacts isolated from host environment.
4. **Enforcement of policies** — the platform gates operations (approve code, create release, deploy) so safety does not depend on human memory.
The pipeline is the enforcement point: roles and policies live in the platform (who can trigger, promote, approve, override), not in a checklist. If an engineer must ask "how do I release this service?", the self-service model has failed — the answer should be a documented command or CI trigger (see [role-and-career.md](./role-and-career.md) for how this shapes the RE role).
## The Hotfix and Emergency Path
Emergencies should not invent a parallel process; they should use a **faster version of the same pipeline**. The recommended model (GoCD, and consistent with Google's release engineering) is a hotfix path that is structurally identical to the production pipeline but *fetches the artifact earlier* — skipping the full promotion ladder — and is kept paused behind strict trigger controls, so it cannot be used casually:
1. Severity justifies an emergency change (SEV-1, active security vulnerability).
2. Emergency change ticket created — abbreviated but auditable.
3. Required approvals obtained via a break-glass path (senior approval, documented; see [change-governance-and-compliance.md](./change-governance-and-compliance.md)).
4. Hotfix branch cut from the production tag; minimal cherry-picked fix.
5. Automated smoke tests + targeted regression on the exact candidate.
6. Deploy to a canary subset, monitor, then full rollout — same gates as any release.
7. Post-implementation review within a bounded window; retroactive full documentation.
The key discipline: **if the pipeline is fast enough, prefer committing the fix through the whole pipeline** so it is fully tested, and spend the postmortem on *how a bad build reached production* rather than on the hotfix mechanics. A hotfix path that is faster but less tested is a standing invitation to skip quality exactly when the stakes are highest.
## Config Strategy in the Pipeline
Configuration is the most common source of production incidents after the code itself, and its treatment in the pipeline is a design decision:
| Strategy | Description | Strengths | Weaknesses | Best for |
|----------|-------------|-----------|------------|----------|
| **Mainline** | Config lives in the app repo and ships inside the artifact | Atomic deploys; version correlation is trivial | Any config change requires a rebuild; cannot toggle independently | Small services |
| **Bundled defaults + runtime overrides** | Artifact carries safe defaults; per-env overrides applied at deploy | Safe-start; env differences explicit | Merge logic for defaults + overrides | Most services |
| **Config-only packages** | Config is its own versioned, promoted artifact | Config promotes/rolls back without redeploying code | Config/code version skew needs management | Large services with frequent config changes |
| **External stores** | Config served at runtime (Consul, etcd, parameter store) | Real-time changes; fine-grained access control; audit logging | Runtime dependency; startup latency; misconfiguration cascades | Multi-service systems, global flags |
The layered recommendation: **bundled defaults inside the artifact, environment overrides in a versioned config artifact or external store, runtime toggles in a feature-flag system.** Whichever strategy you choose, version and review config like code, and snapshot config alongside binaries by build ID so the running binary and its config are always a matched pair — the direct defense against config drift.
## Environment Parity and Config Drift
Staging and pre-prod should mirror production as closely as feasible — same artifact, same config schema, same deployment mechanism. The SRE Workbook is blunt that test environments are **never 100% identical to production**, which is exactly why canarying in real traffic is needed (see [progressive-delivery.md](./progressive-delivery.md)). But parity gaps are also a recurring source of prod-only defects, so name and manage them deliberately:
| Parity dimension | Typical gap | Mitigation |
|------------------|-------------|------------|
| Data | Staging has fake/masked/anonymized data | Refresh from production snapshots (with compliance guards); seed realistic volumes |
| Capacity | Staging runs a fraction of prod capacity | Load-test at prod scale or document the scaling delta |
| Config | Staging config hand-edited, prod config differs | Config as versioned artifacts promoted alongside binaries; drift detection (GitOps reconcile) |
| Dependencies | Staging talks to different upstreams | Same registry/versions; record dependency versions per environment |
**Config drift** — config and binary falling out of sync — is a classic incident cause. Google versions config in VCS with code review and snapshots config *alongside* binaries by build ID, so the running binary and its config are always a matched pair. Treat config as an artifact with the same promotion rules as code: versioned, reviewed, promoted, and rollback-able.
## The Pipeline and Delivery Metrics
The pipeline is also the **measurement instrument** for delivery performance. The five DORA metrics (see [metrics-and-dora.md](./metrics-and-dora.md)) are all derivable from pipeline telemetry — if the pipeline emits it:
- **Deployment frequency** — successful production deploys per time period, from deploy events.
- **Change lead time** — commit timestamp to production deploy, from commit + deploy events.
- **Change-failure rate** — deploys needing immediate remediation, from failed/rolled-back deploy events.
- **Failed deployment recovery time** — failed deploy to next successful deploy, from deploy events.
- **Deployment rework rate** — unplanned deploys caused by a production incident, from incident-linked deploy events.
Design implication: every stage should emit structured events (start/end timestamps, commit SHA, artifact digest, stage result, environment) so the pipeline is not just a delivery mechanism but a source of evidence — for DORA, for audits, and for postmortems. A pipeline that produces no records produces no metrics and no audit trail, no matter how green it looks.
## Secrets and Credentials in the Pipeline
The pipeline holds the keys to production; treat its credentials as a first-class risk surface:
- **Never bake secrets into artifacts.** Secrets are injected at deploy time from a secrets manager (Vault, cloud parameter store), scoped to the environment.
- **Short-lived, least-privilege credentials.** Pipeline credentials with 15-minute expirations and the minimum scope for their stage; a leaked long-lived deploy token is a standing compromise.
- **Separation of duties at the stage level.** Build, promote, and deploy should have distinct authorization — the identity that builds is not the identity that deploys, and neither is the identity that approves (see [change-governance-and-compliance.md](./change-governance-and-compliance.md)).
- **Audit secret access.** Who read which secret, when, from what context — because a pipeline that touches secrets without logging them is a pipeline whose compromise you will never detect.
Supply-chain hygiene (artifact signing, SBOM, provenance) is the other half of this surface; see [supply-chain-security.md](./supply-chain-security.md).
## Limits of the Pipeline
A green pipeline is necessary, not sufficient — it enforces *process*, not *product quality*:
- Tests that assert nothing, or assert the wrong thing, pass green while shipping broken features.
- Requirements errors sail through every gate; the pipeline cannot detect that you built the wrong thing.
- A pipeline cannot fix missing ownership (no on-call, no accountable team) or architectural problems (a service that cannot be deployed independently).
The corollary is positive: because the pipeline *can* enforce process reliably, reserve human attention for what it cannot — verification methodology, code review quality, pre-mortems, and the readiness judgments described in [readiness-and-quality-gates.md](./readiness-and-quality-gates.md). Treat "the pipeline is green" as the floor, not the ceiling.
## Gotchas
> **Gotcha — rebuilding per environment:** The most common pipeline sin. "Production build differs because we set flags at build time" means production runs untested bits. Environment differences belong in *config*, injected at deploy time, never compiled in.
> **Gotcha — green-check theater:** A stage that passes but asserts nothing (a test suite with no assertions, a smoke check that only verifies HTTP 200) looks safe and protects nothing. Validate gates by deliberately breaking them in a rehearsal.
> **Gotcha — flaky-test erosion:** When a suite reruns "until green," the gate is lying. Track rerun rate; a gate that needs retries is a queue of future prod incidents.
> **Gotcha — CI vs CD confusion in metrics:** Pipeline run counts are not deployment frequency; PR merges are not deploys. Measure the metric you actually care about (deployments to production), or your DORA dashboard will flatter you (see [metrics-and-dora.md](./metrics-and-dora.md)).
> **Gotcha — pipeline-as-code not reviewed:** The most privileged code in your system is the pipeline that deploys to production. A compromised or sloppy pipeline definition is worse than a bad app commit. Review it, sign it, audit it.
> **Gotcha — one pipeline, no selection:** If every green build auto-deploys but the org cannot actually absorb that cadence (support, on-call, capacity), push-on-green produces chaos, not velocity. The cadence must match the org's ability to verify and recover.
## Sources and Further Reading
- [Martin Fowler — Continuous Delivery (bliki)](https://martinfowler.com/bliki/ContinuousDelivery.html)
- [Google SRE Book — Release Engineering (ch. 8)](https://sre.google/sre-book/release-engineering/)
- [Google SRE Workbook — Canarying Releases (ch. 16)](https://sre.google/workbook/canarying-releases/)
- [Software Engineering at Google — Continuous Delivery (ch. 24)](https://abseil.io/resources/swe-book/html/ch24.html)
- [Harness — Is a Change Advisory Board Really Needed? (Accelerate evidence)](https://www.harness.io/blog/change-advisory-board-really-needed)
- [DORA — The DORA Metrics guide](https://dora.dev/guides/dora-metrics/)
- [Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Humble & Farley)](https://www.oreilly.com/library/view/continuous-delivery-reliable/9780321670250/)
references/change-governance-and-compliance.md
# Change Governance and Compliance
Release engineering is where change control stops being a bureaucratic form and becomes an **evidence pipeline**. Auditors (SOC 2, SOX, PCI DSS, ISO 27001, EU DORA) do not actually care about your CAB meetings — they care whether every change can be traced, unbroken, from business request through review, test, approval, deployment, and verification. A well-designed pipeline produces that evidence as a **byproduct**; a poorly designed one forces engineers to reconstruct it after the fact. This reference covers evidence-based change control, the artifact-by-artifact evidence chain, emergency/break-glass paths, the major regulatory frameworks, audit-friendly pipeline design, separation of duties, and the pitfalls that destroy evidence.
## Evidence-Based Change Control vs CAB Bureaucracy
The Change Advisory Board (CAB) — ITIL's cross-functional committee that reviews and approves changes — is the traditional governance model. ITIL 4 distinguishes **standard** (pre-authorized, low-risk, repeatable), **normal** (assessed and scheduled, may go to CAB), and **emergency** (expedited) changes.
The empirical case against heavy external approval is decisive. DORA/Accelerate found that external approval (CAB/manager sign-off) is **negatively correlated** with lead time, deployment frequency, and restore time, and has **no correlation** with change failure rate — it slows delivery without improving stability, and is "worse than no approval process." The evidence-backed replacement is **lightweight peer review** (code review, pair programming) **combined with a deployment pipeline that detects and rejects bad changes**.
Modern change control therefore keeps the *evidence*, not the *committee*:
| Dimension | CAB-era governance | Evidence-based control |
|-----------|--------------------|------------------------|
| Gate | Human committee approval | Peer review + automated pipeline gates |
| Evidence | Meeting minutes, sign-off forms | Immutable artifacts: PRs, CI logs, deploy records |
| Cadence | Scheduled change windows | Continuous, small batches |
| Emergency path | Exception to the committee | Pre-documented break-glass procedure with retroactive audit |
| Failure mode | Slow without being safer | Fast with machine-checkable safety |
**Retain a CAB-like review only where it adds value**: high-risk changes (data migrations, security boundaries), regulated launches, and audit-after-the-fact review boards. For everything else, the pipeline is the control. This is not a license to skip documentation — it is a mandate to *automate* it.
## The SOC 2 CC8.1 Evidence Chain
SOC 2's **CC8.1** control requires that the entity "authorizes, designs, develops or acquires, configures, documents, tests, approves, and implements changes to infrastructure, data, software, and procedures to meet its objectives." Its points of focus span the whole SDLC: manage changes, authorize before implementation, design/develop with controls, document for traceability, track to confirm intended outcomes, configure with approved settings, and test before implementation.
For a code change, auditors trace a specific artifact chain. **Every link must exist and connect**:
| # | Artifact | Must Contain | Where Stored |
|---|----------|--------------|--------------|
| 1 | **Change request / ticket** | Business justification, risk assessment, testing plan, requester, timestamp | Ticketing system (Jira, Linear) |
| 2 | **Design / review doc** (significant changes) | Architecture decision, security impact, alternatives | Wiki / RFC repo / ticket attachments |
| 3 | **Peer code review (PR/MR)** | Diff, reviewer comments, approval by an engineer **other than the author**, timestamp | GitHub/GitLab with branch protection enforced |
| 4 | **CI / build / test results** | Unit + integration test results, SAST/DAST scan, lint, build success, **commit SHA tested** | CI platform logs, exported to a centralized store |
| 5 | **Change approval (deployment gate)** | Approver identity (**not** the code author), timestamp, decision, target environment | GitHub Environments required-reviewer, GitLab protected environments, or CAB record |
| 6 | **Deployment log** | Who/what (commit SHA, **artifact digest**), when (UTC), target environment, **pipeline run ID**, deployer identity | CI/CD deployment records, Argo CD sync log, cloud audit logs (CloudTrail, GCP Audit Logs) |
| 7 | **Post-deployment verification** | Smoke/synthetic check results, health metric confirmation, rollback trigger status, ticket closure | Monitoring system, ticketing system status change |
CC8.1 intersects with **CC6.1** (only authorized people approve and deploy) and **CC7.1** (unauthorized changes generate alerts). It is the **second most common source of audit exceptions** in Type II audits — and those exceptions are "almost always process failures rather than technical ones": a missing ticket link, a deleted branch, an unrecorded manual deploy.
### Auditor Sampling
| | Type I | Type II |
|--|--------|---------|
| What it proves | Design exists at a point in time | Controls operated effectively over time |
| Evidence | 1–2 example tickets showing the full workflow | Population of **all** changes over 6–12 months; sample of **25–50** changes checked artifact-by-artifact |
| Exceptions tolerated | Design gaps are findings | **None** — a single missing link in a sampled change is an exception |
| When it applies | Readiness/pre-audit, initial certification | Renewal and continuous assurance |
Sample size guidance: 25 for well-controlled environments, scaling to 50+ when the population is large, high-risk, or control weaknesses are found. Auditors may pull changes directly from version control or request an export — either way, the evidence must be *discoverable without a human tour guide*.
> **Gotcha — The 25-change lie:** If only 24 of your changes have complete evidence chains, an auditor sampling 25 changes can fail you on one gap. The discipline is not "make sampled changes clean" — it is "make the pipeline incapable of producing a change without complete evidence."
### Tracing One Change, End to End
To internalize the chain, trace a single merged PR as an auditor would:
1. **Ticket** `PROJ-2041` exists with justification ("fix rate-limit false positives"), risk tier, and requester.
2. **PR** `#4821` title carries the ticket ID (`[PROJ-2041] fix rate limiting`); branch protection required and recorded one reviewer who is **not** the author; approval timestamped.
3. **CI** run on the PR's commit SHA: unit + integration tests green, SAST scan clean, artifact built and pushed with digest `sha256:9f86d0…`; all logs retained.
4. **Approval** — a GitHub Environments required reviewer (again not the author) approved the production deployment; environment protection rules forbid direct pushes.
5. **Deployment log** — records `commit abc1234`, digest, environment `prod-us-east`, UTC timestamp, pipeline run ID `run-8871`, deployer identity (the pipeline's OIDC role).
6. **Post-deploy verification** — smoke test passed; error-rate dashboard confirmed baseline; ticket closed with verification link.
If any of these six links is missing or unlinkable, that change is an audit exception waiting to be sampled. This is exactly the evidence shape the `templates/change-governance-record.md` template captures per change.
## Emergency Change and Break-Glass
Auditors accept emergencies — but they require the emergency path to be **pre-documented, controlled, and logged in the same system** as normal changes:
| Requirement | Detail |
|-------------|--------|
| **Documented procedure** | Policy defines "emergency" narrowly (security exploit, production outage) *before* an emergency occurs |
| **Approval** | At least one authorized approver (senior manager / on-call lead), even if post-hoc |
| **Written justification** | Why the normal process was bypassed; what the urgency was |
| **Same-system logging** | Emergency changes appear in the same change tracking system as normal changes — no shadow log |
| **Retroactive window** | Post-hoc approval within a defined window (commonly 24–72 hours; set in org policy) |
| **Post-implementation review** | Formal review of root cause and the fix; documented postmortem |
| **Retrospective audit** | Emergency changes reviewed monthly/quarterly; track the ratio |
| **Separation of duties** | Where feasible, implementer ≠ approver ≠ tester; document why if bypassed |
The key constraint: **"CC8.1 requires that all changes are tested; there is no exception for emergencies."** Break-glass is an alternate, audited control path — it does not skip controls. **If everything is an emergency, the normal process is not working**; a high emergency ratio is itself an audit red flag and an organizational signal.
## Regulatory Frameworks
### SOX ITGC (Sarbanes-Oxley IT General Controls)
Applies to US public companies for systems impacting financial reporting (ERP, billing, payroll, AR/AP, financial reporting software). Auditors expect: formal multi-level **change authorization**, fine-grained **segregation of duties** with automated conflict detection, **documented testing** before deployment, comprehensive change documentation, and controlled emergency procedures. The evidence chain has the same shape as SOC 2 but is scoped to financial-reporting-relevant systems, and retention is the longest of any framework: **7 years** for financial system audit logs.
### PCI DSS (Requirement 6 — Change Control)
For cardholder-data environments (PCI DSS v4.0.1), the mandatory change-control sub-requirements include:
| Requirement | Mandate |
|-------------|---------|
| 6.4.1 | Separate dev/test environments from production with access controls |
| 6.4.2 | Separation of duties between development/testing and production personnel |
| 6.4.3 | No live PANs (production card data) in test/dev |
| 6.4.4 | Remove test data and accounts before going live |
| 6.4.5.1 | Document the impact of the change |
| 6.4.5.2 | Documented change approval by authorized parties |
| 6.4.5.3 | Functionality testing proving the change does not adversely affect security |
| 6.4.5.4 | **Establish back-out procedures** for changes |
| 6.4.6 | After significant changes, re-apply all relevant PCI DSS requirements and update documentation |
Additionally, **Req 6.3.2** requires code reviews by someone other than the code developer, with results reviewed and approved by management before publication. PCI DSS 4.0 explicitly covers the CI/CD pipeline itself — QSAs inspect pipeline configuration, not just runtime systems. Log retention: **12 months, with 3 months immediately available**.
### EU DORA (Digital Operational Resilience Act)
In force since 17 January 2025 for EU financial entities and their ICT third-party providers. **Article 9(4)(e)** requires documented policies ensuring that all ICT changes are **"recorded, tested, assessed, approved, implemented and verified"** in a controlled manner, with the process approved by appropriate lines of management. Article 9(4)(c) limits access to what is required for legitimate functions; Article 9(4)(f) requires documented patch/update policies; Article 9(4)(b) requires networks designed to be instantly severed or segmented. DORA is principle-level — but those six verbs map exactly onto the SOC 2 evidence chain, so one well-built pipeline satisfies both.
### EU Cyber Resilience Act (CRA)
Applies to manufacturers of digital products/software sold in the EU. Key deadlines: vulnerability reporting by **11 September 2026**; full compliance including SBOM by **11 December 2027**. Requirements relevant to release engineering: a **mandatory SBOM** for software products (SPDX 2.3 or CycloneDX per BSI TR-03183-2), covering **all components including transitive dependencies**, continuous vulnerability monitoring, secure-by-design development, and mandated vulnerability disclosure timelines. Fines up to €15M or 2.5% of global annual turnover. For release pipelines this makes **SBOM generation a build-time gate** — every release artifact ships with an associated SBOM. See [supply-chain-security.md](./supply-chain-security.md).
### ISO 27001 Annex A.8.32
Requires that changes to information processing facilities and systems are "properly controlled and authorized," with defined responsibilities for planning, evaluating, authorizing, implementing, reviewing, and communicating changes. Less prescriptive than SOC 2/PCI about artifact types, but the evidence expectations converge in practice (ticket, approval, test, deploy record). Adjacent controls: A.8.31 (separation of dev/test/production environments), A.8.33 (test information).
### Retention Requirements Summary
| Framework | Minimum Retention | Notes |
|-----------|-------------------|-------|
| SOC 2 | 12 months (industry standard) | Must cover the full Type II observation period (3–12 months) |
| PCI DSS | 12 months (3 months immediately available) | Req 10.7 |
| SOX | **7 years** | Financial system logs; Section 802 |
| ISO 27001 | Defined by org policy | Must demonstrate control over the certification period |
| DORA | Principle-based | Supervisory authorities may request historical records |
| Multi-regime best practice | 13 months hot + 7 years cold immutable | Stacked regimes, not one number |
## Building Audit-Friendly Pipelines
Audit-friendly means **complete-by-default**: the evidence trail is a byproduct of the pipeline, never reconstructed after the fact.
| Evidence type | Automation mechanism |
|---------------|----------------------|
| Code review record | Branch protection requires PR approval before merge; immutable timestamped record |
| CI/test results | Pipeline runs on every PR/commit; status checks block merge; logs exported to a centralized store |
| Deployment record | GitHub Environments / Argo CD sync / GitLab environments record who/what/when |
| Artifact provenance | SLSA provenance attestations (Sigstore, slsa-github-generator) prove the artifact was built from a specific commit by a specific workflow |
| SBOM | Generated in-pipeline (Syft, Trivy), attached as an attestation |
| Signing | Cosign/sigstore signs images and attestations; verifiable without shared secrets |
| Approval gate | GitHub Environments required-reviewer; GitLab `when: manual` on protected environments |
| Cloud audit correlation | CloudTrail / GCP Audit Logs / Azure Activity Log record every API call by pipeline role; cross-reference by pipeline run ID |
Design principles:
1. **Export configuration as evidence** — branch-protection settings, environment protection rules, and workflow definitions are themselves audit artifacts.
2. **Automate the linkage** — commit message references the ticket (`[PROJ-123]`), PR title carries the ticket ID, deployment metadata references PR + commit. The chain must be reconstructable by following references, not by human memory.
3. **Immutable audit log storage** — write-once, read-many (WORM) storage via S3 Object Lock in compliance mode, in a **separate logging account** with no trust relationship allowing production principals to modify logs; CloudTrail log-file validation adds cryptographic signing of each log file.
4. **Self-audit monthly** — sample your own changes before the auditor does; fix gaps while they are cheap.
5. **Retain deploy logs for the full observation period** — 12+ months hot, 7 years cold for SOX-relevant systems.
### A Stage Map for Evidence-Producing Pipelines
| Pipeline stage | Evidence emitted automatically |
|----------------|--------------------------------|
| PR open → merge | PR metadata, branch-protection enforcement, required review approvals, status checks |
| Build | Commit SHA, builder identity, artifact digest, SBOM, provenance attestation (signed) |
| Test | Test reports, scan results (SAST/DAST/dependency), coverage, all keyed to the commit SHA |
| Promotion to staging | Promotion ledger entry (digest, labels, pipeline run ID) |
| Approval gate | Environment protection approval record (approver ≠ author, timestamped) |
| Deploy to production | Deployment record (who/what/when/env), cloud audit log correlation |
| Post-deploy verification | Smoke/synthetic results, metric snapshots, rollback-trigger status |
| Close | Ticket closure referencing verification evidence |
Each stage writes immutable, linkable records; nothing is "documented" after the fact. A GitOps deploy (Argo CD, Flux) collapses several rows into one reconcilable commit — the git history *is* the approval and deployment record.
## Separation of Duties in Automated Pipelines
The principles translate directly from manual change control:
1. **No single identity builds AND deploys to production.**
2. **Code authors do not approve their own deployments.**
3. **Pipeline definitions are protected from the code they process.**
4. **Build artifacts are immutable once produced.**
| Mechanism | How it implements SoD |
|-----------|-----------------------|
| Per-stage identities | Build, test, sign, stage, deploy each use different IAM roles/service accounts |
| OIDC short-lived credentials | 15–60 minute tokens, no static secrets, scoped to a specific job |
| GitHub Environments | Required reviewers (2+ for production), wait timers, deployment branch restrictions, environment-scoped secrets |
| GitLab protected environments | `when: manual` + protected runners + protected variables (prod creds only on protected branches) |
| CODEOWNERS on workflow files | `.github/workflows/` owned by a platform/security team; their approval required to modify |
| Immutable pipeline templates | Reusable workflows pinned to version tags; teams cannot override protected stages |
| Canary as an SoD control | Second manual approval required to proceed beyond the canary stage |
| Alerting on SoD violations | Alert when approver == author, when a pipeline accesses secrets it should not, when branch protection is modified |
> **Gotcha — Emergency paths that delete controls:** "If a required check is failing, fix the check or use an emergency process that requires multiple approvals and creates an audit trail. Never disable branch protection." Manual SSH to production bypasses every control; when it is unavoidable it must go through the break-glass procedure with a post-incident review, not become a routine habit.
## Pitfalls That Destroy Evidence
| Pitfall | Impact | Mitigation |
|---------|--------|------------|
| **Manual console changes without a ticket** | Change invisible to the auditor; evidence gap | Restrict console write access; require a ticket for any manual change; detect drift (AWS Config) |
| **Overloaded emergency procedure** | Half of all changes labeled "emergency" makes the category meaningless; auditors question each one | Tighten the definition; review emergency changes monthly; track the ratio |
| **No ticket↔deployment linkage** | Auditor cannot trace approval → deployed change; chain breaks | Enforce ticket ID in PR title, commit message, and deploy metadata |
| **Approver-as-author** | Same person approved and deployed; violates SoD | Require separation; at minimum, document why it was not feasible |
| **Deleted branches / PRs** | Evidence destroyed; change unreconstructable | Retain merged PR data; export to an evidence store; never delete main/release branches |
| **Timestamp drift** | Tools on different clocks; events cannot be correlated across systems | UTC everywhere; NTP synchronization; record timezone explicitly |
| **Unrecorded manual deploys** | SSH to production bypasses all controls | Eliminate direct SSH; break-glass with audit trail; alert on direct access |
| **Disabling branch protection "temporarily"** | Unprotected window permits direct pushes bypassing review | Never disable; use the emergency process with multi-approval |
| **Shared runners (PR + production)** | A malicious PR could reach production credentials | Separate runner pools; ephemeral isolated runners for untrusted work |
| **Single admin token ("god token")** | Skeleton key; outlives its creator; never rotated | Per-stage OIDC; no static tokens; regular credential audit |
| **Logs stored in the production account** | An attacker who compromises prod can delete the evidence | Cross-account logging; WORM storage; no delete permission for prod roles |
| **Short log retention** | Evidence gone before the audit window closes | 12 months minimum (SOC 2); 7 years for SOX; multi-regime hot + cold |
## Gotchas
- **Manual evidence is a smell.** If your compliance evidence is a folder of PDF screenshots assembled before each audit, your pipeline is not doing its job. Auditors increasingly expect exported config files and CI logs covering the full window, not screenshots.
- **CC8.1 failures are process failures.** The control text is satisfied by process mechanics (who approved, what was tested, when deployed) — the most common exceptions are missing links, not missing security.
- **"Config as code" is also a change.** Terraform/OpenTofu, feature-flag changes, and pipeline YAML edits are changes under CC8.1 and PCI 6.4 — they need the same review/approval/deploy-record chain, not a separate informal path.
- **Version-pin your workflows and actions.** Pinning pipeline definitions and third-party actions to SHAs is both a security control (see [supply-chain-security.md](./supply-chain-security.md)) and an audit control: the "how" of a deployment must be reproducible from the recorded version.
- **AI-generated code is not a bypass.** If an agent or AI assistant authored a change, the change still needs the same ticket → review → test → approval → deploy chain. Auditors have not yet settled on AI-involvement attestation requirements; the defensible default is to record authorship transparently and apply exactly the same controls as human-authored code.
- **Sample with a calendar, not a feeling.** Run the monthly self-audit as a fixed ritual: pull N random changes from the last month, replay the six-link trace (ticket → PR → CI → approval → deploy → verify), and file the gaps. A standing "evidence debt" list is far cheaper than an audit exception.
- **DORA is not a compliance framework, but it predicts audit outcomes.** Evidence-based change control (peer review + automated gates) is both the high-throughput model and the model that produces the cleanest audit trails — the two goals converge. See [metrics-and-dora.md](./metrics-and-dora.md).
## Sources and Further Reading
- [SOC 2 Change Management Controls (soc2auditors.org)](https://soc2auditors.org/insights/soc-2-change-management-controls/) — CC8.1 interpretation, sampling, exceptions
- [SOC 2 CC8.1 framework guide (episki)](https://episki.com/frameworks/soc2/change-management) — control text and emergency-change requirements
- [PCI DSS Requirement 6 (pcidssguide.com)](https://pcidssguide.com/pci-dss-requirement-6/) — 6.4.x and 6.3.2 change control text
- [DORA Article 9 (digital-operational-resilience-act.com)](https://www.digital-operational-resilience-act.com/Article_9.html) — ICT change management requirement
- [EU CRA SBOM requirements (Anchore)](https://anchore.com/sbom/eu-cra/) — CRA deadlines and SBOM mandates
- [Separation of Duties & Least Privilege in CI/CD (secure-pipelines.com)](https://secure-pipelines.com/ci-cd-security/separation-of-duties-least-privilege/) — pipeline SoD implementation
- [SOC 2 Audit Log Requirements (AuditPath)](https://www.auditpath.io/blog/soc2-audit-log-requirements) — retention regimes, WORM storage
- [Is a Change Advisory Board Really Needed? (Harness)](https://www.harness.io/blog/change-advisory-board-really-needed) — DORA/Accelerate evidence on external approval
references/feature-flag-lifecycle.md
# Feature Flag Lifecycle
Feature flags are **runtime control points** that decouple deployment from release: code ships to production, but behavior is exposed deliberately and reversibly. A flag that is treated as a one-off `if` statement is a liability; a flag managed through a deliberate lifecycle is the fastest rollback mechanism in your toolkit and the backbone of progressive delivery. This reference covers the flag taxonomy, the seven-stage lifecycle, naming and ownership, flag debt, testing, tooling and the OpenFeature standard, SDK key security, anti-patterns, and the precise limits of flags as a rollback lever.
## Flag Taxonomy
Pete Hodgson's canonical taxonomy (martinfowler.com, 2017) classifies flags along two axes — **longevity** (how long the flag lives) and **dynamism** (how often and for whom the value changes):
| Category | Purpose | Typical Lifetime | Dynamism | Cleanup |
|----------|---------|------------------|----------|---------|
| **Release toggles** | Hide incomplete code; enable trunk-based development; decouple deploy from release | Days to weeks (transient) | Static (same for all users per release) | Remove within 1–2 weeks; add a removal task at creation |
| **Experiment toggles** | A/B and multivariate testing; statistically significant cohorts | Hours to weeks (until significance) | High (per-user, per-request) | Remove when the experiment concludes; never let linger past significance |
| **Ops toggles** | Operational control: kill switches, circuit breakers, load shedding | Mostly short-lived; a few permanent kill switches | Very high (reconfigure in seconds, no redeploy) | Retire once confidence is gained; review permanent kill switches quarterly |
| **Permission toggles** | Entitlements, premium features, alpha/beta access | Very long-lived (years) | High (per-user, per-request) | Treat as permanent; review annually; never auto-expire |
A simpler operational split — **temporary vs permanent** — maps directly onto the taxonomy: temporary flags are release, experiment, and interop-testing toggles (created to be removed); permanent flags are entitlements, load shedding, custom branding, and accessibility toggles (created to persist). The two halves have opposite governance: temporary flags demand expiry enforcement; permanent flags demand annual review and careful change control.
## The Seven-Stage Lifecycle
Each stage has distinct best practices. Treat every flag as passing through: **create → guard → evaluate → rollout → verify → remove → expire**.
### 1. Create
- Assign an **owner** at creation — the person responsible for cleanup.
- Choose the category (temporary vs permanent) and set an **expiry date** for temporary flags.
- Follow the naming convention `{type}-{team}-{feature}-{context}` (see below).
- **Create the companion removal PR at the same time** as the feature PR — a documented practice that converts "we'll clean up later" into a scheduled task.
- Keep scope minimal: one flag per feature unit. For multi-part features (e.g., a dashboard plus three widgets), use **prerequisite flags** — a parent flag plus child flags — rather than one sprawling flag or four independent ones.
### 2. Guard (Implement)
- Decouple *toggle points* from *toggle logic* with a centralized abstraction — e.g., `featureDecisions.showNewCheckout()` instead of scattered `flags.isEnabled("next-gen-ecomm")` calls. A single wrapper function means cleanup touches one file, and the code reads as intent, not plumbing.
```python
# One module owns every flag decision for this service.
# Cleanup = delete the wrapper + its tests, then archive the flag key.
class FeatureDecisions:
def show_new_checkout(self, user) -> bool:
return self._flags.get_boolean("release-payments-new-checkout-web",
default=False, context=user)
```
- Convention: **OFF = legacy/old behavior, ON = new behavior**. Inverting this per-feature is how stale flags become dangerous.
- Place **per-user toggles at the edge** (UI layer) and **technical toggles in the core** service; avoid spreading the same flag across layers where a partial rollout can desync the experience.
- For kill switches, use **inverted logic**: kill-switch *disabled* = feature ON (default safe); kill-switch *enabled* = feature OFF. The safe default is the feature on, not off, so a misconfigured switch fails open for users.
### 3. Evaluate
- **Targeting rules** combine attributes (plan, region, cohort) with AND/OR logic.
- **Percentage rollouts** must use **deterministic/consistent hashing** on a stable identifier (user ID) so the same user always lands in the same bucket — otherwise users see the new and old state on alternating refreshes ("flickering"). This is sticky bucketing; no server-side session storage required.
- **Prerequisite flags** define valid dependency chains: the parent must be enabled before children are meaningful, and the SDK short-circuits children when the parent is off. Example: `release-analytics-dashboard` (parent) gates `release-analytics-dashboard-chart-a`, `...-chart-b`, and `...-export` (children); rolling the parent to 10% automatically scopes all three children to that same 10%.
- For experiments, bucket by user ID modulo so cohorts are stable across requests and can be analyzed later.
### 4. Rollout
- Progress through **tiers**: internal testers → beta/canary cohort → full production, and through **percentages**: 1% → 5% → 10% → 50% → 100%.
- Tie every step to **guardrail metrics** (error rate, latency, conversion). Roll back the flag, not the deploy, when guardrails trip. This is the same logic as canary analysis in [progressive-delivery.md](./progressive-delivery.md).
- Pause long enough at each step for the metrics window to be meaningful; a 1% canary for 30 seconds proves nothing.
A concrete rollout gate might look like:
| Step | Exposure | Gate to proceed | On gate failure |
|------|----------|-----------------|-----------------|
| 1 | Internal testers | No new errors in dogfood | Hold |
| 2 | 1% of traffic | Error rate ≤ baseline + 0.5% for 15 min | Toggle off, alert |
| 3 | 10% | P95 latency ≤ baseline + 10% for 30 min | Toggle back to 1% |
| 4 | 50% | Conversion within statistical bounds | Toggle back to 10% |
| 5 | 100% | Stable for 24h; then schedule removal | — |
Each step is a separate, logged event in the flag platform — the audit trail that makes flag rollback defensible to an auditor (see [change-governance-and-compliance.md](./change-governance-and-compliance.md)).
### 5. Verify
- **Test both flag states (ON and OFF)** in CI, plus the production-intended configuration and the fallback. With N flags, exhaustive combinatorial testing is impossible — focus on the flags changed in this release and their known interactions.
- Periodically verify the fallback values of *permanent* flags — a kill switch nobody has exercised in a year is a claim, not a capability.
### 6. Remove (Cleanup)
- Remove **all code references first**, then archive the flag in the flag platform. Deleting the key first strands the code; deleting after code removal is the safe order.
- Use **code-reference scanning** (LaunchDarkly code references, Uber's Piranha) to find every usage — including string literals in non-source files.
- **Archive, do not delete.** A deleted flag key can be recreated later, silently re-enabling old behavior (see Knight Capital below). Archival preserves history and prevents key reuse.
### 7. Expire
- **"Time bombs"** — tests or CI checks that fail if a flag outlives its expiry date — enforce removal mechanically.
- Detect stale flags continuously (GrowthBook auto-flags stale rules; a common target is **stale rate < 15%**, while most orgs run well above 40%).
- General guidance: **archive temporary flags quarterly (90–120 days)**. A healthy project has a high ratio of archived to unarchived flags.
## Naming and Ownership Conventions
| Practice | Convention | Why |
|----------|------------|-----|
| **Naming** | `{type}-{team}-{feature}-{context}`, e.g., `release-payments-new-checkout-web` | Self-documenting; a stale flag's type and owner are visible in its name |
| **Ownership** | Assigned at creation; owner responsible for cleanup | Prevents orphan flags; the flag registry shows who to page |
| **RBAC** | Limit who can toggle sensitive flags (prod kill switches) | A wrong toggle is an incident; scope the blast radius of the button |
| **Expiry** | Set at creation for temporary flags; enforced by time bombs | Removes the "we'll get to it" failure mode |
| **Definition of done** | "A feature is done when the flag is archived" | Pushes cleanup into the feature's own completion criteria, not a later debt cycle |
## Flag Debt and Cleanup
Unremoved flags are **technical debt with a unique compounding cost**: every stale flag doubles the combination space that tests must cover, masks dead code that developers fear to touch, and — in the worst case — becomes a live grenade.
- **Knight Capital (2012):** a reused flag name reactivated an obsolete trading algorithm; the firm lost **$460M in 45 minutes** and was sold days later. The mechanism: an old flag was repurposed, and its previous semantics came back alive. Never reuse flag keys; archive, don't delete.
- **Uber (Piranha):** built automated flag-removal tooling and deleted **2,000 stale flags** that had outlived their purpose. Piranha demonstrates that removal can be mechanized: find references, delete the toggle, migrate the test.
- **LaunchDarkly guidance:** target a 90–120 day time-to-archive for temporary flags; treat deletion of a key as a dangerous operation; use code references and "extinction events" to force cleanup of entire flag cohorts.
> **Gotcha — Flag debt is invisible until it bites:** A flag that has been ON for a year is not "working as intended" — it is dead code plus a test burden plus a future Knight Capital. If a flag's state has not changed in two quarters and it is not a permission/entitlement flag, it should be archived by default and defended on review.
## Testing Both Flag States
Every flag-gated code path must be exercised **ON and OFF** in CI, because the OFF path is what ships if the flag rolls back. Practical guidance:
- Test the production-intended configuration *and* the fallback configuration.
- Focus combinatorial effort on flags changed in the current release plus their prerequisite chains; do not attempt exhaustive N-flag matrix testing.
- For kill switches, include a scheduled drill: toggle, verify the feature actually disappears, toggle back.
- Add the flag-state matrix to the release checklist so a rollout that flips two interacting flags is caught before production.
## Tooling and the OpenFeature Standard
**OpenFeature** (CNCF, incubating since December 2023) is a vendor-neutral SDK standard. It standardizes the *application-facing* surface and deliberately leaves the *platform* surface vendor-specific:
| OpenFeature standardizes | OpenFeature does NOT standardize |
|--------------------------|----------------------------------|
| **Evaluation API** — vendor-agnostic `getBooleanValue("flag", false, context)` | Flag creation UI and admin workflows |
| **Evaluation Context** — key-value container (static global + dynamic per-request) | Targeting rule syntax |
| **Provider interface** — translates API calls to any vendor SDK (LaunchDarkly, Flagsmith, Unleash, ConfigCat, GrowthBook, or a local file) | Percentage rollout algorithms |
| **Hooks** — lifecycle interceptors for logging, telemetry, validation | Flag storage and audit trails |
| **Events** — provider readiness, error, config-change notifications | — |
| **OFREP** — Remote Evaluation Protocol for network evaluation | — |
SDK architecture splits into two evaluation models:
- **Server-side SDKs:** cache the full ruleset locally and evaluate in-process — **sub-millisecond** evaluation, thousands of evals/sec without per-request network cost.
- **Client-side SDKs:** evaluation is delegated to the vendor server per context; the full ruleset (including targeting rules that may embed PII) is **never** downloaded to clients.
### Platform Selection
| Platform | Type | Notable |
|----------|------|---------|
| **LaunchDarkly** | Commercial SaaS | Market leader; code references; flag lifecycle automation; engineering insights |
| **Flagsmith** | Open-source + SaaS | Self-hostable; OpenFeature provider |
| **Unleash** | Open-source + SaaS | 25+ SDKs; dependent flags; kill-switch patterns; GitLab integration |
| **ConfigCat** | SaaS | Budget-friendly; OpenFeature provider |
| **GrowthBook** | Open-source + SaaS | Warehouse-native; experimentation built in; feature evaluation diagnostics |
| **DevCycle / Statsig / Harness** | Commercial | Experimentation-first platforms; governance tooling |
| **GO Feature Flag** | Open-source | Built on OpenFeature; OFREP support |
**Build vs buy:** a config file or environment variable is genuinely enough when you have fewer than ~10 flags, no targeting needs, and no runtime toggling requirement — or in air-gapped environments where no SaaS is acceptable. Beyond that, the operational burden (admin UI, audit trails, RBAC, code-reference scanning, stale detection) makes a dedicated platform — or an OpenFeature-backed integration layer — the cheaper long-term choice. See [toolchain-landscape.md](./toolchain-landscape.md) for the wider deployment/observability tooling context.
### SDK Key Security
The LaunchDarkly key model is representative of the industry:
| Key type | Used by | Security posture | Prefix |
|----------|---------|------------------|--------|
| **SDK key** | Server-side SDKs | **Secret** — grants read access to the full ruleset; rotate if exposed | `sdk-` |
| **Mobile key** | Mobile SDKs (Android, iOS, React Native) | Not secret — only flags marked "available on mobile SDKs" | `mob-` |
| **Client-side ID** | JS client-side SDKs, edge SDKs | Not secret — only flags toggled for client-side use | alphanumeric |
> **Gotcha — SDK key in a client:** Embedding a server-side SDK key in a web or mobile app exposes the full ruleset, including targeting rules and potentially PII-bearing attributes. Client-side IDs exist precisely so you never ship a secret. Rotate any `sdk-` key that appears in a client bundle or a public repo.
Mobile specifics: streaming updates in the foreground, hourly polling in the background (battery/data), cached values served offline, and — critically — **iOS SDKs do not background-fetch**. App store review also treats flag-gated features as live: a hidden-but-functional feature in the binary can trigger review rejection.
## Anti-Patterns
| Anti-pattern | Description | Mitigation |
|--------------|-------------|------------|
| **Flag hell / sprawl** | Hundreds of unowned flags; combinatorial test explosion; nobody knows who owns what | Ownership, naming, expiry, stale detection (<15%), WIP limits on flag count |
| **Flags as permanent config** | Using release flags for static config that rarely changes; using flags as a database or secrets store | Use proper config management and secrets tooling; flags are for runtime behavioral control |
| **Flags masking dead code** | Unremoved flags hide unreachable paths; developers fear removal | Code-reference scanning, extinction events, Piranha-style automated removal |
| **Reused flag keys** | The Knight Capital scenario — old semantics come back alive | Never reuse keys; unique namespacing; archive not delete |
| **Non-sticky percent rollouts** | No consistent hashing → users flicker between states on refresh | Deterministic hashing on stable ID (all major platforms do this) |
| **Remote evaluation latency** | Per-request evaluation from a remote server compounds at thousands of evals/sec | Local/in-process evaluation, CDN caching, edge SDKs; sub-millisecond target |
| **Nested/conflicting flags** | Two flags control overlapping behavior; invalid state combinations | Prerequisite flags; single-purpose flags; document the dependency graph |
| **Client payload leaks** | Full rulesets (and their targeting PII) shipped to clients | Client-side IDs only; mark PII attributes private; never send rulesets to clients |
| **Interaction blindness** | Cannot answer "what does user X see?" across dozens of flags | Per-user flag previews and evaluation diagnostics in the flag platform |
## Flags as the Primary Rollback Lever — and Its Limits
For **code-level behavioral regressions**, a flag rollback beats an artifact rollback on every axis that matters in an incident:
| Dimension | Flag rollback | Artifact rollback |
|-----------|---------------|-------------------|
| Speed | Sub-second (toggle) | Minutes to tens of minutes (redeploy previous version) |
| Scope | Per-feature, per-user-segment | Entire deployment unit |
| Risk | Low — only the toggled feature is affected | Higher — reverts all changes in the artifact |
| Reversibility | Instant re-enable | Requires another deploy |
| Audit | Flag change logged in the flag platform | Deployment logged in CI/CD |
**But flags cannot undo reality.** They are a lever over *code paths*, not over *state*:
1. **Database schema changes** — a flag cannot un-alter a table. Use the **expand/contract pattern**: (a) add the new column additively, (b) deploy code behind a flag that uses the new column, (c) roll the flag out, (d) once stable, deploy code that stops reading the old column, (e) contract — drop the old column in a separate migration. **Wrap flags around code, never around DDL.** See [rollback-and-recovery.md](./rollback-and-recovery.md).
2. **Infrastructure changes** — flags do not control load balancers, DNS, or network policy. Use blue/green, canary infrastructure, or IaC rollback for those.
3. **Data mutations** — if flag-gated code writes data in a new format, toggling off does not undo the writes. Requires dual-write, backfill scripts, or compensating migrations.
4. **External side effects** — emails sent, payments processed, API calls made. Flags prevent future occurrences; they cannot reverse past ones.
5. **Multi-service contracts** — if service A's flag-gated change requires service B to change in lockstep, toggling one without the other breaks the contract. Requires coordinated rollout or API versioning.
6. **Mobile store review** — a flag can gate a feature, but the feature is still *in the binary* and reviewed as live; kill-switch-grade removal of a harmful feature may still require an app update.
> **Gotcha — The flag-rollback reflex:** Toggling a flag off does not "roll back" schema migrations, background jobs that already ran, or writes already committed. Before you tell an incident team "just flip the flag," check what the flag actually guards: code paths (safe to flip) or data/infra state (needs a complementary mechanism).
## Sources and Further Reading
- [Feature Toggles (aka Feature Flags) — Pete Hodgson, martinfowler.com](https://martinfowler.com/articles/feature-toggles.html) — the canonical taxonomy and lifecycle
- [Reducing technical debt from feature flags — LaunchDarkly](https://launchdarkly.com/docs/guides/flags/technical-debt) — lifecycle stages, archival, code references
- [OpenFeature Introduction](https://openfeature.dev/docs/reference/intro) — what the CNCF standard does and does not cover
- [Choosing an SDK type — LaunchDarkly](https://launchdarkly.com/docs/sdk/concepts/client-side-server-side) — SDK key types, client vs server evaluation, mobile behavior
- [4 Types of Feature Flags — Octopus Deploy](https://octopus.com/devops/feature-flags/) — taxonomy confirmation, management best practices
- [How to Implement Feature Flags at Scale — GrowthBook](https://www.growthbook.io/blog/how-to-implement-feature-flags-at-scale) — governance, stale detection, Knight Capital and Piranha case studies
- [Kill Switches Best Practice — Unleash](https://www.getunleash.io/blog/kill-switches-best-practice) — inverted-logic kill switch design
- [Database Migrations with Feature Flags — Harness](https://www.harness.io/blog/database-migration-with-feature-flags) — expand/contract, flags around code not DDL
references/metrics-and-dora.md
# DORA Metrics: Exact Definitions, Benchmarks, and Pitfalls
DORA's software-delivery metrics are the shared scoreboard of release engineering. This file gives the **current five-metric model** with exact definitions, formulas, units, raw data sources, and aggregation rules; the last classic four-tier performance table (2024) with the mandatory 2025 caveat; known vendor-formula divergences; and the pitfalls that make most naive implementations wrong. One-page version: [../assets/dora-metrics-reference.md](../assets/dora-metrics-reference.md).
## The Five Metrics
DORA's metric set evolved over a decade: the original four keys (2014) were joined by a scoped recovery metric (2023), and a fifth delivery metric, **deployment rework rate**, was added in 2024. The current set splits into **throughput** and **instability**:
| Group | Metric | What it measures |
|-------|--------|------------------|
| Throughput | **Change lead time** | Commit → production |
| Throughput | **Deployment frequency** | How often changes ship |
| Throughput | **Failed deployment recovery time** | Time to recover from a failed deploy (formerly MTTR) |
| Instability | **Change failure rate** | Share of deploys needing immediate intervention |
| Instability | **Deployment rework rate** | Share of unplanned, bug-fix deployments |
The metric set's evolution matters when you read older material:
| Year | Change |
|------|--------|
| 2014 | Original four variables: deployment frequency, lead time for changes, MTTR, change fail rate |
| 2015 | Solidified the throughput-vs-stability framing; debunked the speed-vs-stability tradeoff myth |
| 2018 | Added "availability"; renamed the construct "Software Delivery and Operational (SDO) performance" |
| 2021 | Expanded availability → reliability (an *operational* measure, not a delivery metric) |
| 2023 | MTTR renamed and re-scoped to **Failed Deployment Recovery Time** (change-caused failures only) |
| 2024 | **Deployment Rework Rate** added as the fifth metric |
| 2025 | Report renamed "State of AI-assisted Software Development"; tiers replaced by archetypes; the five metrics remain |
## Operationalization: Data Sources and Canonical Scales
**Data-source → metric map** (for building the pipeline that produces these numbers):
| Metric | Primary source | Secondary |
|--------|---------------|-----------|
| Deployment frequency | CI/CD deploy events (env = production) | Git Deployments API / Releases-tags; CD tool records |
| Change lead time | VCS commit timestamps + deploy finish time + deployed SHA | PR merge time (for stage breakdowns) |
| Failed deployment recovery time | Incident management (change-caused incidents only) | Deploy records (failed deploy → restoring deploy) |
| Change failure rate | Deploy records + rollback/roll-forward detection | Incidents linked to deploys |
| Deployment rework rate | PR/branch/label heuristics + incidents | Issue-tracker bug labels |
**DORA's canonical categorical scales** (from the Quick Check — the bands DORA itself uses in surveys):
- **Change lead time:** >6 months | 1–6 months | 1 week–1 month | 1 day–1 week | <1 day | <1 hour
- **Deployment frequency:** <1/6 months | 1/month–1/6 months | 1/week–1/month | 1/day–1/week | 1/hour–1/day | on-demand (multiple/day)
- **Failed deployment recovery:** >6 months | 1–6 months | 1 week–1 month | 1 day–1 week | <1 day | <1 hour
- **Change failure rate:** 0–100% slider
- **Deployment rework rate:** 0–100% slider (last 6 months, unplanned bug-fix deploys)
### Worked Example (30-Day Window)
A service deploys 21 times in 30 days; 20 succeed; 3 of the successful deploys were unplanned hotfixes; 1 deploy fails in production and is rolled back 40 minutes later. A change commits on day 3 and reaches production on day 7.
| Metric | Computation | Result |
|--------|-------------|--------|
| Deployment frequency | 20 successful prod deploys / 30 days | 0.67 deploys/day |
| Change lead time (that change) | day 7 deploy finish − day 3 commit | 4 days (aggregate: median across all changes) |
| Change failure rate | 1 failed deploy / 21 total | 4.8% |
| Failed deployment recovery time | 40 min from failure start to rollback completion | 40 min (aggregate: median) |
| Deployment rework rate | 3 unplanned hotfix deploys / 21 total | 14.3% |
Note the rework rate (14.3%) exceeding CFR (4.8%) — the typical pattern, because unplanned remediation deploys outnumber outright failed deploys.
## Exact Definitions, Formulas, and Data Sources
### Deployment Frequency (DF)
- **Definition:** How often application changes are deployed to production / released to end users.
- **Unit:** successful production deployments per day (a rate), or a categorical band.
- **Formula:** `DF = number of successful production deployments / number of days in window`.
- **Counting:** count *successful* deployments to **production only** (or "release to end users"); staging/test deploys never count. Count per **service**, not per repository — a monorepo deploying five services from one merge produces five deployment events.
- **Raw data sources:** CI/CD deployment events; CD tool records filtered to `environment = production`; Git host Deployments API (e.g., GitHub `/deployments?environment=production`, latest status `state == success`); cluster deploy logs. Proxy: Releases/tags — but this overcounts pre-releases and undercounts untagged hotfixes.
### Change Lead Time (CLT)
- **Definition:** Time from code **committed to version control** to that change **successfully running in production**.
- **Unit:** duration (hours/days).
- **Formula (per change):** `CLT = production_deployment_finish_time − commit_creation_time`.
- **Aggregation:** **median** across changes. Distributions are right-skewed — one long-lived refactor wrecks the mean. Datadog's approach: `git log <prev_deploy_sha>..<this_deploy_sha>` to find the commits in a deployment, drop merge commits (no new code), compute the per-commit duration, then aggregate (median across deployments).
- **Raw data sources:** version-control commit timestamps, deploy records (finish time + deployed SHA), PR merge data for stage breakdowns (time-to-PR-ready, review time, merge time, time-to-deploy, deploy time).
- **Correlation key:** the **commit SHA** (or the PR number stored in deploy metadata). Squash and rebase merges change SHAs and break naive matching — store the PR number in deployment metadata to correlate directly.
### Change Failure Rate (CFR)
- **Definition:** Ratio of deployments that cause a failure in production and **require immediate intervention** (rollback, roll-forward/hotfix, patch).
- **Unit:** percentage (0–100%).
- **Formula:** `CFR = (number of failed deployments / total deployments) × 100`.
- **Counting:** a change is a failure only if it needs *remediation in production*; a defect caught in staging is not CFR.
- **Raw data sources:** deploy records + rollback/roll-forward detection (via git metadata/version tags, or PR-title/branch heuristics like `^revert`, `^rollback`, `^hotfix`, `^emergency`); incidents linked to deployments.
### Failed Deployment Recovery Time (FDRT) — the 2023 MTTR rename
- **Definition:** Time to recover from a **deployment that fails and requires immediate intervention** — i.e., restoring service after a *change to production caused an impairment*.
- **Why the rename:** the old MTTR ("time to restore service") did not distinguish change-caused failures from external causes (data-center outage, network event). In 2023 DORA re-scoped the metric to **change-caused failures only**. Generic incident MTTR is *not* FDRT unless incidents are tagged change-caused.
- **Unit:** duration (minutes/hours).
- **Formula (per incident):** `FDRT = remediation_time − failure_start_time`, where remediation is the rollback or roll-forward deployment that restores service.
- **Aggregation:** **median**; use histograms/scatter, not mean-of-averages (non-normal distribution).
- **Raw data sources:** incident management (PagerDuty/OpsGenie/incident.io: created → resolved), filtered to change-caused incidents; deploy records (failed deploy → restoring deploy); monitoring/alerting (impairment detected → restored).
### Deployment Rework Rate (DRR) — added 2024
- **Definition:** Ratio of deployments that are **unplanned but performed to address a user-facing bug** — reactive remediation rather than planned value delivery. DORA's quickcheck phrasing: "percentage of deployments in the last 6 months that were not planned but were performed to address a user-facing bug."
- **Unit:** percentage (0–100%).
- **Formula:** `DRR = (unplanned/remediation deployments / total deployments) × 100`.
- **Counting:** classify deployments as unplanned via PR title/branch/label heuristics (revert/rollback/hotfix/fix-forward/emergency), linked incidents, or deploy-metadata flags. DORA's survey asks about the **last 6 months** specifically.
- **Distinction from CFR:** CFR counts *failed deployments*; DRR counts *unplanned deployments caused by production issues*. In practice you typically have more unplanned remediation deploys than outright failed deploys, so **DRR ≥ CFR**. DRR captures the "hidden tax" of reactive work that CFR understates.
## Performance Clusters: The 2024 Table (Last Classic Four-Tier Version)
DORA's tiers come from **cluster analysis** — a descriptive pattern-detection method over that year's survey respondents, not fixed prescriptive thresholds — and the values shift every year. **2024 is the last year with the classic Low/Medium/High/Elite table.**
| Level | Change lead time | Deployment frequency | Change failure rate | Failed deployment recovery time |
|-------|------------------|----------------------|---------------------|---------------------------------|
| **Low** | 1 to 6 months | Monthly to biannual | 40% | 1 week to 1 month |
| **Medium** | 1 week to 1 month | Weekly to monthly | 10% | Less than a day |
| **High** | 1 day to 1 week | Daily to weekly | 20% | Less than a day |
| **Elite** | Less than a day | On demand (multiple/day) | 5% | Less than an hour |
- The 2024 table has the famous **inversion**: High shows a *higher* CFR (20%) than Medium (10%) — clusters are descriptive groupings, not a monotonic scorecard.
- Elite vs. Low magnitudes (Octopus's reproduction): elite deploys ~182× more often, ~8× lower CFR, ~127× faster lead time, ~2,293× faster recovery. The Elite cluster has historically been under 20% of organizations.
- 2023 comparison: Elite stayed stable; High's CFR rose; Medium improved CFR and recovery; Low improved stability but worsened throughput.
> **Gotcha — DORA retired the tiers in 2025:** The 2025 report ("State of AI-assisted Software Development") **dropped the Elite/High/Medium/Low tiers entirely**, replacing them with seven qualitative archetypes built on eight measures (throughput, stability, team performance, product performance, individual effectiveness, time on valuable work, friction, burnout). 2025 publishes only metric *distributions*, not tiers. If you quote the four-tier table, **anchor it to 2024** and say the tiers were retired. The five delivery metrics themselves remain the metric set.
**2025 distribution highlights (for context, from the 2025 report):** only 16.2% of respondents deploy on-demand; 23.9% deploy less than once a month; 43.5% have lead times over a week; 39.5% exceed 16% CFR; 56.5% take between a day and a week to recover; only 7.3% have rework below 2%. A 2025 finding with direct release-engineering relevance: AI adoption correlates with higher throughput and individual effectiveness but **higher instability** — stability is the metric AI has not improved.
## Vendor-Formula Divergence
Your tool's numbers will not equal DORA's survey numbers, and different tools disagree with each other. Three known divergences to account for:
- **Datadog** computes CLT from `git log` between deployed SHAs, drops merge commits, and aggregates per-commit values (avg/max/min per deployment, median across deployments); detects CFR via rollback/roll-forward detection; correlates recovery time with incidents.
- **GitLab** measures CLT from **MR merge time** (button clicked) to production — *not* from commit creation as DORA canonically defines it — clamped with `GREATEST(0, deploy_finished − mr_merged)`; uses **mean** for deployment frequency (historical choice) and median for CLT; derives CFR as incidents/deployments with a known double-counting bug for duplicate incidents.
- **Azure DevOps / GitHub-native** tooling covers DF, CLT, and a CFR proxy from their own data but generally **cannot compute FDRT** without an incident-management source (PagerDuty/OpsGenie/incident.io).
> **Gotcha — "your tool's CFR may not equal DORA's CFR":** Vendors operationalize CFR via incident counts (GitLab) or rollback detection (Datadog) rather than the survey's "requires remediation" definition. Always read the vendor's formula before quoting numbers in a report, and note the definition next to the value.
## Pitfalls (Most Implementations Get These Wrong)
1. **Counting PR merges as deployments.** A merge to `main` is not a deployment. If you deploy once a day regardless of merges, DF is once a day.
2. **Mean instead of median** for lead time and recovery time — skewed distributions make the mean meaningless (one 2-week refactor destroys it).
3. **Repository frequency instead of service frequency.** Monorepos must count per service; otherwise one repo's five-service deploy reads as one.
4. **Ignoring rollbacks and roll-forwards.** Without detecting them you undercount CFR and cannot compute FDRT at all.
5. **Counting non-production environments.** Staging failures are not CFR; scope every metric to production/release-to-users.
6. **Generic MTTR vs. change-scoped FDRT.** Including infrastructure/network/hardware outages inflates recovery time and violates the 2023+ definition.
7. **Squash/rebase SHA mismatch** — breaks commit↔deploy correlation; store PR numbers in deploy metadata.
8. **Wrong sampling window and timezones.** Define a fixed window and normalize to the team's local timezone before day-of-week analysis (a Friday 5pm EST deploy looks like Saturday 00:00 UTC).
9. **"Average of averages."** Aggregating deployment-level averages obscures per-commit reality; prefer commit-level CLT then median across deployments.
10. **Disparate comparisons.** Metrics are application/service-level; don't compare a mobile app to a mainframe, and don't build league tables between teams (DORA explicitly cautions against team competition). Compare an application to *itself over time*.
11. **Double-counting duplicate incidents** (a known GitLab CFR bug).
## DORA Metrics Are System Outcomes, Not Individual Metrics
DORA metrics measure an application and its delivery **system**. Attributing them to individuals is a documented misuse: setting DF or CFR as a personal goal invites gaming (splitting deploys to inflate frequency, under-reporting failures) — Goodhart's law in action ("when a measure becomes a target, it ceases to be a good measure"). Share all five metrics across dev, ops, and release rather than assigning single metrics to single teams.
**How release engineers actually use them for process improvement:**
- **Diagnose the bottleneck, not the symptom.** High CLT with low DF → batch-size and trunk-based-development problems; high CFR with low FDRT → weak gates and missing rollback rehearsal; high DRR with normal CFR → reactive workload masking (the hidden tax). DORA's core validated finding: speed and stability are **correlated, not a tradeoff** — top performers do well on both.
- **Drive small batches.** Smaller changes move faster *and* fail/recover better; batch-size reduction is a primary lever.
- **Gate improvement work.** Pair DORA with pipeline health (success rate, mean-time-to-green, flaky-test rate) and supply-chain coverage (signing/SBOM) for a complete release-reliability picture.
- **Compare the application to itself over time.** The defensible improvement loop is: baseline the five metrics → pick one bottleneck (e.g., CLT driven by a slow review stage) → change the process → re-measure on the same definition → keep what moved the number. Never chase another team's number.
- **Bound the measurement cost.** Precise multi-system instrumentation may not pay for itself; start with the Quickcheck conversation or a platform-native dashboard, then invest where the signal is actionable.
## Sources and Further Reading
- [DORA — Software Delivery Performance Metrics (dora.dev guide)](https://dora.dev/guides/dora-metrics/)
- [DORA — A History of DORA's Software Delivery Metrics](https://dora.dev/insights/dora-metrics-history/)
- [DORA Quick Check](https://dora.dev/quickcheck/)
- [Octopus — The 2024 DevOps Performance Clusters](https://octopus.com/blog/2024-devops-performance-clusters)
- [RedMonk — DORA 2025: Measuring Software Delivery After AI](https://redmonk.com/rstephens/2025/12/18/dora2025/)
- [Datadog — DORA Metrics Calculation](https://docs.datadoghq.com/dora_metrics/calculation/)
- [GitLab — DORA Metrics](https://docs.gitlab.com/user/analytics/dora_metrics/)
- [Koalr — How to Calculate DORA Metrics from GitHub Data](https://koalr.com/blog/calculate-dora-from-github)
references/monorepo-polyrepo-release.md
# Monorepo and Polyrepo Release Strategies
Where code lives determines how releases happen. Polyrepos isolate teams but make dependency change a **coordination problem**; monorepos make coordination cheap but push complexity into **build, versioning, and release tooling**. This reference covers the two models, the versioning schemes each enables, affected-build detection, the release tooling landscape, topological publishing, real-world anchors, and decision guidance.
## The Polyrepo Model
In a polyrepo architecture, each repository owns its own CI/CD pipeline, versioning strategy, and release cadence. Independence is the feature: teams get full autonomy over branching, deployment policy, and cadence, and access controls are enforced per project. The cost is that **dependencies must be created deliberately** — every shared library change requires a publish step to a registry, and every consumer must discover, evaluate, and adopt the new version.
### Registry-Mediated Dependency Propagation
Cross-repo coordination flows through registries (npm, PyPI, Maven Central, crates.io). Version pinning strategies — exact pins (`1.2.3`), caret ranges (`^1.2.3`), tilde ranges (`~1.2.3`) — determine how eagerly consumers adopt updates. Polyrepo workflows therefore lean heavily on **automated dependency-update bots**:
- **Renovate** (Mend): 90+ package managers, works across GitHub/GitLab/Bitbucket/Azure DevOps, monorepo-aware grouping and scheduling, auto-merge support.
- **Dependabot** (GitHub): zero-config for GitHub-hosted repos, security alerts and version updates, less flexible for complex or non-GitHub setups.
### Coordination Cost Scaling
The core polyrepo pain is **synchronizing deployments across repositories**. A breaking change to a shared library does not ship — it *cascades*:
- A breaking change in a shared lib triggers **N separate PRs** (one per consumer repo), each with its own review cycle — a process Renovate/Dependabot can generate but not approve for you.
- Update cascades form: service A updates and breaks service B, which blocks service C; the failure is a *topological* problem, not a per-repo one.
- Adoption timing diverges — some repos update immediately, others lag for months, so you permanently run a matrix of library versions in production.
- Atomic cross-service refactors are effectively impossible: there is no single commit that fixes all consumers.
A concrete example: library `auth-core@2.0.0` removes the legacy token format. With 50 consumer services across 50 repos, that is 50 update PRs (Renovate/Dependabot generate them), 50 review cycles, and 50 deployment windows — and service 17's team is on vacation, so for weeks the org runs 49 services on the new contract and one on the old, with the shared identity provider forced to serve both.
**Backporting** is the same problem in miniature: a security fix must be cherry-picked into each repo's stable branches independently. There is no shared stable branch to fix once. GitLab's patch process (backporting security fixes to multiple stable branches) is the canonical monorepo alternative to this per-repo chore.
### Coordination Patterns That Work in a Polyrepo
Polyrepo coordination is a discipline, not a tool. The patterns that keep it tractable:
| Pattern | How it works | Failure mode it prevents |
|----------|--------------|--------------------------|
| **Explicit version pinning policy** | Decide exact-pin vs caret/tilde per artifact class; document it | Silent adoption of breaking versions via loose ranges |
| **Compatibility windows** | Shared libs guarantee N-1 support; majors announce deprecation one cycle ahead | Consumers forced to upgrade on the library's schedule |
| **Dependency-update cadence** | Renovate/Dependabot batches by type (security immediate, minors weekly, majors monthly) | Update PR flood and last-minute major migrations |
| **Contract tests at the boundary** | Consumer-driven contract tests against the shared lib's published API | Cross-repo breakage discovered at deploy time, not CI time |
| **Shared release notes channel** | Every lib publish posts notes consumers can triage | Consumers learning about breaking changes from failing builds |
None of these eliminates the coordination cost — they bound it. If the cost keeps growing regardless, that is the signal that the affected packages belong in a shared repo.
## Monorepo Versioning Models
### Fixed / Synchronized Versioning — The "One Version Rule"
All packages share a **single version number**. Lerna's fixed mode (the default) operates on a single version line: any updated package is released at the new shared version, and `--force-publish` pushes all packages to version together, preventing drift. A major change in any package bumps **all** packages to a new major.
The philosophy's strongest form is Google's **one-version rule**: "There may only be one version of a package in //third_party." Rationale: (1) *maintenance* — multiple copies means multiple locations to keep updated; (2) *security* — vulnerability feeds omit older affected versions, and older versions accumulate latent vulnerabilities; (3) *diamond dependencies* — if two versions exist, eventually a build depends on both, and untangling the conflict can stop an unrelated project dead. Exceptions require formal approval (temporary <1 month auto-approved; >1 month needs director sign-off; permanent is rarely granted). This is the philosophical opposite of npm-style independent versioning.
### Independent Per-Package Versioning
Each package versions on its own SemVer line. **Changesets** is purpose-built for this: developers declare intent per package via small markdown changeset files, and the tool flattens the bump types into a single release per package while handling internal dependencies across a multi-package repository. Lerna's independent mode prompts per-package versions at publish time.
| Dimension | Fixed / one-version | Independent |
|-----------|---------------------|-------------|
| Version count | One for the whole repo | One per package |
| Upgrade story for consumers | Everything moves together; no version matrix | Per-package; consumers choose when to adopt |
| Atomicity | All packages release together | Packages release as they are ready |
| Fit | Google-style source-built monorepos, tight coupling, compliance | OSS libraries (npm ecosystem), decoupled lifecycles |
| Tooling | Lerna fixed mode, release-please `linked-versions` | Changesets, Lerna independent mode, release-please per-package |
### Release Trains
A release train is a **calendar-forced** release: everything merged and deployed by the cut date ships together, whether or not any single feature is "done" — the train leaves on time. GitLab is the canonical example: monthly releases on the third Thursday, with auto-deploy → release candidate (2 days before) → tag day → release day. Features ship only in monthly releases; patch releases carry only fixes; an MR must be merged, deployed to production, and stay deployed without rollback to be included. Trains trade feature timing for coordination cost and predictable dates — see [release-process-models.md](./release-process-models.md) and [release-operations-and-triage.md](./release-operations-and-triage.md).
The opposite end of the spectrum is **publish-on-merge**: every merged change that passes gates becomes part of a release immediately (what Changesets-enabled monorepos like Astro and SvelteKit effectively do per package). The two ends differ on cadence, not on quality gates:
| Dimension | Release train | Publish-on-merge |
|-----------|---------------|------------------|
| Cadence | Fixed calendar (monthly, weekly) | Every merge / every N merges |
| Feature timing | Fixed by the schedule | As soon as it is ready |
| Coordination cost | Low (everything rides the train) | Low per change, but version churn for consumers |
| Rollback | Whole train re-rolled or patch-released | Per-package version pin rollback |
| Fit | Enterprise, regulated, self-managed products | SaaS, libraries, fast-moving OSS |
Most organizations land between the extremes: continuous deployment to internal environments, plus a calendarized *customer-visible* release (train) for self-managed or on-prem consumers — which is exactly GitLab's model.
## Affected-Build Detection
The monorepo build problem: how do you know what to build and test for a given change, when the repo contains dozens or hundreds of projects? The answer is **affected-target analysis** — compute the minimal set of projects affected by a change:
| Tool | Mechanism | Notes |
|------|-----------|-------|
| **Nx** | `nx affected -t <task>`: git-diff changed files → map to projects via the project graph → transitive dependents | Configurable `--base`/`--head` SHAs; `projectsAffectedByDependencyUpdates` controls how dependency updates propagate |
| **Turborepo** | Content hashing of inputs (source, env vars, task dependencies) → cache hits skip unchanged packages | `dependsOn: ["^build"]` ensures dependency tasks run first; remote caching accelerates |
| **Bazel** | Build graph is a DAG of targets with explicit deps; changed-target transitive closure | Google's Piper/Blaze runs presubmit on affected targets only; the deepest form of the model |
> **Gotcha — Affected ≠ unaffected:** If you modify a widely used project, "affected" correctly expands to nearly the whole repo — running tasks for almost all projects is the tool telling you the truth about blast radius. Conversely, a known Nx limitation (nrwl/nx#33276) is that `nx release` has assumed all releaseable projects were built, which conflicts with the affected workflow — validate your release pipeline against the affected set explicitly.
**Changesets takes a different route:** developers *declare* which packages are affected via changeset files (human-declared intent) rather than the tool computing it — trading computation for explicit authorship. The Changesets GitHub Action aggregates pending changesets into a single "version PR" that bumps versions and updates changelogs. A changeset is a small markdown file:
```markdown
---
"@repo/analytics": minor
---
Add session-replay export to the analytics API.
```
The Changesets bot flags PRs that change a package without adding a changeset, keeping the declaration discipline automatic.
### Release-Pipeline Leverage
Affected detection feeds two release strategies:
- **Publish only changed packages** — Nx Release versions and publishes per project; Changesets publishes only packages with pending changesets. Efficient, but consumers must handle partial availability.
- **Release-all on a train** — GitLab ships everything deployed to production in the monthly train; Lerna `--force-publish` publishes all packages regardless of changes. Predictable, but noisy releases for unchanged packages.
The choice mirrors the versioning model: publish-only-changed pairs with independent versioning; release-all pairs with fixed/one-version.
## Release Tooling Fit
| Tool | Model | Best fit |
|------|-------|----------|
| **Changesets** | Human-declared intent (markdown files), version PR, publish | JS/TS monorepos, independent versioning (pnpm, Astro, SvelteKit, Chakra UI, Remix, Firebase JS SDK) |
| **Lerna** | Fixed (default) or independent mode; `lerna version`/`lerna publish`; `from-package` retry | Legacy JS monorepos; teams already on Lerna; task-running delegated to Nx/Turborepo |
| **semantic-release** | Fully automated from conventional commits (analyze → bump → changelog → publish) | Single repos; small monorepos via community plugins (`multi-semantic-release`, `semantic-release-monorepo`) |
| **release-please** | Manifest mode: combined Release PR across configured packages from two config files; plugins `node-workspace`, `cargo-workspace`, `maven-workspace`, `linked-versions`, `group-priority` | Multi-language monorepos, Google-style, hundreds of packages |
| **Nx Release** | Three phases (versioning, changelog, publishing); independent or grouped; npm/Docker/crates targets; programmatic API; `--dry-run` | Nx workspaces needing multi-target publishing |
Automation spectrum worth noting: **semantic-release** (fully automated, no human gate) → **release-please** (auto-generates a release PR, human merges) → **changesets** (developer declares intent per PR). Turborepo itself does not version or publish — its official recommendation is to pair it with Changesets (`turbo run build lint test && changeset version && changeset publish`). For the wider toolchain context (versioning, CI/CD, deployment), see [toolchain-landscape.md](./toolchain-landscape.md).
## Publishing Order and Failure Recovery
### Topological Publish Order
Packages must be published **dependencies-first**. If `@repo/ui` depends on `@repo/utils`, `@repo/utils` must reach the registry before `@repo/ui` publishes, or the registry resolves a version that does not exist. For a dependency chain `app → @repo/ui → @repo/utils → @repo/core`, the publish order is `@repo/core`, then `@repo/utils`, then `@repo/ui`, then `app` — each step's registry range resolving to an already-published version. pnpm recursive commands (`pnpm -r publish`) respect topological order by default; Lerna publishes topologically; Nx Release respects the project graph; release-please's `node-workspace` plugin updates the dependency references in each consumer's manifest as it walks the graph.
### The `workspace:` Protocol
During development, workspace packages reference each other locally via the `workspace:` protocol; at publish time pnpm rewrites it to registry ranges:
| Workspace spec | Published as |
|----------------|--------------|
| `workspace:*` | `1.5.0` (exact version) |
| `workspace:~` | `~1.5.0` |
| `workspace:^` | `^1.5.0` |
| `workspace:^1.5.0` | `^1.5.0` |
This lets you depend on local packages during development and publish without intermediate publish steps. Note `saveWorkspaceProtocol` defaults to `rolling` (saves `workspace:^`).
> **Gotcha — Leaked `workspace:` specifiers:** If a package publishes with an unreplaced `workspace:*` range, consumers cannot resolve it. Verify post-publish that registry metadata contains real versions, and make the rewrite a pipeline assertion, not an assumption.
### Partial-Publish Failure Recovery
A multi-package publish is **not atomic**: if package 3 of 10 fails, packages 1–2 are already on the registry and 4–10 are not. Recovery relies on idempotent retry:
- **Lerna `from-package`:** compares local vs registry versions and publishes only what is missing — re-running after a partial failure converges.
- **Changesets:** `changeset publish` is idempotent; re-running skips already-published versions.
- **release-please:** separates tag creation from publishing — if publish fails, tags exist and the publish step can be retried.
- **npm/pnpm:** `--access public` and OTP/2FA can fail mid-batch; plan for it in the retry loop.
**Cyclic dependencies** break topological ordering: pnpm cannot guarantee script order when workspace cycles exist (`disallowWorkspaceCycles` can make installation fail on cycles instead). Shopify built **Packwerk** specifically to detect and prevent circular dependencies between its monolith components.
## Real-World Anchors
- **Google (one-version monorepo):** ~35,000 developers, billions of lines of code, Piper VCS + Blaze/Bazel build system. One version of every dependency, enforced by policy; all code built from source at HEAD; presubmit on affected targets; large-scale changes via tooling (Rosie). External releases (Go, Angular) use separate processes.
- **GitLab (single repo, release train):** all code in one repository; monthly release train for 178+ consecutive months; auto-deploy daily, RC 2 days before release, tag Wednesday, release Thursday 13:00 UTC; patch releases for security/critical fixes from stable branches; features ship only in monthly releases.
- **Shopify (modular monolith):** a 2.8M+ line Ruby monolith with ~37 components (Rails Engines), Packwerk for dependency enforcement, Sorbet for contracts; selective extraction only for clear reasons (storefront rendering — high-throughput read-only; credit-card vaulting — sensitive data). New Rails apps are "componentized by default": a monorepo with internal modularity rather than npm-style publishing.
## Decision Guidance: Monorepo vs Polyrepo
| Choose monorepo when... | Choose polyrepo when... |
|------------------------|------------------------|
| Packages are tightly coupled and change together | Services are loosely coupled with independent lifecycles |
| Atomic cross-package changes are frequent | Teams need full autonomy over release cadence |
| Shared libraries have many internal consumers | Strict access-control / compliance boundaries per project |
| You want single-version consistency (Google model) | Different compliance/retention rules per product |
| A platform team manages many related packages | Repository size would exceed tooling limits |
| You need system-wide visibility into change impact | You need independent CI/CD scaling per service |
Monorepos excel at code sharing and consistent standards; polyrepos enforce strong per-project security boundaries at the cost of visibility and coordination. **Switching later is expensive and disruptive** — the migration cost argument cuts both ways, so the choice should be deliberate, not accidental.
### Migration Heuristics
- **Toward a monorepo:** if you spend more time coordinating library changes across repos than building features (the "50 PRs for one change" pattern), consolidating the *shared* packages first — while leaving genuinely independent services separate — captures most of the benefit without a big-bang migration. Shopify's modular monolith is the proof that internal modularity can substitute for package-publishing overhead.
- **Away from a monorepo:** if one component's compliance regime (SOX scope, data residency, customer-isolated tenancy) keeps colliding with the rest, extract that component first; the per-repo pipeline cost is a tax you pay to gain the boundary.
- **Never migrate for tooling novelty.** Nx, Turborepo, and Bazel make monorepos *tolerable*; they do not make a loosely coupled polyrepo system better. Measure the actual coordination pain before restructuring.
> **Gotcha — Monorepo ≠ single version:** Choosing a monorepo does not force you into one-version semantics. Independent versioning (Changesets/Lerna independent mode) gives you atomic cross-package *refactors* while keeping per-package *versions* — the two decisions (repo topology vs versioning model) are orthogonal and should be made separately.
## Sources and Further Reading
- [Google One-Version Rule](https://opensource.google/documentation/reference/thirdparty/oneversion) — the one-version philosophy and exception process
- [Changesets documentation](https://changesets-docs.vercel.app/readme.html) — intent-based publishing for JS/TS monorepos
- [Lerna version and publish](https://lerna.js.org/docs/features/version-and-publish) — fixed vs independent mode, `from-package` retry
- [Nx affected](https://nx.dev/docs/features/ci-features/affected) — git-diff + project-graph affected detection
- [release-please manifest mode](https://github.com/googleapis/release-please/blob/main/docs/manifest-releaser.md) — combined release PRs across packages
- [GitLab monthly releases](https://handbook.gitlab.com/handbook/engineering/releases/monthly-releases/) — the canonical release train
- [Shopify's modular monolith](https://shopify.engineering/shopify-monolith) — 37-component Rails monolith with Packwerk
- [Why Google Stores Billions of Lines of Code in a Single Repository (Potvin & Levenberg, CACM 2016)](https://research.google/pubs/why-google-stores-billions-of-lines-of-code-in-a-single-repository) — the monorepo evidence base
references/progressive-delivery.md
# Progressive Delivery
**Progressive delivery** — the term was coined by RedMonk's James Governor around 2018 — is the practice of **decoupling deploy from release**: putting a change into an environment is not the same event as exposing it to users. Deployment becomes a gradual, measured, automatically-evaluated process: you expose a change to a fraction of users or traffic, check that the world is still healthy, then expose more. It wraps canaries, blue/green, rings, feature flags, and A/B testing into one disciplined approach, and it is the primary way modern teams make high deployment frequency compatible with low change-failure rate.
## Decouple Deploy from Release
The single most useful mental model in release engineering:
| Event | Question it answers | Mechanism |
|-------|--------------------|-----------|
| **Deploy** | Is the new version running in the environment? | Pipeline, rollout tooling |
| **Release** | Do users see the new behavior? | Feature flags, traffic routing, store review approval |
Once these are separated, an incomplete or risky change can sit in production dormant (**dark launch**), be shown to a tiny cohort, or be turned off in seconds without any redeploy. This separation is the escape hatch that makes release trains, freezes, and calendar schedules compatible with continuous deployment (see [release-process-models.md](./release-process-models.md) and [feature-flag-lifecycle.md](./feature-flag-lifecycle.md)). It is also the answer to the oldest release dilemma — "the feature is on the train but not ready" — because readiness is now a property of *exposure*, not of *presence*.
## Canary Releases
A **canary release** deploys a new version to a small, time-limited subset of traffic ("canary") while a "control" population stays on the old version, evaluates the canary against the control, and only then proceeds to full rollout. The Google SRE Workbook (ch. 16) is the canonical treatment; its core principles:
- **Canarying conserves the error budget.** Impact scales with exposed traffic: a 5% canary running at a 20% error rate costs roughly 1% of overall error budget. The budget cost of a defective rollout is proportional to the fraction of traffic exposed to the defect — which is precisely why you expose a small fraction first, and why canary analysis can afford to be aggressive: rolling back a 5% canary costs the budget almost nothing.
- **Run one canary at a time.** Concurrent canaries confound attribution — you cannot tell which change caused which effect.
- **Size and duration must be representative.** The canary needs enough traffic volume for statistical significance, and it should span peak-load and time-of-day variation, not just a quiet hour. A canary that runs only off-peak is blind to the failures that matter.
- **Prefer SLI-derived metrics** (HTTP status codes, latency — user-perceivable signals) over noisy resource metrics (CPU, memory), which tend to be ignored or disabled by operators.
- **Compare canary vs. control, never before/after.** Time-based comparison is confounded — the deploy coincides with other events. Concurrent A/B comparison of the canary population against a control population isolates the change.
- **Metrics must be attributable** — isolatable from shared failure domains so a database outage elsewhere does not look like a canary failure.
- **Metric aggregation windows must be ≤ the canary duration**, or the canary ends before the signal resolves.
**Exponential ramp:** Google's production rollouts expand the canary in exponential steps (e.g., 1% → 10% → 50% → 100%, or cluster-by-cluster expansion) with verification between steps. The Rapid rollout system evaluates each step against the service's error budget before proceeding — a step that burns budget pauses the ramp and rolls back rather than continuing blind. This "measure, then expand" loop is the difference between progressive delivery and merely slow delivery.
### Canary analysis in practice
A workable canary procedure, synthesized from the SRE Workbook:
1. **Select SLIs** — a stack-ranked handful of user-perceivable signals (error rate, latency percentiles); no more than ~a dozen, and avoid noisy resource metrics.
2. **Size the canary** — enough traffic for the SLI delta you care about to be statistically detectable within the observation window; account for peak and off-peak variation.
3. **Guarantee attribution** — isolate shared failure domains; label metrics by release version so canary and control are separable.
4. **Run canary vs. control concurrently** — never before/after; ensure aggregation windows are ≤ the canary duration.
5. **Evaluate** — divergence beyond threshold → pause and roll back (or page); healthy → proceed.
6. **Ramp exponentially** — 1% → 10% → 50% → 100% (or cluster-by-cluster), re-evaluating after each step.
7. **Record the verdict** — the canary decision becomes part of the release's audit trail: which SLIs, what delta, what decision.
This is the loop that makes progressive delivery a *safety mechanism* rather than a scheduling preference. See [rollback-and-recovery.md](./rollback-and-recovery.md) for the automatic-rollback side of the loop.
## Blue/Green Deployments
**Blue/green** maintains two full environments — the current (blue) and the candidate (green). Deployment runs green in parallel; release is a router change that cuts traffic from blue to green; rollback is the trivial reversal of that router change. The cost is roughly **2× resources**, and unless both environments run concurrently with split traffic, blue/green is effectively a before/after canary — with the same time-confound risk.
| | Canary | Blue/green |
|---|--------|------------|
| Environments | One, mixed versions | Two full environments |
| Rollback speed | Traffic shift back | Router reversal (seconds) |
| Resource cost | ~1x + canary capacity | ~2× |
| Best for | Continuous delivery, most services | Big-bang migrations, stateful cutovers needing instant revert |
Blue/green shines where instant, whole-stack revert matters more than cost — e.g., a database cutover or a compliance-critical platform where the alternative (re-deploying the old stack) is slower than flipping the router back. Its weakness is the 2× resource bill and the temptation to treat it as a substitute for measurement: the switch is fast, but you still need the same SLI evaluation to know *whether* to switch.
## Ring and Cohort Rollouts
**Ring deployments** roll out to ordered, concentric populations: internal/dogfood first, then early adopters, then broad. **Microsoft's Insider model** is the canonical example — Windows Insider channels (Dev / Beta / Release Preview) and M365 Targeted Release let Microsoft expose builds to progressively larger, more tolerant cohorts before general availability, with enterprise "deployment rings" used for Windows updates. **iOS phased release** is effectively a two-ring model: a small percentage of users first, pausable before broad exposure.
| Ring | Population | Tolerance | Catches |
|------|-----------|-----------|---------|
| Internal/dogfood | Employees, insiders | High — they expect bugs | Integration, internal tooling, gross breakage |
| Early adopters | Beta/preview users | Medium | Real-world hardware/OS/browser diversity |
| Broad | General availability | Low | Long-tail compatibility, scale effects |
The **employee-first cohort** pattern — Meta/Facebook, Google, and LinkedIn have historically rolled changes to their own employees first — is a ring with the most forgiving population. It catches internal-tooling and culture problems early but cannot detect problems only external users hit (browser diversity, third-party integrations, hostile traffic), so employee rings are a complement to, not a substitute for, canary analysis. Rings organize *who* sees the change; canaries measure *how it behaves*.
### Rings vs canaries
| Aspect | Rings | Canaries |
|--------|-------|----------|
| Organizes | Populations, ordered by tolerance | Traffic samples vs. control |
| Cadence | Days–weeks per ring | Minutes–hours per step |
| Question | "Is this population ready for the next ring?" | "Is this version healthy?" |
| Best combined as | The rollout *shape* | The *gate* between steps |
The two compose: roll out in rings, and run a canary analysis inside each ring before expanding to the next. **Choosing ring membership** is itself a decision: order populations from most to least tolerant (internal → beta → new users → existing users → highest-value/least-tolerant last), define explicit exit criteria per ring, and treat "the ring passed" as a metric-backed claim, not a schedule event.
## Percentage Rollouts
A **percentage rollout** incrementally raises the exposed fraction (1% → 5% → 25% → 100%), typically gated on metric health between steps. It is the simplest progressive-delivery mechanism and pairs naturally with feature flags (percentage of users) or traffic routing (percentage of requests). The gate between steps is what makes it progressive rather than just "slow": each step waits for confirmation that the previous step is healthy before expanding. A percentage rollout with no metric gate and a fixed timer is neither fish nor fowl — it does not protect you, it just delays you.
**Sizing the steps:** steps should be small where risk is high and larger where confidence is high — the common shape is roughly logarithmic (1 → 5 → 25 → 100). The step increment is a bet: each step's blast radius should be smaller than the error budget it is allowed to consume, so even a bad step costs little (see [readiness-and-quality-gates.md](./readiness-and-quality-gates.md) for the budget connection).
## Traffic Shadowing
**Traffic shadowing (teeing)** copies live traffic to the candidate version while discarding its responses, so the candidate is exercised with real production load and inputs without any user impact. It is representative and safe for *stateless* systems, but the SRE Workbook warns it is **unreliable for stateful systems** — shared caches and databases can be skewed by the shadow copy, producing results that do not reflect real behavior. Synthetic load has the complementary problem: it "doesn't provide good state coverage" and can be outright dangerous on a billing system. Use shadowing as a rehearsal step, not as a final gate.
## Metric-Gated Progression and Automatic Rollback
Progressive delivery without automated evaluation is just slow delivery. The SRE Workbook's prescription: gate progression on a **stack-ranked set of a few SLIs (no more than ~a dozen)**, comparing canary and control populations; if the canary metric diverges too far from control, **pause and roll back the deployment, or page a human**. The canary analysis must:
- compare populations (canary vs. control), never before/after;
- use intervals ≤ canary duration;
- be attributable to the change under test;
- map to user-perceivable SLIs.
When the gate is wired to an automatic rollback, the pipeline itself becomes the safety net: a bad canary is detected and reverted in minutes without a human noticing first. This is the operational heart of "rollbacks are normal" (see [rollback-and-recovery.md](./rollback-and-recovery.md)). The automatic trigger is sized by the error budget — because a small canary costs little budget, aggressive auto-rollback at canary tier is nearly free insurance.
## Dark Launches and Feature Flags
A **dark launch** deploys code that is inert until toggled: the code path exists in production but is unreachable by users. Combined with **feature flags**, it gives you the full control surface for progressive delivery — percentage of users, targeted cohorts, gradual enablement, and a sub-second kill switch. The flag is what makes "deploy anytime, release when ready" actually true: the deploy and the release are separated by a remote-config decision, not by another release cycle.
Flags are foundational enough that most teams buy them (LaunchDarkly, Flagsmith, Split/Harness, Unleash) rather than build them, and the ecosystem is standardizing around OpenFeature. But flags are also an operational liability when unmanaged: every flag is a decision surface, and flags left in place become permanent ambient state (see [feature-flag-lifecycle.md](./feature-flag-lifecycle.md) for the full lifecycle — including the Knight Capital-style costs of unmanaged toggles). The discipline: name flags for their purpose, assign ownership and expiry, test both states, and remove flags once the rollout completes.
## A/B Testing in Production
**A/B testing** compares business-metric outcomes between exposed and unexposed cohorts — a statistically rigorous experiment layered on top of progressive delivery. The distinction from canarying matters:
| | Canary | A/B test |
|---|--------|----------|
| Question | Is the new version *reliable*? | Does the change *improve outcomes*? |
| Metrics | SLIs (error rate, latency) | Business metrics (conversion, engagement, revenue) |
| Decision | Roll out further or roll back | Keep, revert, or iterate the feature |
| Duration | Minutes-hours | Days-weeks (statistical power) |
Booking.com's experimentation platform automatically detects and reverts a bad change in **~1 second** (Lukas Vermeer) — the canonical example of automated, metric-driven rollback at scale. Walmart's **"Test to Launch"** makes progressive, experiment-gated release the default path to production. Netflix pairs automated canary analysis with experimentation across its fleet. The pattern: pre-register the metric and threshold, let the platform decide, and keep the human out of the per-step judgment call. A/B testing also requires the discipline that canarying does — attributable cohorts, adequate sample size, pre-registered metrics — plus patience: business metrics resolve more slowly than SLIs.
## Field Examples
| Company | Practice | Takeaway |
|---------|----------|----------|
| **Google** | Rapid/Sisyphus rollout system: exponential cluster expansion, canary vs. control evaluation, MPM package labels `dev`/`canary`/`production` | Canary analysis at platform scale, error-budget-conserving ramps |
| **Booking.com** | A/B/experimentation platform with automatic detect-and-revert of a bad change in ~1 second (Lukas Vermeer) | Automated experiment evaluation can revert faster than any human process |
| **Walmart** | "Test to Launch" — progressive exposure tied to automated verification | Retail-scale progressive rollout as the default release path |
| **Netflix** | Automated canary analysis (Kayenta), chaos engineering alongside progressive rollout | Machine-judged canary gates on large fleets |
| **Meta / LinkedIn** | Employee-first cohorts, flag-gated (Gatekeeper-style) rollout | Ring populations ordered by tolerance |
The pattern across all of them: **expose a little, measure against a control, expand on health, revert automatically on divergence.** The specific mechanism matters less than the loop.
## When Not to Use Progressive Delivery
Progressive delivery is not free, and it is not always applicable:
- **When you cannot measure.** A canary gate without SLIs is fiction. If the service has no observability worth the name, fix that first — progressive delivery *requires* a signal to gate on.
- **When cohort isolation is impossible.** If canary and control share state so tightly that attribution is meaningless (e.g., a single shared cache dominating behavior), canary analysis will produce noise. Flags may still work; canary analysis may not.
- **When regulation requires full validation before any exposure.** Certifiable systems (aviation, medical devices) may not legally expose unvalidated builds to any population, however small. There, "validation" happens before deployment, and the rollout shape is dictated by the regulator, not the SLI.
- **When user consent and ethics demand guardrails.** Experiments on users (A/B testing) carry consent and fairness obligations: pre-registered metrics, opt-outs, and limits on who can be assigned to a worse experience.
- **When the change cannot fail.** A documentation-only change or a no-op refactor with zero user-visible risk does not need a canary. Ceremony should track risk; adding ceremony where risk is absent just taxes velocity.
## Choosing a Strategy
The decision axes are **speed of rollout, safety (blast radius of a bad change), rollback time, and complexity/cost**:
| Strategy | Speed | Blast radius at step 1 | Rollback time | Complexity/cost | Default when ... |
|----------|-------|------------------------|---------------|-----------------|------------------|
| Canary (metric-gated) | Minutes-hours | Small fraction of traffic | Minutes | Medium | General-purpose stateless services |
| Blue/green | Seconds (switch) | Full stack at switch | Seconds | 2× resources | Instant whole-stack revert matters |
| Ring/cohort | Days-weeks | Bounded population | Medium | Medium | User-base segmentation (devices, regions, customers) |
| Feature flag | Real-time | Zero (dormant) | Sub-second | Low | Change can be toggled independently of deploy |
| Shadowing | Pre-release | Zero (discarded) | n/a | Medium | Rehearsal of stateless services |
| Percentage | Minutes-hours | Small % of users | Minutes | Low | Simplest option; pairs with flags or routing |
A canary with metric gates is the general-purpose default for stateless services; blue/green wins where instant revert matters more than cost; rings fit user-base segmentation; flags fit anything where the change can be switched on/off independently of deploy; shadowing fits pre-release rehearsal of stateless services. See the [deployment-strategy-matrix](../assets/deployment-strategy-matrix.md) for a side-by-side comparison across these axes.
## A Maturity Ladder
Progressive delivery is a capability you build incrementally, not a binary state. A useful ladder:
1. **Big-bang deploy, human verification** — full rollout; humans notice problems after.
2. **Controlled switch** — blue/green with a manual router flip; rollback is trivial but judgment is human.
3. **Progressive, gated** — canary/rings/percentages with metric gates between steps; humans review the gates.
4. **Automated** — automatic rollback wired to canary analysis; the pipeline enforces the loop without a human in the per-step path.
5. **Self-tuning** — experimentation platform (A/B, automated decisioning); the system decides both reliability and business outcomes.
Most teams should climb deliberately: each rung requires the instrumentation of the previous one (SLIs before gates, gates before automation). Jumping straight to automation without trustworthy SLIs just automates blindness.
## Gotchas
> **Gotcha — canary sized for convenience, not significance:** A 1% canary on a low-traffic service may need days to accumulate signal. Size the canary so the SLI delta you care about is statistically detectable within the observation window, or you will be making decisions on noise.
> **Gotcha — before/after comparisons:** Deploys never happen in a vacuum. Comparing the week before the deploy to the week after attributes unrelated events (marketing pushes, dependency outages) to your change. Always compare concurrent canary vs. control where possible.
> **Gotcha — metrics that cannot be attributed:** If the canary and control share a cache, a database, or a downstream dependency, a shared failure looks like a canary failure. Isolate failure domains before relying on the signal.
> **Gotcha — aggregation window longer than the canary:** The canary ends before the metric resolves, so decisions are made blind. Metric resolution must be faster than rollout steps.
> **Gotcha — shadowing stateful systems:** Teeing traffic into a candidate that writes to shared state corrupts the experiment and can corrupt production data. Restrict shadowing to stateless rehearsal.
> **Gotcha — percentage rollouts without gates:** Raising exposure on a timer rather than on metric health is slow rollout, not progressive delivery. It buys delay without safety.
> **Gotcha — flags as permanent crutches:** Feature flags are the fastest rollback lever, but flags that are never removed become an unmaintained decision surface with its own failure modes. Every flag needs an owner, an expiry, and a cleanup plan (see [feature-flag-lifecycle.md](./feature-flag-lifecycle.md)).
> **Gotcha — canary only at happy hour:** A canary that runs only during quiet hours will miss the load-dependent failures that matter. If you cannot canary across peak traffic, at least size the exposure so the off-peak canary is *statistically meaningful* — and say explicitly which failure modes you are not covering.
## Sources and Further Reading
- [Google SRE Workbook — Canarying Releases (ch. 16)](https://sre.google/workbook/canarying-releases/)
- [Google SRE Book — Release Engineering (ch. 8)](https://sre.google/sre-book/release-engineering/)
- [Google SRE Book — Reliable Product Launches (ch. 27)](https://sre.google/sre-book/reliable-product-launches/)
- [Microsoft Tech Community — Tactical Considerations for Creating Windows Deployment Rings](https://techcommunity.microsoft.com/blog/windows-itpro-blog/tactical-considerations-for-creating-windows-deployment-rings/)
- [Microsoft Windows Insider Blog — Introducing Windows Insider Channels](https://blogs.windows.com/windows-insider/2020/06/15/introducing-windows-insider-channels/)
- [Flagsmith — Progressive Delivery with Feature Flags](https://www.flagsmith.com/blog/progressive-delivery)
- [Martin Fowler — Parallel Change](https://martinfowler.com/bliki/ParallelChange.html)
references/readiness-and-quality-gates.md
# Release Readiness and Quality Gates
A release is not "done" when the code is written; it is done when it is **deployed to production and validated there**. Release readiness is the disciplined answer to "what has to be true before we ship?" — organized so that every requirement has a named owner, an evidence trail, and a gate that actually blocks. The point of a readiness model is not bureaucracy; it is making the implicit explicit before a launch, when a problem costs 100× what it would cost to fix in design.
## Release Readiness as Four Dimensions
Readiness decomposes into four dimensions. Every item in every dimension needs a **named owner** — a person, never a team — plus an evidence link that an auditor (or a skeptical engineer) can check.
| Dimension | Typical items | Named owner example |
|-----------|---------------|---------------------|
| **Functional** | Acceptance tests pass on the *exact* release build; business owner UAT accepted; known defects risk-rated and explicitly accepted | Business/Product owner |
| **Non-functional** | Performance at peak + margin; security review (SAST, DAST, dependency scan); accessibility; resilience/graceful degradation | Engineering lead |
| **Operational** | Monitoring and alerting live *before* go-live; runbooks written and read by on-call; deployment rehearsed; rollback path tested (not assumed); on-call coverage window (24–72h) | Operations/SRE lead |
| **Governance** | Approval recorded (if required); audit artifacts linked (ticket → PR → CI → approval → deploy log → verification); segregation of duties; end-to-end traceability | Release/governance owner |
### A concrete checklist item looks like this
| Item | Owner | Evidence | Gate |
|------|-------|----------|------|
| Acceptance tests pass on exact RC | Product owner | CI run ID + artifact digest | Blocking |
| Performance at peak + margin | Engineering lead | Benchmark report on the RC | Blocking |
| Monitoring/alerting live before go-live | Ops lead | Alert dashboard URL + test page fired | Blocking |
| Rollback path tested (not assumed) | Ops lead | Rehearsal log / game-day record | Blocking |
| Approval recorded + traceability chain | Release owner | Change record with links | Blocking |
| Known defect #42 accepted | Product owner | Risk note + sign-off | Non-blocking |
The format matters as much as the content: each row names a person, points at evidence, and states whether it blocks. Rows without an owner or evidence are how "the checklist passed but the release failed" happens.
> **Gotcha — checklist without owners:** A readiness checklist with checkboxes but no names is theater: "done" means "somebody, somewhere, at some point." Every line item should name the individual who attests to it and link the evidence that backs the attestation.
### The Google Launch Coordination ancestry
The four dimensions map to the launch-coordination structure Google SRE formalizes (SRE Book ch. 27 and the Launch Checklist), which is worth keeping in mind for large launches:
| Launch Checklist area | Maps to dimension |
|-----------------------|-------------------|
| Architecture; failure modes; client behavior | Functional |
| Volume/capacity/performance; security; growth | Non-functional |
| Monitoring; automation & manual tasks; system reliability & failover; rollout plan | Operational |
| External dependencies; schedule; launch processes; sign-offs | Governance |
Google's model adds the launch-coordination view: a named Launch Coordinator, a pre-mortem-style risk review, and kill switches designed in from the start (see [progressive-delivery.md](./progressive-delivery.md)). You can keep the SRE framing for big launches and the four-dimension model for routine releases; the owner-and-evidence discipline is what survives both.
## Release Candidates
A **release candidate (RC)** is a promoted build explicitly labeled as a candidate for release. The label is not cosmetic: it marks the exact artifact whose tests, review, and verification evidence apply, and it is the thing a rollback returns to.
Google's model is instructive: packages are labeled `dev`, `canary`, and `production`, with movable labels pointing at immutable, content-hashed, signed versions. Google also **re-runs unit tests on the release branch** to create an audit trail, because a cherry-picked release branch may contain commits that never existed on mainline — testing only `main` would leave the shipped artifact's exact content unverified. For any release cut from a branch (see [release-process-models.md](./release-process-models.md)), re-verify the branch content itself.
> **Gotcha — testing the wrong build:** Readiness evidence that does not reference the exact artifact digest (SHA, version) being released is worthless. A release plan that says "tests pass" without pinning the build has no way to prove the tested build is the shipped build.
## Staging Parity and Smoke Tests
**Staging parity** is the pursuit of making staging and pre-prod mirror production as closely as feasible — same artifact, same config schema, same deploy mechanism, realistic data and scale. Parity is never perfect (the SRE Workbook states plainly that test environments are not 100% identical to production — which is *why* canarying in real traffic exists, see [progressive-delivery.md](./progressive-delivery.md)). The discipline is to name and manage the gaps: document the capacity delta, refresh data from production with compliance guards, and record dependency versions per environment. Unnamed parity gaps are a recurring source of prod-only defects; named gaps can be weighed, tested around, and closed.
**Smoke tests** are lightweight post-deploy verification that critical paths work — black-box probes (e.g., "the service answers at this URL," "the checkout flow completes") that complement canary metrics. Their value is isolating the canary signal from odd user behavior: a black-box probe exercises a known path deterministically, so a smoke-test failure is a deployment failure, not an artifact of user mix. Smoke tests should be fast, fail loudly, and run in every environment the artifact promotes through — including production immediately after deploy.
## Observability and Error-Budget Gates
The strongest release gate is the production system's own health. Gate promotion on **SLIs** (error rate, latency) and on **error-budget health** — not on test-passing alone, because tests encode your assumptions and production encodes reality.
Google's example **error-budget policy** (SRE Workbook Appendix B) is the template:
- If a service exceeds its **4-week error budget**, **halt all non-P0/security releases** for that service until it is back within SLO.
- A single incident that consumes **>20% of the budget** triggers a postmortem with a P0 action item.
- The policy is justified partly because **~70% of outages are change-induced** — the thing most likely to break a service is a release, so gating releases on reliability health is gating on the actual risk.
Error-budget gates integrate naturally with progressive delivery: the canary's allowed impact is sized by the budget (a 5% canary at 20% error costs ~1% overall), and exceeding the budget halts further promotion. The gate is only as good as the SLI selection behind it — stack-rank a small set of user-perceivable SLIs and feed them into both the canary analysis and the release gate (see [progressive-delivery.md](./progressive-delivery.md)). The error budget is also the objective arbiter between "we need to ship" and "the service is fragile": shipping during a budget burn is spending money you do not have.
## Go/No-Go
A **go/no-go** is a decision point: is this release, at this moment, cleared to proceed? Historically a meeting; in mature CD organizations it is increasingly automated or asynchronous.
**Structure when it is a meeting:**
1. Each dimension owner confirms their section — 30–60 seconds per owner, named individuals, not "the team."
2. The decision is recorded as **GO / NO-GO / GO-WITH-CONDITIONS**, with a timestamp.
3. Conditions have owners and deadlines; a GO-WITH-CONDITIONS is not a free pass — it is a tracked obligation.
**What survives in modern practice:** The ritual shrinks to an exception path. Routine deployments are covered by automated gates — dashboards, policy-as-code, error-budget checks, canary metrics — so the meeting becomes a review of *exceptions*, not a per-deploy ceremony. Asynchronous sign-off (recorded in the change record with evidence links) replaces the room for most releases. The questions that survive are the ones automation cannot answer: *Is the business risk of this launch acceptable? What known defects are we explicitly accepting? What is the customer-communication plan if it fails?*
**When a go/no-go meeting is still warranted:**
- **Compliance-driven releases** where auditors expect a documented approval decision per release (SOC 2, SOX, PCI; see [change-governance-and-compliance.md](./change-governance-and-compliance.md)).
- **Release trains** where a cross-team launch (or rollback) is coordinated on a calendar, and one team's no-go affects the train (see [release-process-models.md](./release-process-models.md)).
- **High-risk, high-blast-radius launches** (billing changes, data migrations, security boundaries) where the cost of being wrong justifies a human checkpoint.
> **Gotcha — go/no-go as rubber stamp:** A meeting that meets, approves, and records nothing is worse than no meeting: it manufactures the appearance of control while adding latency. If the meeting's decisions are never revisited and its conditions are never tracked, automate it away.
## Post-Release Monitoring Window
Readiness does not end at the deploy; it extends through a **post-release monitoring window** (commonly 24–72 hours) during which the release is treated as unproven:
- On-call is explicitly aware a release just shipped and knows which SLIs to watch.
- The rollback runbook is open and rehearsed, not rediscovered (see [rollback-and-recovery.md](./rollback-and-recovery.md)).
- Monitoring compares post-release SLIs and business metrics against the pre-release baseline — the same metrics the canary covered, now at full exposure.
- A known-issue list tracks anything observed in the window, with owners and follow-up.
The window closes when the release has demonstrably held: no SLO violation, no metric regression, no incident. Until then, the release is still *in flight*, and anything discovered in the window is handled by the release's own gates — escalate, roll back, or hotfix — rather than as an unrelated incident. This is the operational completion of "definition of done": a release is done when the window closes, not when the deploy button was pressed.
## Readiness Evidence as Data
The readiness checklist earns its keep when it is **data, not prose**: a structured record (e.g., JSON) where every item carries an ID, dimension, named owner, evidence link, status, and whether it blocks. Machine-readable readiness buys three things:
- **Completeness is checkable.** CI can fail a promotion when a blocking item has no owner or no evidence — the checklist enforces itself instead of relying on a human to read it.
- **Evidence is attached, not remembered.** Each item links its proof (CI run ID, scan report, rehearsal log), which is exactly what an auditor or a skeptical reviewer wants (see [change-governance-and-compliance.md](./change-governance-and-compliance.md)).
- **Automation can consume it.** A data-driven checklist feeds the automated go/no-go (policy-as-code gates, error-budget checks) described below — the same criteria, evaluated by machine for routine releases and by humans for exceptions.
The template lives in the skill's `templates/` directory; the discipline is that the *record* — not the meeting — is the source of truth for "is this release ready?"
An example record for one item:
```json
{
"id": "READ-004",
"dimension": "operational",
"item": "Rollback path tested on exact RC",
"owner": "ops-lead@example.com",
"evidence": "rehearsal://2026-08-01/payments-rc3",
"status": "pass",
"blocks": true
}
```
The gate consumes `status` and `blocks`: a blocking item without `pass` fails the promotion, regardless of how the meeting went.
## Automating the Go/No-Go
In mature CD organizations the go/no-go becomes **policy-as-code**: the criteria that humans used to recite are encoded as automated checks. Dashboards aggregate readiness evidence; pipeline gates evaluate it (see [cd-and-pipeline-stages.md](./cd-and-pipeline-stages.md)); error-budget checks produce an automatic no-go when the budget is burned (see [progressive-delivery.md](./progressive-delivery.md)). The human decision is then limited to what automation cannot answer — business risk acceptance, known-defect sign-off, customer communication — and is recorded asynchronously in the change record with evidence links.
The migration path is deliberate: start with the meeting, then *automate one criterion at a time*, and retire the meeting's agenda items as the checks absorb them. A go/no-go meeting whose entire agenda has been automated is a meeting in search of a purpose — dissolve it and keep the exception path (compliance, trains, high-risk launches) where a human decision genuinely adds value.
## Definition of Done
A release's **definition of done** should be stated explicitly and end in production:
| Stage | Done means |
|-------|-----------|
| Code complete | Merged to trunk, reviewed |
| Build verified | Hermetic build of the exact RC passes unit/integration/security gates |
| Pre-production | Staging deploy green; smoke tests pass; performance evidence recorded |
| Production deployed | Artifact promoted and running in prod |
| Production validated | Canary/metric gates pass; SLIs healthy for the observation window; on-call aware |
Anything less than "production validated" leaves the release in a liminal state — deployed but unverified — which is where post-release incidents start. Note that this definition of done deliberately does not say "QA complete" or "approval granted"; those are inputs, not outcomes. The outcome is a validated, observable production state.
## Readiness Scale: Routine vs Coordinated
Apply readiness in two regimes, and do not confuse them:
| Regime | Applies to | Shape |
|--------|-----------|-------|
| **Routine** | Every deploy in a CD org | Lightweight: automated gates (tests, security, smoke, canary SLIs) + evidence links; no meeting |
| **Coordinated** | Launches, release trains, compliance-gated releases, high-risk changes | Full: four-dimension checklist with named owners, pre-mortem, go/no-go, communication plan, post-release monitoring window |
The failure modes are symmetric: applying *coordinated* ceremony to every routine deploy regresses a CD org to monthly releases; applying *routine* lightness to a coordinated launch (a billing change, a data migration, a regulated release) ships without the checks that justify the risk. The disciplined practice is to define, per release type, which regime applies — and to let risk (blast radius, regulatory exposure, cross-team coordination) drive the classification, not habit.
## Risk-Rated Defect Acceptance
A mature readiness process does not require zero defects — it requires that **known defects be explicit, risk-rated, and accepted by a named owner**. The distinction is between two very different states:
- **Known and accepted:** defect #42 (an obscure settings-page edge case) is logged, risk-rated (probability × impact), assigned an owner, and explicitly signed off as acceptable for this release — visible to everyone.
- **Known and unexamined:** the same defect exists but was never rated or signed off — it ships silently and is discovered by a customer, an auditor, or an incident.
Risk-rated acceptance also covers the *deferral* case: a feature that is ready for deployment but not ready for release can ride the train dormant behind a flag (see [progressive-delivery.md](./progressive-delivery.md)) — that is a different decision from shipping a known defect to everyone. The readiness record should state which defects are accepted, which are deferred behind flags, and which blocked the release.
## The Pre-Mortem
For significant launches, add a **pre-mortem**: before the release, imagine it has failed in six months, and work backward to the plausible causes. Google's launch coordination institutionalized this kind of adversarial review; it complements the checklist by surfacing risks the checklist does not list — the failure mode nobody wrote down. The output is a short list of named risks with mitigations (and often a kill switch, see [progressive-delivery.md](./progressive-delivery.md)). A pre-mortem is cheap, runs in an hour, and consistently finds at least one risk the team had not articulated — which is precisely its point.
## Evidence Retention and the Audit Chain
Governance readiness is only as durable as its evidence chain. The chain is: **ticket → PR review → CI runs → approval → deploy log → verification** — each link timestamped and linked, so an auditor (or a postmortem) can walk a change end to end. SOC 2 Type II auditors sample roughly 25–50 changes and expect exactly this traceability; retention runs 1–7 years depending on regime (see [change-governance-and-compliance.md](./change-governance-and-compliance.md)).
The chain breaks in predictable ways — deleted branches, missing CI logs, manual deploys that leave no record, approvals recorded without evidence. Each break is a readiness defect: a release whose chain cannot be walked is a release whose governance was never demonstrated. Make the pipeline emit the links (see [cd-and-pipeline-stages.md](./cd-and-pipeline-stages.md)) and treat a broken chain as a blocking readiness item, not a bookkeeping annoyance.
**The go/no-go decision record** is itself evidence: decision (GO / NO-GO / GO-WITH-CONDITIONS), timestamp, attendees/approvers, each condition with owner and deadline, and a review date. A decision record that cannot be produced after the fact is a decision that never happened in audit terms.
## Gotchas
> **Gotcha — readiness as a point-in-time artifact:** Readiness rots. Evidence gathered two weeks before launch (performance runs, security scans) describes a build that may no longer be the candidate. Re-run evidence on the exact RC, and re-verify after any branch change.
> **Gotcha — unowned governance evidence:** In regulated environments, "the approval happened" without a timestamped, linked record is not evidence. The change record must chain ticket → PR review → CI runs → approval → deploy log → verification (see [change-governance-and-compliance.md](./change-governance-and-compliance.md)).
> **Gotcha — error-budget gates without teeth:** An error-budget policy that nobody enforces (halts ignored, overrides routine) teaches the org that gates are optional. If you cannot afford to halt releases during an error-budget burn, do not claim you have a policy — you have a wish.
> **Gotcha — smoke tests that only check "site is up":** A probe asserting HTTP 200 on the homepage protects nothing. Smoke the critical user journeys (login, search, checkout, payment), and fail the deploy on probe failure, not just alert.
> **Gotcha — readiness as a single release milestone:** In continuous delivery the "release" is a stream, not an event. Run lightweight readiness checks per deploy and heavyweight checks only for coordinated or compliance-driven launches; a heavyweight ritual on every deploy is how CD orgs regress to monthly releases.
## Sources and Further Reading
- [Google SRE Book — Reliable Product Launches (ch. 27)](https://sre.google/sre-book/reliable-product-launches/)
- [Google SRE Book — Launch Coordination Checklist (Appendix E)](https://sre.google/sre-book/launch-checklist/)
- [Google SRE Workbook — Error Budget Policy (Appendix B)](https://sre.google/workbook/error-budget-policy/)
- [Google SRE Workbook — Canarying Releases (ch. 16)](https://sre.google/workbook/canarying-releases/)
- [RNVATE — Release Readiness Checklist (four-dimension model)](https://rnvate.com/insights/release-readiness-checklist.html)
- [DORA — The DORA Metrics guide](https://dora.dev/guides/dora-metrics/)
references/release-operations-and-triage.md
# Release Operations and Triage
How large projects actually run releases: the ceremony mechanics of four well-documented release trains (Chromium, Firefox, GitLab, Ubuntu), the shared patterns behind them, coordination roles, pipeline debugging, release-infrastructure reliability, environment parity, and the state of AI/agent-assisted release automation in 2025–2026.
## Release Train Ceremony Mechanics
### Chromium: 4-Week Milestones with Weekly Security Refreshes
Chrome ships a new milestone to stable **every 4 weeks**: 4 weeks of development on `main` (starting at the previous milestone's branch point), a **branch cut**, then 4 weeks of stabilization on the milestone branch, then staged rollout to stable. Key mechanics:
- **Channels:** Canary (daily, most unstable) → Dev → Beta → Stable, plus **Extended Stable** — every *other* milestone maintained for 4 extra weeks with backported security fixes (8-week cadence, Windows/Mac enterprise only, biweekly refreshes).
- **Weekly stable "refreshes"** carry security fixes forward to keep the *patch gap* short.
- **Staged rollout:** a release generally reaches all users within 1–2 weeks unless major issues arise; staged rollouts compare two identical builds that differ only in build number for statistical signal.
- **Cycle checkpoints:** **Branch Point** (features must be code-complete, strings landed, beta blockers addressed; incomplete features are punted) → **Beta Promotion** (4 weeks in beta with weekly builds) → **Early Stable Cut** (early release candidate to a small % of stable users; all stable blockers fixed) → **Stable Cut/Promotion** → weekly **Stable Refresh**.
- **Dates flex for coverage:** branch-point dates are fixed but adjusted to avoid shipping around major holidays so coverage is maintained.
- **Merge (cherry-pick) governance:** all code lands on trunk; every merge to a release branch is gated because it "introduces risk and costs time." **Release managers (and security delegates) review all merges**, with criteria that tighten as the stable date approaches: beta phase (Finch-gated fixes, new regressions, release blockers, security issues, emergency string changes) → stable phase (urgent regressions, release blockers, medium+ security) → extended phase (medium+ security only). Automation (Blintz) does first-pass triage and may auto-approve; release managers answer within 2 business days; missed merges of release-blocking fixes are flagged.
### Firefox: 2-Week Trains with Flag-Driven Uplifts
- **Cadence:** the standard release interval is **two weeks** (may be lengthened around holidays). Nightly builds flow from `firefox-main` roughly every 12 hours; **every 2 weeks, main merges to `firefox-beta`**, after which the beta branch takes stabilization patches only.
- **Beta releases** ship ~3x/week for Desktop → about **5 betas per cycle**, unless emergency "chemspills" (unplanned betas) are needed. At cycle end, the **final build is QA-validated and tagged** into `firefox-release`.
- **Channels/repos:** `firefox-main` (Nightly), `firefox-beta` (+ Devedition), `firefox-release`, `firefox-esr` (ESR for enterprises).
- **Uplift process is flag-driven:** a developer sets `tracking-firefoxXX: ?` to nominate; Release Management sets `tracking-firefoxXX: +` if it should block the release; the patch is nominated with `approval-mozilla-beta/release: ?`; on `+`, **sheriffs or release managers land it** on the branch and ensure Treeherder is green. `relnote-firefox` nominates release-note changes; "nag emails" chase owners of tracked blockers.
- **Merge day:** at the end of the beta cycle, release management emails release-drivers requesting the main→release merge; anything needing uplift must be nominated before the RC build. Security bugs follow a separate security-approval process.
- **Mobile:** Android RC builds push to production at **5% rollout after QA signoff**, bumped to 25% on the official release date to match Desktop.
### GitLab: Monthly Self-Managed Releases Off Continuous Delivery
- **Two-part model:** a **monthly self-managed release** (`XX.YY.0`) plus **continuous delivery on GitLab.com**, where auto-deploy packages built from `master` deploy multiple times per day.
- **Release day:** the self-managed release ships on the **third Thursday** of each month (with a one-week delay if needed); patches land twice a month (scheduled) plus unplanned critical patches as needed; the stated priority of both processes is "GitLab availability & security."
- **Flow:** engineer merge → pipeline packages an **auto-deploy package** (multiple/day) → deployed to GitLab.com if no Production Change Locks or unhealthy environments → changes that succeed on GitLab.com become the **release candidate** for self-managed → RC runs automated QA in test environments → RC tagged and published. **All changes must deploy to GitLab.com before they are considered for a self-managed release** — dogfooding is the gate.
- **Patch ceremony:** e.g., an Early Merge Phase on Mondays where release managers deploy security fixes to GitLab.com; MRs labeled `~"security-target"` link to the security tracking issue.
### Ubuntu: 6-Month Cycle with a Graduated Freeze Ladder
- **Cadence:** strict time-based release every **6 months** (since 2004), LTS every ~2 years. The freeze ladder is a sequence of progressively tighter gates, with exceptions granted only by the Release Team:
1. **Debian Import Freeze** — automatic imports from Debian `unstable` stop; imports must be explicitly requested.
2. **Feature Freeze** — no new features, packages, or API/ABI changes; bug-fix-only uploads allowed if documented.
3. **UI Freeze** — default-app UI, artwork, and user-visible strings frozen (for docs/translation).
4. **Documentation String Freeze** and **Kernel Feature Freeze** — docs frozen for translation; kernel feature-complete.
5. **Hardware Enablement Freeze** → **Beta Freeze** — all uploads queued and subject to manual Release Team approval; after beta ships, the archive rolls back to Feature + UI freeze status.
6. **Kernel Freeze** → translation deadlines → **Final Freeze** — "extremely high-caution": only release-critical, security-critical, or exceptional fixes, confirmed by the Release Team; near release, uploads go to the `-proposed` pocket and the Release Team cherry-picks into `-release`.
7. **Release Candidate** — images built Monday of release week; ideally RC == final release.
- **Post-release:** the **SRU (Stable Release Update)** process governs fixes; each upload must reference at least one bug.
### Shared Mechanics Across All Four
| Lever | Pattern |
|-------|---------|
| **Cadence choice** | Time-based (fixed calendar) beats feature-based: Chromium 4w, Firefox 2w, GitLab monthly, Ubuntu 6m. Dates fixed; **scope flexes** |
| **Branch cut** | A cut from the development line creates a stabilization branch (Chromium milestone branch, Firefox `firefox-beta`, GitLab RC tag, Ubuntu archive freezes). New features stop at the cut |
| **Stabilization window** | Dedicated period where only bug-fix/stabilization patches land, gated by criteria that tighten over time (Chromium beta→stable→extended; Ubuntu graduated freezes; Firefox beta uplift rules) |
| **Backport approval** | Formal nomination + approval gate with **named approvers** (Chromium release managers + Blintz automation; Firefox tracking/approval flags + sheriffs; Ubuntu Release Team exceptions; GitLab `security-target` label + release managers). Criteria escalate with risk and branch age |
| **Readiness / go-no-go** | Staged rollout with explicit signoffs (Chromium Stable Cut, Firefox QA on RC, Ubuntu Final Freeze + RC, GitLab automated QA + Production Change Locks) |
| **Staged rollout** | Ship to a fraction first, monitor, expand (Chromium staged %, Firefox Android 5%→25%, GitLab.com before self-managed) |
| **Coverage awareness** | Dates adjusted for holidays and staffing coverage (Chromium explicit; Firefox "lengthened for holidays") |
## Coordination Roles and Ceremonies
- **Release Manager (RM) / DRI:** the operational single-threaded owner. GitLab runs RMs on a **rotation/schedule** with explicit escalation paths and public permissions; Chromium release managers review/approve/reject every branch merge within 2 business days; Firefox Release Management sets tracking flags and lands approved uplifts via sheriffs.
- **Sheriffs / release drivers:** Firefox sheriffs land approved patches and keep Treeherder green; a `release-drivers` channel and mailing lists coordinate across teams.
- **Delegates:** the security team acts as a merge-approval delegate for security fixes (Chromium).
- **Ceremonies:** milestone kickoff/planning → recurring status (GitLab's **weekly delivery-metrics review** — MTTP, deployment blockers, Deployment SLO, DORA metrics, auto-deploy dashboards; Firefox channel meetings and nag emails) → branch cut → stabilization with readiness tracking → **go/no-go** (cut/RC signoff) → staged promotion → refresh/patch cadence.
- **Communication surfaces:** Slack/Matrix channels (`#releases`, `#release-drivers`), dashboards (Chromium Dash, GitLab Grafana + DORA analytics, Firefox release calendar), and issue-tracker queries for blockers.
The recurring roles across all four projects:
| Role | What they do | Example |
|------|--------------|---------|
| **Release Manager (RM) / DRI** | Single-threaded owner of a given release; approves backports, drives go/no-go, owns escalation | GitLab rotating RMs; Chromium RMs approving every branch merge; Firefox Release Management |
| **Sheriff / release driver** | Lands approved patches, keeps the tree green, chases blockers | Firefox sheriffs; Chromium build sheriffs |
| **Security delegate** | Approves/lands security fixes on release branches under a faster path | Chromium security team as merge delegates |
| **QA sign-off** | Validates the RC before promotion | Firefox final-build QA; GitLab automated QA on RC |
| **Automation (first-pass triage)** | Screens merge requests against criteria, flags missed merges | Chromium Blintz (formerly Sheriffbot) |
**Ceremony lifecycle (the recurring rhythm):** kickoff/milestone planning → recurring status (metrics review, channel meetings) → **branch cut** (feature-complete gate) → stabilization with readiness tracking → **go/no-go** (cut/RC signoff) → staged promotion → refresh/patch cadence → next cycle kickoff.
## Debugging Release Pipelines
### Failure Taxonomy
| Failure mode | Symptom | First response |
|--------------|---------|----------------|
| **Flaky tests** | Intermittent pass/fail unrelated to the change — the #1 time sink | Quarantine, retry-with-backoff, root-cause (timing, shared state, resource contention) |
| **Cache poisoning / stale cache** | Corrupt or stale build/dependency cache produces failures or wrong artifacts | Invalidate and rebuild; pin cache keys |
| **Version / dependency drift** | Unpinned deps or toolchains resolve differently across runs ("works on my machine") | Pin versions; enforce lockfiles |
| **Secret rotation** | Expired/rotated tokens cause auth failures mid-pipeline | Scoped tokens, rotation runbooks, secret lifecycle management |
| **Runner exhaustion** | Saturated runner pool, pod timeouts, resource contention | Capacity planning, queue observability, autoscaling |
| **Registry/rate limits** | Docker Hub/registry pull limits, API timeouts | Proxy/mirror registries, retry policy, quota monitoring |
| **Environment mismatch / permissions** | Env vars, file permissions, OS differences between runner and prod | Enforce parity (see below); CI matrix vs. prod |
### Change-vs-Pipeline Triage
The core question for any pipeline failure: **is it the change or the pipeline?** Workflow:
1. **Reproduce/replay** the run; compare against a known-green baseline run.
2. **Read logs first** — find the *first* failure (earliest `level=error`), trace backward via correlation IDs / job IDs.
3. **Bisect** — `git bisect` for code; re-run individual jobs/stages to isolate the failing step.
4. **Correlate across systems** — match CI logs → container logs → deploy logs with shared correlation IDs.
5. **Classify transient vs. systemic** — roughly 60% of CI failures are transient (timeouts, rate limits, network, resource contention) → retry-with-backoff and record the rationale; systemic → fix.
6. **Watch intermittent signals** — warnings/retries/degraded performance preceding a failure often point to environment/config issues.
7. **Check external dependencies** — third-party API and cloud timeouts.
**Rerun vs. fix:** rerun only when the failure is transient/flaky and isolated; fix when systemic, reproducible, or tied to the change. Never blind-rerun — classify with observability first. The observability substrate: structured JSON logs (`timestamp/level/service/message/correlation_id`) shipped to Loki/Promtail or ELK; Prometheus metrics + Grafana dashboards; OpenTelemetry traces; and exemplars that jump from a metrics spike straight to the relevant log lines. Retention and rotation (logrotate, Loki retention, ES ILM) are part of the design, not an afterthought — you will need history when a release incident surfaces days later.
### Deploy-Time Failure Recovery
When a deployment fails in production, the recovery options in rough preference order:
1. **Revert / roll back** — simplest, *when a viable rollback target exists*. Disable the feature flag, remove the new version from the load-balancer pool, restore the previous artifact. "Wherever possible, reverting a change causing a customer incident should be the initial plan of attack."
2. **Failover to secondary/DR environment** — when there is no change in play (external cause) or rollback is onerous: promote the DB leader, reroute traffic.
3. **Fix forward** — last resort: customers stay impacted until you diagnose, remediate, and redeploy.
**Why rollback fails (and what to design around):** protocol changes clients cannot revert from; destructive DB schema changes; components that do not blue-green well (databases, message brokers). Identify non-rollback-able areas *in advance* and redesign them; keep the previous version retained until the next deploy so a half-complete rollout can be reverted. **Timebox fix-forward attempts** (e.g., 30 minutes) then roll back; document the decision path in runbooks so tribal knowledge becomes institutional steps.
**The emergency release path** deserves the same rigor as the normal path, not less: model a hotfix as a near-identical production pipeline that fetches the artifact earlier (skipping full promotion), kept paused with strict trigger controls and break-glass approvals. If your pipeline is fast enough, prefer committing the fix through the *whole* pipeline so it is fully tested — then do RCA on how the bad build reached production in the first place. The "shortcut" should be the exception with a post-implementation review attached, never the routine.
## Release Infrastructure Reliability and DR
The release toolchain — CI servers, artifact storage, registries, deploy tooling — **is critical infrastructure**. Treat it that way:
- **Artifact storage:** a checksum-deduplicated store (e.g., Artifactory's SHA1 filestore + metadata DB) requires backing up **both** the filestore and the metadata DB, or the filestore is "just a folder with files named after their checksum" — unidentifiable. Snapshot the DB *before* copying the filestore to avoid dangling references; use federated/replicated second sites and restore drills.
- **Signing keys and master keys are the crown jewels:** loss of the master key means loss of all encrypted secrets/passwords at recovery time; treat key escrow, rotation, and a recovery runbook as a top DR item. Same logic applies to artifact signing keys — losing them bricks future releases and undermines supply-chain trust.
- **Runner fleet:** capacity planning, autoscaling, multi-region, queue observability; treat runners as cattle.
- **DR fundamentals:** multi-region deployment, automated failover, IaC for reproducible recovery, defined RTO/RPO, and *tested* failover. DR is also the enabler for safe maintenance windows.
**Disaster scenarios and mitigations:**
| Scenario | Mitigation |
|----------|-----------|
| Registry/artifact corruption | Checksum-based dedup + federated/replicated second site + regular restore drills |
| Signing/master-key loss | Key escrow and backup; loss of the master key means loss of all encrypted secrets at recovery; rotation + recovery runbook |
| Build-cache loss | Cache is regenerable but expensive: pin cache keys, keep warm caches in multiple regions, budget cold-rebuild capacity |
| CI server / runner fleet outage | Fleet capacity planning, autoscaling, multi-region, queue observability; treat runners as cattle |
| Compliance/tamper challenge | Immutable/WORM artifact storage for tamper evidence and retention |
| Maintenance windows | Active-passive DR enables taking the registry/CI down for hardware maintenance safely |
## Config Drift and Environment Parity
Despite decades of best practices, teams still hit the **"repro gap"**: features work locally but break in staging/prod because environments are maintained separately and drift in service versions, configurations, and environment variables. Key failure drivers: the shared "staging queue" (manual hotfixes applied to staging that never flow back), **Docker being insufficient** (Compose handles local service relationships but not cloud routing/lifecycle), and **stale or stubbed data** — the leading cause of late-stage deployment failures.
Mitigations: generate every environment (local, preview, prod) from **one declarative, version-controlled manifest** so drift is structurally impossible; branch the whole environment on git branch; use byte-for-byte production clones into isolated preview environments with automated sanitization; manage secrets with scoped tokens and runtime injection (never store real secrets in non-prod configs); audit rollback data for environment-diff-caused failures; tear ephemeral preview environments down after merge.
## AI/Agent-Assisted Release Automation (2025–2026)
The CI/CD layer is where agentic adoption lags most: AI adoption among individual developers crossed 90% by early 2026, but only ~13% of organizations have AI across the full delivery lifecycle. The emerging paradigm is **CA/CD (Continuous Agentic/Continuous Deployment)**: agents observe pipeline state, reason about whether failures are transient vs. systemic and whether a deploy window is safe, act autonomously on low-risk decisions, and escalate high-risk ones — "risk-aware releases rather than pass/fail gates."
**What exists today:**
- **GitHub Agentic Workflows** (technical preview, 2026): automation written in plain Markdown instead of YAML, compiled to standard Actions running coding agents; handles issue triage, PR review, CI failure root-cause analysis, and repo maintenance. Security-first: read-only by default, sandboxed, network-isolated, SHA-pinned dependencies, sanitized writes. GitHub's **Copilot Coding Agent** opens PRs, and **CI checks do not run on agent-authored PRs until a human approves**.
- **MCP as the integration standard:** CircleCI ships a production MCP server exposing pipeline graphs, build history, and failure logs to agents; Dagger agents monitor pipelines, generate patches, and submit them **through the same review process as human code** with a "rationale diff"; Nx offers AI self-healing CI (analyze → propose fix → apply → re-run affected checks, with a decision trace).
- **Self-healing** is the most mature capability: failure classification (high reported F1 on flaky tests, runner pod timeouts, dependency install failures), transient-vs-systemic routing (~60% transient → auto retry), policy-gated fix generation, and outcome learning.
**Governance: tiered autonomy** is the dominant architecture — match agent authority to action risk:
| Tier | Actions | Authority |
|------|---------|-----------|
| **Low** | Retry transient failures, update docs, reorder tests | Fully autonomous |
| **Medium** | Revert a failing deploy, scale ahead of predicted load | Autonomous + logging + notify |
| **High** | Merge to main, modify security policy, infra changes | Human approval required |
| **Critical** | Architectural changes, prod data migrations | Formal review gate |
Companion controls: **immutable audit trails** (what/when/why/what-changed, for debugging and compliance), **policy-as-code** validation of agent decisions (quotas, security, blast radius), **confidence-gated autonomy** (low-confidence/high-impact → human), and **human checkpoints at high-blast-radius transitions** (merge to release branch, deploy to prod, access-control changes).
> **Gotcha — agents amplify whatever exists:** "High-velocity, high-confidence mistakes." Fragile pipelines get broken faster; thin test coverage ships untested code at higher velocity. The teams advancing fastest defined their autonomy tiers clearly, built observable audit trails, and expanded agent authority only as confidence grew. **Trust calibration, not tooling, is the bottleneck** — and governance must be built in from the start, not retrofitted after agents run autonomously. Vendor-reported numbers (e.g., ~94% automatic failure resolution, ROI figures) should be attributed, not treated as independently validated.
**Implication for release engineers:** the job shifts from *executing* releases to *defining autonomy tiers, reviewing agent rationale diffs, owning audit trails, and calibrating trust*. Routine toil (retrying flakes, triaging failures, drafting changelogs, dependency updates) is delegated; humans concentrate on go/no-go judgment, security, and high-blast-radius approvals. The human gate is deliberately preserved.
## Running a Release Train: The Operational Checklist
Synthesis of the four projects' mechanics into a reusable operating rhythm for any train-based release:
1. **Publish the calendar first.** Fixed dates for branch cut, RC, and stable; adjust for holidays/coverage; communicate in a shared channel.
2. **Set the feature-complete gate at the branch cut.** Code-complete, strings landed, blockers addressed; anything incomplete is punted, not carried.
3. **Publish merge criteria that tighten over time.** Permissive early (beta phase), restrictive as stable approaches; named approvers with an SLA; automation for first-pass triage.
4. **Run a dedicated stabilization window.** Only bug-fix/stabilization patches land; track readiness on a shared dashboard.
5. **Gate promotion with explicit signoffs.** RC validated by QA, staged rollout to a small cohort, monitor, expand.
6. **Keep a refresh/patch cadence** (weekly security refreshes, scheduled patches) so fixes don't wait for the next milestone.
7. **Review delivery metrics weekly** (deployment frequency, lead time, blockers) and use the review to improve tooling and process, not to blame.
## Gotchas
- **Cadence values move:** Chromium moved from 6-week to 4-week milestones; Firefox from 4-week to 2-week releases. Teach cadence as a *design choice* (fixed calendar, flexible scope) and cite current values as examples, not doctrine.
- **A branch cut is a feature-complete gate, not a suggestion:** every project above that enforces it (Chromium punts incomplete features; Ubuntu freezes; Firefox stops features at beta).
- **Merge criteria must tighten over time:** permissive early, restrictive as the stable date approaches — the opposite order is how regressions ship.
- **Never treat your registry/CI as disposable:** they are critical infrastructure with backup, DR, and key-escrow requirements; see [toolchain-landscape.md](./toolchain-landscape.md) and [metrics-and-dora.md](./metrics-and-dora.md) for adjacent operational context.
## Sources and Further Reading
- [Chrome Release Cycle (chromium.googlesource.com)](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/process/release_cycle.md)
- [Chromium Merge Request Process](https://chromium.googlesource.com/chromium/src.git/+/refs/heads/main/docs/process/merge_request.md)
- [GitLab Handbook — Deployments and Releases](https://handbook.gitlab.com/handbook/engineering/deployments-and-releases/)
- [MozillaWiki — Firefox Release Process (RapidRelease)](https://wiki.mozilla.org/RapidRelease)
- [Ubuntu Project Docs — Release Team Freezes](https://documentation.ubuntu.com/project/release-team/freezes/)
- [JFrog — Best Practices for Artifactory Backups and Disaster Recovery](https://jfrog.com/whitepaper/best-practices-for-artifactory-backups-and-disaster-recovery/)
- [xMatters — After a Deployment Error: Fix Forward or Roll Back](https://www.xmatters.com/blog/after-a-deployment-error-should-you-fix-forward-or-roll-back)
- [Zylos Research — Agentic CI/CD (2026)](https://zylos.ai/research/2026-05-12-agentic-cicd-ai-driven-delivery-pipelines/)
references/release-process-models.md
# Release Process Models
How a team branches, integrates, and cuts releases is the first structural decision in release engineering. The model you choose determines merge complexity, how defects flow toward production, whether a bad change can be isolated, and — per DORA's research — how fast and how stable your delivery actually is. The models below are not a menu of equal options: the evidence and the practitioner consensus point hard at trunk-based development for anything cloud-hosted, with the others surviving where constraints (packaged software, regulation, hardware) genuinely demand them.
## Trunk-Based Development — the Modern Default
**Trunk-based development (TBD)** means all developers collaborate on one long-lived branch (`main`/`trunk`). Feature branches are short-lived — ideally less than a day and holding a single developer's work — and exist only to carry a code review through CI before merging back. No long-lived feature or release branches are used for artifact creation. Incomplete work is hidden behind **feature flags** or **branch-by-abstraction**, never parked on a branch.
Why this matters (and why DORA evidence favors it):
| Effect | Mechanism |
|--------|-----------|
| Small batches | Merging to trunk daily forces small, reviewable increments; DORA correlates small batches with higher performance and lower failure impact |
| No merge hell | Conflicts are resolved continuously instead of accumulated for a big merge |
| One source of truth | Every deploy candidate comes from the same integrated line; no "is this fix on the branch?" archaeology |
| Fast feedback | Every change is integration-tested the moment it lands, not weeks later |
TBD is the default recommendation for cloud/SaaS teams that can automate build, test, and deploy, and that can use flags or branch-by-abstraction to decouple deploy from release. If you cannot merge to trunk at least daily, you are not really doing TBD — you are doing batched integration with a trunk-shaped label.
### Merge Queues
Under high PR volume, keeping `main` green is itself a bottleneck. **Merge queues** (GitHub merge queue, Graphite, Trunk.io) dynamically group ready PRs, run CI against the combined set, and only merge when the group passes — replacing fragile "rebase-and-retest" manual rituals. GitHub's own data is the reference point: its merge queue ships **2,500+ PRs/month into its monorepo from 500+ engineers**, and reduced average wait time by ~33% after launch. A merge queue is an enabler of TBD, not a substitute for it: it keeps the trunk green while preserving small, frequent merges.
### Branch Protection and Review Workflow
The branching model is only as good as its enforcement layer — the branch-protection rules that make the model structural rather than aspirational:
| Rule | Protects against | Applies to |
|------|------------------|-----------|
| Required status checks (CI must pass) | Merging broken code | All merges to `main`/trunk |
| Required PR review (peer review) | Untested, unreviewed changes | All merges to `main`/trunk |
| No direct pushes to protected branches | Bypassing the model | `main`/trunk, release branches |
| Restrict pushes to release branches | Unauthorized hotfixes | `release/*` during stabilization |
| Merge queue on protected branches | CI races and red `main` | High-volume repos |
Branch protection is policy-as-code applied to the repository: it encodes "small, reviewed, CI-passing changes only" so the process does not depend on memory or goodwill. It pairs with the pipeline's own gates (see [cd-and-pipeline-stages.md](./cd-and-pipeline-stages.md)) — the repo protects the merge, the pipeline protects the deploy.
### Named example — Google
Roughly **35,000 developers** collaborate in a single monorepo on a shared trunk. Most major projects branch from mainline *at a revision* for a release and **never merge back**; fixes are cherry-picked forward into the release branch and periodically returned to mainline. This works because builds are hermetic, tests are fast, and feature flags cover in-flight work. Google's scale is the stress test for TBD: if it breaks down anywhere, it is at merge-request review and CI capacity, not at the branching model itself.
## GitHub Flow — Lightweight Middle Ground
**GitHub Flow** is `main` plus short-lived feature branches merged via pull request, with deployment happening from `main`. The trunkbaseddevelopment.com reference notes it is "quite similar" to TBD; the difference is mostly *where you release from* — GitHub Flow keeps the option of releasing straight off `main` with no release-branch ceremony.
| Aspect | GitHub Flow | Trunk-based |
|--------|-------------|-------------|
| Long-lived branch | `main` only | `main`/trunk only |
| Feature branches | Short-lived, PR-merged | Short-lived, PR-merged |
| Release point | Tag or deploy from `main` | Tag from trunk, or cut a short-lived release branch |
| Best fit | Small-to-mid teams, SaaS, low ceremony | Teams needing strict mainline discipline at scale |
GitHub Flow is the right default when a team wants most of TBD's benefits but has not yet built the flag/testing machinery to keep incomplete work safely on trunk. The migration path is usually GitHub Flow → TBD as flags and pipeline maturity arrive, not the reverse.
## GitFlow — the Legacy Model (When It Still Fits)
**GitFlow** adds long-lived `develop` and `release` branches alongside `main`, with `hotfix` branches for urgent patches. It was designed for versioned, packaged software with scheduled releases and parallel maintenance of multiple released versions. That is also its cost: every change travels `feature → develop → release → main`, and fixes must be merged in multiple directions, which accumulates merge overhead and slows lead time. Practitioner consensus is to prefer TBD or GitHub Flow unless you genuinely ship boxed/versioned software that customers run without your control.
When GitFlow still fits:
- **On-premise or customer-managed software** where several released major versions are supported in parallel and each must receive security patches.
- **Regulated environments** that mandate a formal release branch as the artifact source for audit.
- **Hardware-coupled or firmware products** where field units cannot be upgraded arbitrarily.
> **Gotcha — GitFlow as cargo cult:** Teams adopt GitFlow "because the enterprise template says so," then pay merge-tax on every release while gaining nothing. If you cannot name the customer constraint that requires parallel version maintenance, use trunk-based or GitHub Flow instead.
## Release Branches Cut Just-in-Time
Release branches remain a valid tool when they are **cut just-in-time from trunk**, hardened, and **deleted after release** — or omitted entirely when you release from trunk with a fix-forward strategy. The just-in-time pattern gives you a stable stabilization surface without the long-lived-branch overhead of GitFlow:
1. **Branch cut** — at the release-candidate point, create `release/v<major>.<minor>.x` from a trunk revision you trust.
2. **Stabilization** — cherry-pick only critical fixes (P0/P1 defects, security, customer-blocking issues) from trunk into the branch.
3. **Release** — tag and build the artifact from the branch; re-run the full CI suite on the branch, because cherry-picks bypass the integration testing that happened on trunk.
4. **Patches** — hotfixes go through the same branch, tagged `v2.3.1`, `v2.3.2`, ...
5. **End of life** — archive or delete the branch once all consumers have migrated.
**Cherry-pick discipline:** every cherry-pick is code that skipped trunk's integration tests. Require a tracking issue, enforce the same code-review standard, re-run CI on the branch after each pick, and keep a log (SHA + description) attached to the release notes. Google rebuilds at the original release revision and pins the *build toolchain* to that revision, so a compiler change cannot silently alter a hotfixed in-production release.
**The merge-back decision:** at release time, choose explicitly among *merge the branch back, cherry-pick fixes forward, or archive it*. Google's model is branch-from-mainline-at-a-revision, never merge back, cherry-pick fixes forward and periodically return them to mainline. The unforgivable option is *no decision*: a branch that lingers indefinitely, silently accumulating divergence until "which branch has the fix?" becomes archaeology. If a branch will outlive one release, it must have a merge-back or archiving plan before it is cut.
## Release Trains and Calendar Releases
**Release trains** ship on a fixed cadence regardless of which features are ready — the train leaves the station on schedule. This converts coordination cost into a predictable date: teams know when the branch cuts, when stabilization starts, and when the release ships, and can plan cross-team dependencies against it. The tradeoff is that incomplete work must be explicitly deferred to the next train, which only works when features are flag-gated or genuinely shippable in pieces.
| Example | Cadence | Notes |
|---------|---------|-------|
| **Ubuntu** | 6-month cycle (`YY.MM` versions) | Feature freeze and release freeze ladder leading up to each release |
| **Chromium** | 4-week branch cadence | Branch point every 4 weeks, stabilization to stable, patches on the branch |
| **GitLab** | Monthly (`YY.MM`), self-managed release on the third Thursday of the month (one-week delay if needed) | Patch releases for regressions and security fixes between majors |
| **Firefox** | 2-week cycle (`main` → `beta` merge every 2 weeks) | Version-numbered trains with a single beta channel feeding stable; ~5 betas per cycle, RC QA-tagged, uplifts via tracking/approval flags |
The freeze ladder is the train's engine: each milestone (feature freeze → release candidate → final release) converts "should we include this?" from a debate into a date. What is not frozen by the freeze date rides the next train — and the only way that is acceptable is feature flags, which let a feature land on the train while *exposure* waits (see [progressive-delivery.md](./progressive-delivery.md)).
**SAFe Agile Release Trains (ARTs)** are the scaled-agile variant: 5–12+ teams align on a common **Program Increment** cadence (typically 8–12 weeks) with PI Planning and periodic sync events, coordinated by a Release Train Engineer. ARTs are an *organizational scaling construct* — they schedule and synchronize many teams — not a deployment technique. They fit large, multi-team, regulated, or hardware-coupled environments where independent per-team cadence would fragment the product. Treat the specific PI length and role names as SAFe-version-dependent rather than fixed doctrine.
> **Gotcha — the train metaphor taken literally:** A release train that ships regardless of readiness forces you to either ship broken features or hold finished ones. The escape valve is feature flags: ride the train, but expose features only when each is ready. Trains without flags are how "it's on the train, so we shipped it" accidents happen.
## Feature-Driven vs Time-Based Releases
| Model | Trigger | Pros | Cons | Fits |
|-------|---------|------|------|------|
| **Feature-driven** | Ship when a feature is done | User value arrives immediately; no artificial wait | No predictable dates; feature scope creep delays everything | SaaS with flag-based dark launches, small teams |
| **Time-based (train/calendar)** | Ship on schedule | Predictable dates; coordination cost falls; scope control via deferral | Ships whatever is ready; needs flags/deferral discipline | Multi-team products, regulated releases, hardware, enterprises |
The two are not mutually exclusive: mature teams use **calendar cadence for the container and feature-driven release inside it** — the train ships on schedule, but each feature is flag-gated and turned on when its own quality bar is met. This is precisely how "deploy on the train, release features on demand" reconciles the models. The container handles coordination; the flags handle readiness.
## Single-Repo vs Multi-Repo Cadence
The branching model interacts with repository topology:
- **Multi-repo (polyrepo):** each repo owns its pipeline and versioning; cross-repo dependencies are mediated by artifact registries and version ranges. Coordination cost moves into dependency upgrades — every breaking change triggers a cascade of downstream releases. Releases are per-service, which is good for independence and bad for atomic cross-cutting changes.
- **Monorepo:** one repository, one (or per-service) pipeline, atomic cross-cutting changes, synchronized builds. Releases may be **fixed/one-version** (everything ships together — Google's one-version rule) or **independently versioned** (changesets/release-please) with affected-build detection to avoid rebuilding everything (see [monorepo-polyrepo-release.md](./monorepo-polyrepo-release.md)).
The cadence question follows: polyrepo teams need explicit cross-repo release coordination (version ranges, deprecation windows); monorepo teams can release atomically and are the natural home for TBD and release trains.
## Deployment Risk Profiles
The process model also determines *how* a release reaches production, and Google's risk-profiled deployment is the reference pattern: most services roll out via **exponential cluster expansion** (small canary → doubling exposure on health), while **sensitive infrastructure** (billing, data, anything where a mistake is expensive) extends the rollout over several days, **interleaving across geographic regions** so a regional failure does not become a global one. The risk profile is a property of the release's blast radius, not of the team's mood: a config change to a payment path and a new README deploy do not deserve the same rollout shape. Choosing the profile is part of choosing the release model (see [progressive-delivery.md](./progressive-delivery.md) for the mechanisms).
## Ownership and Ceremony
Every process model implies an ownership and ceremony structure, and the two must match:
| Model | Ceremony | Owner |
|-------|----------|-------|
| Trunk-based | None per deploy — the pipeline is the gate | The deploying team; no release manager |
| GitHub Flow | PR review + CI; tag-and-deploy on demand | The deploying team |
| Release branches (JIT) | Branch cut, stabilization, patch flow | Release manager or designated DRI per release |
| Release train | Branch cut, freeze ladder, go/no-go, comms | Release manager DRI who owns the train's schedule |
The rule: **ceremony should scale with coordination need.** Per-deploy ceremony in a trunk-based team is waste; the absence of ceremony on a multi-team train is how trains derail. The train's DRI owns the calendar, the branch cut, and the go/no-go (see [readiness-and-quality-gates.md](./readiness-and-quality-gates.md)); the trunk-based team's "ceremony" is the pipeline itself. When in doubt, put the ceremony in the pipeline (automated gates) rather than in the calendar (meetings) — the pipeline enforces 24×7; the meeting enforces once a month. And whatever the model, name the DRI: an unnamed owner is an unowned process.
## Decision Guidance
| Situation | Model |
|-----------|-------|
| SaaS/high-throughput team with automated tests and flags | Trunk-based |
| Small team, low ceremony, deploy from `main` | GitHub Flow |
| Versioned, packaged software, parallel major-version support | GitFlow (or just-in-time release branches from trunk) |
| Regulated enterprise needing a formal artifact branch for audit | Trunk + just-in-time release branches |
| Multi-team product needing predictable dates | Release train / calendar (SAFe ART at scale) |
| Firmware/hardware with field units | Time-based trains + just-in-time release branches |
| High PR volume breaking `main` | TBD + merge queue (GitHub merge queue, Graphite, Trunk.io) |
A useful litmus test: *can the current `main` head ship to production right now?* If not, your process model is forcing batches larger than your quality gates can absorb — shrink the model, not the gates.
## Choosing: a Checklist
When selecting or auditing a process model, run the team through these questions:
1. Can the head of `main` ship to production today? (If not, batches are too large or gates too weak.)
2. Can incomplete work live safely on trunk (feature flags / branch-by-abstraction)? (If not, you will be tempted to park it on branches.)
3. How many released versions must be maintained in parallel, and by what obligation (customer contract, regulation, device fleet)?
4. Does any regulator or customer contract require a release-branch artifact as the audit source?
5. Do multiple teams need a common schedule (trains), or does per-team cadence fragment the product?
6. What is the actual merge cost today — conflict rate, PR wait time, stabilization length? (Measure it; do not argue about it.)
7. What ceremony does the model demand, and does the team have the ownership structure to sustain it?
The answers to 1–2 determine whether trunk-based is viable; 3–4 determine whether GitFlow-style parallel-version support is genuinely required; 5 determines whether trains are warranted; 6–7 tell you whether the team is actually operating the model it thinks it has.
## Signs the Model Is Wrong
Process-model problems announce themselves before they cause incidents. Watch for these symptoms:
| Symptom | Likely cause | Fix |
|---------|--------------|-----|
| Long-lived branches accumulate divergent fixes that never reach mainline | Branch lifecycle unenforced | Schedule branch deletion/merge-back; track every backport |
| `main` is frequently red | Batches too large; incomplete work not flag-gated | Shrink batches; hide in-flight work behind flags |
| PRs sit unmerged for days | No merge queue; PRs too big | Add a merge queue; split PRs |
| Every release needs weeks of stabilization | Cadence mismatch or oversize batches | Smaller trains; flag-gated deferral of unfinished work |
| Cherry-pick rate on release branches stays high for months | Fixes are not merging back to trunk | Enforce merge-back as a release criterion |
| Merges regularly conflict | Long-lived feature branches | Move to trunk-based / short-lived branches |
Any of these persisting is a process-model defect, not a personnel problem — the model is generating the friction.
## Gotchas
> **Gotcha — long-lived release branches:** A release branch kept alive past its release accumulates divergent fixes that never reach mainline. Either schedule the branch's deletion, or treat every backport as a first-class change with a tracking issue.
> **Gotcha — trunk-based without the prerequisites:** Merging to trunk daily without fast automated tests and feature flags just moves the pain: broken `main` becomes everyone's problem. TBD is a contract — merge small, keep tests green, hide incomplete work.
> **Gotcha — merge queues treated as a magic shield:** Merge queues keep `main` green under volume but do not reduce batch size. If PRs still sit unmerged for days, the queue is masking an integration bottleneck, not fixing one.
> **Gotcha — cherry-pick archaeology:** Release branches that are never merged back force engineers to re-apply fixes by hand. Track every cherry-pick with a ticket and a changelog entry so the "did this fix ship?" question has a machine-answerable record.
> **Gotcha — trains without a deferral mechanism:** A calendar release with no way to say "this feature is not ready" degrades into a quality lottery. The deferral mechanism (flag, exemption process, scope review) is part of the model, not an add-on.
## Sources and Further Reading
- [Trunk-Based Development (trunkbaseddevelopment.com)](https://trunkbaseddevelopment.com/)
- [Google SRE Book — Release Engineering (ch. 8)](https://sre.google/sre-book/release-engineering/)
- [DORA — Working in Small Batches capability](https://dora.dev/capabilities/working-in-small-batches/)
- [DORA — The DORA Metrics guide](https://dora.dev/guides/dora-metrics/)
- [GitHub Blog — How GitHub uses merge queue to ship hundreds of changes every day](https://github.blog/engineering/engineering-principles/how-github-uses-merge-queue-to-ship-hundreds-of-changes-every-day/)
- [Chromium — Release Process](https://www.chromium.org/developers/release-process/)
- [Ubuntu Project Docs — Release Team Freezes](https://documentation.ubuntu.com/project/release-team/freezes/)
- [SAFe — Agile Release Train](https://scaledagileframework.com/agile-release-train/)
references/role-and-career.md
# Release Engineering as a Discipline and Career
## What Release Engineering Is
Release engineering (RE) is the software-engineering discipline of turning source code into **reliable, reproducible, repeatable, and safe releases** — Google's SRE book defines it bluntly as "building and delivering software." A release engineer holds working knowledge of source-code management, compilers, build configuration, automated build tools, package managers, and installers, and connects otherwise separate worlds: development, configuration management, test integration, system administration, and customer support.
The discipline is guided by four operating principles, all from Google SRE chapter 8:
1. **Self-service model** — teams run their own releases through shared, best-practice tooling so release engineering effort scales with the org, not linearly with headcount.
2. **High velocity** — frequent, small releases are *safer* than occasional big-bang ones ("Push on Green": deploy every build that passes all tests).
3. **Hermetic builds** — reproducible, dependency-pinned builds insensitive to the build machine, enabling exact rebuilds, cherry-picking, and trustworthy hotfixes.
4. **Enforcement of policies and procedures** — gated operations (code review, branch creation, deploy, build-config changes) with a complete audit trail of every change in a release.
A distinctive practitioner mindset follows from these principles: **"Where others see features, we see release challenges. Where others count change lists, we count how long it took for a change from submission until it was in front of the customer."** Release engineers are expected to be emotionally unattached to any particular code change — the job is getting the pipeline and the process right, including saying "you can't put that feature in because it will break everything."
## Release Engineering vs. Adjacent Disciplines
The boundaries are genuinely blurry and company-dependent, but the four neighbors can be distinguished cleanly:
| Discipline | Owns | Primary lens | Typical artifacts |
|-----------|------|--------------|-------------------|
| **Release engineering** | The release process from source to deployment: build, package, sign, promote, version, rollback | The *means of production* of the artifact | Pipelines, artifacts, release branches, rollback runbooks, release notes |
| **SRE** | Production reliability of the *running system*: SLIs/SLOs, error budgets, on-call, capacity | The deployed service | SLOs, dashboards, incident responses, capacity plans |
| **DevOps / platform engineering** | The automation platform and "paved roads": source-control workflows, CI/CD runners, IaC, secrets, observability guardrails | The developer experience of shipping | Internal developer platform, golden templates, runner fleets |
| **Release management** | Process, scheduling, and coordination: release calendars, train coordination, go/no-go decisions, stakeholder comms | The *when and who* of shipping | Release schedules, sign-off records, comms plans |
- **RE vs. SRE:** RE owns *getting a change from source to deployment* so that what SRE deploys is reproducible and never a "unique snowflake." SRE owns *keeping the deployed system reliable*. They collaborate closely on canarying, safe rollout, and rollback; configuration management is an area of "particularly close collaboration" between the two.
- **RE vs. DevOps/platform:** A build-and-release engineer "standardizes builds, generates artifacts, versions and signs them, and promotes releases across dev → staging → production using controlled pipelines," while a DevOps/platform engineer "designs and automates the delivery platform" (source control, CI, IaC, observability). Practitioner framing: release engineering is a *spoke* of the DevOps hub; modern platform engineering absorbs much of RE into golden pipelines and developer portals.
- **RE vs. release management:** The cleanest real-world split is at **Mozilla**: "release engineers don't monitor the quality of the release; we have a team called Release Management to perform that function." Release *engineering* is technical execution; release *management* is coordination and judgment. **GitLab's Release Manager** is a rotating operational role that drives the monthly release, approves patches, and makes hard calls (refuse a feature, revert work) — a coordination DRI, not a pipeline engineer.
## Where Release Engineering Sits in an Organization
Two dominant placement models, often combined:
- **Central release/platform team** — a named function that builds shared tooling, standards, and pipelines for all product teams (Google's RE group, GitLab's Delivery team, Mozilla's RelEng). This is the scaling model: one team, org-wide leverage.
- **Embedded release coordination** — release engineers or release managers attached to a specific product line (e.g., Search, YouTube) to run that product's trains and ceremonies.
The coordination mechanism that makes either model work at scale is the **release train**: a fixed, predictable release schedule where "if you're late for the release train, it will leave without you." Trains trade feature flexibility for coordination cost and calendar predictability. Real examples:
- **Chromium**: canary (daily-ish) → dev (weekly) → beta → stable every 4 weeks; branches live roughly 18 weeks; every change lands on trunk first, then is cherry-picked to branches.
- **Mozilla/Firefox**: features land on Nightly, get uplifted branch → branch, and bake several weeks per channel; the standard release interval is two weeks.
- **GitLab**: a monthly self-managed release plus twice-monthly patch releases and ad-hoc security releases, run by rotating release managers; GitLab.com itself ships continuously from `master`.
The two placement models trade off differently:
| Model | Strengths | Weaknesses | Fits |
|-------|-----------|------------|------|
| **Central release/platform team** | Leverage, consistent standards, career ladder, shared tooling | Can become a bottleneck or "ticket desk" if self-service fails; distance from product nuance | Scaling orgs (Google RE, GitLab Delivery, Mozilla RelEng) |
| **Embedded release coordination** | Deep product context, tight relationship with the release train's consumers | Duplication, inconsistent practices, thin career path | Large product lines with distinct release needs (Search, YouTube) |
Most mature orgs run a hybrid: a central team owns the *platform* (golden pipelines, standards, tooling) while embedded release engineers own *product trains* on top of it. When do companies create a dedicated RE function at all? Practitioner evidence says it happens as a consequence of growth: "once they start to grow, they look for a person to do release work" — the function is usually born from the pain of manual, error-prone shipping, not from a strategy memo.
## What Senior Release Engineers Do Day-to-Day
A senior release engineer operates at **project scope**: they own one product's release pipeline end-to-end and still write code every week. Concrete activities:
- **Pipeline design and maintenance** — architect and maintain CI/CD build/test/deploy pipelines; tune caching, parallelization, and deterministic/hermetic builds; drive down flaky-test rates and build times.
- **Artifact management** — version, package, sign, and promote artifacts through dev → staging → prod; operate artifact repositories (Artifactory, Nexus, ECR, GHCR, Harbor); enforce immutability, retention, and SBOM generation.
- **Release coordination** — operate release trains and release windows; cut release branches; manage cherry-picks; produce change reports and release notes.
- **Release readiness and go/no-go** — run release checklists; verify test coverage and quality signals; assemble readiness evidence; in regulated contexts, produce change records and approvals.
- **Rollout safety and rollback** — implement canary, blue-green, and progressive-delivery strategies; configure automated rollback triggers; write, rehearse, and execute rollback runbooks.
- **Toil automation** — automate repetitive release steps, eliminate fragile manual steps, and build small CLIs and templates.
- **Dependency hygiene** — pinning, updating, and toolchain/runner upgrades.
- **Mentoring** — help product teams troubleshoot their own pipelines and adopt best practices.
A representative senior week decomposes into daily and weekly rhythms:
| Rhythm | Activities |
|--------|------------|
| **Daily** | CI/CD health checks (pipeline success rate, queue times, flaky signals); triage and fix pipeline failures; review release-related PRs and config changes; support releases and hotfixes for high-impact services; verify artifacts are versioned, signed, and published correctly |
| **Weekly** | Release-reliability review (top failure modes, toil, automation backlog); meet product teams adopting new release patterns; tune release gates with SRE and Security (what blocks vs. warns vs. approves); publish release status updates; hold developer office hours |
The common thread: a senior release engineer is a *firefighter and a builder* — unblocking today's release while making tomorrow's release not need unblocking.
> **Gotcha — senior is a "career level":** Being a senior release engineer is a terminal destination, not a waiting room. The leveling gate that follows is not about doing *more* delivery; it is about changing what kind of work you do.
## Staff Scope: Leverage, Golden Pipelines, and Influence
Staff release engineers own **product or org-level release strategy**. The staff brief is to stop shipping releases yourself and start making every team able to ship well:
- Define the release-engineering strategy for the developer-platform roadmap; set standard release patterns (branching, versioning, artifact management, deployment strategy) that scale across stacks.
- Design **governance models**: release trains vs. continuous delivery, approval gates, risk tiers, change-management integration.
- Build **self-service "golden pipelines"**: reusable templates, libraries, CLI tooling, and developer-portal integrations so teams ship without opening a platform ticket.
- Find and fix **systemic bottlenecks** — the cross-cutting, multi-quarter improvements that unblock dozens of teams.
- Implement **supply-chain integrity controls** (SBOM, signing, provenance, SLSA alignment) and release-safety mechanisms (feature flags, canary, automated rollback) as shared platforms.
- Drive adoption **without authority**: RFCs, architecture reviews, workshops, and cross-team mentoring.
Staff-level evidence is about leverage, not output: a golden pipeline with measurable adoption (e.g., 60–80% of services), standards with high adoption and low friction, DORA improvements you caused, cross-team initiatives you led, and teams that stopped filing platform tickets because they became self-sufficient. A recurring staff duty is turning release incidents into systemic fixes: post-release reviews, trend reporting, pipeline hardening, rollback game days, and the authority to **stop the line** when release risk is high (widespread flaky tests, a compromised dependency).
## Principal Scope: Operating Model, Governance, and Multi-Year Direction
Principal release engineers set the **enterprise release operating model** and multi-year direction across dozens or hundreds of services:
- Define the enterprise release operating model and roadmap with adoption paths per maturity level (teams start at different places).
- Architect scalable CI/CD and release-orchestration patterns across monorepo/polyrepo, microservices, and shared platform components.
- Make **progressive delivery the default** — deploy continuously, release deliberately, roll back by flag or traffic shift.
- Define **risk-based, not bureaucracy-based** quality gates; embed risk tiers in automation rather than in meetings.
- Own release governance: change-management alignment, evidence collection, segregation of duties, audit readiness — and ensure every exception is time-bound and reviewed.
- Influence platform investment with data (toil metrics, incident trends, cycle time) and coach teams out of anti-patterns: manual releases, snowflake pipelines, environment drift.
The Staff+ archetypes from Will Larson's *Staff Engineer* map cleanly onto release engineering: the **Architect** (owns the direction and quality of the release/CI-CD domain — the most natural fit), the **Solver** (drops into the worst release bottleneck or incident hotspot), the **Tech/Team Lead** (guides a release-platform team), and the **Right Hand** (extends an infra/engineering executive across a large org).
### "Protect the Product from the Developers"
A recurring staff+ responsibility — stated bluntly in *Software Engineering at Google* (ch. 24) — is to **protect the product from the developers**: the urgency of new features must never trump the existing user experience. Concretely this means holding a release even when a highly visible feature is at stake, enforcing quality gates against pressure to "just ship it," and running post-release reviews that turn incidents into systemic fixes: trend reporting, pipeline hardening, rollback game days, and "stop the line" authority when release risk is high (widespread flaky tests, a compromised dependency). The classic historical example is YouTube's manual release process: a release involved a **50-hour manual regression** and a "Build Cop" gatekeeper — precisely the toil and risk that release engineering exists to replace with automation and evidence.
## Leveling Evidence and Scope Progression
The single cleanest differentiator between levels is **scope**: which organizational unit's release outcomes are your responsibility.
| Level | Scope | What "good" looks like | Typical horizon |
|-------|-------|------------------------|-----------------|
| **Senior** | Project / one product | The product's release pipeline is reliable, fast, and safe; you personally maintain it and its runbooks | One release cycle to a year |
| **Staff** | Product / org-wide release patterns | Many teams self-serve on release patterns you designed; DORA and toil improve org-wide | Multi-quarter |
| **Principal** | Organization / multiple products | Enterprise operating model, risk-tiered governance, supply-chain controls at scale | Multi-year |
| **Distinguished** | Department / company | Release capability is a competitive advantage; org-wide policy and investment decisions | 3+ years |
**Evidence separates the levels, not titles.** Senior → Staff: from "I delivered the release pipeline for this product" to "many teams self-serve on release patterns I designed" — cite reusable golden pipelines with adoption numbers, standards with high adoption, DORA improvements you caused, and cross-team initiatives you led. Staff → Principal: from "product release platform" to "enterprise release operating model" — cite org-wide standards and governance, risk-tiered policy embedded in automation, supply-chain controls at scale, and durable reductions in org-wide CFR, toil, and incidents driven by your roadmap.
> **Gotcha — the "Engineer 2.5" trap:** Going senior → staff is "almost a different job." The common failure is treating staff as senior-plus-more-delivery — "Engineer 2.5." At staff and above, individual output stops being the lever; **leverage** (self-service tooling, standards, influence) is. If you are the only person who can run your product's release, that is evidence you have not staffed up: you built a dependency, not a platform.
**Public RE-specific leveling rubrics are scarce.** No major company publishes a release-engineering-specific promotion packet; RE is usually leveled under the general software-engineering or platform-engineering ladder. At Google, "release engineers are software engineers; there is no difference." The closest role-specific public artifacts are the DevOps School Staff/Principal release-engineer blueprints and GitLab's public Release Manager docs (which describe a coordination role, not a pipeline-engineering one). Generic IC-ladder variants matter for expectations: Google and Meta add a "senior staff" rung; Amazon has no staff level (principal and senior principal instead); most open ladders (career-ladders.dev, Levels.fyi's SWE framework) cover the generic progression but not RE specifics.
### The Self-Taught Discipline
Release engineering is rarely taught in school; practitioners describe hiring as "finding unicorns" — people with utilitarian programming ability, architecture knowledge, and release judgment developed on the job. This has two consequences for careers. First, **the discipline is portable**: the build/release skills you master at one company (hermeticity, promotion, progressive delivery, rollback) transfer across stacks and industries. Second, **breadth is the differentiator**: engineers who combine deep Git/build knowledge with supply-chain security and release-operations judgment are the ones who clear the staff+ gate. One structural caveat to plan around: **mobile and embedded release engineering is harder than server-side CD** — app-store distribution, review times, and device fragmentation constrain how much of the continuous-deployment playbook applies.
## KPIs: The Shared Scoreboard
Release engineers are judged on system outcomes. The full scoreboard spans five categories:
| Category | KPIs | What they signal |
|----------|------|------------------|
| **Throughput** | Deployment frequency, change lead time | How fast the delivery system ships |
| **Stability** | Change failure rate (CFR), failed deployment recovery time, deployment rework rate | How safe it is to ship |
| **Pipeline health** | Pipeline success rate, mean-time-to-green, pipeline duration p50/p90, flaky-test rate | How reliable the pipeline itself is |
| **Supply chain** | Artifact signing coverage, SBOM coverage, policy-compliance rate, exception aging | How trustworthy the artifacts are |
| **Operational** | Manual steps per release, automation coverage, golden-pipeline adoption, rollback readiness, release-incident recurrence | How much toil remains and whether incidents repeat |
For a single metric, the de-facto scoreboard is **DORA**; for exact definitions and the vendor-divergence caveats, see [metrics-and-dora.md](./metrics-and-dora.md).
> **Gotcha — DORA metrics are not individual performance metrics:** They measure an application's delivery *system*, not a person. Setting deployment frequency or CFR as an individual goal invites gaming (splitting deploys, under-reporting failures) and is an explicit misuse DORA warns against. Use them to steer process improvement and to show the system-level impact of your work, never as a per-engineer score.
## Transitions and Adjacent Career Paths
Release engineering is a good base for several adjacent roles because it is the junction of development, operations, and security:
- **Platform engineering** — the natural move for staff-level release engineers who build golden pipelines and developer platforms; the boundary between the two is genuinely blurry in modern orgs.
- **SRE** — the other side of the deploy boundary; RE engineers who move into SRE bring rollout and rollback depth to incident response.
- **Software supply-chain security** — REs already own signing, SBOM, and provenance; a security specialization formalizes it.
- **Engineering management / release management** — for engineers whose strength is coordination, go/no-go judgment, and stakeholder communication.
- **DevOps / delivery consulting** — the discipline's portability makes RE a strong consulting specialty (assess → redesign pipelines → coach the team).
## Hiring: What to Assess
Because public RE-specific leveling rubrics are scarce, interviews lean on scenario signals. Useful probes: *Walk me through the last release you owned — where were the manual steps and how did you remove them?* (automation judgment); *A canary deploy starts throwing errors at 5% — walk me through your decisions* (rollback judgment under uncertainty); *A team wants to skip the gate to ship a feature for a customer — how do you respond?* (protect-the-product + negotiation); *Show me a pipeline you designed and the metrics that prove it works* (evidence-based claims). The strongest predictor across all of these is a candidate's demonstrated *reduction in toil and risk over time*, not their tool familiarity.
## Named Examples and Role Models
- **Google SRE book ch. 8** — the canonical treatment: the four principles, Rapid/Blaze/MPM tooling, package labels (`dev`/`canary`/`production`), and the self-service scaling model.
- **Chromium** — a 4-week train with an elaborate merge-approval process: release managers review *every* cherry-pick to release branches, with criteria that tighten as the stable date approaches, and automation (Blintz) does first-pass triage.
- **GitLab** — a rotating **Release Manager DRI** owns each monthly self-managed release, twice-monthly patches, and the weekly delivery-metrics review; the role is documented publicly with permissions and escalation paths.
- **Mozilla** — the explicit split between release *engineering* (RelEng builds and operates the pipeline) and release *management* (owns quality monitoring and go/no-go) — the cleanest public separation of the two functions.
For the broader skills that carry an engineer through these levels, see [skills-competency-model.md](./skills-competency-model.md). For the tooling a release engineer operates, see [toolchain-landscape.md](./toolchain-landscape.md); for the metrics scoreboard, see [metrics-and-dora.md](./metrics-and-dora.md).
## Sources and Further Reading
- [Google SRE Book — Release Engineering (ch. 8)](https://sre.google/sre-book/release-engineering/)
- [Software Engineering at Google — Continuous Delivery (ch. 24)](https://abseil.io/resources/swe-book/html/ch24.html)
- [The Practice and Future of Release Engineering (IEEE Software / CMU SEI)](https://www.infoq.com/articles/practice-and-future-of-release-engineering/)
- [GitLab Release Documentation — Release Manager](https://gitlab-org.gitlab.io/release/docs/release_manager/)
- [Chromium — Release Process](https://www.chromium.org/developers/release-process/)
- [Software Engineer Career Levels (End of Line Blog)](https://www.endoflineblog.com/software-engineer-career-levels)
- [Staff Engineer Archetypes (Will Larson)](https://lethain.com/staff-engineer-archetypes/)
- [Staff Release Engineer Role Blueprint (DevOps School)](https://www.devopsschool.com/blog/staff-release-engineer-role-blueprint-responsibilities-skills-kpis-and-career-path/)
references/rollback-and-recovery.md
# Rollback and Recovery
Rollback is not one operation. The single most common release-engineering failure is treating "undo the deploy" as a single button when it is really a family of mechanisms with wildly different speeds, blast radii, and risks. The second most common failure is assuming rollback is possible at all. Google SRE's framing is the right starting point: **rollback capability is a precondition for rollout, not an afterthought** — "there's no good rollout unless you have a corresponding rollback ready to do" — and the hardest lesson is that *reverting code is not rolling back a deploy*.
## The Four Undo Operations (Never Conflate)
| Operation | What it does | Speed | Risk | Notes |
|-----------|--------------|-------|------|-------|
| **Artifact rollback** | Redeploy the previous immutable, known-good artifact | Minutes | Low — deterministic, previously healthy | Requires retained, immutable artifacts and hermetic builds |
| **Roll-forward (hotfix)** | Build new release = old release + minimal fix, deploy it | Hours (build + test + deploy) | Medium — the new artifact has never run in prod | Google: discourages as first response for user-visible bugs |
| **Git revert** | Source-control revert that produces *new code* | Slow — must rebuild, retest, redeploy | High — new artifact untested in prod; reverts code only, not schema/data/config/flags | Reverting a commit with a destructive migration leaves old code + new schema = broken |
| **Feature-flag rollback** | Toggle the offending behavior off via remote config | Seconds (sub-second) | Very low — reversible, auditable | Only works if the change is behind a flag |
**Decision criteria** (synthesized across Google, GoCD, and mobile practitioners):
1. Is the defect behind a **feature flag**? → flag off. Seconds, no redeploy.
2. Otherwise, is it **user-visible or severe**? → artifact rollback. Minutes, back to a known-good state.
3. Is it **minor with a trivial, low-risk fix**? → roll-forward. A quick roll-forward is often preferable when the fix is genuinely small and well-tested.
4. Never `git revert` and redeploy as a "rollback" — it produces a *new* artifact that has never run in production and does nothing about migrations, data, config, or flag state.
The decision axes are **change-failure expectations, time-to-fix, blast radius, and user impact**. The two extremes are the ones teams get wrong: reaching for `git revert` (slow, new untested artifact) when an artifact rollback exists, and reaching for roll-forward under incident pressure when the fix is non-trivial.
> **Gotcha — git revert ≠ rollback:** `git revert` only reverts code. It does nothing about schema migrations, data changes, config, or feature-flag state that shipped alongside the commit. If the commit contained a destructive migration, reverting the code leaves an incompatible schema in place — the worst of both worlds.
## Rollback by System Type
Rollback difficulty tracks **reversibility**: from nearly trivial (stateless services) through hard (shared databases) to impossible (shipped device binaries). Pick the mechanism by the layer you touched.
| Layer | Reversibility | Primary mechanism |
|-------|---------------|-------------------|
| Feature-flagged behavior | Near-total | Toggle off (seconds) |
| Stateless services | High | Redeploy prior artifact, re-point traffic |
| Microservices | High (with N-1 compatibility) | Independent rollback, consumers before producers |
| Stateful / databases | Bounded | Expand/contract; safe only before finalization |
| Mobile / desktop | Low–none | Forward-fix, flags, phased-release pause |
| IoT / firmware | None once flashed | A/B partitions, watchdog auto-revert |
### Stateless Services
For a service behind a load balancer, artifact rollback is: redeploy the prior artifact across instances and re-point traffic. It looks trivial and hides four operational steps that teams routinely omit:
1. **Detection by version** — keep error/latency summaries **broken down by binary release version**. Subtle failures (e.g., errors only for "users whose name contains an apostrophe") surface in aggregate monitoring only once the majority of instances are upgraded; per-version metrics are what distinguish a bad canary from the control.
2. **Connection draining / graceful stop** — stop accepting new connections, finish in-flight requests, then terminate, so rollback causes no mid-request failures.
3. **Cache warming** — after redeploying the prior artifact, warm caches before full traffic; the previous version's caches are cold or evicted, and a cold-cache thundering herd is a classic post-rollback incident.
4. **Verification** — confirm per-version error rates return to the healthy baseline using the same SLI set as canary gating (HTTP status + latency; CPU/memory are noisy and unreliable signals) before declaring the rollback complete.
**Blue/green** makes stateless rollback near-instant: rollback is a trivial reversal of the router change, at 2× resource cost (see [progressive-delivery.md](./progressive-delivery.md)).
### Stateful Services and Databases
This is the hard case. The core problem: a schema change and a binary change can desynchronize. Google's canonical trap — you release the new binary, upgrade the schema, then find a problem and roll back the binary — leaves you with "a binary that doesn't expect the new schema, and hasn't been tested with it." Destructive changes (drop column, rename, add non-null) are **irreversible at the data level**: you cannot `git revert` dropped rows.
The discipline is **expand/contract (parallel change)** — the backbone of safe schema evolution (Fowler; operationalized in production by Bitwarden):
| Phase | What happens | Releases |
|-------|--------------|----------|
| **Expand** | Add the new structure (column/table) alongside the old; **both old and new code work** | Release X |
| **Migrate/Transition** | Backfill data (batched, as a background task to avoid load); update clients/code incrementally; old and new coexist and stay in sync (e.g., dual-write) | Release X → X+1 |
| **Contract** | Only after nothing depends on the old, remove it (drop the column) — **in a later release, never the deploy that introduced the change** | Release X+2 |
Bitwarden's **release support matrix** shows the invariant — the schema must always support the previous release of the server, so code can be rolled back:
| Database phase | Release X | Release X+1 | Release X+2 |
|----------------|-----------|-------------|-------------|
| Start (initial migration adds new, keeps old) | ✅ supported | ❌ | ❌ |
| Transition (dual-write, backfill) | ✅ | ✅ | ❌ |
| End (finalization drops old) | ❌ | ✅ | ✅ |
Three migration types enforce this:
- **Initial migration** (before code deploy): adds support for the new release *without breaking the old*; must be fast/cheap for zero downtime.
- **Transition migration** (background task during dual-write): batched data backfill only — **no schema changes in this phase**.
- **Finalization migration** (runs as part of the next deploy): drops the old structure; the schema now supports only the new release.
**Rollback is safe only before finalization.** The state machine is explicit: old code + old schema → initial migration → old code + new schema (both supported) — roll code back or forward safely. Then new code + new schema (both supported) — roll back safely. Then finalization → new code + finalized schema — **old code + finalized schema = broken**: rollback is no longer an option; you must roll *forward* with a new migration. On a safe rollback, "it should be as simple as just re-deploying the previous version again," with the database staying in transition until a patch ships. Fully pulling a feature after finalization requires writing a *new forward* migration to undo the change — generally not recommended, since pending migrations and the rollout need revisiting.
Supporting rules:
- **Forward-only migrations:** migrations are append-only, sequenced, version-controlled with app code, tracked in a changelog table (Flyway/Liquibase). You never un-apply a migration in production; you add a new one. Migrations should be idempotent — safe to run multiple times.
- **Feature-free release for schema-coupled changes (Google):** ship release v+1 = v but *able to safely handle the new schema* (no new features); upgrade the schema; then release v+2 that *uses* the schema. Now either binary can be rolled back without rolling back the schema.
- **Backup/restore and point-in-time recovery (PITR) as last resort:** restore when data is corrupted or lost and no forward path exists. Governed by **RPO** (max tolerable data loss — PITR to 5 minutes ago loses ≤5 minutes of writes) and **RTO** (max tolerable downtime — how long the restore takes). Restore is slow (violates fast rollback), lossy (violates known-good), and can clobber good post-backup data — hence *last resort*.
> **Gotcha — destructive migration in the same deploy:** The single most common stateful-release disaster is dropping the old column in the same release that introduces the new one. Contract must always be a later release — and if you need to roll back after a finalization, you have already lost that option.
### Microservices
Rollback ordering follows **dependency direction**, under one ecosystem assumption (Google): *any service could be rolled back by one version.* Operationally this is **N-1 compatibility**: your service must tolerate its dependency being one version behind what you built against, because that dependency may roll back. If your launch waits for dependency S to move from r to r+1, be sure S will "stick" at r+1 — otherwise wait for r+2 before depending on r+1 features.
| Direction | Deploy order | Rollback order |
|-----------|--------------|----------------|
| **Producers (providers)** | Ship backward-compatibly *first*: expand the API, keep old behavior working | Remove the capability *last* |
| **Consumers (callers)** | Deploy *after* the producer's additive change exists | **Roll back first** — undo the caller's use of the new behavior before the provider loses it |
This mirrors expand/contract: contract (remove old) only after all consumers have migrated; un-migrate (roll back) consumers first. **Version-skew tolerance** makes this practical: Tolerant Reader and Postel's Law ("be conservative in what you send, liberal in what you accept") let consumers ignore unknown fields so a provider can expand without breaking them. If every service is N-1 compatible, services roll back **independently** — no orchestration needed; coordinated/sequenced rollback is only required when compatibility windows are violated, which is itself a design smell (if deploying one service requires deploying others, you have hidden coupling). Independent deployability is also what makes **partial rollback** possible: revert the single offending service while others stay forward.
### Mobile, Desktop, and IoT
**Mobile:** a true rollback is **impossible** — once a build is installed on a device, the only way to change it is to distribute a new one (per iOS-factor: the only way to change an installed build is a new version with an updated version/build number). The approximation: re-submit the last stable binary as a "new" version with a **higher build number**, re-signed (modifying a build invalidates its signature), re-submitted to **store review** (hours to days). Store constraints compound it:
- Build numbers must monotonically increase (Apple and Google both enforce this).
- Apple will not let you create a new version until the current one is live — a timing trap when you need to replace a bad release immediately.
- Google allows only one draft release on the Production track.
And there is a **version long-tail**: even after a hotfix ships, some users keep running the bad version indefinitely — you cannot uninstall a bad build from a device you do not own.
Levers that substitute for rollback on mobile:
- **Phased/staged rollout** — iOS phased release can be *paused* before reaching most users, then the binary replaced — the closest thing to a real rollback.
- **Feature flags / remote config / kill switches** — disable the broken feature on already-installed binaries without any store interaction. You cannot force users to update, so **build a kill switch in from day one**.
- **Forced update** — gate the app behind a minimum version; used sparingly, as it is hostile UX.
> **Gotcha — mobile rollback + irreversible client migration:** If the faulty release included a client-side database migration that cannot be reversed, deploying a rollback build can corrupt user data. Client rollback safety depends on server/schema backward compatibility — the same rule as servers, now enforced on software you cannot reach.
**Desktop:** auto-update channels (stable/beta/canary/dev rings) give staged exposure; rollback is publishing the prior version to the channel or promoting a fix forward. Easier than mobile — no store review, no marketplace-imposed build monotonicity — but still forward-push, not reach-back.
**IoT / embedded / firmware:** rollback is a *hardware-architecture* concern, not a deploy concern:
- **A/B (dual-bank) partitions:** new firmware installs to the inactive partition; the boot switch happens only after validation; if the update fails, the system automatically reverts to the previous partition — blue/green at the firmware level, and the mechanism that makes device rollback possible at all.
- **Watchdog timers:** if the device hangs during/after update, trigger recovery to the good partition.
- **Power-loss resumption:** resume an interrupted OTA rather than corrupting the active image (a bricked, offline device may be unrecoverable remotely).
- **Anti-downgrade checks:** signed firmware with version anti-rollback prevents installing a known-vulnerable older image — a security-vs-rollback-freedom tension to resolve explicitly.
- Automotive/standards context: UNECE R156 and ISO 24089 require robust update management including anti-bricking.
**When devices are offline or bricked, rollback is genuinely impossible** — design for forward-fix and fail-safe (safe-state) behavior. The mobile/IoT rule generalizes: the less reach you have over the artifact, the more your "rollback" strategy must be *forward* strategy — flags, staged rollout, and safe degradation.
## Preconditions: Artifact Retention and Immutability
Rollback is only as good as your ability to *re-deploy the exact previous artifact*. The preconditions are cheap and routinely neglected:
- **Immutability:** never mutate a published artifact. Content-addressed storage and signing make "the same version always means the same bytes" verifiable (see [cd-and-pipeline-stages.md](./cd-and-pipeline-stages.md)).
- **Retention:** keep the last N known-good artifacts for every environment, with retention tied to your recovery objectives and compliance obligations. An artifact you deleted cannot save a Monday-morning rollback.
- **Movable promotion labels:** promote by moving labels (`dev`, `canary`, `production`) that point at immutable versions, so "roll back to production" means "point the label at the previous version" — a deterministic, scriptable operation.
- **Reproducibility:** hermetic builds mean a lost artifact can be rebuilt byte-for-byte from its revision — the safety net under retention.
These preconditions are also the answer to the question "how fast can we roll back?" The answer is bounded by what you retained and what you tested — which is why retention, rehearsal, and rollback speed are the same conversation.
## Operationalizing Rollback
- **Rehearsed rollbacks:** Google's practice — roll back "just because" every few weeks, to find traps (incompatible versions, broken automation, broken tests) *while the new release is healthy*, which is "better by far" than discovering them while the service is on fire. If the rollback works, roll forward again; if it breaks, roll forward to remove the breakage and then diagnose. Rollback drills are fire drills; skipping them is a failure mode.
- **Runbooks:** every service needs a rollback runbook with: decision thresholds (which metric/canary signal triggers rollback), the exact commands/automation to redeploy the prior artifact, verification steps (per-version SLIs back to baseline), an escalation path, and **manual checkpoints for data-sensitive operations** before anything touching a destructive migration or backup/restore. The runbook is the tested artifact; a runbook that has not been executed is a hypothesis.
- **Time-boxed decisions:** decide rollback-vs-roll-forward within a bounded window (minutes), rather than debugging while users burn. This is the operational expression of "rollback first, investigate second."
- **Automatic triggers:** wire canary analysis to automatic action — if the canary metric diverges too far from control, pause and roll back the deployment or page a human. Gate on a stack-ranked top-few SLIs (≤ ~a dozen), compare canary vs. control populations (never before/after), and size the canary by error budget: a 5% canary at 20% error costs ~1% overall, so auto-rollback at that tier costs almost nothing (see [progressive-delivery.md](./progressive-delivery.md)).
- **Quarantine + postmortem:** after a rollback, quarantine the bad artifact (label/remove it so it cannot be re-promoted), open a blameless postmortem, and capture a rollback changelist describing the observed problem. The postmortem's real output is fixing the *pipeline* that let the bad build through — thresholds, tooling, runbooks — not assigning blame.
**Rollback communications:** the rollback itself needs a communication plan, not just commands. Announce the detection and decision on the status page and internal channels early (users prefer an honest "we rolled back a release" to silent breakage); keep the rollback changelist attached to the incident so anyone can see *what was observed*; and after verification, publish the all-clear with the postmortem link. The communication is part of the rollback because trust is part of recovery — and "rollbacks are normal" only holds if the org treats them as routine, which means announcing them as routine.
**Culture:** Google treats rollbacks as normal — "rollbacks are normal." When an error is found or suspected in a new release, the releasing team rolls back first and investigates second; a rollback request "is not interpreted as an attack on the releasing team." Rollback must be *easy to perform* and *trusted to be low-risk*; rehearsal is what keeps it trusted.
## Rollback Decision Authority
Pre-decide **who can call a rollback and at what threshold** — this is a release-time decision that should not require a meeting. Google's cultural norm is the useful default: rollback authority is broad and exercised without stigma — "a rollback request is not interpreted as an attack on the releasing team." The practical rule: any engineer with evidence (canary metric divergence, error spike tied to the release) can initiate a rollback of a stateless service; the rollback changelist records the observed problem.
Authority narrows where risk concentrates:
- **Data-touching operations** (destructive migrations, backup/restore, DB cutovers) need explicit authority and **manual checkpoints** before execution — the operator who touches data is not the operator who casually flips a load balancer.
- **Cross-service coordinated rollbacks** (when N-1 compatibility is violated and consumers must roll back before producers) need a coordinator, because ordering mistakes compound the incident.
- **Time-boxed decisions** bound the window: decide within X minutes, then execute — the decision authority is exercised within the box, not after it (see [progressive-delivery.md](./progressive-delivery.md) for the automatic-trigger version of this).
## Rollback vs Roll-Forward
The guidance converges on a layered answer rather than a universal rule:
| Situation | Choice | Why |
|-----------|--------|-----|
| Change behind a flag | Flag off | Seconds, no deploy risk |
| User-visible/severe defect, or any significant bug | **Artifact rollback first** | Known-good state; Google warns a hasty roll-forward under incident pressure either fails to fix the problem or makes it worse — "you're taking yourself further from a known-good state" |
| Minor issue with a trivial, low-risk fix | Roll-forward | Quick; GoCD: a quick roll-forward is generally preferable, and *frequent* rollbacks signal weak pipeline gates |
| After DB finalization / on shipped binaries | Roll-forward (or forward-migration / new build) only | Rollback is no longer available |
Reconciling the philosophies: for fast-reversible layers (stateless services, flagged features) rollback is cheap and should be a reflex; for slow or irreversible layers (finalized schemas, shipped binaries) you *cannot* rely on rollback, so invest in progressive delivery to avoid needing it. Both are true — at different layers. Progressive delivery shrinks the set of changes that require the expensive kinds of rollback (see [progressive-delivery.md](./progressive-delivery.md)), and rehearsed rollback capability covers the ones that still need it.
## Gotchas
> **Gotcha — rollback that was never rehearsed:** The rollback that has not been run in months will fail at the worst moment — wrong artifact retention, broken automation, incompatible versions. Rehearse on a schedule, not under fire.
> **Gotcha — rolling back code past a destructive migration:** Rolling the binary back after finalization of a schema change leaves old code on a new schema. Verify the migration phase before any rollback touches a stateful layer.
> **Gotcha — cold caches after stateless rollback:** Redeploying the prior artifact without warming its caches trades one incident for a thundering-herd latency spike. Warm, then release traffic.
> **Gotcha — consumers rolling forward past a reverted producer:** If a producer rolls back and a consumer keeps calling the new API, you get production errors from a service you did not change. Roll consumers back first; enforce N-1 compatibility.
> **Gotcha — no version-labeled metrics:** Without per-version error/latency breakdowns you cannot tell *which* release is misbehaving in aggregate monitoring. Version labels on metrics are the prerequisite for both canary analysis and rollback verification.
> **Gotcha — anti-downgrade vs rollback freedom:** Signed firmware with anti-rollback protection blocks installing known-vulnerable images — but also blocks your rollback. Resolve the tension explicitly in the update architecture, not during an incident.
> **Gotcha — rolling back the wrong release:** Without version-labeled metrics, an error spike can be attributed to the latest release when the culprit is two releases back. Verify attribution from per-version SLIs *before* rolling back; rolling back the wrong version while the real offender stays deployed doubles the incident.
## Sources and Further Reading
- [Google Cloud — Reliable Releases and Rollbacks (CRE Life Lessons)](https://cloud.google.com/blog/products/gcp/reliable-releases-and-rollbacks-cre-life-lessons)
- [Google SRE Workbook — Canarying Releases (ch. 16)](https://sre.google/workbook/canarying-releases/)
- [Google SRE Book — Release Engineering (ch. 8)](https://sre.google/sre-book/release-engineering/)
- [Martin Fowler — Parallel Change](https://martinfowler.com/bliki/ParallelChange.html)
- [Martin Fowler — Evolutionary Database Design](https://martinfowler.com/articles/evodb.html)
- [Bitwarden — Evolutionary Database Design (production engineering docs)](https://contributing.bitwarden.com/contributing/database-migrations/edd/)
- [iOS-Factor — Rollbacks](https://ios-factor.com/rollbacks)
- [Redstone OTA — Anti-Bricking OTA: Failure Recovery & Safe-Fail Design](https://www.redstoneota.com/anti-bricking-ota-failure-recovery-safe-fail-design/)
references/skills-competency-model.md
# The Release Engineer's Skills and Competency Model
This file is the competency map behind [role-and-career.md](./role-and-career.md): the mastered skills of a release engineer, organized into a **technical core** (baseline for every level), **technical advanced/differentiating** (the sharp edge at senior+), and **professional** skills (the true gate between senior and staff). Each skill lists what mastery looks like and where it shows up in leveling evidence.
## Technical Core
These are the baseline skills. A release engineer at any level is expected to be competent, not merely aware, in each.
### CI/CD System Design
Designing and maintaining build/test/deploy pipelines end to end: stage topology, artifact promotion between environments, caching and parallelization, and failure isolation. Mastery means you can explain the trade-offs of a pipeline design (what runs on PR vs. merge vs. schedule), tune it (cache keys, parallelism, runner sizing), and reduce mean-time-to-green without weakening gates.
### Build and Hermeticity
**Hermetic builds** are the release engineer's core guarantee: the same revision and inputs produce the same artifact regardless of the machine that builds it. This requires pinned toolchains, locked dependencies, and reproducible packaging. Mastery includes knowing what breaks hermeticity (network fetches at build time, timestamps, machine-specific paths, nondeterministic ordering) and how to enforce it (containerized builds, Bazel/Nix, remote execution, build verification).
### Artifact and Registry Management
Versioning, packaging, signing, and promoting immutable artifacts through a registry. Mastery covers immutability and movable promotion pointers (a `latest` or `production` label that points at an immutable version), retention and cleanup policies, registry topology (proxy/mirror for supply-chain hygiene), and SBOM attachment. The guiding rule: **build once, promote many** — never rebuild per environment.
### Versioning and Changelogs
SemVer (with its rules, `0.x` caveats, and build metadata), CalVer for time-based products, and Conventional Commits as the machine-readable history that drives automated bumps and changelogs. Mastery means you can pick the right versioning scheme per artifact type (SemVer for libraries/APIs, CalVer for applications with time-based releases) and operate automated tooling (semantic-release, release-please, changesets, git-cliff) without losing the human changelog (Keep a Changelog) that customers read.
### Scripting and Automation
Python, Bash, and often Go for pipeline glue, CLIs, and release tooling. Mastery includes writing idempotent, testable scripts; handling exit codes and structured output; and knowing when a script belongs in the pipeline versus in a proper tool.
### Containers and Kubernetes
Image builds, registry operations, and Kubernetes rollout mechanics (Deployments, rollouts, probes, HPA). Critical in cloud-native organizations; important everywhere else. Mastery includes multi-arch builds, image signing, and understanding how K8s-native deployment strategies (rolling, canary via Rollouts/Flagger) interact with the release pipeline.
### Observability
Pipeline and deployment metrics, logs, traces, and dashboards used both to *verify releases* (release markers, deploy annotations, smoke checks) and to *debug the pipeline itself* (structured logs with correlation IDs, first-failure triage). Mastery means a release engineer can tell, minutes after a deploy, whether it improved or degraded the service.
### Configuration Management and Cloud
IaC (Terraform/OpenTofu, Helm, Kustomize), configuration-as-code, and environment parity. Mastery includes treating config like code — versioned, reviewed, and promoted with the artifact — so config drift cannot silently fork environments. See [release-operations-and-triage.md](./release-operations-and-triage.md) for the drift failure modes.
### Observable Mastery of the Technical Core
Skills lists are hard to evaluate; observable behaviors are not. Use these probes when assessing (self or others):
| Skill | Novice tells | Mastery tells |
|-------|--------------|---------------|
| CI/CD design | Rebuilds the pipeline for every new project; cannot explain cache invalidation | Can articulate stage topology trade-offs, tune caching/parallelism, and reduce MTG without weakening gates |
| Hermeticity | "Works on my machine" is a recurring excuse | Reproduces any artifact from a revision pin; can bisect which build input broke reproducibility |
| Artifact/registry | Deploys whatever the latest build produced; mutates tags | Enforces immutability, uses movable promotion labels, and can answer "what exactly is in prod, from which commit?" |
| Versioning | Manual version bumps, changelogs written from memory at release time | Versioning is automated from commit history; changelog is a byproduct of the process |
| Scripting | One-off imperative scripts with copy-paste errors | Idempotent, tested, flag-driven CLIs with structured output |
| Containers/K8s | Treats images as opaque artifacts | Can explain rollout mechanics, probe failures, and image signing end to end |
| Observability | Deploys then hopes | Every deploy creates a marker; can judge deploy impact from dashboards within minutes |
| Config management | Edits prod config by hand in a console | Config is versioned, reviewed, and promoted alongside the artifact; drift is detected, not discovered |
## Technical Advanced / Differentiating
These skills separate a competent pipeline operator from a senior+ release engineer. They are the differentiators recruiters and leveling panels actually look for.
### Software Supply-Chain Security
The sharpest modern differentiator: SBOM generation (Syft, Trivy, CycloneDX/SPDX), artifact signing (Cosign/sigstore, including keyless OIDC-based signing), provenance attestations, SLSA level mapping, and dependency trust decisions. Mastery means the supply chain is *verifiable end to end* — every artifact in production can be traced to source, build, and signer, and the pipeline itself verifies provenance (rather than trusting the registry). Regulatory pressure (US EO 14028, EU Cyber Resilience Act) is making this table stakes for enterprise work. See [supply-chain-security.md](./supply-chain-security.md).
### Policy-as-Code
Encoding release policy (who may promote, what gates block, what risk tier applies) in executable form — OPA/Gatekeeper/Conftest and pipeline policy engines — instead of in human approval meetings. Mastery means risk-tiered policy embedded in automation: low-risk changes flow automatically, high-risk changes route to named approvers, and exceptions are time-bound and audited. This is the technical heart of "risk-based, not bureaucracy-based" release governance.
### Large-Scale CI Optimization
Cache architecture, remote execution, parallelization, and runner-fleet management at the point where naive pipelines stop scaling: tens of thousands of builds, monorepo-wide impact analysis, and build-performance engineering. Mastery includes the economics (managed compute vs. engineer time), queue observability, and treating runners as cattle.
### Microservice Release Architecture
Coordinating releases across many interdependent services: dependency direction, N-1 compatibility between versions, contract testing, dependency-aware rollback ordering, and coordinated multi-service rollouts. Mastery means you can answer "which services can ship together, in what order, and what do we revert first if it goes wrong?" See [rollback-and-recovery.md](./rollback-and-recovery.md).
### Progressive-Delivery Engineering
Canary, blue-green, rings/cohorts, percentage rollouts, traffic shadowing, feature-flag-driven release, and **metric-gated auto-rollback**. Mastery means deploy and release are decoupled as a matter of architecture, not ceremony: the pipeline ships continuously, and exposure to users is controlled by flags and traffic shaping with automated evaluation. See [progressive-delivery.md](./progressive-delivery.md) and [feature-flag-lifecycle.md](./feature-flag-lifecycle.md).
## Professional Skills
The senior → staff jump is explicitly *not* about more delivery; it is gated on these capabilities.
| Skill | What mastery looks like |
|-------|-------------------------|
| **Communication with developers, ops, and stakeholders** | Crisp release notes, change summaries, standards docs, and dashboards; translating between product urgency and release risk |
| **Change management** | Running release trains, branch cuts, and go/no-go; turning approval into evidence-based, risk-tiered decisions |
| **Incident leadership** | Operational calm; decisive rollback guidance; structured triage and comms under time pressure |
| **Negotiation** | Aligning SRE, security, and product on acceptable risk; defending "stop the line" decisions; trading guardrails for team autonomy |
| **Teaching and mentoring** | Office hours, pairing, and "paved road" design that makes teams self-sufficient rather than ticket-dependent |
| **Systems thinking** | Connecting pipeline failures to upstream causes — test strategy, dependency churn, ownership gaps — and fixing the durable cause |
| **Empathy for developers and operators** | Designing pipelines that respect both the dev's flow and the on-call engineer's night; treating developer experience as a release requirement |
| **Influence without authority** | RFCs, data-driven persuasion, stakeholder alignment — the #1 staff+ essential |
| **Data literacy and storytelling** | Tying improvements to toil and incident reduction to secure investment and adoption |
**Growing the professional skills** is deliberate practice, not personality: influence-without-authority grows by writing RFCs that change decisions (start small: propose a pipeline standard, measure adoption); incident leadership grows by taking the triage lead in rehearsed game days before real incidents; negotiation grows by running the go/no-go meeting yourself with a pre-written decision framework; teaching grows by running office hours and recording what questions recur (those questions are your backlog).
## How the Model Is Used
The competency model is not an academic taxonomy — it is the operating manual for three concrete artifacts:
- **Hiring rubrics.** Score candidates against the technical core (all levels) plus the differentiating skills (senior+) with the observable-mastery probes above; weight professional skills heavily for staff+ candidates, since the senior→staff gate is primarily non-technical.
- **Promotion documents.** Structure leveling packets as scope + evidence + skills: which scope you now own (project/product/org), which evidence proves it (adoption, DORA delta, standards), and which skills you demonstrably mastered (with artifacts, not adjectives).
- **Team composition.** A healthy release/platform team mixes profiles: operators (technical core depth), builders (pipeline/platform construction), and diplomats (professional skills) — plus at least one person with real supply-chain security depth, the current differentiator. Teams that hire only "pipeline coders" staff up at senior and stall at staff.
## Mapping Skills to Scope
| Skill cluster | Senior (project) | Staff (product/org) | Principal (org/enterprise) |
|---------------|------------------|---------------------|----------------------------|
| CI/CD design, hermeticity, artifact/registry | Masters them on one product's pipeline | Encodes them as reusable golden pipelines | Sets the org-wide pipeline architecture |
| Versioning/changelogs, scripting, containers/K8s | Operates daily | Standardizes patterns across stacks | Establishes versioning policy org-wide |
| Observability, config management | Builds release dashboards, keeps parity on own product | Defines shared release-observability standards | Owns release governance and audit evidence |
| Supply-chain security, policy-as-code | Signs and SBOMs own artifacts | Ships org-wide signing/SBOM/provenance platforms | Aligns supply-chain controls with regulation and audit |
| Large-scale CI optimization | Optimizes own pipeline | Runs the shared fleet and caching strategy | Owns CI/release capacity economics |
| Microservice release architecture | Runs one service's rollouts safely | Defines dependency-aware release patterns | Sets coordinated multi-service rollout doctrine |
| Progressive delivery | Implements canary/flags for own product | Makes progressive delivery the default org-wide | Defines risk-based release policy and exception governance |
| Professional skills | Communicates well within the team | Leads cross-team initiatives without authority | Negotiates at the leadership/audit level |
## How to Evaluate Growth: Evidence-Based Leveling
Leveling is a portfolio of evidence, not a skills checklist. For each level boundary, collect artifacts that prove the *scope* of impact:
- **Senior evidence:** pipeline designs and runbooks you own; measured improvements (mean-time-to-green down, CFR down, flaky rate down) on your product; release incidents you triaged and root-caused.
- **Staff evidence:** golden pipelines with adoption numbers (e.g., 60–80% of services), standards docs with measurable uptake, DORA improvements you caused across teams, cross-team initiatives you led, teams that became self-sufficient (platform ticket volume down).
- **Principal evidence:** org-wide standards and governance model, risk-tiered policy embedded in automation, supply-chain controls at scale (SBOM/signing/provenance coverage percentages), a multi-year roadmap with adoption paths per maturity level, and durable reduction in org-wide CFR/toil/incidents.
**Build the portfolio deliberately, and review it on a cadence.** Keep a living leveling document updated each quarter: (1) *scope statement* — which organizational unit's release outcomes you own; (2) *artifacts* — links to pipelines, standards docs, RFCs, and adoption metrics; (3) *system outcomes* — DORA and pipeline-health before/after for changes you caused; (4) *influence evidence* — cross-team initiatives, mentorships, and decisions you changed with data. The portfolio answers the question "what changed because you existed?" with receipts, not adjectives.
**Skill-development paths** for the differentiators: supply-chain security (climb the SLSA levels on your own org's builds — SBOM → signed artifacts → provenance attestations → policy verification), policy-as-code (encode your own approval matrix into OPA/Conftest), large-scale CI (own the caching/remote-execution strategy for the fleet), microservice release architecture (lead a dependency-aware multi-service rollout), and progressive delivery (make canary + flags the default on one product, then org-wide).
Self-assessment questions worth asking each cycle: *How many teams can ship without talking to me? What percentage of services use the patterns I published? What would break if I were hit by a bus — and who owns the bus-factor mitigation? Which of my last ten improvements were leverage, and which were just more delivery?*
> **Gotcha — "release is the product":** The identity that distinguishes strong release engineers comes from the IEEE/CMU roundtable: **"where others see features, we see release challenges."** If you evaluate yourself only on feature-delivery output, you will undershoot at staff and misread the job at every level. The release process, its reliability, and its safety *are* the product a release engineer builds — measure yourself against the delivery system's outcomes, not against shipped features.
> **Gotcha — skills lists are not leveling rubrics:** Competence in a skill does not earn a promotion; evidence of org-level *leverage* does. Two engineers can both master canary rollouts; only one can show that their rollout framework cut org-wide CFR by half. Collect the evidence, not the checkboxes.
## Sources and Further Reading
- [Google SRE Book — Release Engineering (ch. 8)](https://sre.google/sre-book/release-engineering/)
- [The Practice and Future of Release Engineering (IEEE Software / CMU SEI)](https://www.infoq.com/articles/practice-and-future-of-release-engineering/)
- [Staff Release Engineer Role Blueprint (DevOps School)](https://www.devopsschool.com/blog/staff-release-engineer-role-blueprint-responsibilities-skills-kpis-and-career-path/)
- [Principal Release Engineer Role Blueprint (DevOps School)](https://www.devopsschool.com/blog/principal-release-engineer-role-blueprint-responsibilities-skills-kpis-and-career-path/)
- [Staff Engineer Archetypes (Will Larson)](https://lethain.com/staff-engineer-archetypes/)
- [Software Engineer Career Levels (End of Line Blog)](https://www.endoflineblog.com/software-engineer-career-levels)
- [SLSA Specification v1.0](https://slsa.dev/spec/v1.0/)
references/supply-chain-security.md
# Supply-Chain Security in Release Engineering
Supply-chain security is **release engineering** — not a separate security discipline bolted on afterwards. The artifact your pipeline builds is a bundle of provenance questions: what source produced it, what dependencies it contains, who signed it, and how a consumer verifies any of that. This reference covers SLSA levels, SBOM generation, software signing, dependency-update automation, registries as a security boundary, regulatory drivers, provenance attestations, and the incident playbook for a compromised dependency in a shipped release. For the audit-evidence side of these same artifacts, see [change-governance-and-compliance.md](./change-governance-and-compliance.md).
## SLSA Levels
**SLSA** (Supply-chain Levels for Software Artifacts, v1.0) is a graduated framework that answers: *how much can I trust that this artifact was built from this source by this process?* It has a build track (levels 1–3) and a source track, with **provenance attestations** as the core mechanism — machine-readable statements (in-toto format) recording source repo, commit, build workflow, and builder identity.
| Level | What it proves | Practical meaning |
|-------|----------------|-------------------|
| **L1** | Provenance exists | Documentation-only; the statement is recorded but not strongly protected |
| **L2** | Provenance is generated by a hosted build platform and **signed** | An attestation exists that the artifact came from a specific build service; verifiable without shared secrets (sigstore keyless) |
| **L3** | The build platform **prevents tampering**; artifact traceable to a specific source commit and build environment | The builder isolates the build and signs provenance that cannot be forged by the build's own steps |
SLSA is a *ratchet*: you cannot skip to L3, and consumers can demand a minimum level from dependencies. For most teams, **L2 is achievable with default GitHub Actions/GitLab CI** (hosted, signed provenance via Sigstore and `slsa-github-generator`); L3 requires stronger build isolation. Higher levels (beyond L3) cover more of the supply chain (source integrity, dependency integrity) and are the direction of the framework's continued work.
### Reaching Each Level in Practice
| Target | Minimum work | Check that proves it |
|--------|--------------|----------------------|
| **L1** | Generate a provenance statement in the pipeline | Provenance file exists for the artifact |
| **L2** | Use a hosted builder's signed provenance (slsa-github-generator, OIDC) | `cosign verify-attestation` succeeds against the expected workflow identity |
| **L3** | Isolate the build (ephemeral, no ambient creds), enforce source pins | Provenance builder ID + hermetic build inputs match policy; no untrusted steps can alter the attestation |
An **OpenSSF Scorecard** check on each direct dependency (CI presence, code review, pinned deps, fuzzing) is a cheap complement: it grades the *source-level* hygiene of the projects you depend on, which SLSA build-track provenance does not cover.
> **Gotcha — Provenance is only as good as its builder:** Provenance signed by a self-hosted runner you do not control, or by a workflow that could be modified by untrusted PRs, proves little. This is why SLSA levels and the "pipeline definitions protected from the code they process" principle in [change-governance-and-compliance.md](./change-governance-and-compliance.md) go together.
## SBOM Generation
A **Software Bill of Materials** is an inventory of the components in an artifact — names, versions, licenses, and (ideally) dependency relationships — in a standardized format (CycloneDX or SPDX). It is the foundation for vulnerability queries, license compliance, and incident response.
### Tooling
| Tool | Strength | Notes |
|------|----------|-------|
| **Syft** (Anchore) | Deepest cataloger coverage; best-in-class binary analysis (Go, Rust) | Generates SPDX + CycloneDX; does **not** scan for vulnerabilities — pair with Grype |
| **Trivy** (Aqua) | All-in-one: SBOM + vulnerability + misconfig + secrets scanning | Broadest integration (GitHub Action, Kubernetes operator, SARIF); depth slightly lower than Syft in edge cases |
| **CycloneDX CLI** | Build-time generation, SBOM merge/diff/validate, native VEX | Most accurate build-time resolution; requires per-language plugin setup; good for merging per-package SBOMs in monorepos |
### CycloneDX vs SPDX
| Dimension | CycloneDX | SPDX |
|-----------|-----------|------|
| Orientation | Security-focused (vulnerability, VEX, attestation data models) | License/legal-focused; Linux ecosystem heritage |
| Current | 1.6 | 3.0 |
| Regulatory pick | Explicitly accepted for EU CRA (alongside SPDX) | The other explicitly accepted format (SPDX 2.3 per BSI TR-03183-2) |
| Best when | You need vulnerability-matching and security tooling | You need license compliance and legal review |
### Build-Time vs Runtime Accuracy
Generate the SBOM **at build time** from the actual dependency resolution (lockfiles, package manifests, resolved module graphs), not by scanning a deployed container. Build-time SBOMs reflect what was linked; runtime scans can misattribute layers, miss stripped binaries, or include build-only tooling. CycloneDX CLI's build-time resolution is the most accurate of the three tools for this reason. Attach the SBOM as an artifact or attestation beside the image, and record its reference in the deployment record — this is the artifact that makes [change-governance-and-compliance.md](./change-governance-and-compliance.md)'s evidence chain supply-chain-aware.
### SBOM Quality
An SBOM is only as good as its **depth and fidelity**. Minimum expectations for an SBOM that will be queried in an incident:
- **Component identity** that survives across ecosystems: purl (package URL) or similar canonical identifiers, not display names.
- **Full transitive coverage** — the EU CRA explicitly requires "all components, including dependencies"; a first-level-only SBOM will miss the vulnerable transitive package.
- **Relationship data** (component A depends on B), not just a flat list — blast-radius analysis needs the graph.
- **Regeneration on every release** — an SBOM generated once and reused across versions is stale by definition. Make it a pipeline step that fails the build if missing.
A minimal CycloneDX component entry shows the fidelity expected — canonical identity, version, and purl are the fields an incident query keys on:
```json
{
"type": "library",
"name": "requests",
"version": "2.31.0",
"purl": "pkg:pypi/requests@2.31.0",
"licenses": [ { "license": { "id": "Apache-2.0" } } ]
}
```
Where a known vulnerability exists in a component with no fix available, attach a **VEX (Vulnerability Exploitability eXchange)** statement recording the exploitability assessment — it is the difference between "this component is present" and "this component is known-vulnerable and we have decided how to treat it."
## Software Signing
Signing answers "**who vouches for this artifact and its metadata?**" and enables verification at deploy time.
| Mechanism | Status | Model |
|-----------|--------|-------|
| **Cosign / sigstore** | Current standard for OCI signing | **Keyless signing** via OIDC (no long-lived private keys); signatures + attestations published to a transparency log (Rekor); verifiable without shared secrets |
| **Notary / Docker Content Trust** | Legacy, declining | Key-based image signing; being superseded by sigstore/Cosign; Notary v2 (notation) retains some traction |
| **Code signing** | Complementary | Platform trust (macOS notarization, Windows Authenticode, mobile provisioning); proves publisher identity to end-user OSes |
| **Verification at deploy time** | Practice | Gate deployments on signature + attestation verification: reject unsigned images, mismatched digests, or provenance from unapproved builders (admission controllers like Kyverno, cosign verify in the pipeline) |
A minimal signing workflow: (1) build the artifact and generate the SBOM; (2) sign the artifact (image/package) with Cosign; (3) sign the SBOM and the provenance attestation; (4) push artifact + signatures + attestations to the registry; (5) at deploy time, **verify** the signature, the attestation policy (workflow identity, source repo), and the digest before the artifact can run. Steps 2–3 are where "keyless" matters most: the build platform's OIDC identity is the signer, so there is no human-held key to leak and no key rotation treadmill.
> **Gotcha — Signing without verifying:** A pipeline that signs images but never verifies signatures at deploy time has produced ceremony, not security. The verification step (admission controller or pipeline gate) is what converts signing into a control; it belongs in the same pipeline stage as digest pinning.
## Dependency Update Automation
Keeping dependencies current is both a vulnerability mitigation and a release-process requirement (a stale dependency matrix makes backports and emergency respins expensive):
| Dimension | Renovate (Mend) | Dependabot (GitHub) |
|-----------|-----------------|---------------------|
| Platforms | GitHub, GitLab, Bitbucket, Azure DevOps | GitHub only |
| Package managers | 90+ | Broad but narrower, GitHub-first |
| Monorepo awareness | Grouping, scheduling, auto-merge, dependency dashboard | Basic per-repo PRs |
| Complexity | Higher configuration surface | Zero-config |
| Fit | Complex monorepos, multi-platform orgs | Simple GitHub-hosted repos |
Both generate per-repo PRs. In a polyrepo with 50 services on a shared library, a breaking change triggers 50 PRs — automation generates them, but humans still review and merge (see [monorepo-polyrepo-release.md](./monorepo-polyrepo-release.md) for the coordination angle). Set update policy deliberately: security-patch auto-merge (fast), minor bumps in batches (reviewed), majors as scheduled upgrades (planned breaking-change handling). Pin everything that can be pinned — base images by digest, actions by SHA, dependencies via lockfiles.
## Registries as a Security Boundary
The registry is where the supply chain meets the release pipeline — treat it as a security control, not a file server:
- **Digest-pinned base images:** reference base images by digest, not mutable `latest` tags, so a republished tag cannot silently change what your builds run on. This mirrors the artifact-immutability rules in [versioning-and-artifacts.md](./versioning-and-artifacts.md).
- **Registry scanning:** scan images at push time (Harbor+Trivy, ECR scanning, JFrog Xray) and block or quarantine images failing policy (critical CVEs, unknown provenance, unsigned).
- **Proxy/mirror for hermetic builds:** a registry proxy/mirror (or an internal mirror with allowlisting) keeps builds hermetic — they resolve from a controlled source rather than the public internet, which is both a reproducibility and a poisoning-defense measure (a compromised upstream package can be blocked centrally).
- **RBAC and audit:** registry access is scoped per team/service account; every push is logged with its pipeline identity so a malicious artifact can be traced to its build (which is exactly the trace [change-governance-and-compliance.md](./change-governance-and-compliance.md) demands).
## Regulatory Drivers
- **US EO 14028 (2021):** SBOM required for software sold to the US federal government; drove SBOM generation into the mainstream of build pipelines.
- **EU Cyber Resilience Act (CRA):** SBOM mandatory for digital products sold in the EU — vulnerability reporting by **11 September 2026**, full compliance (including SBOM) by **11 December 2027**; fines up to €15M or 2.5% of global annual turnover. Requires SBOMs covering all components including transitive dependencies, plus continuous vulnerability monitoring.
- **Direction of travel:** SBOMs move from optional to a build-step requirement; provenance attestations (SLSA + in-toto) connect source → build → artifact; dependency pinning (SHA-pinned actions, lockfiles) becomes standard practice; OpenSSF Scorecard provides a health score for OSS dependencies.
## Provenance Attestations and Build Integrity
Provenance is the connective tissue of the supply chain: **in-toto attestations** (signed statements, typically SLSA provenance) link the source commit → build workflow → artifact digest → SBOM. Build integrity underneath them requires **hermetic, reproducible builds**: same source revision and inputs yield identical outputs, independent of build-host state (this is also what makes rollback trustworthy — see [rollback-and-recovery.md](./rollback-and-recovery.md)). Practical build-time checks: pin the toolchain, disable network access during the build where possible, run builds in isolated ephemeral environments, and sign the provenance from the build platform, not from a step the build itself controls.
## When a Dependency in a Shipped Release Is Compromised
The incident playbook when a dependency in a released artifact is found compromised:
1. **Query the SBOM:** find every artifact that contains the affected component (name + version range). This is the moment an accurate build-time SBOM pays for itself — scanning containers post-hoc is slower and less reliable. Concretely: query the SBOM store by purl or component name, list every artifact digest that embeds the vulnerable version, and map each digest to the releases/environments it reached via the deployment records.
2. **Assess blast radius:** which releases, environments, and consumers are affected; did the vulnerable version reach production; does the compromise require an active exploit path or mere presence? Distinguish *shipped* (in production) from *published* (in the registry) — the registry may hold vulnerable versions that never ran anywhere.
3. **Determine remediation per artifact:** upgrade the dependency (roll-forward with a new build) vs. emergency respin of the affected release vs. flag-gate the vulnerable code path (see [feature-flag-lifecycle.md](./feature-flag-lifecycle.md) — but remember flags cannot undo state).
4. **Respin through the pipeline:** publish a fixed build with a new digest, update promotion pointers, and redeploy — through the *normal* pipeline, not a side channel, so the evidence chain stays intact (see [change-governance-and-compliance.md](./change-governance-and-compliance.md)).
5. **Enforce forward:** add the compromised version to a denylist/policy (block at the registry or dependency-update gate), verify signatures on everything republished, and run a postmortem on why the vulnerable version shipped and whether the SBOM matched reality.
> **Gotcha — The legacy-release blind spot:** Old releases are where compromised dependencies bite hardest: their SBOMs were never generated, their builds are unreproducible, and nobody remembers the promotion path. For anything long-lived (mobile apps with store-review lag, embedded/OT devices, LTS products), retroactively generate SBOMs and keep old builds reproducible — the CRA's December 2027 deadline will make this mandatory for EU-market software.
## Where the Controls Live
Supply-chain controls belong at a specific layer; putting them in the wrong layer is the usual failure. Map each control to where it can actually fire:
| Layer | Controls | Enforced when |
|-------|----------|---------------|
| **Source** | Branch protection, CODEOWNERS, signed commits, OpenSSF Scorecard checks on deps | Commit/PR time |
| **Build** | Lockfiles, digest-pinned base images, hermetic build, SBOM generation, signing | Pipeline time |
| **Registry** | Push-time scanning, allowlist/denylist policy, provenance verification on push, RBAC, audit | Push time |
| **Deploy** | Signature + attestation verification (admission controller), digest-pinned manifests, image age policy | Deploy time |
| **Runtime** | Continuous vulnerability monitoring, drift detection, quarantine of running vulnerable images | Post-deploy |
A control that only exists at one layer (e.g., scanning only at the registry) leaves the other layers unguarded — the respin and the admission gate both need to verify, not just the scanner.
## Gotchas
- **SBOM without verification is inventory, not security.** A list of components is useful for querying; the control comes from scanning those components continuously and gating on findings.
- **Keyless is not key-free.** Sigstore keyless signing still has a trust model — verify that the OIDC identity (workflow, repo, environment) in the signature is actually the one you approve, via policy rules in the verifier.
- **Scanning the final image misses the build.** Build-time SBOMs capture the resolved dependency graph; post-push scans may include or exclude layers differently. Publish the build-time SBOM as the canonical artifact and treat registry scans as a second opinion.
- **Dependency bots are not a policy.** Auto-merge ranges are how a compromised patch propagates fastest. Gate automated merges behind scan results and allowlist behavior.
- **Supply-chain incidents are release incidents.** The respin is an emergency release; run it through the break-glass/emergency path with the same evidence discipline, not as an unrecorded hotfix. See [change-governance-and-compliance.md](./change-governance-and-compliance.md).
## Sources and Further Reading
- [SLSA Specification v1.0](https://slsa.dev/spec/v1.0/) — the levels framework and provenance model
- [SBOM Tools Compared: Syft vs Trivy vs CycloneDX CLI](https://secure-pipelines.com/ci-cd-security/sbom-tools-compared-syft-trivy-cyclonedx-cli/) — tool-by-tool comparison with build-time guidance
- [EU CRA SBOM requirements (Anchore)](https://anchore.com/sbom/eu-cra/) — CRA deadlines, formats, and enforcement
- [Renovate vs Dependabot](https://docs.renovatebot.com/bot-comparison/) — dependency-update automation comparison
- [CycloneDX](https://cyclonedx.org/) and [SPDX](https://spdx.dev/) — the two SBOM formats
- [sigstore / Cosign](https://docs.sigstore.dev/) — keyless signing and verification
- [Google SRE Book — Release Engineering (Chapter 8)](https://sre.google/sre-book/release-engineering/) — hermetic builds and signed artifacts (CC BY-NC-ND: cite, do not copy)
references/toolchain-landscape.md
# The Release Engineering Toolchain Landscape (2025–2026)
A practitioner's catalog of the tools release engineers actually use, organized by category, with current status, alternatives, and selection guidance. Adoption figures come from the JetBrains State of Developer Ecosystem 2025 survey (cited below) and vendor/CNCF documentation; treat percentages as directional, not precise. For a one-screen cheat sheet, see [../assets/release-toolchain-cheatsheet.md](../assets/release-toolchain-cheatsheet.md).
## CI/CD Orchestration
| Tool | Status (2025–2026) | Best for | Alternatives |
|------|--------------------|----------|--------------|
| **GitHub Actions** | Leader: ~33% org adoption; 20K+ marketplace actions; Copilot integration | Default for GitHub-hosted code and new projects | CircleCI, Buildkite (speed); GitLab CI (integrated security) |
| **Jenkins** | ~28% org adoption; actively maintained; biggest CloudBees update in a decade | Air-gapped, regulated, legacy environments; Groovy pipelines | GitHub Actions Importer supports migration (70–90% accuracy) |
| **GitLab CI/CD** | ~19% org adoption; leader in integrated DevSecOps (SAST/DAST/dependency scanning in Ultimate); native DORA metrics | Teams wanting a single-platform VCS+CI+security | GitHub Actions, Azure DevOps |
| **CircleCI** | Active; ML-powered test splitting cuts build times 50–70% (vendor-reported) | Test-heavy pipelines (Ruby, Python, JS) | GitHub Actions, Buildkite |
| **Buildkite** | Active; hybrid model (managed control plane + self-hosted agents) | Enterprises wanting control without full self-hosting | GitHub Actions self-hosted runners |
| **Azure DevOps Pipelines** | Active; deep Azure/.NET integration | Microsoft-stack enterprises | GitHub Actions (increasingly preferred even at Microsoft) |
| **Tekton** | Active CNCF project; K8s-native building blocks | Platform teams building *custom* CI on Kubernetes | Jenkins X (built on Tekton), Dagger |
| **Dagger** | Active; pipelines-as-code in Go/Python/TS; containerized execution; absorbing Earthly users | Portable, testable pipeline logic that runs on any CI | Earthly (winding down), Bazel for builds |
**Trends:** pipeline-as-code in version control is non-negotiable; self-hosted runners for cost/control with cache optimization as the top lever; DORA dashboards built into platforms (GitLab native, Datadog, CircleCI); AI entering pipeline authoring and debugging. Note that ~18% of organizations still use no CI/CD at all — the adoption gap is real and is the first thing to fix in low performers.
> **Gotcha — the "era of defaulting to Jenkins for everything is over":** Managed CI compute is cheaper than engineer time for most teams, and Jenkins' strength (total self-hosting control) is its weakness (fleet maintenance, plugin sprawl). Default to managed CI unless you have a hard reason (air-gap, regulation, existing investment) to self-host.
**Quick decision rule for CI/CD selection:**
| If you are... | Default choice | Reconsider when |
|---------------|----------------|-----------------|
| On GitHub, starting fresh | **GitHub Actions** | You need specialized test compute (CircleCI/Buildkite) or cross-repo pipeline portability (Dagger) |
| On GitLab, wanting one platform | **GitLab CI/CD** | You need the deepest multi-platform runner control |
| Air-gapped / regulated / legacy | **Jenkins** (or Buildkite hybrid) | Migration tools (GitHub Actions Importer) now cover 70–90% of pipelines |
| Building a custom K8s-native CI platform | **Tekton** or **Dagger** | Your team lacks the platform-engineering depth to operate the layer |
| .NET/Azure enterprise | **Azure DevOps Pipelines** | Your org is standardizing on GitHub anyway |
## Release Automation and Versioning
| Tool | What it automates | Model | Notes |
|------|-------------------|-------|-------|
| **semantic-release** | Version bump + changelog + publish + GitHub release/tag | Fully automated, no human gate | Mature plugin ecosystem; npm/JS-native but extensible |
| **release-please** | Release PRs with changelog + version bump | Auto-generates PR, **human merges** | Google's model; strong monorepo support; GitHub Action |
| **changesets** | Developers write changeset files; tooling aggregates into changelog + bump | Intentional, per-PR declaration | Popular in pnpm/yarn workspace monorepos |
| **release-drafter** | Drafts GitHub release notes from PR labels/titles | Notes only, no versioning | Simple; pairs with manual tagging |
| **git-cliff** | Generates CHANGELOG.md from git history (conventional commits) | Changelog only, highly configurable | Rust-based; language-agnostic |
The ecosystem is converging on the **"generate a release PR, human approves"** model (release-please) over fully automatic publishing — the human gate stays on the merge, not on the version math. Conventional Commits is the shared foundation underneath all of them. The three leading tools occupy a useful automation spectrum:
| Tool | Human gate | Who declares impact | Monorepo story | Best when |
|------|-----------|---------------------|----------------|-----------|
| **semantic-release** | None (publishes automatically) | The commit message | Via plugins | You trust your commit discipline completely and want zero-touch npm/package publishing |
| **release-please** | Yes — a release PR to merge | The commit message | Strong (manifest mode) | You want automated changelog + version math but a human to eyeball the release PR |
| **changesets** | Yes — per-PR changeset files | The developer, at PR time | Strong (pnpm/yarn workspaces) | You want intentional, developer-declared impact and grouped monorepo releases |
A common trap is reaching for semantic-release when the team's commit hygiene cannot support it — if `feat:`/`BREAKING CHANGE:` are not enforced at review time, the automation will silently produce wrong versions. Enforce Conventional Commits in CI before automating on top of it.
## Build Systems and Dependency Managers
- **Bazel** — hermetic, reproducible builds with remote execution and caching; the standard for large polyglot monorepos; steep learning curve. Alternatives for JS/polyglot: Nx, Turborepo, Pants, Buck2.
- **Nix** — declarative, reproducible system-level packaging; complements Bazel (Nix for system-level environments, Bazel for project builds); `nix develop` is displacing Docker for dev environments in some shops.
- **uv** — the rising Python package/project manager (Rust-based; 85K+ GitHub stars; 8–100x faster than pip per Astral). It is displacing pip/Poetry/pip-tools/pyenv as a single tool and can import existing Poetry projects. Poetry remains active but is being challenged.
- **Gradle** — JVM build system with build cache and configuration cache; preferred for new Android/JVM projects over Maven.
- **Lockfiles are universal:** package-lock.json, Cargo.lock, go.sum, uv.lock — every ecosystem now expects them. **Reproducible builds are a requirement, not a nice-to-have** — but note the caveat: an IEEE Software 2025 study found even Bazel-using OSS projects rarely achieve full hermeticity, so verify, don't assume.
## Artifact Registries
| Tool | Status | Best for |
|------|--------|----------|
| **JFrog Artifactory** | Market leader; 40+ formats incl. OCI; Xray for security | Enterprise single source of truth across formats; on-prem/multi-cloud |
| **Sonatype Nexus** | Active; OSS + Pro | Java-heavy enterprises; open-source alternative to Artifactory |
| **GitHub Container Registry (GHCR)** | Active; free tier; integrated with Actions | GitHub-native teams; displacing Docker Hub for OSS |
| **Docker Hub** | Active; anonymous-pull rate limits | Public default image distribution |
| **AWS ECR / Google Artifact Registry / Azure ACR** | Active; cloud-native | Cloud-locked teams; built-in scanning |
| **Harbor** | CNCF graduated; self-hosted OCI registry with Trivy scanning, RBAC, replication | Self-hosted registries with security built in |
**The convergence point is OCI:** containers, Helm charts, SBOMs, WASM, and even ML models are all moving to OCI format, so registry selection is increasingly a question of *where* your OCI artifacts live, not *which formats* you support. Registry-level security (scanning, SBOM attach, policy) is becoming the norm.
> **Gotcha — the registry is a security boundary and a DR liability:** A registry without backup/DR is a single point of failure for every release. If you use a checksum-deduplicated store (like Artifactory's), you must back up both the metadata database *and* the filestore — and the master/signing keys — or recovery is impossible. See [release-operations-and-triage.md](./release-operations-and-triage.md).
## Deployment and Progressive Delivery
- **Argo CD and Flux** — both CNCF graduated; the de facto standard for GitOps CD on Kubernetes. Argo CD: web UI, multi-cluster, RBAC/SSO; Flux: lighter, controller-based, CLI-driven, good for air-gapped. **GitOps is winning for K8s; push-based CD (pipeline deploys to targets) still dominates for VMs, serverless, and legacy.**
- **Argo Rollouts** — progressive delivery for Argo users (canary, blue-green, experiments, metric-based analysis); integrates with service meshes.
- **Flagger** — progressive delivery operator for Flux users; metric-based canary analysis with Istio/Linkerd/NGINX/etc.
- **Spinnaker** — **declining**: Netflix-originated, complex to operate, community-maintained; being displaced by Argo CD/Harness/GitOps patterns.
- **Harness** — enterprise CD with AI deployment verification and auto-rollback, feature flags, and policy/governance; overkill for simple setups.
- **Octopus Deploy** — strong in the .NET/Windows enterprise; multi-environment orchestration and runbooks; less relevant for cloud-native/K8s.
**Trends:** canary + blue-green with metric-gated auto-rollback is no longer optional for production services; progressive delivery is the default pattern, and feature flags decouple deploy from release (deploy continuously, release deliberately).
## Feature Flags
- **LaunchDarkly** — enterprise market leader; server-side + client-side SDKs; experimentation platform; expensive and proprietary.
- **Flagsmith, Unleash** — open-source, self-hostable flag platforms; Unleash has strong community and GitLab integration; both are OpenFeature-compatible.
- **ConfigCat** — budget-friendly managed flags with 10+ SDKs.
- **OpenFeature** (CNCF) — the vendor-neutral SDK standard; providers implement backends, so application code is not locked to a vendor. Adopt it if you want to avoid flag-tool lock-in.
See [feature-flag-lifecycle.md](./feature-flag-lifecycle.md) for the lifecycle discipline (naming, ownership, expiry, cleanup) that makes any flag tool safe.
## Release Observability
- **Release markers:** Sentry releases (error-to-release correlation, crash-free rate), Datadog deployment markers (version-tagged traces, DORA metrics product), Grafana annotations (deploy events on dashboards).
- **Metric gates:** Argo Rollouts/Flagger analysis, error-budget-based release gating (halt non-P0 releases when a service exceeds its 4-week error budget — the Google SRE pattern).
- **DORA dashboards** are now built into GitLab, Datadog, CircleCI, and Harness; manual metric tracking is being replaced by platform-native reporting. See [metrics-and-dora.md](./metrics-and-dora.md) for the exact definitions and the vendor-formula caveats.
The common pattern across every tool: **every deploy creates a marker**, and release health questions ("is the error rate up since 14:02?") are answered from version-tagged data rather than hunches.
| Tool | Marker type | Typical use |
|------|-------------|-------------|
| **Sentry releases** | Release versions on error events | Regression detection, crash-free rate, suspect commits |
| **Datadog deployments** | Version-tagged traces + dashboard markers | Performance impact per deploy; DORA product |
| **Grafana annotations** | Time-series deploy events | Visual correlation of deploys with metric changes |
| **Argo Rollouts / Flagger analysis** | Metric queries against rollout phases | Automated canary pass/fail and auto-rollback |
## Security Tooling
- **SBOM generation:** Syft (deepest cataloger coverage for binary analysis), Trivy (all-in-one: SBOM + vulnerability + misconfig + secrets), CycloneDX CLI (build-time accurate, monorepo merging, native VEX).
- **Signing:** Cosign/sigstore (keyless OIDC signing + Rekor transparency log — the standard for OCI); Notary/Notary v2 (legacy Docker content trust, declining).
- **Dependency updates:** Renovate (90+ package managers, multi-platform, grouping/scheduling) vs. Dependabot (GitHub-only, zero-config). Every active project should have one.
- **Standards:** SLSA v1.0 (build/source provenance levels), SPDX 3.0 / CycloneDX 1.6 formats, OpenSSF Scorecard.
> **Gotcha — SBOMs are now mandatory, not aspirational:** US EO 14028 requires SBOMs for software sold to the federal government, and the EU Cyber Resilience Act requires them for digital products with enforcement ramping 2026–2027. Treat SBOM generation as a build step and signing as a pipeline gate, not a nice-to-have. Full detail in [supply-chain-security.md](./supply-chain-security.md).
## Infrastructure
- **Terraform / OpenTofu** — Terraform remains the most widely used IaC tool but moved to the BSL license in 2023 (and HashiCorp was acquired by IBM), which drove the **OpenTofu** fork under the Linux Foundation; OpenTofu is compatible with Terraform providers/state and is gaining enterprise adoption.
- **Docker** — image building and local dev (BuildKit standard, multi-arch builds); **containerd/CRI-O** run production K8s; Podman as a rootless alternative.
- **Kubernetes** — CNCF graduated, the de facto orchestration standard; managed services (EKS, GKE, AKS) dominate.
- **Ephemeral preview environments** — per-PR environments (Vercel Previews, Railway, GitLab Review Apps, Harness CIE, and peers) are maturing into a standard practice: "every PR gets an environment," torn down on merge.
## How to Choose: A Selection Framework
1. **Start from your delivery model, not the tool catalog.** Trunk-based + continuous deployment → managed CI + GitOps + progressive delivery. Scheduled/versioned software → release trains + a release-automation tool + a branch-cut ceremony.
2. **Pick the CI your repo already lives in** unless you have a concrete reason not to (GitHub → Actions; GitLab → GitLab CI; legacy/air-gapped → Jenkins).
3. **Adopt OCI as the artifact lingua franca** — it keeps registry, signing, and SBOM choices interoperable.
4. **Buy progressive delivery and feature flags; build the release process.** Flags, canary analysis, and registries are commodity capabilities; your differentiation is the governance, gates, and runbooks wrapped around them.
5. **Optimize the bottleneck first:** cache strategy and runner economics for CI cost; SBOM/signing coverage for compliance; rollback time for incident risk.
6. **Revisit quarterly.** The toolchain moves fast: Spinnaker declined, Earthly wound down, uv displaced Poetry, OpenTofu rose — a tool picked three years ago may now be a maintenance liability.
> **Gotcha — dead and dying tools:** Do not recommend tools on momentum. **Earthly's container-native CI business wound down in 2025** (Dagger runs an official migration program); **Spinnaker** is in organic decline; **Notary** is legacy; **Poetry** is being displaced by uv for new Python work. Check maintenance status before recommending anything in this list's "declining" column.
## Trends
- **GitOps vs. push:** GitOps (Argo CD, Flux) is winning for Kubernetes; push-based CD remains necessary for non-K8s targets. The two coexist; treat Git as the reconciliation source of truth where it fits.
| Dimension | GitOps (Argo CD, Flux) | Push-based (Jenkins, CI pipelines, Harness) |
|-----------|------------------------|----------------------------------------------|
| Model | Git is the source of truth; controllers reconcile cluster state | Pipeline pushes artifacts/changes to targets |
| Strengths | Auditability, drift detection, self-healing, declarative | Simplicity for non-K8s targets; familiar mental model |
| Best for | Kubernetes-native workloads | VMs, serverless, multi-target, legacy |
| Trajectory | Winning for K8s; becoming the default | Still necessary for non-K8s; declining for K8s |
- **Platform engineering convergence:** Backstage (CNCF's #5 project by velocity) and internal developer platforms (IDPs from Harness, Humanitec, Port, Cortex) are absorbing CI/CD, IaC, observability, and service catalogs behind developer-facing abstractions — "deployment pipelines as product," with SLAs, docs, and support.
- **Ephemeral environments:** per-PR preview environments are becoming a standard practice ("every PR gets an environment," torn down on merge), replacing the cost and drift of idle permanent staging.
- **AI-assisted pipelines:** Copilot for workflow authoring, GitLab Duo for pipeline debugging, CircleCI ML test splitting, Harness AI deployment verification, and Dagger + LLM pipeline functions. Trajectory: from passive suggestions to active pipeline participation to autonomous remediation — see the tiered-autonomy model in [release-operations-and-triage.md](./release-operations-and-triage.md).
## Sources and Further Reading
- [Best CI/CD Tools in 2026 — What the Data Actually Shows (JetBrains 2025 data)](https://www.awsquality.com/best-ci-cd-tools-what-the-data-actually-shows/)
- [A Soft Landing for Earthly Users (Dagger)](https://dagger.io/blog/earthly-to-dagger-migration/)
- [SBOM Tools Compared: Syft vs Trivy vs CycloneDX CLI](https://secure-pipelines.com/ci-cd-security/sbom-tools-compared-syft-trivy-cyclonedx-cli/)
- [Argo CD vs Flux CD: Complete GitOps Comparison](https://devtron.ai/blog/gitops-tool-selection-argo-cd-or-flux-cd/)
- [SLSA Specification v1.0](https://slsa.dev/spec/v1.0/)
- [uv — Astral Docs](https://docs.astral.sh/uv/)
- [Renovate Bot Comparison (vs Dependabot)](https://docs.renovatebot.com/bot-comparison/)
- [Datadog DORA Metrics Documentation](https://docs.datadoghq.com/dora_metrics/)
references/versioning-and-artifacts.md
# Versioning and Artifacts
Version numbers are a **contract with every consumer** of your software — downstream teams, registries, release tooling, auditors, and users. Artifacts are the physical embodiment of a release: they must be **immutable once built**, **promotable** across environments, and **provably traceable** to the source and build that produced them. This reference covers version schemes (SemVer, CalVer), the commit conventions that drive automated bumps, changelog conventions, artifact immutability and promotion, naming/registry conventions, and provenance.
## Semantic Versioning (SemVer 2.0.0)
SemVer encodes API-compatibility intent in a three-part number: `MAJOR.MINOR.PATCH`, optionally followed by a `-prerelease` identifier and a `+build` metadata suffix.
| Component | Meaning | Example |
|-----------|---------|---------|
| **MAJOR** | Incompatible API change (consumers must change code) | `2.0.0` |
| **MINOR** | Backward-compatible addition of functionality | `1.4.0` |
| **PATCH** | Backward-compatible bug fix | `1.4.1` |
| **Prerelease** | Pre-release build, lower precedence than release | `1.4.0-rc.1` |
| **Build metadata** | Build-specific info; **ignored in precedence** | `1.4.0+20260801` |
### The Core Rules
- Once a version is published, its content **must not change**. Any modification requires a new, higher version.
- Version numbers must increment by the largest changed component: a MINOR change also resets PATCH to zero (`1.4.1` → `1.5.0`, not `1.5.1`); a MAJOR change resets MINOR and PATCH.
- Precedence is resolved left to right: `1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta < 1.0.0-rc.1 < 1.0.0`. Numeric identifiers compare numerically, alphanumeric identifiers lexically.
- Build metadata (`+sha.abc123`) is **not** part of precedence: `1.0.0+build1` and `1.0.0+build2` are equal in precedence, so two different builds of the same version are indistinguishable to dependency resolvers that sort by precedence.
> **Gotcha — Build metadata is invisible to precedence:** If you append a commit SHA as build metadata, semver-aware tools cannot tell two builds apart. Use prerelease identifiers (`1.0.0-alpha.1`) for ordered builds, or pin by **digest** (see Provenance) rather than by version string.
### Precedence in Practice
Precedence is a total order over releases and a partial order over prereleases of the same release:
| Version | Position |
|---------|----------|
| `1.0.0-alpha` | Earliest — fewer identifiers sorts before more (`alpha` < `alpha.1`) |
| `1.0.0-alpha.1` | |
| `1.0.0-beta` | |
| `1.0.0-rc.1` | |
| `1.0.0` | Latest — any prerelease sorts below its release |
Two rules that commonly surprise: numeric identifiers compare **numerically** (`alpha.10` > `alpha.9`), and a shorter identifier list sorts **below** a longer one sharing its prefix (`1.0.0-alpha` < `1.0.0-alpha.1`). Tooling such as `semver_check.py` in `scripts/` implements this ordering; get it wrong in your own sort and prerelease promotion logic will promote `1.0.0-rc.9` over `1.0.0-rc.10`.
### 0.x Semantics
`0.y.z` means **initial development**: the spec explicitly states that "anything may change at any time" and the public API should not be considered stable. Practical consequences:
- `0.1.0` → `0.2.0` usually signals breaking changes in pre-1.0 libraries — the `1.0.0` MAJOR convention is effectively deferred.
- This skill's default Release Please-compatible policy maps `feat` and breaking commits to MINOR in 0.x (`0.5.0` → `0.6.0`); fixes and other commit types remain PATCH (`0.5.0` → `0.5.1`). This keeps pre-1.0 release lines meaningful while preserving normal SemVer behavior at 1.0+.
- Consumers pinning `0.x` with caret ranges (`^0.3.1`) get **no automatic updates** in most package managers (npm, for example, treats caret on `0.x` as `>=0.3.1 <0.4.0`), which is exactly the behavior you want for a pre-stable API.
Declare a policy so both humans and automation agree on what a `0.x` bump means:
| Policy | Rule | Consequence |
|--------|------|-------------|
| **Strict** | Breaking changes bump MAJOR even in 0.x | Version jumps `0.3.0` → `1.0.0` early; signals commitment before the API is ready |
| **Deferred** (skill default) | Features and breaking changes bump MINOR until 1.0; fixes and other changes bump PATCH | `0.5.0` → `0.6.0` for `feat`, `0.5.0` → `0.5.1` for `fix`; 1.0 arrives when the API stabilizes |
| **Tooling-default** | Whatever your release tool computes from commits | Usually `feat`→MINOR; document that pre-1.0 MINOR may break |
> **Gotcha — The 0.x trap:** A 1.0.0 release is a promise about API stability. If your library has public consumers, treat 1.0.0 as a deliberate commitment — and conversely, do not stay in `0.x` forever because bumping to 1.0 feels risky; consumers already treat `0.x` as unstable either way.
### When SemVer Fits
SemVer is the right tool for **libraries, SDKs, and APIs** where consumers depend on compatibility contracts and need machine-readable signals for safe upgrades. It is a poor fit for applications or products that ship on a time cadence and whose "API" is a UI or an internal contract — see CalVer.
## Calendar Versioning (CalVer)
CalVer encodes **time** in the version, communicating freshness, support windows, and external-change-driven release schedules. Schemes combine date segments: `YYYY`, `YY`, `0M`, `MM`, `WW`, `DD` — plus an optional `.micro` or `.patch` suffix.
| Project | Scheme | Example |
|---------|--------|---------|
| Ubuntu | `YY.0M` | `26.04` |
| Twisted | `YY.MM.MICRO` | `24.7.0` |
| pip | `YY.MINOR.MICRO` | `24.2` |
| certifi | `YYYY.MM.DD` | `2025.01.01` |
| Stripe API | `YYYY-MM-DD` | `2025-07-15` |
The available date segments (per [CalVer](https://calver.org/)): `YYYY`/`YY` (year), `0M`/`MM` (month with/without zero-padding), `0W`/`WW` (ISO week), `0D`/`DD` (day with/without zero-padding) — combined with an optional `.micro`/`.patch` counter for multiple releases in the same period. Choosing `YYYY.MM.DD` buys maximum granularity but forces awkward micro-versions if you ever ship twice in a day; `YY.0M` (Ubuntu-style) is the common enterprise cadence.
### When CalVer Fits
- **Time-sensitive products** where "how fresh is this?" matters more than "what changed in the API?" — browsers, OS releases, compliance certificate bundles, data snapshots.
- **Large or frequently changing scope** where semantic bumps become meaningless (a browser's "minor" version says nothing about compatibility).
- **Externally driven releases** — e.g., a TLS certificate bundle that must be re-released when certificates rotate, or security tooling that tracks a moving threat landscape.
- **Applications** where the version is a support-window marker (Ubuntu LTS = "supported until YYYY.MM + N years").
### SemVer vs CalVer — Selection Guidance
| Dimension | SemVer | CalVer |
|-----------|--------|--------|
| Signal | API compatibility | Time / freshness |
| Best for | Libraries, SDKs, APIs | Applications, OSes, time-bound products |
| Consumers | Downstream code (automated resolution) | Humans, support contracts, infosec |
| Bump driven by | Commit semantics (Conventional Commits) | Calendar (plus manual patch bumps) |
| Common hybrid | SemVer for libraries, CalVer for the app that bundles them | — |
Many organizations run both: libraries version with SemVer while the product line uses CalVer (or a train name). The skill's `assets/versioning-decision-table.md` gives a structured comparison including fixed/one-version monorepo schemes (see [monorepo-polyrepo-release.md](./monorepo-polyrepo-release.md)).
## Conventional Commits
[Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) is a lightweight specification for **machine-readable commit messages** that makes automated version bumps and changelog generation deterministic. Structure:
```
<type>[optional scope]: <description>
[optional body]
[optional footer(s)]
```
The three types that map directly to SemVer:
| Commit | SemVer Effect | Example |
|--------|---------------|---------|
| `fix` | PATCH | `fix: correct retry backoff calculation` |
| `feat` | MINOR | `feat(api): add list-users endpoint` |
| `feat!` / `fix!` or **BREAKING CHANGE** footer | MAJOR | `feat!: drop support for v1 auth` |
- The **scope** is an optional noun in parentheses (`feat(api): ...`) that groups related changes for changelog and release-notes purposes; it does **not** change the bump.
- The **`!`** after the type/subject, or a footer line reading `BREAKING CHANGE: <description>`, both signal a MAJOR bump. Prefer the footer form: it forces you to write the migration note that changelogs and consumers need.
- Other types — `build`, `chore`, `ci`, `docs`, `style`, `refactor`, `perf`, `test` — carry **no bump** by default (you can configure `perf` to bump PATCH). This is what keeps documentation-only PRs from inflating versions.
- A breaking change made without `!` or the footer is the most common source of **silent MAJOR drift** — the version bumps PATCH/MINOR while consumers break.
The full type set and its default bump effect:
| Type | Default bump | Typical use |
|------|--------------|-------------|
| `feat` | MINOR | New user-visible capability |
| `fix` | PATCH | Bug fix |
| `perf` | none (configurable) | Performance improvement |
| `refactor` | none | Internal restructuring, no behavior change |
| `docs` | none | Documentation only |
| `style` | none | Formatting, whitespace |
| `test` | none | Test additions/fixes |
| `build` | none | Build system changes |
| `ci` | none | CI configuration changes |
| `chore` | none | Maintenance, tooling |
Consistency matters more than the exact mapping: if `perf` bumps PATCH in one repo and nothing in another, consumers cannot predict release behavior from commit history. Choose a mapping once, encode it in the release tool config, and document it in `CONTRIBUTING`.
> **Gotcha — Squash-merge hygiene:** If you squash-merge, the PR title becomes the commit message. A PR titled `Update auth library` produces a commit with **no type**, which either fails the lint gate or defaults to no-bump, so the change ships inside whatever the last real bump was. Require conventional titles (via a lint action or merge-queue check) and keep the `BREAKING CHANGE:` footer in the squashed body.
Automated bump tooling reads this history: **semantic-release** analyzes commits and performs version + changelog + publish with no human gate; **release-please** generates a release PR (version bump + changelog) that a human merges; **git-cliff** generates the changelog from commits without publishing. For monorepo specifics (per-package vs combined releases), see [monorepo-polyrepo-release.md](./monorepo-polyrepo-release.md).
## Changelogs (Keep a Changelog and Release Please)
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) is the de-facto human changelog format, designed for **people, not machines** (machines read commits). Core conventions:
- **`## [Unreleased]`** section at the top for pending changes, replaced by a versioned header on release.
- Version headers linkable and date-stamped: `## [1.2.0] - 2026-08-01`, with a `[1.2.0]: https://...` reference link at the bottom.
- Change types grouped and consistent: **Added, Changed, Deprecated, Removed, Fixed, Security** — the same six groups in every entry.
- Latest version first; **`[YANKED]`** marks a version that must be pulled (e.g., `## [1.2.0] - 2026-08-01 [YANKED]`).
- Advertise SemVer adherence in the README or the changelog itself.
Anti-patterns: dumping the raw commit log as the changelog, ignoring deprecations, inconsistent date formats, and failing to link to diffs. If you use Conventional Commits, changelog sections can be generated automatically, but a human should review the "breaking changes and migration" entry for every MAJOR — that prose is what consumers actually read.
A minimal conforming structure:
```markdown
# Changelog
## [Unreleased]
### Added
- ... (pending changes)
## [2.1.0] - 2026-08-01
### Added
- ...
### Fixed
- ...
[Unreleased]: https://github.com/acme/app/compare/v2.1.0...HEAD
[2.1.0]: https://github.com/acme/app/compare/v2.0.0...v2.1.0
```
The `scripts/changelog_check.py` in this skill validates this shape with `--format keep-a-changelog` (or safe `--format auto` detection). It also validates Release Please output with linked dated headers such as `## [0.6.0](https://github.com/example/proj/compare/v0.5.0...v0.6.0) (2026-08-03)`, conventional `###` sections such as `Features`, `Bug Fixes`, and `Reverts`, and `*` bullets with inline links. Release Please section labels are configurable. Release Please files do not use an `Unreleased` section; select that format explicitly in CI when auto-detection is not appropriate.
## Artifact Immutability and Promotion
The single most important artifact rule: **build once, promote the same artifact** through every environment. Never rebuild per environment — a rebuild introduces nondeterminism (time stamps, dependency drift, build-host state) and breaks the guarantee that what you tested is what you deployed.
- The artifact (image, package, binary) is **immutable** — identified by a content digest or a versioned tag that never changes.
- **Promotion** is a *pointer move*, not a rebuild: label the same artifact as `dev`, then `canary`, then `prod`. Google's internal package manager (MPM) does exactly this — content-hashed, versioned, **signed** packages with movable labels (`dev`, `canary`, `production`) pointing at immutable versions.
- This decouples "the build passed" from "the rollout reached production" and makes rollback trivial: point the label back at the previous known-good version (see [rollback-and-recovery.md](./rollback-and-recovery.md)).
### Movable Labels vs Immutable Tags
| Kind | Examples | Mutability | Use |
|------|----------|------------|-----|
| **Immutable** | `v1.2.3`, `sha-abc1234`, digest | Never rewritten | Point-in-time identity, provenance, rollback target |
| **Movable label** | `stable`, `latest`, `canary`, `prod` | Rewritten on promotion | "What should env X run now?" pointer |
> **Gotcha — Mutable tags in production manifests:** Pointing deployments at `latest` or `stable` means the running version is whoever promoted last, and rollback "to the previous version" is ambiguous. Reference the immutable tag or digest in your deployment manifests and GitOps repo; use the mutable label only for "current" semantics. Also see the supply-chain risk of digest drift in [supply-chain-security.md](./supply-chain-security.md).
### The Promotion Ledger
Every promotion is an auditable event. Record at minimum: the artifact digest, the version label moved, the source environment, the target environment, the actor (person or pipeline identity), the timestamp in UTC, and the pipeline run ID that authorized the move. This ledger is the same artifact an auditor samples under SOC 2 CC8.1 — a promotion with no ledger entry is indistinguishable from a manual deploy, which is an audit exception (see [change-governance-and-compliance.md](./change-governance-and-compliance.md)). GitOps systems (Argo CD, Flux) produce this record naturally: the promotion is a commit changing the environment's desired digest; anything that happens outside that commit is drift.
## Artifact Naming and Registry Conventions
### OCI (Container/Registry) Tags
- **Immutable tags:** git SHA (`sha-abc1234` or full SHA), SemVer (`v1.2.3`), or a monotonically increasing build number. These are the only tags safe to pin.
- **Mutable tags:** `latest`, `stable`, `canary`, `dev` — promoted as the label moves. Used for convenience and for environment pointers.
- **Promotion pattern:** build → tag with SHA + SemVer → test → promote the mutable label (`rc.1` → `stable`) without touching the immutable tag.
- **GitOps constraint:** Flux/GitLab-style operators can resolve semver ranges on tags (e.g., `>=1.0.0 <2.0.0`), but range resolution against *mutable* tags is a footgun — pin by digest in critical environments.
- **OCI convergence:** the OCI spec is absorbing every artifact type — container images, Helm charts, WASM modules, ML models, and SBOMs all publish as OCI artifacts, so a single registry with a single signing/scanning posture covers the whole catalog.
### Language-Registry Conventions
- **Maven/Gradle:** `1.0-SNAPSHOT` marks development; release versions are **immutable once deployed** to a central repository; a **BOM (Bill of Materials)** aligns dependency versions across modules.
- **npm/pnpm:** lockfiles (`package-lock.json`, `pnpm-lock.yaml`) pin the exact resolved tree; published versions are immutable (npm rejects republishing an identical version number with different content, though yanking is possible — avoid it).
- **Go modules:** `go.sum` pins module hashes; semantic import versioning (`/v2` suffix) is the convention for MAJOR breaks.
- **Python:** `uv`/pip lockfiles pin exact versions; `1.0-SNAPSHOT`-style pre-releases use PEP 440 prerelease tags (`1.0.0rc1`).
> **Gotcha — Version collisions:** Publishing a SemVer tag and a CalVer tag for the same artifact, or a mutable tag that shadows a SemVer tag (`latest` pointed at `v1.2.3` while `v2.0.0` ships), creates two "current" versions. Enforce one canonical identity per artifact: the digest is truth, tags are aliases.
## Provenance
Provenance answers "**what exactly is this artifact, and where did it come from?**" with machine-verifiable evidence:
- **Content digest** — the artifact's immutable identity (e.g., `sha256:...` for container images). Two artifacts with the same digest are byte-identical; promotion and rollback should operate on digests.
- **Build metadata** — commit SHA, build ID, source repo, build timestamp, build platform, build environment inputs. This connects the artifact to the exact code and pipeline run that produced it.
- **SBOM linkage** — an inventory of components (with versions and licenses) attached to or published alongside the artifact, enabling vulnerability queries after release (see [supply-chain-security.md](./supply-chain-security.md)).
- **SLSA provenance attestations** — signed statements (in-toto format) proving the artifact was built from a specific source commit by a specific workflow on a specific platform, enabling level-graded trust (see [supply-chain-security.md](./supply-chain-security.md)).
A provenance attestation is only as useful as the fields it carries. Minimum viable set:
| Field | Example | Why it matters |
|-------|---------|----------------|
| Subject digest | `sha256:9f86d08...` | Identifies the exact artifact being attested |
| Predicate type | SLSA provenance v1.0 | Says what the attestation claims |
| Builder identity | `https://github.com/acme/app/.github/workflows/release.yml@refs/heads/main` | Who built it (OIDC-verifiable) |
| Source location | `git+https://github.com/acme/app@<sha>` | Where the code came from |
| Materials | list of input digests (base images, deps) | What went into the build |
| Build invocation | command + env | How the build ran (hermeticity evidence) |
In practice: generate the SBOM **at build time** (not by scanning a running container), sign the image *and* the attestation with Cosign/sigstore, and record the digest + SBOM reference in the deployment record. This is also the evidence an auditor samples for SOC 2 CC8.1 — see [change-governance-and-compliance.md](./change-governance-and-compliance.md).
> **Gotcha — Provenance as an afterthought:** Attaching provenance after the artifact has been deployed means you are reconstructing evidence instead of producing it. Generate digests, SBOM, and attestations inside the build pipeline as steps, and fail the pipeline if they are missing — the same posture as [change-governance-and-compliance.md](./change-governance-and-compliance.md) takes toward audit evidence.
## Sources and Further Reading
- [Semantic Versioning 2.0.0](https://semver.org/) — the specification (CC BY 3.0)
- [Conventional Commits v1.0.0](https://www.conventionalcommits.org/en/v1.0.0/) — commit message spec driving automated bumps (CC BY 3.0)
- [Keep a Changelog 1.1.0](https://keepachangelog.com/en/1.1.0/) — changelog format conventions (MIT)
- [CalVer](https://calver.org/) — calendar versioning schemes and rationale
- [Google SRE Book — Chapter 8: Release Engineering](https://sre.google/sre-book/release-engineering/) — build-once, signed, labeled packages (CC BY-NC-ND: cite, do not copy)
- [semantic-release](https://github.com/semantic-release/semantic-release) — fully automated version + changelog + publish
- [googleapis/release-please](https://github.com/googleapis/release-please) — release-PR generation from conventional commits
- [pnpm Workspaces](https://pnpm.io/workspaces) — workspace protocol and monorepo publishing
scripts/changelog_check.py
#!/usr/bin/env python3
"""Validate Keep a Changelog or Release Please CHANGELOG.md files.
Checks performed:
Keep a Changelog validation requires the first non-empty line to be the
`# Changelog` title, an `## [Unreleased]` section, dated version headers,
allowed change types, and reference links. Release Please validation accepts
linked dated headers such as `## [1.2.0](https://...) (2026-08-03)`, conventional
changelog sections such as `Features`, `Bug Fixes`, and `Reverts`, and star
bullets; it rejects an `Unreleased` section. Release Please section labels are
configurable, so the validator checks that non-empty `###` headings contain the
bullets.
With `--format auto` (the default), a valid Release Please header selects the
Release Please validator; otherwise the strict Keep a Changelog validator is
used. Explicit `--format` selection is available for CI gates.
Keep a Changelog checks:
- Version headers use the form `## [X.Y.Z] - YYYY-MM-DD` with a strict
SemVer version (pre-releases allowed) and a valid ISO-8601 date
(YYYY-MM-DD or YYYY-MM); the optional `[YANKED]` marker is allowed.
- Subsection headings use one of the six Keep a Changelog change types
(Added/Changed/Deprecated/Removed/Fixed/Security), and standalone
bullets (not under a categorized subsection) name a change type in
their first word. Bullets under a categorized subsection are
free-form, matching the canonical Keep a Changelog layout.
- Every version header (including [Unreleased]) has a matching
reference link definition (`[X.Y.Z]: https://...`).
Each problem is reported with its line number. Exit 0 when the file is
clean, exit 1 when any problem is found.
Arguments: [changelog.md] (default: CHANGELOG.md), --format, --json.
Exit codes:
0 changelog is valid
1 changelog has problems, or the file cannot be read
2 usage error (argparse)
"""
import argparse
import datetime
import json
import re
import sys
TITLE_RE = re.compile(r"^#\s+Changelog\s*$")
UNRELEASED_HEADER_RE = re.compile(r"^##\s+\[Unreleased\](\s+\[YANKED\])?\s*$")
VERSION_HEADER_RE = re.compile(
r"^##\s+\[([^\]]+)\]\s*-\s*(\d{4}-\d{2}-\d{2}|\d{4}-\d{2})(\s+\[YANKED\])?\s*$"
)
RELEASE_PLEASE_HEADER_RE = re.compile(
r"^##\s+\[([^\]]+)\]\(([^)]+)\)\s+\((\d{4}-\d{2}-\d{2})\)\s*$"
)
BULLET_RE = re.compile(r"^-\s+(\S+)")
STAR_BULLET_RE = re.compile(r"^\*\s+(\S+)")
LINK_REF_RE = re.compile(r"^\[([^\]]+)\]:\s+(\S+)")
CHANGE_TYPES = ("Added", "Changed", "Deprecated", "Removed", "Fixed", "Security")
FORMATS = ("auto", "keep-a-changelog", "release-please")
SEMVER_RE = re.compile(
r"^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)"
r"(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)"
r"(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
r"(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$"
)
def parse_args(argv=None):
"""Parse command-line arguments."""
parser = argparse.ArgumentParser(
prog="changelog_check.py",
description=(
"Validate a CHANGELOG.md as Keep a Changelog or Release Please; "
"auto-detect the format unless explicitly selected."
),
epilog=(
"Exit codes: 0 valid, 1 problems found / file unreadable, "
"2 usage error.\n\n"
"Examples:\n"
" changelog_check.py\n"
" changelog_check.py CHANGELOG.md\n"
" changelog_check.py CHANGELOG.md --format release-please\n"
" changelog_check.py CHANGELOG.md --json\n"
),
)
parser.add_argument(
"changelog",
nargs="?",
default="CHANGELOG.md",
metavar="CHANGELOG.md",
help="Path to the changelog file (default: CHANGELOG.md).",
)
parser.add_argument(
"--format",
choices=FORMATS,
default="auto",
dest="format",
help=(
"Changelog format: auto (default), keep-a-changelog, or "
"release-please."
),
)
parser.add_argument(
"--json",
action="store_true",
dest="json_output",
help="Output results as machine-parseable JSON instead of text.",
)
return parser.parse_args(argv)
def valid_semver(text):
"""Return True if text is a strict SemVer version (pre-releases ok)."""
return SEMVER_RE.match(text.strip()) is not None
def valid_date(text):
"""Return True if text is a valid YYYY-MM-DD or YYYY-MM date."""
if len(text) == 10:
try:
datetime.date.fromisoformat(text)
return True
except ValueError:
return False
if len(text) == 7:
try:
datetime.datetime.strptime(text, "%Y-%m")
return True
except ValueError:
return False
return False
def detect_format(text):
"""Detect Release Please only from its unambiguous linked header shape."""
for line in text.splitlines():
if RELEASE_PLEASE_HEADER_RE.match(line.rstrip()):
return "release-please"
return "keep-a-changelog"
def check_keep_a_changelog(text):
"""Validate changelog text; returns (valid, problems).
problems is a list of {"line": int, "message": str} dicts.
"""
problems = []
lines = text.splitlines()
non_empty = [i for i, line in enumerate(lines) if line.strip()]
if not non_empty:
return False, [{"line": 1, "message": "file is empty"}]
if not TITLE_RE.match(lines[non_empty[0]].strip()):
problems.append(
{
"line": non_empty[0] + 1,
"message": "expected '# Changelog' title as the first non-empty line",
}
)
# Pass 1: section headers, subsection headings, bullets, link refs.
in_section = False
subsection = None
has_unreleased = False
seen_headers = []
seen_links = set()
for idx, raw in enumerate(lines):
line = raw.rstrip()
stripped = line.strip()
lineno = idx + 1
if line.startswith("## "):
# A version section resets any subsection context.
in_section = False
subsection = None
if stripped.startswith("## ["):
if UNRELEASED_HEADER_RE.match(stripped):
in_section = True
has_unreleased = True
seen_headers.append(("Unreleased", lineno))
continue
match = VERSION_HEADER_RE.match(stripped)
if not match:
problems.append(
{
"line": lineno,
"message": (
"malformed version header (expected "
"'## [Unreleased]' or '## [X.Y.Z] - YYYY-MM-DD'): "
"'{}'".format(stripped)
),
}
)
continue
version, date_text = match.group(1), match.group(2)
if not valid_semver(version):
problems.append(
{
"line": lineno,
"message": (
"version '{}' is not strict SemVer".format(version)
),
}
)
if not valid_date(date_text):
problems.append(
{
"line": lineno,
"message": (
"invalid release date '{}' in header".format(date_text)
),
}
)
seen_headers.append((version, lineno))
in_section = True
continue
if not in_section:
continue
if line.startswith("### "):
heading_word = ""
remainder = stripped[len("### "):]
if remainder.split():
heading_word = remainder.split()[0].rstrip(":,").strip()
if heading_word and heading_word not in CHANGE_TYPES:
problems.append(
{
"line": lineno,
"message": (
"subsection heading '{}' is not a Keep a Changelog "
"type (Added/Changed/Deprecated/Removed/Fixed/Security)"
).format(stripped),
}
)
subsection = heading_word
continue
if stripped.startswith("- "):
bullet_match = BULLET_RE.match(stripped)
if not bullet_match:
continue
first_word = bullet_match.group(1).rstrip(":,").strip()
# Bullets under a categorized subsection (e.g. ### Added) are
# already categorized and are free-form; standalone bullets must
# name the change type themselves.
if subsection is None and first_word not in CHANGE_TYPES:
problems.append(
{
"line": lineno,
"message": (
"bullet '{}' is not under a change-type subsection "
"and does not start with a Keep a Changelog type "
"(Added/Changed/Deprecated/Removed/Fixed/Security)"
).format(stripped),
}
)
continue
if stripped.startswith("["):
link_match = LINK_REF_RE.match(stripped)
if link_match:
seen_links.add(link_match.group(1))
if not has_unreleased:
problems.append(
{"line": 1, "message": "missing '## [Unreleased]' section"}
)
# Pass 2: every version header needs a matching reference link.
for header, lineno in seen_headers:
if header not in seen_links:
problems.append(
{
"line": lineno,
"message": (
"missing reference link for '[{}]' "
"(add a '[{}]: <url>' definition)".format(header, header)
),
}
)
return len(problems) == 0, problems
def check_release_please(text):
"""Validate the Release Please changelog format."""
problems = []
lines = text.splitlines()
non_empty = [i for i, line in enumerate(lines) if line.strip()]
if not non_empty:
return False, [{"line": 1, "message": "file is empty"}]
if not TITLE_RE.match(lines[non_empty[0]].strip()):
problems.append(
{
"line": non_empty[0] + 1,
"message": "expected '# Changelog' title as the first non-empty line",
}
)
in_section = False
subsection = None
release_count = 0
for idx, raw in enumerate(lines):
line = raw.rstrip()
stripped = line.strip()
lineno = idx + 1
if line.startswith("## "):
in_section = False
subsection = None
if stripped == "## [Unreleased]" or stripped.startswith("## [Unreleased]"):
problems.append(
{"line": lineno, "message": "Release Please files must not contain an '## [Unreleased]' section"}
)
continue
match = RELEASE_PLEASE_HEADER_RE.match(stripped)
if not match:
problems.append(
{
"line": lineno,
"message": "malformed Release Please version header (expected '## [X.Y.Z](<url>) (YYYY-MM-DD)')",
}
)
continue
version, url, date_text = match.groups()
release_count += 1
in_section = True
if not valid_semver(version):
problems.append(
{"line": lineno, "message": "version '{}' is not strict SemVer".format(version)}
)
if not url.startswith(("https://", "http://")):
problems.append(
{"line": lineno, "message": "Release Please header link must be an http(s) URL"}
)
if not valid_date(date_text):
problems.append(
{"line": lineno, "message": "invalid release date '{}' in header".format(date_text)}
)
continue
if not in_section:
continue
if line.startswith("### "):
subsection = stripped[len("### "):].strip()
if not subsection:
problems.append(
{
"line": lineno,
"message": "Release Please subsection headings must not be empty",
}
)
continue
if stripped.startswith("*"):
if not STAR_BULLET_RE.match(stripped):
problems.append(
{"line": lineno, "message": "malformed Release Please bullet (expected '* <change>')"}
)
elif subsection is None:
problems.append(
{"line": lineno, "message": "Release Please bullets must be under a subsection heading"}
)
continue
if stripped.startswith("-"):
problems.append(
{"line": lineno, "message": "Release Please bullets must use '*' rather than '-'"}
)
if release_count == 0:
problems.append({"line": 1, "message": "missing Release Please version section"})
return len(problems) == 0, problems
def check_changelog(text, format="auto"):
"""Validate text using an explicit format or safe auto-detection."""
selected = detect_format(text) if format == "auto" else format
if selected == "release-please":
return check_release_please(text)
return check_keep_a_changelog(text)
def main(argv=None):
"""Entry point."""
args = parse_args(argv)
try:
with open(args.changelog, "r", encoding="utf-8") as fh:
text = fh.read()
except OSError as exc:
print("error: cannot read '{}': {}".format(args.changelog, exc), file=sys.stderr)
return 1
detected_format = detect_format(text)
selected_format = detected_format if args.format == "auto" else args.format
valid, problems = check_changelog(text, selected_format)
if args.json_output:
print(
json.dumps(
{
"file": args.changelog,
"format": selected_format,
"detected_format": detected_format,
"valid": valid,
"problem_count": len(problems),
"problems": problems,
},
indent=2,
)
)
else:
if not problems:
print("valid: {} conforms to {} (detected: {})".format(
args.changelog, selected_format, detected_format
))
else:
for problem in problems:
print(
"{}:{}: {}".format(
args.changelog, problem["line"], problem["message"]
)
)
print(
"invalid: {} problem{} found in {}".format(
len(problems), "" if len(problems) == 1 else "s", args.changelog
)
)
return 0 if valid else 1
if __name__ == "__main__":
sys.exit(main())
scripts/dora_metrics.py
#!/usr/bin/env python3
"""Compute the five current DORA metrics from a deployment events file.
Input: --events <file.json> with the schema:
{
"deployments": [
{"started_at": "<iso>", "finished_at": "<iso>", "commit_sha": "<sha>",
"environment": "prod", "success": true, "unplanned": false}
],
"commits": [{"sha": "<sha>", "created_at": "<iso>"}]
}
Metrics (scoped to --environment, default "prod"):
1. Deployment frequency Successful deployments per day. The
observation window spans the earliest
deployment start to the latest deployment
finish among deployments in the selected
environment (minimum 1 day), so mixed-
environment events files do not widen the
window with other environments' activity.
If the environment has zero deployments,
the window is unavailable and the metric
is reported as unavailable.
2. Change lead time Median of (finished_at - commit created_at)
over successful deployments whose commit_sha
resolves to a commit with a timestamp. A
negative duration (a deployment finishing
before its commit was created) is clamped
to 0. Reported as unavailable when no
deployment has commit timestamp data.
3. Change failure rate (failed / total) * 100.
4. Failed deployment recovery time
Median of (next successful deployment
finished_at - failed deployment started_at)
over failed deployments that have a later
successful deployment. A recovery
deployment counts only if it starts at or
after the failed deployment finished (a
success that began mid-failure is not a
recovery). Reported as unavailable when a
failed deployment has not yet been
recovered (or there are none).
5. Deployment rework rate (unplanned / total) * 100.
Timestamps are ISO-8601 strings ('Z' suffix accepted; naive timestamps are
treated as UTC). Metrics whose computation is not meaningful for the data
are reported with "available": false and a reason, and the script still
exits 0 for well-formed input.
Output is a human-readable table by default, or structured JSON with --json.
Exit codes:
0 success (metrics computed or reported unavailable)
1 input error (unreadable file, invalid JSON, malformed events)
2 usage error (argparse)
"""
import argparse
import json
import statistics
import sys
from datetime import datetime, timezone
SECONDS_PER_DAY = 86400.0
def parse_args(argv=None):
"""Parse command-line arguments."""
parser = argparse.ArgumentParser(
prog="dora_metrics.py",
description=(
"Compute the five current DORA metrics (deployment frequency, "
"change lead time, change failure rate, failed deployment "
"recovery time, deployment rework rate) from an events JSON file."
),
epilog=(
"Exit codes: 0 success, 1 input error, 2 usage error.\n\n"
"Examples:\n"
" dora_metrics.py --events events.json\n"
" dora_metrics.py --events events.json --environment staging\n"
" dora_metrics.py --events events.json --json\n"
),
)
parser.add_argument(
"--events",
metavar="FILE",
required=True,
help="Path to the deployment events JSON file.",
)
parser.add_argument(
"--environment",
metavar="NAME",
default="prod",
help="Environment to scope metrics to (default: prod).",
)
parser.add_argument(
"--json",
action="store_true",
dest="json_output",
help="Output results as machine-parseable JSON instead of a table.",
)
return parser.parse_args(argv)
def parse_iso(value):
"""Parse an ISO-8601 timestamp; returns datetime or None."""
if not isinstance(value, str) or not value.strip():
return None
text = value.strip().replace("Z", "+00:00")
try:
dt = datetime.fromisoformat(text)
except ValueError:
return None
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt
def load_events(path):
"""Load and validate the events file.
Returns (events, None) on success or (None, error_message).
events = {"deployments": [...], "commits": {sha: created_at_datetime}}
"""
try:
with open(path, "r", encoding="utf-8") as fh:
data = json.load(fh)
except OSError as exc:
return None, "cannot read events file: {}".format(exc)
except ValueError as exc:
return None, "invalid JSON in events file: {}".format(exc)
if not isinstance(data, dict):
return None, "events must be a JSON object with 'deployments' and 'commits'"
deployments = data.get("deployments", [])
commits = data.get("commits", [])
if not isinstance(deployments, list):
return None, "'deployments' must be an array"
if not isinstance(commits, list):
return None, "'commits' must be an array"
normalized = []
for index, deploy in enumerate(deployments):
if not isinstance(deploy, dict):
return None, "deployment {} is not an object".format(index)
started = parse_iso(deploy.get("started_at"))
if started is None:
return None, "deployment {} has an invalid 'started_at'".format(index)
finished = parse_iso(deploy.get("finished_at"))
if finished is None:
return None, "deployment {} has an invalid 'finished_at'".format(index)
environment = deploy.get("environment")
if not isinstance(environment, str) or not environment.strip():
return None, "deployment {} has an invalid 'environment'".format(index)
success = deploy.get("success")
if not isinstance(success, bool):
return None, "deployment {} field 'success' must be a boolean".format(index)
unplanned = deploy.get("unplanned")
if not isinstance(unplanned, bool):
return None, "deployment {} field 'unplanned' must be a boolean".format(index)
commit_sha = deploy.get("commit_sha")
if commit_sha is not None and not isinstance(commit_sha, str):
return None, "deployment {} field 'commit_sha' must be a string".format(index)
normalized.append(
{
"started_at": started,
"finished_at": finished,
"environment": environment,
"success": success,
"unplanned": unplanned,
"commit_sha": commit_sha,
}
)
commit_map = {}
for index, commit in enumerate(commits):
if not isinstance(commit, dict):
return None, "commit {} is not an object".format(index)
sha = commit.get("sha")
if not isinstance(sha, str) or not sha.strip():
return None, "commit {} has an invalid 'sha'".format(index)
created = parse_iso(commit.get("created_at"))
if created is None:
# Commit without a usable timestamp: contributes no lead-time
# data (reported as unavailable rather than failing the file).
continue
commit_map[sha] = created
return {"deployments": normalized, "commits": commit_map}, None
def observation_window(deployments):
"""Return (window_days, min_start, max_finish) over the given deployments.
The window spans the earliest started_at to the latest finished_at;
it is clamped to a minimum of one day so frequency is well-defined
for single-deployment windows. Pass the environment-filtered
deployment set so mixed-environment events files do not widen the
window with other environments' activity.
"""
if not deployments:
return 1.0, None, None
min_start = min(d["started_at"] for d in deployments)
max_finish = max(d["finished_at"] for d in deployments)
span = (max_finish - min_start).total_seconds()
window_days = max(1.0, span / SECONDS_PER_DAY)
return window_days, min_start, max_finish
def compute_metrics(events, environment):
"""Compute the five DORA metrics for an environment.
Returns a dict of metric payloads (each with an 'available' flag).
"""
deployments = [d for d in events["deployments"] if d["environment"] == environment]
total = len(deployments)
successful = [d for d in deployments if d["success"]]
failed = [d for d in deployments if not d["success"]]
unplanned = [d for d in deployments if d["unplanned"]]
no_deploys_reason = "no deployments in environment '{}'".format(environment)
# 1. Deployment frequency: successful prod deploys per day, over a
# window scoped to this environment's deployments.
window_days, _, _ = observation_window(deployments)
if total == 0:
deployment_frequency = {"available": False, "reason": no_deploys_reason}
else:
deployment_frequency = {
"available": True,
"deployments_per_day": _round(len(successful) / window_days),
"successful_deployments": len(successful),
"window_days": _round(window_days),
}
# 2. Change lead time: median finished_at - commit created_at.
lead_times = []
without_commit_data = 0
for deploy in successful:
sha = deploy.get("commit_sha")
created = events["commits"].get(sha) if sha else None
if created is None:
without_commit_data += 1
continue
# Clamp negative durations (a deployment finishing before its
# commit was created) to zero rather than reporting negative time.
lead_times.append(
max(0.0, (deploy["finished_at"] - created).total_seconds())
)
if not lead_times:
change_lead_time = {
"available": False,
"reason": (
"no successful deployments with commit timestamp data "
"in environment '{}'".format(environment)
),
"deployments_measured": 0,
"deployments_without_commit_data": without_commit_data,
}
else:
change_lead_time = {
"available": True,
"seconds": _round(statistics.median(lead_times)),
"human": format_duration(statistics.median(lead_times)),
"deployments_measured": len(lead_times),
"deployments_without_commit_data": without_commit_data,
}
# 3. Change failure rate: failed / total as a percentage.
if total == 0:
change_failure_rate = {"available": False, "reason": no_deploys_reason}
else:
change_failure_rate = {
"available": True,
"percent": _round(len(failed) / total * 100.0),
"failed": len(failed),
"total": total,
}
# 4. Failed deployment recovery time: median next-success - failed start.
# A recovery deployment must start at or after the failed deployment
# finished, so a success that began mid-failure is not counted.
ordered = sorted(deployments, key=lambda d: (d["started_at"], d["finished_at"]))
recoveries = []
unrecovered = 0
for index, deploy in enumerate(ordered):
if deploy["success"]:
continue
next_success = None
for candidate in ordered[index + 1:]:
if candidate["success"] and candidate["started_at"] >= deploy["finished_at"]:
next_success = candidate
break
if next_success is None:
unrecovered += 1
else:
recoveries.append((next_success["finished_at"] - deploy["started_at"]).total_seconds())
if len(failed) == 0:
failed_deployment_recovery_time = {
"available": False,
"reason": "no failed deployments to recover from",
"recovered": 0,
"unrecovered": 0,
}
elif unrecovered > 0:
failed_deployment_recovery_time = {
"available": False,
"reason": "{} failed deployment(s) not yet recovered".format(unrecovered),
"recovered": len(recoveries),
"unrecovered": unrecovered,
}
else:
failed_deployment_recovery_time = {
"available": True,
"seconds": _round(statistics.median(recoveries)),
"human": format_duration(statistics.median(recoveries)),
"recovered": len(recoveries),
"unrecovered": 0,
}
# 5. Deployment rework rate: unplanned / total as a percentage.
if total == 0:
deployment_rework_rate = {"available": False, "reason": no_deploys_reason}
else:
deployment_rework_rate = {
"available": True,
"percent": _round(len(unplanned) / total * 100.0),
"unplanned": len(unplanned),
"total": total,
}
return {
"environment": environment,
"window_days": _round(window_days),
"deployments": {
"total": total,
"successful": len(successful),
"failed": len(failed),
"unplanned": len(unplanned),
},
"deployment_frequency": deployment_frequency,
"change_lead_time": change_lead_time,
"change_failure_rate": change_failure_rate,
"failed_deployment_recovery_time": failed_deployment_recovery_time,
"deployment_rework_rate": deployment_rework_rate,
}
def _round(value):
"""Round a float to 6 decimal places for deterministic output."""
return round(value, 6)
def format_duration(seconds):
"""Format a duration in seconds as a compact human string."""
total = int(round(seconds))
days, rem = divmod(total, 86400)
hours, rem = divmod(rem, 3600)
minutes, secs = divmod(rem, 60)
parts = []
if days:
parts.append("{}d".format(days))
if hours:
parts.append("{}h".format(hours))
if minutes:
parts.append("{}m".format(minutes))
if secs or not parts:
parts.append("{}s".format(secs))
return " ".join(parts)
def format_table(metrics):
"""Format metrics as a human-readable table."""
counts = metrics["deployments"]
lines = []
lines.append("DORA metrics (environment: {})".format(metrics["environment"]))
lines.append("-" * 72)
df = metrics["deployment_frequency"]
if df["available"]:
lines.append(
"{:<34} {} per day ({} successful over {} days)".format(
"deployment frequency",
df["deployments_per_day"],
df["successful_deployments"],
df["window_days"],
)
)
else:
lines.append(
"{:<34} unavailable ({})".format("deployment frequency", df["reason"])
)
clt = metrics["change_lead_time"]
if clt["available"]:
lines.append(
"{:<34} {} (median over {} deployment{})".format(
"change lead time",
clt["human"],
clt["deployments_measured"],
"" if clt["deployments_measured"] == 1 else "s",
)
)
else:
lines.append(
"{:<34} unavailable ({})".format("change lead time", clt["reason"])
)
cfr = metrics["change_failure_rate"]
if cfr["available"]:
lines.append(
"{:<34} {}% ({} of {})".format(
"change failure rate", cfr["percent"], cfr["failed"], cfr["total"]
)
)
else:
lines.append(
"{:<34} unavailable ({})".format("change failure rate", cfr["reason"])
)
fdrt = metrics["failed_deployment_recovery_time"]
if fdrt["available"]:
lines.append(
"{:<34} {} (median over {} recovery{})".format(
"failed deployment recovery time",
fdrt["human"],
fdrt["recovered"],
"" if fdrt["recovered"] == 1 else "ies",
)
)
else:
lines.append(
"{:<34} unavailable ({})".format(
"failed deployment recovery time", fdrt["reason"]
)
)
drr = metrics["deployment_rework_rate"]
if drr["available"]:
lines.append(
"{:<34} {}% ({} of {})".format(
"deployment rework rate", drr["percent"], drr["unplanned"], drr["total"]
)
)
else:
lines.append(
"{:<34} unavailable ({})".format("deployment rework rate", drr["reason"])
)
lines.append("-" * 72)
lines.append(
"deployments in environment: {} total, {} successful, {} failed, {} unplanned".format(
counts["total"],
counts["successful"],
counts["failed"],
counts["unplanned"],
)
)
return "\n".join(lines)
def main(argv=None):
"""Entry point."""
args = parse_args(argv)
events, err = load_events(args.events)
if err:
print("error: {}".format(err), file=sys.stderr)
return 1
metrics = compute_metrics(events, args.environment)
if args.json_output:
print(json.dumps(metrics, indent=2))
else:
print(format_table(metrics))
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/release_plan_scaffold.py
#!/usr/bin/env python3
"""Scaffold a release plan markdown document.
Fills the release-plan template structure (see templates/release-plan.md)
from flags: metadata table, overview and scope (with an in-scope table),
versioning and artifacts, timeline and milestones, owners and RACI, risks
and mitigations, rollout plan, rollback contingency, communication plan,
post-release monitoring, and sign-offs.
Scope items come either from --scope (semicolon-separated) or
--git-range (commit subjects and short SHAs read via `git log`).
Output is written to --output <file> if given, otherwise to stdout.
--json emits a machine-parseable object containing the parsed fields and
the fully rendered document (the rendered markdown is also written to
--output when given). Output is deterministic: no timestamps, no
environment-dependent defaults.
Exit codes:
0 success
1 input error (git failure, cannot write output file)
2 usage error (argparse)
"""
import argparse
import json
import re
import subprocess
import sys
# Conventional-commit prefix used to infer the Type column in the
# in-scope table (e.g. "feat(api): add endpoint" -> "feat").
CONVENTIONAL_PREFIX_RE = re.compile(r"^([a-z]+)(?:\([^)]*\))?!?:\s*")
def parse_args(argv=None):
"""Parse command-line arguments."""
parser = argparse.ArgumentParser(
prog="release_plan_scaffold.py",
description=(
"Scaffold a release plan markdown document following the "
"release-plan template structure, filling in the provided fields."
),
epilog=(
"Exit codes: 0 success, 1 input error, 2 usage error.\n\n"
"Examples:\n"
" release_plan_scaffold.py --version 1.2.3 --name 'Acme 1.2.3' \\\n"
" --date 2026-08-15 --owner 'Jane Doe' \\\n"
" --milestones 'Branch cut;Release candidate;GA' \\\n"
" --scope 'feat: new API;fix: retry bug' \\\n"
" --risks 'DB migration late;API contract change' \\\n"
" --output release-plan.md\n"
" release_plan_scaffold.py --version 1.2.3 --git-range main..HEAD --json\n"
),
)
parser.add_argument(
"--version",
metavar="X.Y.Z",
required=True,
help="Version this release plan covers (required).",
)
parser.add_argument(
"--name",
metavar="NAME",
default=None,
help="Release name (default: 'Release <version>').",
)
parser.add_argument(
"--date",
metavar="YYYY-MM-DD",
default="TBD",
help="Target release date (default: TBD).",
)
parser.add_argument(
"--owner",
metavar="NAME",
default="TBD",
help="Release manager / accountable owner (default: TBD).",
)
parser.add_argument(
"--milestones",
metavar="M1;M2",
help="Semicolon-separated timeline milestones.",
)
scope_source = parser.add_mutually_exclusive_group()
scope_source.add_argument(
"--scope",
metavar="ITEM1;ITEM2",
help="Semicolon-separated scope items.",
)
scope_source.add_argument(
"--git-range",
metavar="FROM..TO",
help=(
"Git revision range; commit subjects and short SHAs become the "
"scope items (via git log)."
),
)
parser.add_argument(
"--risks",
metavar="R1;R2",
help="Semicolon-separated risks with mitigations.",
)
parser.add_argument(
"--output",
metavar="FILE",
help="Write the rendered markdown to this file instead of stdout.",
)
parser.add_argument(
"--json",
action="store_true",
dest="json_output",
help=(
"Output the parsed fields and rendered document as JSON on "
"stdout (the markdown is still written to --output if given)."
),
)
return parser.parse_args(argv)
def parse_semicolon_list(value):
"""Split a semicolon-separated string into a trimmed, non-empty list."""
if not value:
return []
return [item.strip() for item in value.split(";") if item.strip()]
def fetch_git_scope(git_range):
"""Fetch (short_sha, subject) pairs for a git revision range.
Returns (items, None) on success or (None, error_message).
items = [{"sha": str, "subject": str}, ...]
"""
try:
proc = subprocess.run(
["git", "log", "--reverse", "--format=%h%x00%s", git_range],
capture_output=True,
text=True,
timeout=30,
)
except OSError as exc:
return None, "cannot run git: {}".format(exc)
if proc.returncode != 0:
message = proc.stderr.strip() or "git exited {}".format(proc.returncode)
return None, "git log failed for range '{}': {}".format(git_range, message)
items = []
for line in proc.stdout.splitlines():
if not line.strip():
continue
parts = line.split("\x00", 1)
sha = parts[0].strip() if parts else ""
subject = parts[1].strip() if len(parts) > 1 else ""
if subject:
items.append({"sha": sha, "subject": subject})
return items, None
def infer_type(subject):
"""Infer a change type from a conventional-commit subject prefix."""
match = CONVENTIONAL_PREFIX_RE.match(subject)
if not match:
return "TBD"
ctype = match.group(1)
if ctype == "revert":
return "fix"
return ctype
def render_plan(fields):
"""Render the release plan markdown from parsed fields (deterministic)."""
lines = []
lines.append("# Release Plan: {}".format(fields["name"]))
lines.append("")
lines.append("## Metadata")
lines.append("")
lines.append("| Field | Value |")
lines.append("|-------|-------|")
lines.append("| Release name | {} |".format(fields["name"]))
lines.append("| Version | {} |".format(fields["version"]))
lines.append("| Target date (GA) | {} |".format(fields["date"]))
lines.append("| Release manager (DRI) | {} |".format(fields["owner"]))
lines.append("| Status | Draft |")
lines.append("")
lines.append("## 1. Overview and Scope")
lines.append("")
lines.append("### Objective")
lines.append("")
lines.append(
"_TBD - one or two sentences: the user/business outcome this release "
"delivers and how success is measured._"
)
lines.append("")
lines.append("### In Scope")
lines.append("")
lines.append("| ID | Item | Type (feat/fix/chore/security) | Source (PR/commit) |")
lines.append("|----|------|--------------------------------|--------------------|")
if fields["scope"]:
for index, item in enumerate(fields["scope"], start=1):
source = fields["scope_sources"][index - 1] or "—"
lines.append(
"| S-{} | {} | {} | {} |".format(
index, item, infer_type(item), source
)
)
else:
lines.append("| S-1 | _TBD_ | TBD | — |")
lines.append("")
lines.append("### Out of Scope")
lines.append("")
lines.append("- _TBD - list items excluded from this release and why._")
lines.append("")
lines.append("## 2. Versioning and Artifacts")
lines.append("")
lines.append("| Field | Value |")
lines.append("|-------|-------|")
lines.append("| Version scheme | TBD (SemVer / CalVer) |")
lines.append("| Primary artifact(s) | TBD |")
lines.append("| Artifact digest(s) | TBD |")
lines.append("| SBOM / provenance | TBD |")
lines.append("| Promotion policy | TBD (build once, promote the same artifact) |")
lines.append("")
lines.append("## 3. Timeline and Milestones")
lines.append("")
lines.append("| Milestone | Date | Owner | Exit Criteria |")
lines.append("|-----------|------|-------|---------------|")
if fields["milestones"]:
for milestone in fields["milestones"]:
lines.append("| {} | TBD | TBD | TBD |".format(milestone))
else:
lines.append("| _TBD_ | TBD | TBD | TBD |")
lines.append("")
lines.append("## 4. Owners and RACI")
lines.append("")
lines.append("| Activity | R | A | C | I |")
lines.append("|----------|---|---|---|---|")
for activity in (
"Scope definition",
"Build & test",
"Deploy",
"Rollback decision",
"Comms",
"Post-release monitoring",
):
lines.append("| {} | TBD | TBD | TBD | TBD |".format(activity))
lines.append("")
lines.append("## 5. Risks and Mitigations")
lines.append("")
lines.append("| ID | Risk | Probability (H/M/L) | Impact (H/M/L) | Mitigation | Owner |")
lines.append("|----|------|---------------------|----------------|------------|-------|")
if fields["risks"]:
for index, risk in enumerate(fields["risks"], start=1):
lines.append("| R-{} | {} | TBD | TBD | TBD | TBD |".format(index, risk))
else:
lines.append("| R-1 | _TBD_ | TBD | TBD | TBD | TBD |")
lines.append("")
lines.append("## 6. Rollout Plan")
lines.append("")
lines.append(
"- _TBD - strategy (canary / blue-green / rolling / ring / feature-flag), "
"staged progression, gates, and auto-rollback triggers._"
)
lines.append("")
lines.append("## 7. Rollback Contingency")
lines.append("")
lines.append(
"- _TBD - decision authority, time-box, known-good artifact, and "
"special cases; see rollback-runbook.md._"
)
lines.append("")
lines.append("## 8. Communication Plan")
lines.append("")
lines.append(
"- _TBD - internal announcements, support handoff, customer comms, "
"and status page cadence._"
)
lines.append("")
lines.append("## 9. Post-Release Monitoring")
lines.append("")
lines.append(
"- _TBD - metrics to watch (DORA, SLIs) and on-call coverage window._"
)
lines.append("")
lines.append("## 10. Sign-offs")
lines.append("")
lines.append("| Role | Name | Date | Decision |")
lines.append("|------|------|------|----------|")
lines.append("| Engineering lead (quality) | TBD | TBD | Approved / Not approved |")
lines.append("| SRE / operations (rollback + monitoring ready) | TBD | TBD | Approved / Not approved |")
lines.append("| Product owner (scope + comms) | TBD | TBD | Approved / Not approved |")
lines.append(
"| Release manager (final) | {} | TBD | GO / NO-GO / GO WITH CONDITIONS |".format(
fields["owner"]
)
)
return "\n".join(lines) + "\n"
def main(argv=None):
"""Entry point."""
args = parse_args(argv)
name = args.name if args.name else "Release {}".format(args.version)
milestones = parse_semicolon_list(args.milestones)
risks = parse_semicolon_list(args.risks)
if args.git_range:
scope_items, err = fetch_git_scope(args.git_range)
if err:
print("error: {}".format(err), file=sys.stderr)
return 1
scope_source = "git-range"
else:
scope_items = [
{"sha": None, "subject": item}
for item in parse_semicolon_list(args.scope)
]
scope_source = "flags"
fields = {
"version": args.version,
"name": name,
"date": args.date,
"owner": args.owner,
"milestones": milestones,
"scope": [item["subject"] for item in scope_items],
"scope_sources": [item["sha"] for item in scope_items],
"risks": risks,
"scope_source": scope_source,
}
document = render_plan(fields)
if args.output:
try:
with open(args.output, "w", encoding="utf-8") as fh:
fh.write(document)
except OSError as exc:
print("error: cannot write '{}': {}".format(args.output, exc), file=sys.stderr)
return 1
if args.json_output:
payload = dict(fields)
payload["document"] = document
print(json.dumps(payload, indent=2))
elif not args.output:
print(document, end="")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/semver_check.py
#!/usr/bin/env python3
"""Strict SemVer 2.0.0 validation and precedence tooling.
Three mutually exclusive modes:
--check VERSION Validate a version; exits 0 if valid, 1 if invalid
(the reason is printed either way).
--compare A B Compare two versions; prints lt, gt, or eq.
--sort V1 V2 ... Sort versions ascending by SemVer precedence.
Precedence follows SemVer 2.0.0 section 11: MAJOR.MINOR.PATCH compared
numerically; pre-release versions sort below the same core without a
pre-release; numeric pre-release identifiers sort below alphanumeric
ones; build metadata is ignored for precedence.
Output is human-readable by default, or machine-parseable JSON with
--json (a single JSON object on stdout; errors still go to stderr).
Exit codes:
0 success
1 invalid version (--check) or invalid input (--compare/--sort)
2 usage error (argparse)
"""
import argparse
import json
import re
import sys
# Strict SemVer 2.0.0 grammar (semver.org): no leading zeros, optional
# pre-release and build metadata.
SEMVER_RE = re.compile(
r"^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)"
r"(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)"
r"(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
r"(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$"
)
NUMERIC_RE = re.compile(r"^\d+$")
BUILD_IDENT_RE = re.compile(r"^[0-9a-zA-Z-]+$")
def parse_args(argv=None):
"""Parse command-line arguments."""
parser = argparse.ArgumentParser(
prog="semver_check.py",
description=(
"Validate strict SemVer 2.0.0 versions and apply SemVer "
"precedence (compare and sort)."
),
epilog=(
"Exit codes: 0 success, 1 invalid version / invalid input, "
"2 usage error.\n\n"
"Examples:\n"
" semver_check.py --check 1.2.3\n"
" semver_check.py --check 01.2.3\n"
" semver_check.py --compare 1.2.3 1.2.4\n"
" semver_check.py --sort 1.0.0-beta.2 1.0.0-alpha.1 1.0.0 1.0.0+build\n"
" semver_check.py --check 1.2.3-alpha.1 --json\n"
),
)
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument(
"--check",
metavar="VERSION",
help=(
"Validate a version: exits 0 and prints 'valid' if it conforms "
"to SemVer 2.0.0, exits 1 and prints a reason otherwise."
),
)
mode.add_argument(
"--compare",
nargs=2,
metavar=("A", "B"),
help="Compare two versions; prints lt, gt, or eq.",
)
mode.add_argument(
"--sort",
nargs="+",
metavar="VERSION",
help="Sort the given versions ascending by SemVer precedence.",
)
parser.add_argument(
"--json",
action="store_true",
dest="json_output",
help="Output results as machine-parseable JSON instead of text.",
)
return parser.parse_args(argv)
def parse_semver(text):
"""Parse a SemVer string into components, or None if invalid."""
match = SEMVER_RE.match(text.strip())
if not match:
return None
return {
"major": int(match.group(1)),
"minor": int(match.group(2)),
"patch": int(match.group(3)),
"prerelease": match.group(4),
"build": match.group(5),
}
def invalid_reason(text):
"""Produce a human-readable reason why a version is invalid.
Returns a generic message when the specific defect is not obvious.
"""
value = text.strip()
if not value:
return "version is empty"
match = re.match(r"^([0-9]*)\.([0-9]*)\.([0-9]*)(.*)$", value)
if not match:
return "expected MAJOR.MINOR.PATCH (e.g. 1.2.3)"
major_raw, minor_raw, patch_raw, rest = match.groups()
for name, raw in (("MAJOR", major_raw), ("MINOR", minor_raw), ("PATCH", patch_raw)):
if raw == "":
return "{} component is missing".format(name)
if not raw.isdigit():
return "{} component must be numeric, got '{}'".format(name, raw)
if len(raw) > 1 and raw.startswith("0"):
return "{} component must not have leading zeros, got '{}'".format(
name, raw
)
if not rest:
return "invalid SemVer 2.0.0 version"
if rest.startswith("+"):
build = rest[1:]
if not _valid_build(build):
return "invalid build metadata: '{}'".format(build)
return "invalid SemVer 2.0.0 version"
if rest.startswith("-"):
prerelease = rest[1:]
build = None
if "+" in prerelease:
prerelease, build = prerelease.split("+", 1)
if not _valid_prerelease(prerelease):
return "invalid pre-release identifiers: '{}'".format(prerelease)
if build is not None and not _valid_build(build):
return "invalid build metadata: '{}'".format(build)
return "invalid SemVer 2.0.0 version"
return "unexpected characters after PATCH: '{}'".format(rest)
def _valid_prerelease(identifiers):
"""Validate dot-separated pre-release identifiers."""
if not identifiers:
return False
for ident in identifiers.split("."):
if ident == "":
return False
if ident.isdigit():
if len(ident) > 1 and ident.startswith("0"):
return False
elif not re.match(r"^[0-9a-zA-Z-]+$", ident) or not re.search(r"[a-zA-Z-]", ident):
# Alphanumeric identifiers must contain at least one letter/hyphen.
return False
return True
def _valid_build(identifiers):
"""Validate dot-separated build metadata identifiers."""
if not identifiers:
return False
return all(BUILD_IDENT_RE.match(ident) for ident in identifiers.split("."))
def compare_precedence(a, b):
"""Compare two parsed versions by SemVer precedence.
Returns -1 if a < b, 0 if equal, 1 if a > b. Build metadata is ignored.
"""
for key in ("major", "minor", "patch"):
if a[key] != b[key]:
return -1 if a[key] < b[key] else 1
pa, pb = a["prerelease"], b["prerelease"]
if pa == pb:
return 0
if pa is None:
return 1
if pb is None:
return -1
ia, ib = pa.split("."), pb.split(".")
for x, y in zip(ia, ib):
xn, yn = x.isdigit(), y.isdigit()
if xn and yn:
if int(x) != int(y):
return -1 if int(x) < int(y) else 1
elif xn != yn:
# Numeric identifiers always sort below alphanumeric ones.
return -1 if xn else 1
else:
if x != y:
return -1 if x < y else 1
if len(ia) != len(ib):
return -1 if len(ia) < len(ib) else 1
return 0
def relation_word(a, b):
"""Map compare_precedence output to lt/gt/eq."""
cmp_result = compare_precedence(a, b)
return "eq" if cmp_result == 0 else ("lt" if cmp_result < 0 else "gt")
def _parsed_payload(parsed):
"""Serialize a parsed version dict for JSON output."""
return {
"major": parsed["major"],
"minor": parsed["minor"],
"patch": parsed["patch"],
"prerelease": parsed["prerelease"],
"build": parsed["build"],
}
def handle_check(args, version_text):
"""Run --check mode; returns exit code."""
parsed = parse_semver(version_text)
if parsed is not None:
if args.json_output:
print(
json.dumps(
{
"version": version_text,
"valid": True,
"reason": None,
"parsed": _parsed_payload(parsed),
},
indent=2,
)
)
else:
print("valid: {}".format(version_text))
return 0
reason = invalid_reason(version_text)
if args.json_output:
print(
json.dumps(
{
"version": version_text,
"valid": False,
"reason": reason,
"parsed": None,
},
indent=2,
)
)
else:
print("invalid: {} ({})".format(version_text, reason))
return 1
def handle_compare(args):
"""Run --compare mode; returns exit code."""
a_text, b_text = args.compare[0], args.compare[1]
a, b = parse_semver(a_text), parse_semver(b_text)
if a is None or b is None:
bad = a_text if a is None else b_text
print(
"error: invalid version '{}': {}".format(bad, invalid_reason(bad)),
file=sys.stderr,
)
return 1
relation = relation_word(a, b)
if args.json_output:
print(
json.dumps(
{"a": a_text, "b": b_text, "relation": relation}, indent=2
)
)
else:
print(relation)
return 0
def handle_sort(args):
"""Run --sort mode; returns exit code."""
parsed = {}
for version in args.sort:
item = parse_semver(version)
if item is None:
print(
"error: invalid version '{}': {}".format(
version, invalid_reason(version)
),
file=sys.stderr,
)
return 1
parsed[version] = item
# Stable sort by precedence; equal-precedence versions keep input order.
sorted_versions = sorted(parsed.keys(), key=lambda v: _sort_key(parsed[v]))
if args.json_output:
print(json.dumps({"sorted": sorted_versions}, indent=2))
else:
for version in sorted_versions:
print(version)
return 0
def _sort_key(parsed):
"""Sort key: numeric core, then pre-release mapping for precedence."""
# Map pre-release identifiers to a comparable tuple:
# pre-release versions sort below no-pre-release (2^63 sentinel),
# numeric identifiers sort below alphanumeric ones.
prerelease = parsed["prerelease"]
if prerelease is None:
return (parsed["major"], parsed["minor"], parsed["patch"], 2 ** 63, ())
key = []
for ident in prerelease.split("."):
if ident.isdigit():
key.append((0, int(ident)))
else:
key.append((1, ident))
return (parsed["major"], parsed["minor"], parsed["patch"], -1, tuple(key))
def main(argv=None):
"""Entry point."""
args = parse_args(argv)
if args.check is not None:
return handle_check(args, args.check)
if args.compare is not None:
return handle_compare(args)
return handle_sort(args)
if __name__ == "__main__":
sys.exit(main())
scripts/version_bump.py
#!/usr/bin/env python3
"""Compute the next SemVer version from Conventional Commits.
Reads the current version either from --current-version or --from-file
(package.json / pyproject.toml) and the commit history either from a
--commits-file (one commit message per line, or full message blocks in
the format "type(scope): subject" with an optional "BREAKING CHANGE:"
footer) or from a --git-range (runs `git log`).
Bump rules (Conventional Commits 1.0.0):
- BREAKING CHANGE footer, or a `!` after the type/scope -> MAJOR
- feat -> MINOR
- fix and all other types -> PATCH
- 0.x versions (initial development): MAJOR and MINOR bumps both become
MINOR under the documented Release Please-compatible policy, while
PATCH bumps remain PATCH.
Pre-release handling (--pre-release alpha|beta|rc):
- If the current version is already a pre-release with the SAME tag
and a numeric suffix (e.g. 1.2.0-alpha.1), the next pre-release
increments the number without bumping the core (1.2.0-alpha.2).
- Otherwise the core is bumped per the rules above and the tag is
applied with a fresh ".1" suffix (e.g. 1.3.0-beta.1).
- When --pre-release is omitted, the core is bumped per the rules
above and a stable version is emitted; a pre-release current
version is not carried into the result (e.g. 1.2.0-alpha.1 with a
feat commit -> 1.3.0).
Output is a single line with the next version, or structured JSON with
--json.
Exit codes:
0 success
1 input error (unreadable file, invalid current version, no commits,
git failure)
2 usage error (argparse)
"""
import argparse
import json
import re
import subprocess
import sys
# Strict SemVer 2.0.0 (from semver.org): no leading zeros, optional
# pre-release and build metadata.
SEMVER_RE = re.compile(
r"^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)"
r"(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)"
r"(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?"
r"(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$"
)
# Conventional Commit header: type[!][(scope)!]: description
CONVENTIONAL_RE = re.compile(r"^([a-zA-Z]+)(!)?(?:\(([^)]+)\))?(!)?:(.*)$")
# Breaking-change footer marker (BREAKING CHANGE or the deprecated
# BREAKING-CHANGE alias), matched case-insensitively for robustness.
BREAKING_FOOTER_RE = re.compile(r"breaking[- ]change\s*:", re.IGNORECASE)
PRE_RELEASE_TAGS = ("alpha", "beta", "rc")
def parse_args(argv=None):
"""Parse command-line arguments."""
parser = argparse.ArgumentParser(
prog="version_bump.py",
description=(
"Compute the next SemVer version from Conventional Commits. "
"BREAKING CHANGE (or type!/scope!) bumps MAJOR, feat bumps "
"MINOR, everything else bumps PATCH, with 0.x handling."
),
epilog=(
"Exit codes: 0 success, 1 input error, 2 usage error.\n\n"
"Examples:\n"
" version_bump.py --current-version 1.2.3 --commits-file commits.txt\n"
" version_bump.py --current-version 1.2.3 --git-range main..HEAD\n"
" version_bump.py --from-file package.json --commits-file commits.txt\n"
" version_bump.py --current-version 1.2.0-alpha.1 --commits-file commits.txt --pre-release alpha\n"
" version_bump.py --current-version 1.2.3 --commits-file commits.txt --json\n"
),
)
version_source = parser.add_mutually_exclusive_group(required=True)
version_source.add_argument(
"--current-version",
metavar="X.Y.Z",
help="Current version to bump (strict SemVer, optional pre-release/build).",
)
version_source.add_argument(
"--from-file",
metavar="FILE",
help=(
"Read the current version from a package.json or pyproject.toml "
"file (the 'version' field)."
),
)
commits_source = parser.add_mutually_exclusive_group(required=True)
commits_source.add_argument(
"--commits-file",
metavar="FILE",
help=(
"File with one commit message per line (or message blocks) in "
"Conventional Commits format; optional 'BREAKING CHANGE:' footer."
),
)
commits_source.add_argument(
"--git-range",
metavar="FROM..TO",
help="Git revision range (e.g. 'main..HEAD'); subjects are read via git log.",
)
parser.add_argument(
"--pre-release",
choices=PRE_RELEASE_TAGS,
metavar="TAG",
help=(
"Emit a pre-release with the given tag (alpha, beta, or rc) and "
"an incrementing numeric suffix."
),
)
parser.add_argument(
"--json",
action="store_true",
dest="json_output",
help="Output results as machine-parseable JSON instead of text.",
)
return parser.parse_args(argv)
def parse_current_version(text):
"""Parse a SemVer string into components, or None if invalid."""
match = SEMVER_RE.match(text.strip())
if not match:
return None
return {
"major": int(match.group(1)),
"minor": int(match.group(2)),
"patch": int(match.group(3)),
"prerelease": match.group(4),
"build": match.group(5),
}
def split_commits(text):
"""Split raw text into commit message blocks.
A new commit block starts at any line matching the Conventional
Commits header format; all following lines (body/footer) belong to
that commit until the next header line. This accepts both
one-message-per-line files and full `git log --format=%B` output.
"""
commits = []
current = None
for line in text.splitlines():
if CONVENTIONAL_RE.match(line):
if current is not None:
commits.append(current)
current = {"header": line, "body": []}
elif current is not None:
current["body"].append(line)
if current is not None:
commits.append(current)
return commits
def classify_commit(commit):
"""Classify one commit block; returns a dict or None if not a CC commit."""
match = CONVENTIONAL_RE.match(commit["header"])
if not match:
return None
ctype = match.group(1).lower()
breaking = bool(match.group(2)) or bool(match.group(4))
if not breaking:
for line in commit["body"]:
if BREAKING_FOOTER_RE.search(line):
breaking = True
break
return {"type": ctype, "breaking": breaking, "header": commit["header"]}
def compute_bump(commits):
"""Determine the bump level (major/minor/patch) from classified commits."""
if any(c["breaking"] for c in commits):
return "major"
if any(c["type"] == "feat" and not c["breaking"] for c in commits):
return "minor"
return "patch"
def bump_core(current, level):
"""Bump MAJOR.MINOR.PATCH per level, with 0.x handling."""
major, minor, patch = current["major"], current["minor"], current["patch"]
if level == "major":
if major == 0:
return (major, minor + 1, 0)
return (major + 1, 0, 0)
if level == "minor":
if major == 0:
return (major, minor + 1, 0)
return (major, minor + 1, 0)
return (major, minor, patch + 1)
def compute_next(current, level, prerelease_tag):
"""Compute the next version string from current, bump level, and tag."""
if prerelease_tag is not None and current["prerelease"]:
parts = current["prerelease"].split(".")
if (
len(parts) == 2
and parts[0] == prerelease_tag
and parts[1].isdigit()
):
# Same pre-release series: increment the numeric suffix without
# bumping the core (e.g. 1.2.0-alpha.1 -> 1.2.0-alpha.2).
return "{}.{}.{}-{}.{}".format(
current["major"],
current["minor"],
current["patch"],
prerelease_tag,
int(parts[1]) + 1,
)
core = bump_core(current, level)
if prerelease_tag is None:
return "{}.{}.{}".format(*core)
return "{}.{}.{}-{}.1".format(core[0], core[1], core[2], prerelease_tag)
def read_version_from_file(path):
"""Read the current version from package.json or pyproject.toml.
Returns (version, None) on success or (None, error_message).
"""
try:
with open(path, "r", encoding="utf-8") as fh:
content = fh.read()
except OSError as exc:
return None, "cannot read '{}': {}".format(path, exc)
if path.endswith(".json"):
try:
data = json.loads(content)
except ValueError as exc:
return None, "invalid JSON in '{}': {}".format(path, exc)
if not isinstance(data, dict):
return None, "'{}' must be a JSON object".format(path)
version = data.get("version")
if not isinstance(version, str) or not version.strip():
return None, "no string 'version' field in '{}'".format(path)
return version.strip(), None
if path.endswith(".toml"):
for line in content.splitlines():
match = re.match(r"^\s*version\s*=\s*['\"]([^'\"]+)['\"]\s*$", line)
if match:
return match.group(1), None
return None, "no 'version' field found in '{}'".format(path)
return None, "unsupported file type (expected .json or .toml): '{}'".format(path)
def fetch_git_log(git_range):
"""Fetch commit messages for a git revision range.
Returns (text, None) on success or (None, error_message).
"""
try:
proc = subprocess.run(
["git", "log", "--reverse", "--pretty=format:%B", git_range],
capture_output=True,
text=True,
timeout=30,
)
except OSError as exc:
return None, "cannot run git: {}".format(exc)
if proc.returncode != 0:
message = proc.stderr.strip() or "git exited {}".format(proc.returncode)
return None, "git log failed for range '{}': {}".format(git_range, message)
return proc.stdout, None
def main(argv=None):
"""Entry point."""
args = parse_args(argv)
if args.current_version:
current_text = args.current_version
else:
current_text, err = read_version_from_file(args.from_file)
if err:
print("error: {}".format(err), file=sys.stderr)
return 1
current = parse_current_version(current_text)
if current is None:
print(
"error: invalid current version: '{}'".format(current_text),
file=sys.stderr,
)
return 1
if args.commits_file:
try:
with open(args.commits_file, "r", encoding="utf-8") as fh:
raw = fh.read()
except OSError as exc:
print("error: cannot read commits file: {}".format(exc), file=sys.stderr)
return 1
source = "commits-file"
else:
raw, err = fetch_git_log(args.git_range)
if err:
print("error: {}".format(err), file=sys.stderr)
return 1
source = "git-range"
commits = split_commits(raw)
classified = [c for c in (classify_commit(c) for c in commits) if c is not None]
if not classified:
print("error: no conventional commits found in input", file=sys.stderr)
return 1
level = compute_bump(classified)
next_version = compute_next(current, level, args.pre_release)
breaking_count = sum(1 for c in classified if c["breaking"])
commit_count = len(classified)
result = {
"current_version": current_text,
"next_version": next_version,
"bump": level,
"commit_count": commit_count,
"breaking_count": breaking_count,
"prerelease": args.pre_release,
"source": source,
}
if args.json_output:
print(json.dumps(result, indent=2))
else:
print("current version: {}".format(current_text))
print("next version: {}".format(next_version))
print(
"bump: {} ({} commit{}, {} breaking)".format(
level,
commit_count,
"" if commit_count == 1 else "s",
breaking_count,
)
)
return 0
if __name__ == "__main__":
sys.exit(main())
SKILL.md
---
name: release-engineering
description: >-
Design, automate, and operate end-to-end software releases: release process
models and pipelines (trunk-based development, CD stages, release trains),
progressive delivery and feature flags, versioning and artifact management
(SemVer, conventional commits, changelogs, SBOM/provenance), readiness and
quality gates, rollback and recovery planning, change-management and audit
compliance (SOC 2, SOX, PCI), DORA metrics, and multi-team release
coordination. Do not use for application feature implementation
(backend-engineering/frontend-engineering), production incident root-cause
debugging or on-call/SLO operations (systematic-debugging /
site-reliability-engineering), security implementation or threat modeling
(secure-software-engineering), or internal developer platform construction
(platform-engineering).
license: MIT
compatibility: >-
Platform-agnostic methodology. Scripts require Python 3.8+ (stdlib only).
No CI platform, deployment tool, or version control mandate.
metadata:
source_repo: https://github.com/magnus919/agent-skills
skill_version: "1.0.0"
tags: release-engineering, releases, ci-cd, continuous-delivery, progressive-delivery,
feature-flags, semver, changelog, rollback, release-trains, dora, sbom, change-management,
deployment, artifacts, release-coordination
---
# Release Engineering
Senior-to-principal release engineering methodology: designing and operating the pipelines, processes, artifacts, gates, compliance evidence, and metrics that move software from commit to customer safely, predictably, and fast.
## Ownership
| You own | You don't own |
|---------|--------------|
| Release process design — branch strategy, process model, release cadence | Application feature implementation — route to [backend-engineering](../backend-engineering/SKILL.md) / [frontend-engineering](../frontend-engineering/SKILL.md) |
| CD pipeline architecture — build-once promotion stages, gates, push-on-green | Production incident root-cause debugging — route to [systematic-debugging](../systematic-debugging/SKILL.md) |
| Progressive delivery — canaries, rings, percentage rollouts, feature flags | On-call and SLO operations — route to [site-reliability-engineering](../site-reliability-engineering/SKILL.md) |
| Versioning and artifacts — SemVer, conventional commits, changelogs, immutability, provenance | Security implementation and threat modeling — route to [secure-software-engineering](../secure-software-engineering/SKILL.md) |
| Readiness and quality gates — release candidates, go/no-go, sign-off | Internal developer platform construction — route to [platform-engineering](../platform-engineering/SKILL.md) |
| Rollback and recovery planning — runbooks, rehearsals, recovery targets | Data pipeline operations and schema migration engineering — route to [data-engineering](../data-engineering/SKILL.md) |
| Change governance and audit compliance — SOC 2 / SOX / PCI evidence chains | Spec authoring and SDD gate mechanics — route to [spec-driven-development](../spec-driven-development/SKILL.md) |
| DORA metrics — definitions, computation, thresholds | API contract design and versioning policy — route to [api-design-and-evolution](../api-design-and-evolution/SKILL.md) |
| Multi-team release coordination — trains, calendars, stabilization | Evidence collection and verdicts against explicit criteria — route to [verification-methodology](../verification-methodology/SKILL.md) |
| Release operations — branch cuts, pipeline triage, emergency releases | Runtime monitoring and alerting — that's SRE |
## Core Principles
**Build once, promote many.** The artifact that passed every gate is the only artifact that ships. Rebuilding per environment reintroduces risk and invalidates what was verified.
**Small batches ship faster and safer.** Trunk-based development with frequent small merges outperforms long-lived branches on every DORA metric — deployment frequency, lead time, change failure rate, and recovery time.
**Decouple deploy from release.** Deploying code to production is not the same as exposing it to users. Feature flags and progressive rollout separate the two so each can happen on its own timeline and either can be reversed independently.
**Automated safety beats approval bureaucracy.** Evidence-based controls and pipeline gates outperform change-advisory-board sign-off, which research correlates with slower, less stable delivery. Automate the checks; reserve human approval for genuinely exceptional change.
**Rehearse rollback.** A rollback you haven't practiced will fail under pressure. Rollback is a first-class release artifact with its own runbook, trigger thresholds, and rehearsal log.
**Verify in production.** Staging parity helps, but production canaries, smoke tests, and observability gates are where readiness is actually proven. Real traffic, real telemetry, real decision points.
**Protect the supply chain.** Sign every artifact, record provenance and SBOM data, and treat registries as a security boundary. Integrity is part of the release, not an add-on.
**DORA is a system outcome.** Speed and stability emerge from how the process is designed — batch size, architecture, automation, safety culture — not from chasing metric targets.
## Loading Guide
| File | Load when |
|------|-----------|
| `references/role-and-career.md` | Understanding the release engineering role — org placement, day-to-day, Senior/Staff/Principal leveling, and evidence separating levels |
| `references/skills-competency-model.md` | Mapping the technical and professional skills release engineers master, and which ones differentiate at senior and above |
| `references/release-process-models.md` | Choosing a process model — trunk-based, GitFlow, GitHub Flow, release branches, release trains — and matching cadence to the org |
| `references/cd-and-pipeline-stages.md` | Designing CD pipelines — build-once promotion, stage gates, push-on-green, pipeline-as-code, hermetic builds, environment parity |
| `references/progressive-delivery.md` | Planning canary, blue/green, ring, or percentage rollouts; metric and error-budget gates; auto-rollback triggers |
| `references/change-governance-and-compliance.md` | Building audit-ready change control — SOC 2 CC8.1, SOX ITGC, PCI back-out, evidence chains, separation of duties, emergency change |
| `references/readiness-and-quality-gates.md` | Defining readiness dimensions, release candidates, go/no-go structure, error-budget release policy, definition of done |
| `references/rollback-and-recovery.md` | Planning rollback vs roll-forward vs flag recovery per system type; rehearsed rollbacks; time-boxed decisions; quarantine |
| `references/versioning-and-artifacts.md` | Setting SemVer/CalVer policy, conventional commits, changelog conventions, artifact immutability and provenance |
| `references/feature-flag-lifecycle.md` | Running flags through their full lifecycle — create, guard, rollout, verify, remove, expire — and avoiding flag debt |
| `references/monorepo-polyrepo-release.md` | Choosing mono vs polyrepo release strategy, affected-build detection, topological publish order, release tooling |
| `references/toolchain-landscape.md` | Selecting tools by category — CI/CD, release automation, artifact repos, GitOps, feature flags (2025-2026 status) |
| `references/supply-chain-security.md` | Hardening the supply chain — SLSA, SBOM, sigstore keyless signing, dependency updates, provenance attestations |
| `references/metrics-and-dora.md` | Defining and computing the five DORA metrics, thresholds, vendor divergence, and measurement pitfalls |
| `references/release-operations-and-triage.md` | Running release trains end-to-end — branch cuts, stabilization, go/no-go, pipeline triage, release infra reliability, agentic automation |
| `templates/release-plan.md` | Producing a release plan — scope, milestones, owners, risks, rollout, rollback contingency, comms |
| `templates/release-readiness-checklist.md` | Running a readiness review — checkbox table by dimension with owner + evidence, go/no-go block |
| `templates/rollback-runbook.md` | Writing a rollback runbook — triggers, per-layer steps, verification, ordering, comms, rehearsal log |
| `templates/release-notes.md` | Drafting release notes — version, date, change-type sections, breaking changes, migration steps |
| `templates/change-governance-record.md` | Recording an audit-ready change — ticket, PR, CI runs, artifact digest, deploy timestamp, verification, emergency flag |
| `templates/hotfix-emergency-release-plan.md` | Planning an emergency release — severity, break-glass approvals, expedited path, post-implementation review |
| `assets/dora-metrics-reference.md` | Looking up the five DORA metrics — formulas, units, data sources, thresholds, pitfalls — in one page |
| `assets/versioning-decision-table.md` | Choosing a versioning scheme — SemVer vs CalVer vs independent vs one-version, bump rules |
| `assets/deployment-strategy-matrix.md` | Comparing deployment strategies — rolling, blue-green, canary, ring, flag, shadow — on speed, safety, rollback |
| `assets/release-toolchain-cheatsheet.md` | Quick tool lookup by category — one-liner and when-to-pick per tool |
| `scripts/version_bump.py` | Computing the next SemVer from conventional commits or a git range |
| `scripts/semver_check.py` | Validating, comparing, or sorting strict SemVer versions |
| `scripts/changelog_check.py` | Validating Keep a Changelog or Release Please CHANGELOG.md files |
| `scripts/dora_metrics.py` | Computing the five DORA metrics from deployment and commit event data |
| `scripts/release_plan_scaffold.py` | Scaffolding a release plan document from flags or a git range |
| `evals/evals.json` | Running output-quality evals for this skill (schema v1, 8 cases) |
## Adjacent Skills
- Use [qa-methodology](../qa-methodology/SKILL.md) to define test strategy, regression scope, flake policy, and quality-gate semantics. This skill combines the resulting evidence with operational and governance evidence for a release decision.
- For AI-agent releases, use [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md) to design and interpret behavioral, trajectory, and model-regression evidence before incorporating it into a release gate.
## Scripts
| Script | Invocation | Purpose |
|--------|-----------|---------|
| version_bump | `python3 scripts/version_bump.py --current-version 1.4.0 --git-range v1.4.0..HEAD` | Computes the next SemVer from Conventional Commits — breaking → major (minor in 0.x), feat → minor, fix → patch; optional prerelease tag |
| semver_check | `python3 scripts/semver_check.py --check 1.2.3-beta.1` | Validates strict SemVer 2.0.0, compares two versions, or sorts a list |
| changelog_check | `python3 scripts/changelog_check.py CHANGELOG.md [--format auto|keep-a-changelog|release-please]` | Validates supported changelog headers, dates, sections, bullets, and links; reports selected/detected format |
| dora_metrics | `python3 scripts/dora_metrics.py --events events.json` | Computes all five DORA metrics from deployment and commit event data |
| release_plan_scaffold | `python3 scripts/release_plan_scaffold.py --version 2.0.0 --owner alice --output plan.md` | Scaffolds a release plan document with filled placeholders from flags or a git range |
## Triggers
Load this skill when the task involves:
- **Release planning** — designing a release plan, timeline, milestones, or rollout strategy
- **Pipeline design** — architecting or reviewing CD pipelines, promotion stages, or push-on-green
- **Version bumps** — computing the next version, SemVer validation, conventional-commit classification
- **Changelogs** — authoring or validating a changelog
- **Readiness review** — go/no-go calls, release candidates, readiness checklists
- **Rollback planning** — writing or rehearsing rollback runbooks and recovery plans
- **Progressive delivery** — canaries, rings, percentage rollouts, traffic shadowing
- **Feature flags** — flag rollout, lifecycle, and cleanup
- **Release trains** — calendar releases, branch cuts, stabilization windows
- **DORA metrics** — defining, computing, or reporting the five DORA metrics
- **Compliance evidence** — SOC 2 / SOX / PCI change records, audit trails, separation of duties
- **Artifacts and supply chain** — SBOM, provenance, signing, registry hygiene
- **Emergency releases** — hotfixes, break-glass changes, expedited release paths
- **Multi-team coordination** — cross-team release calendars and integration stabilization
## When not to use
Route to the named sibling skill instead:
- [backend-engineering](../backend-engineering/SKILL.md) / [frontend-engineering](../frontend-engineering/SKILL.md) — implementing application features. Release engineering designs the pipeline that ships code; it does not implement the code itself.
- [systematic-debugging](../systematic-debugging/SKILL.md) — root-cause analysis of production incidents, fault localization, bug reproduction. Release engineering plans recovery; debugging finds the cause.
- [site-reliability-engineering](../site-reliability-engineering/SKILL.md) — on-call operations, SLO/error-budget management, capacity planning, and day-to-day incident response.
- [secure-software-engineering](../secure-software-engineering/SKILL.md) — security implementation and threat modeling. Release engineering consumes security controls (SBOM, signing, dependency scanning) but does not design them.
- [platform-engineering](../platform-engineering/SKILL.md) — building the internal developer platform: CI infrastructure, GitOps controllers, self-service tooling, and golden paths.
- [data-engineering](../data-engineering/SKILL.md) — data pipeline operations, schema migration engineering, and data quality monitoring.
- [spec-driven-development](../spec-driven-development/SKILL.md) — writing specifications and running SDD pipeline gates and verdicts.
- [api-design-and-evolution](../api-design-and-evolution/SKILL.md) — API contract design and versioning policy. Release engineering version-bumps artifacts; it does not set API evolution rules.
- [verification-methodology](../verification-methodology/SKILL.md) — collecting evidence and rendering verdicts against explicit pass/fail criteria.
## Stop and Exit Conditions
- **Release plan complete when:** the plan names version, target date, milestones and branch cut, owners (RACI), scope items, risks with mitigations, rollout plan, rollback contingency, comms plan, and sign-offs.
- **Readiness checklist complete when:** every item in each dimension (functional, non-functional, operational, governance) has an owner and evidence, and a go/no-go decision is recorded.
- **Rollback runbook complete when:** trigger thresholds, impact assessment, decision criteria, per-layer rollback steps, verification SLIs, rollback ordering, comms, and a rehearsal log are all in place.
- **Version bump complete when:** the next version is computed from actual commit history (or an explicit file), validated against strict SemVer, and the bump rule is justified by the conventional-commit types.
- **Compliance record complete when:** the change chain — ticket → PR review → CI → approval → deploy log → verification — is complete, sampled, and retained per policy, with separation of duties respected.
- **Bounded escalation:** stop after three non-converging diagnostic passes and report the evidence collected so far.
templates/change-governance-record.md
# Change Governance Record — [CHANGE ID]
> One record per production change. This is the artifact an auditor samples (SOC 2 CC8.1: request → review → test → approve → deploy → verify). Every field must be populated for the change to be audit-ready; leave nothing implied. Retention: at least 12 months (SOC 2 / PCI DSS); 7 years if this touches a financial-reporting system (SOX).
## 1. Change Identification
| Field | Value |
|-------|-------|
| Change ID | [e.g., CHG-2026-0417 — unique, sequential] |
| Change title | [short description] |
| Ticket link | [ticketing link — contains business justification, risk assessment, test plan] |
| Risk tier | Low / Medium / High / Emergency (see section 6) |
| Type | Feature / Fix / Security / Configuration / Infrastructure / Migration |
| Requester | [name] |
| Date requested | [YYYY-MM-DD] |
## 2. Description and Justification
### Description
[What changed and why. Enough context that a reviewer with no memory of this change can evaluate it.]
### Business justification
[Problem being solved, expected outcome, success criteria.]
## 3. Design and Review
| Field | Value |
|-------|-------|
| Design / RFC link | [link for significant changes — ADR, design doc] |
| PR / MR link(s) | [link to the pull request containing the diff] |
| Author | [name] |
| Reviewer(s) | [name(s) — at least one engineer other than the author] |
| Review approval time (UTC) | [YYYY-MM-DD HH:MM UTC] |
| Branch protection enforced | Yes / No |
> Branch protection (required peer review, status checks) is the control that makes the review record trustworthy. Never disable it "temporarily" — use the emergency path in section 6 instead.
## 4. Testing and CI
| Field | Value |
|-------|-------|
| CI pipeline run ID(s) | [e.g., Actions run #12345 / GitLab pipeline 67890] |
| Commit SHA tested | [full SHA] |
| Unit / integration results | [pass link] |
| Security scans (SAST / dependency / secret) | [report links + pass/fail] |
| Test environment(s) | [staging, perf, ...] |
| Test data used | [synthetic / masked subset — no production PII in test environments] |
## 5. Approval (Deployment Gate)
| Field | Value |
|-------|-------|
| Approver | [name — MUST NOT be the code author] |
| Approval decision | Approved / Rejected |
| Approval time (UTC) | [YYYY-MM-DD HH:MM UTC] |
| Approval mechanism | [e.g., GitHub Environments required reviewer, CAB record] |
| Deployer | [name or pipeline identity — separate from the author where required] |
## 6. Emergency Change (Break-Glass) — only if applicable
| Field | Value |
|-------|-------|
| Emergency flag | Yes / No |
| Emergency justification | [why the normal path was bypassed — security exploit / production outage] |
| Authorized approver | [senior manager / on-call lead] |
| Retroactive approval due | [within 24–72 h per org policy] |
| Retroactive approval date | [YYYY-MM-DD] |
| Post-implementation review due | [YYYY-MM-DD — within X days] |
> Emergency ≠ uncontrolled. The break-glass path is an alternate, audited path: it still requires approval, testing (as far as possible), and full documentation in the same tracking system as normal changes.
## 7. Deployment
| Field | Value |
|-------|-------|
| Environment(s) | [staging → production (list all)] |
| Deploy timestamp (UTC) | [YYYY-MM-DD HH:MM UTC per environment] |
| Artifact (name + version) | [e.g., api:2.4.0] |
| Artifact digest | [sha256:...] |
| SBOM / provenance | [link — SLSA attestation] |
| Deployment mechanism | [pipeline deploy / Argo CD sync / manual] |
| Pipeline run ID (deploy) | [run ID] |
| Rollout strategy | [canary / blue-green / rolling / feature-flag — see deployment-strategy-matrix.md] |
## 8. Post-Deployment Verification
| Field | Value |
|-------|-------|
| Smoke / synthetic checks | [results link] |
| Health / SLI confirmation | [dashboard snapshot or link, e.g., error rate below threshold] |
| Rollback status | [not needed / available / triggered — see rollback-runbook.md if triggered] |
| Verification time (UTC) | [YYYY-MM-DD HH:MM UTC] |
| Ticket closed | [date, by whom] |
## 9. Evidence Traceability (audit summary)
| Chain step | Artifact | Location / link |
|------------|----------|-----------------|
| Request & authorization | Ticket | [link] |
| Design & review | PR with approvals | [link] |
| Testing | CI runs + scan reports | [link] |
| Final approval | Approval record (approver ≠ author) | [link] |
| Implementation | Deploy log + artifact digest | [link] |
| Post-implementation review | Verification + ticket closure | [link] |
> Timestamps must be UTC everywhere, and tools must be NTP-synchronized so the chain correlates. Keep the evidence chain intact after merge — never delete merged PRs, release branches, or deploy logs.
templates/hotfix-emergency-release-plan.md
# Hotfix / Emergency Release Plan — [HOTFIX VERSION]
> For SEV-1 incidents, security exploits, or critical data issues that cannot wait for the normal release train. Fill in the placeholders and obtain the required approvals BEFORE deploying. The normal governance chain is abbreviated — not skipped. See [change-governance-record.md](change-governance-record.md) section 6 for the emergency evidence requirements.
## 1. Emergency Details
| Field | Value |
|-------|-------|
| Severity | SEV-1 / SEV-2 / SEV-3 |
| Incident link | [incident ticket / war-room channel] |
| Hotfix version | [e.g., 2.4.1 — must be greater than the broken 2.4.0] |
| Broken version being fixed | [e.g., 2.4.0] |
| Incident commander | [name] |
| Release manager (hotfix DRI) | [name] |
| Target deploy time | [YYYY-MM-DD HH:MM UTC] |
## 2. Problem Statement
[What is broken, the observed impact (users/data/revenue), and the minimal fix that addresses it. One or two sentences plus the error/alert IDs.]
## 3. Hotfix Branch and Cherry-Picks
| Step | Detail |
|------|--------|
| Branch cut from | [production tag or last known-good release branch, e.g., release/2.4] |
| Hotfix branch | [e.g., hotfix/2.4.1-cve-auth-bypass] |
| Cherry-pick list | [commit SHAs, one per line, each with its original PR] |
| Excluded from hotfix | [anything in main that must NOT ride along — keep the diff minimal] |
> A hotfix must contain ONLY the fix. Do not pull in unrelated merges — every extra line is new risk under pressure.
## 4. Expedited Pipeline Path
| Stage | Expedited step | Who |
|-------|----------------|-----|
| Build | Hermetic build of the hotfix branch | [CI] |
| Tests | Targeted regression + smoke + tests covering the fixed path (full suite if time allows) | [eng lead] |
| Security | [scans if applicable — security fixes get SAST + dependency re-scan] | [sec eng] |
| Deploy | Canary subset → monitor [X min] → full rollout (never straight to 100% without a canary step) | [deployer] |
## 5. Required Approvals (Break-Glass)
> Same tracking system as normal changes — no shadow logs. Approvals may be post-hoc but are mandatory and time-stamped.
| Approval | Approver (≠ author) | Method | Deadline |
|----------|---------------------|--------|----------|
| Emergency change approval | [senior manager / on-call lead] | [chat + ticket, recorded] | Before deploy |
| Retroactive documentation | [release manager] | [change-governance-record.md](change-governance-record.md) section 6 | Within [24–72] h |
| Post-implementation review | [eng lead] | [postmortem] | Within [7] days |
## 6. Rollback Plan
- Known-good artifact (pre-incident): [e.g., api:2.3.1 + digest sha256:...]
- Rollback trigger: [e.g., hotfix error rate > 2× baseline for 10 min]
- Rollback mechanism: [re-deploy known-good artifact / flag off / forward migration — see rollback-runbook.md]
- Special caution: [e.g., "if the incident involved a data migration, verify the schema supports the rollback target before rolling code back"]
## 7. Communication Plan
| Audience | Channel | When |
|----------|---------|------|
| Incident team | #incident | Continuously |
| Engineering | #releases | On branch cut and deploy |
| Support | #support | Before user-facing impact is seen |
| Customers / status page | status page | Per severity SLA |
## 8. Post-Implementation Review
- [ ] Root cause documented in a blameless postmortem by [date — within 7 days]
- [ ] Permanent fix tracked for the next regular release
- [ ] Emergency procedure reviewed: was the break-glass path used correctly? Is the emergency ratio healthy (< [X]% of changes)?
- [ ] Monitoring/alerts added or tuned to catch this class of failure earlier
- [ ] Hotfix diff reviewed and merged back to main / release branches
## 9. Sign-Offs
| Role | Name | Date | Decision |
|------|------|------|----------|
| Incident commander | | | Deploy approved / Rejected |
| Release manager | | | Deploy approved / Rejected |
| Senior approver (emergency) | | | Retro-approval granted / Pending |
templates/release-notes.md
# Release Notes — v[VERSION]
> Fill in for human readers. Use Keep a Changelog change types. These notes are the customer-facing view of the same content tracked in CHANGELOG.md — keep them consistent. Delete the guidance notes once populated.
| Field | Value |
|-------|-------|
| Version | [e.g., 2.4.0] |
| Release date | [YYYY-MM-DD] |
| Release type | Major / Minor / Patch (per SemVer) |
| Status | Draft / In Review / Published |
## Summary
[2–3 sentences for a busy reader: what changed, why it matters, and whether any action is required.]
## Added
- [New capability — e.g., "Export to CSV is now available on the reports page."]
## Changed
- [Behavior change — e.g., "Default page size increased from 20 to 50."]
## Deprecated
- [Being phased out — e.g., "The v1 reports endpoint is deprecated and will be removed in v3.0."]
## Removed
- [Removed feature — e.g., "Legacy SMS notifications have been removed."]
## Fixed
- [Bug fix — e.g., "Fixed an issue where session tokens expired mid-checkout."]
## Security
- [Security fix — e.g., "Updated the auth library to address CVE-2026-XXXX."]
## Known Issues
- [Issue with a workaround — e.g., "On iOS < 17 the export button is hidden; upgrade the OS or use the web app."]
- [Issue being tracked for the next release — e.g., "The spinner does not render on very slow connections; tracked in #2345."]
## Breaking Changes and Migration
> Only fill this section if the version has breaking changes; otherwise state "No breaking changes in this release."
### Breaking changes
- [What is incompatible — e.g., "The `customer.list` endpoint now requires a `region` query parameter."]
### Migration steps
1. [Step 1 — e.g., "Update clients to send `region`."]
2. [Step 2 — e.g., "Run the provided migration script on any cached data."]
## Upgrade Instructions
1. [e.g., "Pull the new image: `docker pull registry.example.com/api:2.4.0`."]
2. [e.g., "Apply the schema migration: `migrate up` — expand-only, safe to run during the deploy."]
3. [e.g., "Deploy, then confirm the health endpoint returns 200."]
4. [e.g., "Monitor the error-rate dashboard for 48 hours."]
## Links
| Item | Link |
|------|------|
| Changelog | [link to CHANGELOG.md] |
| Diff (previous → this) | [link to compare v2.3.1...v2.4.0] |
| Artifact(s) | [registry / artifact URL] |
| Artifact digest(s) | [sha256:...] |
| SBOM | [link] |
| Migration scripts | [link] |
| Documentation | [link] |
| Security advisories | [link] |
| Support / feedback | [link] |
## Versioning
This release follows Semantic Versioning (MAJOR.MINOR.PATCH). Patch releases contain backward-compatible fixes; minor releases add backward-compatible features; major releases may contain breaking changes documented above.
templates/release-plan.md
# Release Plan — [RELEASE NAME] v[VERSION]
> Fill in every `[PLACEHOLDER]` and delete the italic guidance notes before publishing. Keep this file in the release branch and link it from the release ticket so the audit chain ([change-governance-record.md](change-governance-record.md)) can reference it.
## Metadata
| Field | Value |
|-------|-------|
| Release name | [e.g., Aurora — a short, memorable label for the release train] |
| Version | [e.g., 2.4.0 — must match version-control tags and artifact names] |
| Target date (GA) | [YYYY-MM-DD] |
| Release manager (DRI) | [name — exactly one accountable owner] |
| Status | Draft / Frozen / In Flight / Shipped / Rolled Back |
| Changelog / commit source | [link to CHANGELOG.md or the commit range, e.g., main...release/2.4] |
## 1. Overview and Scope
### Objective
[One or two sentences: the user/business outcome this release delivers and how success is measured. Make it verifiable — "reduce checkout p95 latency by 20%", "GA the billing API v2", "remediate the CVE-2026-XXXX dependency".]
### In Scope
| ID | Item | Type (feat/fix/chore/security) | Source (PR/commit) |
|----|------|--------------------------------|--------------------|
| S-1 | [item] | feat | [#1234] |
| S-2 | [item] | fix | [abc1234] |
> Pull scope from the changelog or `git log --oneline <from>..<to>`; every item should trace to a merged PR or commit. The `release_plan_scaffold.py` script generates this table from a git range.
### Out of Scope
- [item — and why it is excluded: next release, blocked, product decision]
- [item]
## 2. Versioning and Artifacts
| Field | Value |
|-------|-------|
| Version scheme | SemVer / CalVer (see [versioning-decision-table.md](../assets/versioning-decision-table.md)) |
| Primary artifact(s) | [e.g., image registry.example.com/api:2.4.0, dist/api-2.4.0.tar.gz] |
| Artifact digest(s) | [sha256:... — of the exact artifact promoted to production] |
| SBOM / provenance | [link to SBOM (CycloneDX/SPDX) and SLSA provenance attestation] |
| Promotion policy | [build once, promote the same immutable artifact dev → staging → prod] |
## 3. Timeline and Milestones
| Milestone | Date | Owner | Exit Criteria |
|-----------|------|-------|---------------|
| Branch cut | [YYYY-MM-DD] | [name] | release/[version] branch created from main; CI green on branch |
| Code freeze | [YYYY-MM-DD] | [name] | only blockers merge; freeze announced in release channel |
| Release candidate build | [YYYY-MM-DD] | [name] | RC artifact built once, signed, digest recorded |
| Full test pass / UAT | [YYYY-MM-DD] | [name] | acceptance tests + UAT sign-off on the RC artifact |
| Staging deploy | [YYYY-MM-DD] | [name] | staging parity confirmed; readiness checklist run |
| GA / go-live | [YYYY-MM-DD] | [name] | go/no-go passed; rollout started |
| End of rollout | [YYYY-MM-DD] | [name] | 100% of target population on new version |
| Monitoring window closes | [YYYY-MM-DD + 48–72 h] | [name] | post-release monitoring done; release ticket closed |
> Freeze dates are a commitment: any change after branch cut needs the release manager's explicit sign-off and a re-run of the affected gates.
## 4. Owners and RACI
| Activity | R (Responsible) | A (Accountable) | C (Consulted) | I (Informed) |
|----------|-----------------|-----------------|---------------|--------------|
| Scope definition | [eng lead] | [PM] | [eng team] | [stakeholders] |
| Build & test | [CI owner] | [eng lead] | [QA] | [release manager] |
| Deploy | [deployer] | [release manager] | [SRE] | [eng team] |
| Rollback decision | [release manager] | [on-call lead] | [SRE, eng lead] | [all] |
| Comms | [comms owner] | [release manager] | [PM] | [customers] |
| Post-release monitoring | [SRE] | [SRE lead] | [eng lead] | [release manager] |
> Every row needs exactly one Accountable. The approver must never equal the deployer (separation of duties — see [change-governance-record.md](change-governance-record.md)).
## 5. Risks and Mitigations
| ID | Risk | Probability (H/M/L) | Impact (H/M/L) | Mitigation | Owner |
|----|------|---------------------|----------------|------------|-------|
| R-1 | [e.g., new dependency unavailable in prod registry] | M | H | [e.g., pre-push artifact to prod registry during staging] | [name] |
| R-2 | [e.g., schema migration locks the payments table] | M | H | [e.g., expand/contract migration; backfill in background; see rollback-runbook.md] | [name] |
| R-3 | [e.g., third-party API rate limit during launch] | L | M | [e.g., staged rollout + circuit breaker] | [name] |
## 6. Rollout Plan
| Strategy | [Canary / blue-green / rolling / ring / feature-flag — see deployment-strategy-matrix.md] |
|----------|--------------------------------------------------------------------------------------------|
| Staged progression | [e.g., canary 5% (4 h) → 25% (24 h) → 50% (24 h) → 100%; or ring: internal → beta → 10% → 100%] |
| Gate between stages | [SLI thresholds that must hold before the next stage, e.g., error rate < X%, p95 latency < Y ms] |
| Feature flags | [list flags toggled as part of this release and who flips them] |
| Auto-rollback trigger | [e.g., canary error rate > 2× baseline for 10 min → automatic rollback of canary] |
| Go/no-go authority | [who decides to pause or abort the rollout] |
## 7. Rollback Contingency
- Decision path, commands, and verification steps: see [rollback-runbook.md](rollback-runbook.md).
- Decision authority: [name] (on-call lead) — decision time-boxed to [X] minutes after a confirmed signal.
- Known-good artifact: [e.g., registry.example.com/api:2.3.1 with digest sha256:...]
- Special cases: [list anything unusual — irreversible migrations, client-side changes that cannot be recalled, data changes]
## 8. Communication Plan
| Audience | When | Channel | Message Owner |
|----------|------|---------|---------------|
| Internal engineering | [branch cut / RC / GA] | [#releases] | [release manager] |
| Support / on-call | [before GA] | [#support-handoff] | [comms owner] |
| Customers / status page | [GA + during rollout] | [status page, release notes] | [PM] |
| Executives | [GA] | [email / weekly] | [release manager] |
## 9. Post-Release Monitoring
| What | Where | Window | Alert Threshold |
|------|-------|--------|-----------------|
| [e.g., deployment frequency & change failure rate] | [DORA dashboard] | [48–72 h] | [n/a — trend] |
| [e.g., checkout p95 latency] | [Grafana] | [72 h] | [> 300 ms for 15 min] |
| [e.g., payment error rate] | [Sentry / Datadog] | [72 h] | [> 0.5% for 10 min] |
| [e.g., support tickets mentioning new feature] | [ticketing] | [1 week] | [n/a — triage] |
## 10. Sign-offs
| Role | Name | Date | Decision |
|------|------|------|----------|
| Engineering lead (quality) | | | Approved / Not approved |
| SRE / operations (rollback + monitoring ready) | | | Approved / Not approved |
| Product owner (scope + comms) | | | Approved / Not approved |
| Release manager (final) | | | GO / NO-GO / GO WITH CONDITIONS |
> GO WITH CONDITIONS requires named conditions, owners, and deadlines in the table below.
| Condition | Owner | Deadline | Status |
|-----------|-------|----------|--------|
| [condition] | [name] | [date] | Open / Done |
templates/release-readiness-checklist.md
# Release Readiness Checklist — [RELEASE NAME] v[VERSION]
> One row per item. Every item needs a **named owner** (a person, not a team) and **evidence** (a link, log, or artifact). Do not mark an item done without evidence — the go/no-go call is only as strong as the checklist behind it. Target date: [YYYY-MM-DD]. Release manager: [name].
## Functional
| # | Check | Owner | Evidence | Done |
|---|-------|-------|----------|------|
| F-1 | Acceptance tests pass on the exact release-candidate artifact (not on latest main) | [name] | [CI run URL] | ☐ |
| F-2 | End-to-end smoke test of critical user journeys passes against the RC | [name] | [test report link] | ☐ |
| F-3 | UAT accepted by the business owner | [name] | [UAT sign-off ticket] | ☐ |
| F-4 | Known defects triaged: risk-rated and explicitly accepted or deferred | [name] | [ticket list] | ☐ |
| F-5 | Backward compatibility verified (API consumers, data contracts, N-1 services) | [name] | [contract test output] | ☐ |
## Non-Functional
| # | Check | Owner | Evidence | Done |
|---|-------|-------|----------|------|
| N-1 | Performance verified at peak load + margin (p95/p99 targets met) | [name] | [load test report] | ☐ |
| N-2 | Security scans clean or exceptions approved: SAST, dependency/CVE, secret scan | [name] | [scan reports] | ☐ |
| N-3 | Accessibility checks pass for UI changes | [name] | [a11y report] | ☐ |
| N-4 | Resilience validated: failover, circuit breakers, graceful degradation | [name] | [chaos / game-day record] | ☐ |
| N-5 | Capacity verified: no new scaling limits hit at projected traffic | [name] | [capacity plan / load test] | ☐ |
## Operational
| # | Check | Owner | Evidence | Done |
|---|-------|-------|----------|------|
| O-1 | Monitoring + alerting live for new metrics before go-live | [name] | [dashboard link] | ☐ |
| O-2 | Runbooks exist and were read by on-call (deploy, rollback, incident) | [name] | [runbook links, read receipts] | ☐ |
| O-3 | Deployment rehearsed in staging; exact steps executed once | [name] | [staging deploy log] | ☐ |
| O-4 | Rollback path tested, not assumed (time-boxed rehearsal) | [name] | [rollback-runbook.md](rollback-runbook.md) rehearsal log | ☐ |
| O-5 | On-call coverage confirmed for the post-release window (48–72 h) | [name] | [roster] | ☐ |
| O-6 | Data safety checkpoints set for any migration or destructive step | [name] | [migration plan + RPO/RTO values] | ☐ |
## Governance
| # | Check | Owner | Evidence | Done |
|---|-------|-------|----------|------|
| G-1 | Change record created and linked to this release ([change-governance-record.md](change-governance-record.md)) | [name] | [change ID / ticket] | ☐ |
| G-2 | Approval recorded by a person who is not the author | [name] | [approval record] | ☐ |
| G-3 | Separation of duties enforced: deployer ≠ author ≠ approver | [name] | [deploy log] | ☐ |
| G-4 | Artifact digest + SBOM/provenance recorded for the promoted artifact | [name] | [digest + SBOM link] | ☐ |
| G-5 | Audit artifacts linked end-to-end: ticket → PR → CI → approval → deploy → verify | [name] | [traceability export] | ☐ |
## Go / No-Go Decision
### Conditions for GO
All of the following must hold; record evidence next to each:
- [ ] All Functional items F-1..F-5 complete with evidence
- [ ] All Non-Functional items N-1..N-5 complete with evidence
- [ ] All Operational items O-1..O-6 complete with evidence
- [ ] All Governance items G-1..G-5 complete with evidence
- [ ] No open Critical or High severity defects; accepted risks documented in [release-plan.md](release-plan.md) section 5
- [ ] Rollback rehearsal completed within the last [30] days
- [ ] On-call coverage confirmed for the monitoring window
### Decision
| Field | Value |
|-------|-------|
| Decision | GO / NO-GO / GO WITH CONDITIONS |
| Meeting time (UTC) | [YYYY-MM-DD HH:MM UTC] |
| Conditions (if GO WITH CONDITIONS) | [condition — owner — deadline] |
| Decision record link | [link to meeting notes / recorded decision] |
### Signatories
| Role | Name | Signature / Date |
|------|------|------------------|
| Engineering lead | | |
| SRE / operations lead | | |
| Product owner | | |
| Release manager | | |
> The go/no-go decision is a time-stamped, recorded call by named individuals — not a round of applause. If conditions are attached, the release does not proceed past the next stage until they are closed.
## Checklist Hygiene
- Keep every item's evidence URL live until the release ticket closes; stale links break the audit chain.
- Re-run the checklist as a whole after any change to the release candidate — partial re-runs miss interactions.
- Record the checklist result (with the go/no-go decision) in the release ticket so the call is traceable.
- Evidence types to prefer: CI run URLs, scan reports, dashboard snapshots, deploy logs, and sign-off tickets — each maps to a row above.
templates/rollback-runbook.md
# Rollback Runbook — [SERVICE / SYSTEM] v[VERSION]
> One runbook per service or release. Fill in the placeholders, then rehearse it — an unrehearsed runbook is fiction. Keep this next to the deploy runbook. Last reviewed: [YYYY-MM-DD] by [name].
## 1. When This Runbook Applies
| Field | Value |
|-------|-------|
| Service / system | [name] |
| Release(s) covered | [e.g., v2.4.0 and any patch on top of it] |
| Known-good artifact | [image:tag + sha256 digest] |
| Deploy mechanism | [Argo CD sync / pipeline deploy / manual steps] |
| Rollback decision authority | [name — on-call lead; decision time-boxed to X minutes] |
## 2. Trigger and Detection Thresholds
Initiate rollback (or flag-off) when any of these holds for the stated window. Base thresholds on pre-release baselines, not guesses.
| Signal | Threshold | Window | Tool / Alert |
|--------|-----------|--------|--------------|
| Error rate (HTTP 5xx) | [e.g., > 1.0%, or > 2× baseline] | [10 min] | [Datadog alert] |
| Latency p95 | [e.g., > 300 ms, or > 1.5× baseline] | [15 min] | [Grafana] |
| Error budget burn | [e.g., > 2% of monthly budget consumed in 1 h] | [1 h] | [burn-rate alert] |
| Saturation / capacity | [e.g., CPU/memory > 85% on > 50% of instances] | [15 min] | [infra alert] |
| Data integrity | [e.g., migration verification query fails] | [immediately] | [migration log] |
| Business signal | [e.g., support-ticket spike about the new feature] | [1 h] | [ticketing] |
> **Gotcha —** compare canary vs. control populations, never "before vs. after" (time is a confound). If the defect is gated behind a feature flag, flip the flag off first — it is the fastest and least risky lever.
## 3. Impact Assessment
| Question | Answer |
|----------|--------|
| Who is affected? | [users / segments / internal teams] |
| What is the blast radius? | [service, downstream dependencies, data, clients] |
| Severity | [SEV-1 / SEV-2 / SEV-3] |
| Is data at risk? | [yes/no — if yes, stop and involve the DB owner before acting] |
| Was a schema migration deployed with this release? | [yes/no — if yes, see section 5.3; rollback may be unsafe after finalization] |
| Is the change client-side (mobile/desktop/IoT)? | [yes/no — if yes, rollback is forward-only; use a kill switch / phased release] |
## 4. Decision Matrix — Rollback vs. Roll-Forward vs. Flag-Off
| Situation | Recommended action | Why |
|-----------|--------------------|-----|
| Defect is behind a feature flag | **Flag off** | Seconds, no redeploy, fully reversible, auditable |
| User-visible or severe defect, flag not involved | **Artifact rollback** | Returns to a known-good state that has run in production |
| Minor defect with a trivial, low-risk fix | **Roll-forward (hotfix)** | Faster than rollback if the fix is certain; still build + test + stage it |
| Destructive schema change already finalized | **Roll-forward with a new migration** | Code rollback is broken after finalization — never combine old code with a finalized schema |
| Data corruption / loss, no forward path | **Backup / point-in-time restore (last resort)** | Slow and lossy; governed by RPO/RTO — escalate first |
> **Gotcha —** `git revert` is not a rollback. It produces new code that must be rebuilt, retested, and redeployed, and it does not undo migrations, data changes, or flag state that shipped with the reverted commit.
## 5. Step-by-Step Rollback
### 5.1 Ordering (microservices)
- Roll **consumers back before producers**: undo the caller's use of the new behavior before removing the provider's capability.
- Assume any service may roll back one version (N-1 compatibility): never depend on a service that could roll back under you.
- With N-1 contracts in place, services roll back independently — no orchestration needed. Coordinated rollback across services is a design smell.
### 5.2 Stateless services / artifacts
1. [Announce in #incident / #releases: "Rolling back <service> to <known-good version> — reason: <observed signal>".]
2. [Re-point the deploy to the known-good artifact — e.g., `kubectl set image deployment/<svc> <svc>=<registry>/<svc>:<good-tag>`, or Argo CD sync to the previous tag, or pipeline "redeploy release <previous>".]
3. [Enable connection draining / graceful termination so in-flight requests finish.]
4. [Warm caches before restoring full traffic to avoid a latency spike.]
5. [Confirm new pods healthy and traffic shifted.]
6. [Blue/green: rollback is a router change — cut traffic back to the blue environment, verify, then keep the bad green environment for inspection.]
### 5.3 Stateful services / databases
1. [Identify the migration phase: initial / transition / finalization. Never roll code back past a finalized schema.]
2. [If code rollback is safe (schema still supports the previous release): redeploy the previous binary; the database stays in the transition phase until a patch is released.]
3. [For feature removal: prefer a new forward migration (append-only, idempotent) over "un-applying" the old one.]
4. [If data is corrupted: escalate to the DB owner; plan backup / point-in-time restore with RPO [X min] and RTO [Y min]; get approval before restoring.]
5. [Manual checkpoint before any destructive step — pause and confirm with the on-call lead.]
### 5.4 Clients / devices (mobile, desktop, IoT)
1. [Rollback is not possible for shipped binaries — use a kill switch / remote config / feature flag to disable the broken behavior.]
2. [Mobile: pause a phased release, then publish the last stable build as a new version with a higher build number, re-signed and re-submitted.]
3. [IoT: rely on A/B (dual-bank) partitions + watchdog auto-revert; validate post-install before switching the active bank.]
4. [Document the version long-tail: some users will keep the bad version for days or indefinitely.]
## 6. Verification (Rollback Is Complete When ...)
| SLI | Target after rollback | Check |
|-----|----------------------|-------|
| Error rate | [back to baseline, e.g., < 0.5%] | [dashboard link] |
| Latency p95 | [back to baseline, e.g., < 250 ms] | [dashboard link] |
| Error budget | [no longer burning] | [budget dashboard] |
| Version breakdown | [100% of traffic on the known-good version] | [version-labeled metrics] |
| Data integrity | [migration / consistency checks green] | [check output] |
> **Gotcha —** verify per-version metrics, not aggregate: subtle failures (e.g., errors only for a subset of users) surface only when most instances run the bad version.
## 7. Communication Plan
| Audience | Channel | Message | When |
|----------|---------|---------|------|
| Internal (eng + on-call) | #incident | Decision + observed signal | Immediately |
| Support | #support | User-facing impact + ETA | Within [15] min |
| Customers / status page | status page | Outage / degradation notice | Within [30] min |
| Post-incident | #postmortems | Rollback changelist + timeline | After resolution |
## 8. Post-Rollback Activities
- [ ] **Quarantine the bad artifact**: label/deny it so it cannot be re-promoted (e.g., remove the tag, blocklist the digest).
- [ ] **Record the rollback changelist** describing the observed problem (see [change-governance-record.md](change-governance-record.md)).
- [ ] **Open a blameless postmortem** within [2] business days; rollbacks are normal, not a failure of the team.
- [ ] Fix the pipeline / thresholds that let the defect through (CI gap, missing canary stage, wrong threshold).
- [ ] Update this runbook and the readiness checklist with lessons learned.
## 9. Rehearsal Log
> Rehearse "just because" every few weeks — find traps (incompatible versions, broken automation) while the release is healthy. If rehearsal breaks, roll forward and fix the cause.
| Date | Rehearsed by | Scenario | Result | Traps found | Follow-up |
|------|--------------|----------|--------|-------------|-----------|
| [YYYY-MM-DD] | [name] | [e.g., canary error-rate spike] | Pass / Fail | [none / description] | [ticket] |
| [YYYY-MM-DD] | [name] | [e.g., migration rollback window] | Pass / Fail | [none / description] | [ticket] |
tests/test_changelog_check.py
"""Tests for changelog_check.py.
Covers: Keep a Changelog and Release Please validation, format selection,
--json output, and exit codes.
Discoverable by both pytest and unittest (unittest.TestCase classes).
"""
import json
import os
import subprocess
import sys
import tempfile
import unittest
SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "scripts")
CHANGELOG_CHECK = os.path.join(SCRIPTS_DIR, "changelog_check.py")
VALID_CHANGELOG = """\
# Changelog
All notable changes to this project will be documented in this file.
## [Unreleased]
### Added
- New dashboard widget.
## [1.1.0] - 2026-07-01
### Added
- Export to CSV.
### Fixed
- Fixed the retry bug.
[Unreleased]: https://github.com/example/proj/compare/v1.1.0...HEAD
[1.1.0]: https://github.com/example/proj/releases/tag/v1.1.0
"""
VALID_RELEASE_PLEASE = """\
# Changelog
## [0.6.0](https://github.com/magnus919/agent-skills/compare/v0.5.0...v0.6.0) (2026-08-03)
### Features
* add a feature ([abc123](https://github.com/example/proj/commit/abc123))
### Bug Fixes
* fix a bug ([def456](https://github.com/example/proj/commit/def456))
### Reverts
* revert an earlier change ([fedcba](https://github.com/example/proj/commit/fedcba))
"""
def run_changelog(args, cwd=None):
"""Run changelog_check.py with given args, return (returncode, stdout, stderr)."""
cmd = [sys.executable, CHANGELOG_CHECK] + args
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=30, cwd=cwd)
return proc.returncode, proc.stdout, proc.stderr
def write_changelog(content):
"""Write changelog content to a temp file and return its path."""
with tempfile.NamedTemporaryFile(mode="w", suffix=".md", delete=False) as tmp:
tmp.write(content)
return tmp.name
class TestChangelogCheckValid(unittest.TestCase):
"""Valid changelogs pass."""
def test_valid_canonical_changelog(self):
"""Canonical Keep a Changelog file exits 0."""
path = write_changelog(VALID_CHANGELOG)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 0)
self.assertIn("valid", out.lower())
finally:
os.unlink(path)
def test_standalone_type_bullets_valid(self):
"""Bullets that name the change type are valid without subsections."""
content = """\
# Changelog
## [Unreleased]
- Added: widget.
- Fixed: crash.
## [1.0.0] - 2026-01-01
- Added: thing.
[Unreleased]: https://example.com/x
[1.0.0]: https://example.com/x
"""
path = write_changelog(content)
try:
rc, _, _ = run_changelog([path])
self.assertEqual(rc, 0)
finally:
os.unlink(path)
def test_prerelease_version_header_valid(self):
"""Pre-release version headers are allowed."""
content = """\
# Changelog
## [Unreleased]
- Added: x.
## [1.2.0-rc.1] - 2026-07-10
- Added: y.
[Unreleased]: https://example.com/x
[1.2.0-rc.1]: https://example.com/x
"""
path = write_changelog(content)
try:
rc, _, _ = run_changelog([path])
self.assertEqual(rc, 0)
finally:
os.unlink(path)
def test_default_changelog_filename(self):
"""Without a path argument, CHANGELOG.md in the cwd is checked."""
with tempfile.TemporaryDirectory() as tmp:
with open(os.path.join(tmp, "CHANGELOG.md"), "w") as fh:
fh.write(VALID_CHANGELOG)
rc, _, _ = run_changelog([], cwd=tmp)
self.assertEqual(rc, 0)
def test_yanked_header_valid(self):
"""A [YANKED] marker is accepted."""
content = """\
# Changelog
## [Unreleased]
- Added: x.
## [1.0.0] - 2026-01-01 [YANKED]
- Added: broken.
[Unreleased]: https://example.com/x
[1.0.0]: https://example.com/x
"""
path = write_changelog(content)
try:
rc, _, _ = run_changelog([path])
self.assertEqual(rc, 0)
finally:
os.unlink(path)
def test_release_please_format_valid(self):
"""Release Please's dated linked headers and star bullets pass."""
path = write_changelog(VALID_RELEASE_PLEASE)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 0)
self.assertIn("release-please", out.lower())
finally:
os.unlink(path)
def test_release_please_explicit_format_valid(self):
"""The Release Please format can be selected explicitly."""
path = write_changelog(VALID_RELEASE_PLEASE)
try:
rc, out, _ = run_changelog([path, "--format", "release-please"])
self.assertEqual(rc, 0)
self.assertIn("release-please", out.lower())
finally:
os.unlink(path)
def test_release_please_custom_section_valid(self):
"""Custom Release Please section names remain valid."""
content = VALID_RELEASE_PLEASE.replace("### Features", "### Documentation")
path = write_changelog(content)
try:
rc, _, _ = run_changelog([path, "--format", "release-please"])
self.assertEqual(rc, 0)
finally:
os.unlink(path)
class TestChangelogCheckProblems(unittest.TestCase):
"""Each Keep a Changelog violation is reported with exit 1."""
def test_missing_title_exit_1(self):
"""A missing '# Changelog' title is a problem."""
content = VALID_CHANGELOG.replace("# Changelog\n\n", "", 1)
path = write_changelog(content)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 1)
self.assertIn("title", out.lower())
finally:
os.unlink(path)
def test_missing_unreleased_exit_1(self):
"""A missing '## [Unreleased]' section is a problem."""
content = VALID_CHANGELOG.replace("## [Unreleased]\n\n### Added\n- New dashboard widget.\n\n", "")
path = write_changelog(content)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 1)
self.assertIn("unreleased", out.lower())
finally:
os.unlink(path)
def test_non_semver_version_exit_1(self):
"""A non-strict-SemVer version header is a problem."""
content = VALID_CHANGELOG.replace("## [1.1.0] - 2026-07-01", "## [1.1] - 2026-07-01")
path = write_changelog(content)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 1)
self.assertIn("semver", out.lower())
finally:
os.unlink(path)
def test_invalid_date_exit_1(self):
"""An impossible calendar date is a problem."""
content = VALID_CHANGELOG.replace("## [1.1.0] - 2026-07-01", "## [1.1.0] - 2026-02-30")
path = write_changelog(content)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 1)
self.assertIn("date", out.lower())
finally:
os.unlink(path)
def test_malformed_header_exit_1(self):
"""A version header missing its date is malformed."""
content = VALID_CHANGELOG.replace("## [1.1.0] - 2026-07-01", "## [1.1.0]")
path = write_changelog(content)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 1)
self.assertIn("malformed", out.lower())
finally:
os.unlink(path)
def test_bad_subsection_heading_exit_1(self):
"""A subsection heading outside the six change types is a problem."""
content = VALID_CHANGELOG.replace("### Fixed", "### Improvements")
path = write_changelog(content)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 1)
self.assertIn("improvements", out.lower())
finally:
os.unlink(path)
def test_standalone_bullet_without_type_exit_1(self):
"""A standalone bullet not naming a change type is a problem."""
content = """\
# Changelog
## [Unreleased]
- Improved speed.
[Unreleased]: https://example.com/x
"""
path = write_changelog(content)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 1)
self.assertIn("added/changed/deprecated/removed/fixed/security", out.lower())
finally:
os.unlink(path)
def test_missing_reference_link_exit_1(self):
"""A version header without a matching link reference is a problem."""
content = VALID_CHANGELOG.replace(
"[1.1.0]: https://github.com/example/proj/releases/tag/v1.1.0\n", ""
)
path = write_changelog(content)
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 1)
self.assertIn("reference link", out.lower())
finally:
os.unlink(path)
def test_empty_file_exit_1(self):
"""An empty changelog is a problem."""
path = write_changelog("")
try:
rc, out, _ = run_changelog([path])
self.assertEqual(rc, 1)
self.assertIn("empty", out.lower())
finally:
os.unlink(path)
def test_malformed_release_please_header_exit_1(self):
"""A Release Please header without its compare link is invalid."""
content = VALID_RELEASE_PLEASE.replace(
"## [0.6.0](https://github.com/magnus919/agent-skills/compare/v0.5.0...v0.6.0) (2026-08-03)",
"## [0.6.0] (2026-08-03)",
)
path = write_changelog(content)
try:
rc, out, _ = run_changelog([path, "--format", "release-please"])
self.assertEqual(rc, 1)
self.assertIn("header", out.lower())
finally:
os.unlink(path)
def test_release_please_rejects_unreleased_section(self):
"""Release Please files must not contain Keep a Changelog Unreleased."""
path = write_changelog(VALID_RELEASE_PLEASE.replace(
"# Changelog\n", "# Changelog\n\n## [Unreleased]\n"
))
try:
rc, out, _ = run_changelog([path, "--format", "release-please"])
self.assertEqual(rc, 1)
self.assertIn("unreleased", out.lower())
finally:
os.unlink(path)
class TestChangelogCheckExitCodes(unittest.TestCase):
"""Exit codes for missing files and usage errors."""
def test_missing_file_exit_1(self):
"""A missing changelog file is an input error."""
rc, _, err = run_changelog(["/nonexistent/CHANGELOG.md"])
self.assertEqual(rc, 1)
self.assertIn("error", err.lower())
def test_bogus_flag_exit_2(self):
"""An unknown flag is a usage error (exit 2)."""
rc, _, err = run_changelog(["--bogus"])
self.assertEqual(rc, 2)
self.assertIn("usage", err.lower())
def test_no_traceback_on_bad_file(self):
"""Errors never produce a traceback."""
rc, out, err = run_changelog(["/nonexistent/CHANGELOG.md"])
self.assertEqual(rc, 1)
self.assertNotIn("Traceback", err)
self.assertNotIn("Traceback", out)
def test_help_exits_0(self):
"""--help exits 0 and describes usage."""
rc, out, _ = run_changelog(["--help"])
self.assertEqual(rc, 0)
self.assertIn("usage", out.lower())
self.assertIn("changelog_check", out.lower())
class TestChangelogCheckJsonOutput(unittest.TestCase):
"""--json output is machine-parseable."""
def test_json_valid(self):
"""Valid changelog --json parses with valid=true."""
path = write_changelog(VALID_CHANGELOG)
try:
rc, out, _ = run_changelog([path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertTrue(data["valid"])
self.assertEqual(data["problem_count"], 0)
self.assertEqual(data["problems"], [])
self.assertEqual(data["format"], "keep-a-changelog")
finally:
os.unlink(path)
def test_json_release_please_reports_format(self):
"""JSON reports the selected and detected Release Please format."""
path = write_changelog(VALID_RELEASE_PLEASE)
try:
rc, out, _ = run_changelog([path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["format"], "release-please")
self.assertEqual(data["detected_format"], "release-please")
finally:
os.unlink(path)
def test_explicit_keep_format_rejects_release_please(self):
"""Explicit Keep a Changelog validation remains strict."""
path = write_changelog(VALID_RELEASE_PLEASE)
try:
rc, out, _ = run_changelog([path, "--format", "keep-a-changelog"])
self.assertEqual(rc, 1)
self.assertIn("unreleased", out.lower())
finally:
os.unlink(path)
def test_json_invalid(self):
"""Invalid changelog --json lists per-line problems."""
path = write_changelog(VALID_CHANGELOG.replace("## [Unreleased]", "## Missing"))
try:
rc, out, _ = run_changelog([path, "--json"])
self.assertEqual(rc, 1)
data = json.loads(out)
self.assertFalse(data["valid"])
self.assertGreater(data["problem_count"], 0)
for problem in data["problems"]:
self.assertIn("line", problem)
self.assertIn("message", problem)
finally:
os.unlink(path)
def test_json_deterministic(self):
"""Two runs produce identical JSON."""
path = write_changelog(VALID_CHANGELOG)
try:
rc1, out1, _ = run_changelog([path, "--json"])
rc2, out2, _ = run_changelog([path, "--json"])
self.assertEqual(rc1, 0)
self.assertEqual(rc2, 0)
self.assertEqual(out1, out2)
finally:
os.unlink(path)
if __name__ == "__main__":
unittest.main()
tests/test_dora_metrics.py
"""Tests for dora_metrics.py.
Covers: the five DORA metrics (deployment frequency, change lead time,
change failure rate, failed deployment recovery time, deployment rework
rate), environment scoping, unavailable-metric guards, --json output, and
exit codes.
Discoverable by both pytest and unittest (unittest.TestCase classes).
"""
import json
import os
import subprocess
import sys
import tempfile
import unittest
SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "scripts")
DORA_METRICS = os.path.join(SCRIPTS_DIR, "dora_metrics.py")
HAPPY_EVENTS = {
"deployments": [
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T00:30:00Z",
"commit_sha": "aaa",
"environment": "prod",
"success": True,
"unplanned": False,
},
{
"started_at": "2026-01-02T00:00:00Z",
"finished_at": "2026-01-02T00:30:00Z",
"commit_sha": "bbb",
"environment": "prod",
"success": False,
"unplanned": True,
},
{
"started_at": "2026-01-02T01:00:00Z",
"finished_at": "2026-01-02T01:15:00Z",
"commit_sha": "ccc",
"environment": "prod",
"success": True,
"unplanned": True,
},
{
"started_at": "2026-01-03T00:00:00Z",
"finished_at": "2026-01-03T00:30:00Z",
"commit_sha": "ddd",
"environment": "prod",
"success": True,
"unplanned": False,
},
],
"commits": [
{"sha": "aaa", "created_at": "2025-12-31T12:00:00Z"},
{"sha": "bbb", "created_at": "2026-01-01T12:00:00Z"},
{"sha": "ccc", "created_at": "2026-01-02T00:45:00Z"},
{"sha": "ddd", "created_at": "2026-01-03T00:00:00Z"},
],
}
def run_dora(args):
"""Run dora_metrics.py with given args, return (returncode, stdout, stderr)."""
cmd = [sys.executable, DORA_METRICS] + args
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
return proc.returncode, proc.stdout, proc.stderr
def write_events(events):
"""Write an events dict to a temp JSON file and return its path."""
with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as tmp:
json.dump(events, tmp)
return tmp.name
class TestDoraMetricsHappyPath(unittest.TestCase):
"""All five metrics compute correctly on well-formed input."""
def setUp(self):
self.path = write_events(HAPPY_EVENTS)
def tearDown(self):
os.unlink(self.path)
def test_json_metrics_values(self):
"""Each metric matches the hand-computed expectation."""
rc, out, _ = run_dora(["--events", self.path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
# Deployment frequency: 3 successful over the observation window.
self.assertTrue(data["deployment_frequency"]["available"])
self.assertEqual(data["deployment_frequency"]["successful_deployments"], 3)
self.assertAlmostEqual(
data["deployment_frequency"]["deployments_per_day"],
3.0 / 2.020833,
places=4,
)
# Change lead time: median of finished_at - commit created_at over
# successful deploys with commit data: [12.5h, 30m, 30m] -> 30m.
self.assertTrue(data["change_lead_time"]["available"])
self.assertAlmostEqual(data["change_lead_time"]["seconds"], 1800.0, places=3)
self.assertEqual(data["change_lead_time"]["deployments_measured"], 3)
# Change failure rate: 1 failed of 4.
self.assertTrue(data["change_failure_rate"]["available"])
self.assertAlmostEqual(data["change_failure_rate"]["percent"], 25.0, places=4)
# Failed deployment recovery time: failed bbb -> next success ccc
# finished 01:15 Jan 2, bbb started 00:00 Jan 2 -> 75 minutes.
self.assertTrue(data["failed_deployment_recovery_time"]["available"])
self.assertAlmostEqual(
data["failed_deployment_recovery_time"]["seconds"], 4500.0, places=3
)
# Deployment rework rate: 2 unplanned of 4.
self.assertTrue(data["deployment_rework_rate"]["available"])
self.assertAlmostEqual(data["deployment_rework_rate"]["percent"], 50.0, places=4)
def test_counts(self):
"""Counts reflect the input deployments."""
rc, out, _ = run_dora(["--events", self.path, "--json"])
data = json.loads(out)
counts = data["deployments"]
self.assertEqual(
counts, {"total": 4, "successful": 3, "failed": 1, "unplanned": 2}
)
def test_human_output_default(self):
"""Without --json, output is a human-readable table."""
rc, out, _ = run_dora(["--events", self.path])
self.assertEqual(rc, 0)
self.assertIn("deployment frequency", out)
self.assertIn("change lead time", out)
self.assertIn("change failure rate", out)
self.assertIn("failed deployment recovery time", out)
self.assertIn("deployment rework rate", out)
class TestDoraMetricsGuards(unittest.TestCase):
"""Unavailable-metric guards."""
def test_unrecovered_failed_deploy_fdrt_unavailable(self):
"""FDRT is unavailable when a failed deploy has no later success."""
events = {
"deployments": [
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T00:10:00Z",
"commit_sha": None,
"environment": "prod",
"success": True,
"unplanned": False,
},
{
"started_at": "2026-01-02T00:00:00Z",
"finished_at": "2026-01-02T00:10:00Z",
"commit_sha": None,
"environment": "prod",
"success": False,
"unplanned": True,
},
],
"commits": [],
}
path = write_events(events)
try:
rc, out, _ = run_dora(["--events", path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
fdrt = data["failed_deployment_recovery_time"]
self.assertFalse(fdrt["available"])
self.assertIn("not yet recovered", fdrt["reason"])
finally:
os.unlink(path)
def test_no_failed_deployments_fdrt_unavailable(self):
"""FDRT is unavailable when there are no failed deployments."""
events = {
"deployments": [
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T00:10:00Z",
"commit_sha": None,
"environment": "prod",
"success": True,
"unplanned": False,
}
],
"commits": [],
}
path = write_events(events)
try:
rc, out, _ = run_dora(["--events", path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
fdrt = data["failed_deployment_recovery_time"]
self.assertFalse(fdrt["available"])
self.assertIn("no failed deployments", fdrt["reason"])
finally:
os.unlink(path)
def test_missing_commit_timestamps_clt_unavailable(self):
"""CLT is unavailable when no deployment has commit timestamp data."""
events = {
"deployments": [
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T00:10:00Z",
"commit_sha": "aaa",
"environment": "prod",
"success": True,
"unplanned": False,
}
],
"commits": [],
}
path = write_events(events)
try:
rc, out, _ = run_dora(["--events", path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
clt = data["change_lead_time"]
self.assertFalse(clt["available"])
self.assertEqual(clt["deployments_measured"], 0)
self.assertEqual(clt["deployments_without_commit_data"], 1)
finally:
os.unlink(path)
def test_no_deployments_all_unavailable(self):
"""An empty deployments array yields unavailable metrics, exit 0."""
path = write_events({"deployments": [], "commits": []})
try:
rc, out, _ = run_dora(["--events", path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
for key in (
"deployment_frequency",
"change_lead_time",
"change_failure_rate",
"failed_deployment_recovery_time",
"deployment_rework_rate",
):
self.assertFalse(data[key]["available"], key)
finally:
os.unlink(path)
def test_environment_scoping(self):
"""Deployments in other environments are excluded."""
events = {
"deployments": [
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T00:10:00Z",
"commit_sha": None,
"environment": "staging",
"success": False,
"unplanned": True,
}
],
"commits": [],
}
path = write_events(events)
try:
rc, out, _ = run_dora(["--events", path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["deployments"]["total"], 0)
self.assertFalse(data["deployment_frequency"]["available"])
finally:
os.unlink(path)
def test_custom_environment(self):
"""--environment selects a non-default environment."""
path = write_events(HAPPY_EVENTS)
try:
rc, out, _ = run_dora(
["--events", path, "--environment", "staging", "--json"]
)
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["environment"], "staging")
self.assertEqual(data["deployments"]["total"], 0)
finally:
os.unlink(path)
class TestDoraMetricsEdgeBehavior(unittest.TestCase):
"""Edge behavior locked by review fixes: environment-scoped window,
change-lead-time clamping, and recovery-candidate ordering."""
def test_mixed_environment_window_scoped_to_environment(self):
"""DF window covers only the selected environment's deployments."""
events = {
"deployments": [
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T00:30:00Z",
"commit_sha": None,
"environment": "prod",
"success": True,
"unplanned": False,
},
{
"started_at": "2026-01-03T00:00:00Z",
"finished_at": "2026-01-03T00:30:00Z",
"commit_sha": None,
"environment": "prod",
"success": True,
"unplanned": False,
},
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T00:30:00Z",
"commit_sha": None,
"environment": "staging",
"success": True,
"unplanned": False,
},
{
"started_at": "2026-01-10T00:00:00Z",
"finished_at": "2026-01-10T00:30:00Z",
"commit_sha": None,
"environment": "staging",
"success": True,
"unplanned": False,
},
],
"commits": [],
}
path = write_events(events)
try:
rc, out, _ = run_dora(["--events", path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
df = data["deployment_frequency"]
self.assertTrue(df["available"])
# Prod window is Jan 1 00:00 -> Jan 3 00:30 (2 days + 30 min),
# NOT the all-environments window that extends to Jan 10.
self.assertAlmostEqual(df["window_days"], 2.020833, places=4)
self.assertAlmostEqual(
df["deployments_per_day"], 2.0 / 2.020833, places=4
)
self.assertEqual(df["successful_deployments"], 2)
finally:
os.unlink(path)
def test_negative_change_lead_time_clamped_to_zero(self):
"""A deploy finishing before its commit was created yields CLT 0."""
events = {
"deployments": [
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T00:30:00Z",
"commit_sha": "aaa",
"environment": "prod",
"success": True,
"unplanned": False,
}
],
"commits": [{"sha": "aaa", "created_at": "2026-01-02T12:00:00Z"}],
}
path = write_events(events)
try:
rc, out, _ = run_dora(["--events", path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
clt = data["change_lead_time"]
self.assertTrue(clt["available"])
self.assertEqual(clt["seconds"], 0.0)
finally:
os.unlink(path)
def test_recovery_must_start_after_failed_deploy_finished(self):
"""A success that began mid-failure is not counted as recovery."""
events = {
"deployments": [
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T01:00:00Z",
"commit_sha": None,
"environment": "prod",
"success": False,
"unplanned": True,
},
{
# Started mid-failure (before the failed deploy
# finished at 01:00) — must NOT count as recovery.
"started_at": "2026-01-01T00:30:00Z",
"finished_at": "2026-01-01T00:45:00Z",
"commit_sha": None,
"environment": "prod",
"success": True,
"unplanned": False,
},
{
"started_at": "2026-01-01T01:30:00Z",
"finished_at": "2026-01-01T02:00:00Z",
"commit_sha": None,
"environment": "prod",
"success": True,
"unplanned": False,
},
],
"commits": [],
}
path = write_events(events)
try:
rc, out, _ = run_dora(["--events", path, "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
fdrt = data["failed_deployment_recovery_time"]
self.assertTrue(fdrt["available"])
# Recovery = post-failure success finished (02:00) minus failed
# started (00:00) = 2 hours; the mid-failure success is excluded.
self.assertAlmostEqual(fdrt["seconds"], 7200.0, places=3)
self.assertEqual(fdrt["recovered"], 1)
self.assertEqual(fdrt["unrecovered"], 0)
finally:
os.unlink(path)
class TestDoraMetricsExitCodes(unittest.TestCase):
"""Exit codes for malformed input and usage errors."""
def test_missing_events_exit_2(self):
"""Missing --events is a usage error (exit 2)."""
rc, _, err = run_dora([])
self.assertEqual(rc, 2)
self.assertIn("usage", err.lower())
def test_malformed_json_exit_1(self):
"""Invalid JSON is an input error."""
with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as tmp:
tmp.write("{not json")
path = tmp.name
try:
rc, _, err = run_dora(["--events", path])
self.assertEqual(rc, 1)
self.assertIn("invalid json", err.lower())
finally:
os.unlink(path)
def test_missing_file_exit_1(self):
"""A missing events file is an input error."""
rc, _, err = run_dora(["--events", "/nonexistent/events.json"])
self.assertEqual(rc, 1)
self.assertIn("error", err.lower())
def test_success_must_be_boolean_exit_1(self):
"""A non-boolean success field is an input error."""
events = {
"deployments": [
{
"started_at": "2026-01-01T00:00:00Z",
"finished_at": "2026-01-01T00:10:00Z",
"commit_sha": None,
"environment": "prod",
"success": "true",
"unplanned": False,
}
],
"commits": [],
}
path = write_events(events)
try:
rc, _, err = run_dora(["--events", path])
self.assertEqual(rc, 1)
self.assertIn("boolean", err.lower())
finally:
os.unlink(path)
def test_invalid_timestamp_exit_1(self):
"""An unparseable timestamp is an input error."""
events = {
"deployments": [
{
"started_at": "not-a-date",
"finished_at": "2026-01-01T00:10:00Z",
"commit_sha": None,
"environment": "prod",
"success": True,
"unplanned": False,
}
],
"commits": [],
}
path = write_events(events)
try:
rc, _, err = run_dora(["--events", path])
self.assertEqual(rc, 1)
self.assertIn("started_at", err.lower())
finally:
os.unlink(path)
def test_no_traceback_on_malformed(self):
"""Malformed input never produces a traceback."""
with tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False) as tmp:
tmp.write("{broken")
path = tmp.name
try:
rc, out, err = run_dora(["--events", path])
self.assertNotEqual(rc, 0)
self.assertNotIn("Traceback", err)
self.assertNotIn("Traceback", out)
finally:
os.unlink(path)
def test_help_exits_0(self):
"""--help exits 0 and describes usage."""
rc, out, _ = run_dora(["--help"])
self.assertEqual(rc, 0)
self.assertIn("usage", out.lower())
self.assertIn("dora_metrics", out.lower())
class TestDoraMetricsDeterminism(unittest.TestCase):
"""Output is deterministic across runs."""
def test_json_deterministic(self):
"""Two runs produce identical JSON."""
path = write_events(HAPPY_EVENTS)
try:
rc1, out1, _ = run_dora(["--events", path, "--json"])
rc2, out2, _ = run_dora(["--events", path, "--json"])
self.assertEqual(rc1, 0)
self.assertEqual(rc2, 0)
self.assertEqual(out1, out2)
finally:
os.unlink(path)
if __name__ == "__main__":
unittest.main()
tests/test_release_plan_scaffold.py
"""Tests for release_plan_scaffold.py.
Covers: rendered document structure, --output file writing, --json output
with parsed fields, --git-range scope source, determinism, and exit codes.
Discoverable by both pytest and unittest (unittest.TestCase classes).
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "scripts")
PLAN_SCAFFOLD = os.path.join(SCRIPTS_DIR, "release_plan_scaffold.py")
BASE_ARGS = [
"--version", "1.2.3",
"--name", "Acme 1.2.3",
"--date", "2026-08-15",
"--owner", "Jane Doe",
"--milestones", "Branch cut;Release candidate;GA",
"--scope", "feat: new API;fix: retry bug",
"--risks", "DB migration late;API contract change",
]
def run_plan(args, cwd=None):
"""Run release_plan_scaffold.py with given args, return (rc, stdout, stderr)."""
cmd = [sys.executable, PLAN_SCAFFOLD] + args
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=30, cwd=cwd)
return proc.returncode, proc.stdout, proc.stderr
class TestReleasePlanScaffoldRender(unittest.TestCase):
"""Rendered document structure."""
def test_document_contains_fields(self):
"""The rendered markdown contains the provided fields."""
rc, out, _ = run_plan(BASE_ARGS)
self.assertEqual(rc, 0)
self.assertIn("# Release Plan: Acme 1.2.3", out)
self.assertIn("| Version | 1.2.3 |", out)
self.assertIn("| Target date (GA) | 2026-08-15 |", out)
self.assertIn("| Release manager (DRI) | Jane Doe |", out)
def test_document_contains_lists(self):
"""Milestones, scope, and risks are rendered as tables."""
rc, out, _ = run_plan(BASE_ARGS)
self.assertEqual(rc, 0)
self.assertIn("| Branch cut | TBD | TBD | TBD |", out)
self.assertIn("| S-1 | feat: new API | feat | — |", out)
self.assertIn("| R-1 | DB migration late | TBD | TBD | TBD | TBD |", out)
def test_standard_sections_present(self):
"""The release plan template structure is complete."""
rc, out, _ = run_plan(BASE_ARGS)
self.assertEqual(rc, 0)
for section in (
"## Metadata",
"## 1. Overview and Scope",
"## 2. Versioning and Artifacts",
"## 3. Timeline and Milestones",
"## 4. Owners and RACI",
"## 5. Risks and Mitigations",
"## 6. Rollout Plan",
"## 7. Rollback Contingency",
"## 8. Communication Plan",
"## 9. Post-Release Monitoring",
"## 10. Sign-offs",
):
self.assertIn(section, out)
def test_default_name_and_placeholders(self):
"""Unspecified fields fall back to deterministic TBD placeholders."""
rc, out, _ = run_plan(["--version", "1.2.3"])
self.assertEqual(rc, 0)
self.assertIn("# Release Plan: Release 1.2.3", out)
self.assertIn("| Target date (GA) | TBD |", out)
self.assertIn("| Release manager (DRI) | TBD |", out)
def test_empty_lists_render_tbd(self):
"""No scope/milestones/risks render TBD placeholder rows."""
rc, out, _ = run_plan(["--version", "1.2.3"])
self.assertEqual(rc, 0)
self.assertIn("_TBD_", out)
def test_deterministic_output(self):
"""Two runs on identical input produce identical output."""
rc1, out1, _ = run_plan(BASE_ARGS)
rc2, out2, _ = run_plan(BASE_ARGS)
self.assertEqual(rc1, 0)
self.assertEqual(rc2, 0)
self.assertEqual(out1, out2)
class TestReleasePlanScaffoldOutput(unittest.TestCase):
"""--output and --json."""
def test_output_writes_file(self):
"""--output writes the rendered markdown to the file."""
with tempfile.TemporaryDirectory() as tmp:
target = os.path.join(tmp, "release-plan.md")
rc, out, _ = run_plan(BASE_ARGS + ["--output", target])
self.assertEqual(rc, 0)
self.assertEqual(out, "")
with open(target, "r") as fh:
content = fh.read()
self.assertIn("# Release Plan: Acme 1.2.3", content)
def test_json_parseable(self):
"""--json emits parsed fields plus the rendered document."""
rc, out, _ = run_plan(BASE_ARGS + ["--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["version"], "1.2.3")
self.assertEqual(data["name"], "Acme 1.2.3")
self.assertEqual(data["date"], "2026-08-15")
self.assertEqual(data["owner"], "Jane Doe")
self.assertEqual(data["milestones"], ["Branch cut", "Release candidate", "GA"])
self.assertEqual(data["scope"], ["feat: new API", "fix: retry bug"])
self.assertEqual(data["risks"], ["DB migration late", "API contract change"])
self.assertEqual(data["scope_source"], "flags")
self.assertIn("# Release Plan: Acme 1.2.3", data["document"])
def test_json_with_output_writes_both(self):
"""--json plus --output writes the file and prints JSON."""
with tempfile.TemporaryDirectory() as tmp:
target = os.path.join(tmp, "plan.md")
rc, out, _ = run_plan(BASE_ARGS + ["--json", "--output", target])
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["version"], "1.2.3")
with open(target, "r") as fh:
self.assertIn("# Release Plan: Acme 1.2.3", fh.read())
def test_json_deterministic(self):
"""Two --json runs produce identical output."""
rc1, out1, _ = run_plan(BASE_ARGS + ["--json"])
rc2, out2, _ = run_plan(BASE_ARGS + ["--json"])
self.assertEqual(rc1, 0)
self.assertEqual(rc2, 0)
self.assertEqual(out1, out2)
class TestReleasePlanScaffoldGitRange(unittest.TestCase):
"""--git-range pulls scope from commit subjects."""
@unittest.skipUnless(shutil.which("git"), "git not available")
def test_git_range_scope(self):
"""Commit subjects become the scope items."""
with tempfile.TemporaryDirectory() as tmp:
subprocess.run(["git", "init", "-q"], cwd=tmp, check=True)
subprocess.run(
["git", "-c", "user.name=t", "-c", "user.email=t@t",
"commit", "--allow-empty", "-q", "-m", "chore: init"],
cwd=tmp, check=True,
)
first_sha = subprocess.run(
["git", "rev-parse", "HEAD"], cwd=tmp, check=True,
capture_output=True, text=True,
).stdout.strip()
for message in ("feat: widget", "fix: retry"):
subprocess.run(
["git", "-c", "user.name=t", "-c", "user.email=t@t",
"commit", "--allow-empty", "-q", "-m", message],
cwd=tmp, check=True,
)
rc, out, _ = run_plan(
["--version", "1.2.3", "--git-range", "{}..HEAD".format(first_sha),
"--json"],
cwd=tmp,
)
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["scope_source"], "git-range")
self.assertEqual(data["scope"], ["feat: widget", "fix: retry"])
# Short commit SHAs are surfaced for the in-scope table.
self.assertEqual(len(data["scope_sources"]), 2)
self.assertTrue(all(data["scope_sources"]))
self.assertIn("feat: widget", data["document"])
self.assertIn("| feat |", data["document"])
def test_git_range_failure_exit_1(self):
"""An invalid git range is an input error."""
rc, _, err = run_plan(
["--version", "1.2.3", "--git-range", "nope..HEAD"]
)
self.assertEqual(rc, 1)
self.assertIn("error", err.lower())
class TestReleasePlanScaffoldExitCodes(unittest.TestCase):
"""Exit codes and usage errors."""
def test_missing_version_exit_2(self):
"""Missing --version is a usage error (exit 2)."""
rc, _, err = run_plan(["--name", "X"])
self.assertEqual(rc, 2)
self.assertIn("usage", err.lower())
def test_scope_and_git_range_conflict_exit_2(self):
"""--scope and --git-range together is a usage error."""
rc, _, _ = run_plan(
["--version", "1.2.3", "--scope", "a;b", "--git-range", "main..HEAD"]
)
self.assertEqual(rc, 2)
def test_unwritable_output_exit_1(self):
"""An unwritable output path is an input error."""
rc, _, err = run_plan(BASE_ARGS + ["--output", "/nonexistent/dir/plan.md"])
self.assertEqual(rc, 1)
self.assertIn("error", err.lower())
def test_no_traceback_on_error(self):
"""Errors never produce a traceback."""
rc, out, err = run_plan(
["--version", "1.2.3", "--git-range", "nope..HEAD"]
)
self.assertNotEqual(rc, 0)
self.assertNotIn("Traceback", err)
self.assertNotIn("Traceback", out)
def test_help_exits_0(self):
"""--help exits 0 and describes usage."""
rc, out, _ = run_plan(["--help"])
self.assertEqual(rc, 0)
self.assertIn("usage", out.lower())
self.assertIn("release_plan_scaffold", out.lower())
if __name__ == "__main__":
unittest.main()
tests/test_semver_check.py
"""Tests for semver_check.py.
Covers: strict SemVer validation (--check), precedence comparison
(--compare), precedence-aware sorting (--sort), --json output, and exit
codes.
Discoverable by both pytest and unittest (unittest.TestCase classes).
"""
import json
import os
import subprocess
import sys
import unittest
SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "scripts")
SEMVER_CHECK = os.path.join(SCRIPTS_DIR, "semver_check.py")
def run_semver(args):
"""Run semver_check.py with given args, return (returncode, stdout, stderr)."""
cmd = [sys.executable, SEMVER_CHECK] + args
proc = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
return proc.returncode, proc.stdout, proc.stderr
class TestSemverCheckValidation(unittest.TestCase):
"""--check validates strict SemVer 2.0.0."""
def test_valid_version_exit_0(self):
"""A conforming version exits 0."""
rc, out, _ = run_semver(["--check", "1.2.3"])
self.assertEqual(rc, 0)
self.assertIn("valid", out)
def test_valid_with_prerelease(self):
"""Pre-release versions are valid."""
rc, out, _ = run_semver(["--check", "1.2.3-alpha.1"])
self.assertEqual(rc, 0)
self.assertIn("valid", out)
def test_valid_with_build_metadata(self):
"""Build metadata is valid."""
rc, _, _ = run_semver(["--check", "1.2.3+build.5"])
self.assertEqual(rc, 0)
def test_zero_version_valid(self):
"""0.x.y versions are valid."""
rc, _, _ = run_semver(["--check", "0.0.0"])
self.assertEqual(rc, 0)
def test_leading_zero_invalid_exit_1(self):
"""Leading zeros are rejected."""
rc, out, _ = run_semver(["--check", "01.2.3"])
self.assertEqual(rc, 1)
self.assertIn("invalid", out)
self.assertIn("leading zeros", out)
def test_missing_patch_invalid_exit_1(self):
"""A two-part version is invalid."""
rc, out, _ = run_semver(["--check", "1.2"])
self.assertEqual(rc, 1)
self.assertIn("invalid", out)
def test_non_numeric_component_invalid(self):
"""Non-numeric components are invalid."""
rc, out, _ = run_semver(["--check", "1.x.3"])
self.assertEqual(rc, 1)
self.assertIn("invalid", out)
def test_empty_version_invalid(self):
"""An empty version is invalid."""
rc, out, _ = run_semver(["--check", ""])
self.assertEqual(rc, 1)
self.assertIn("invalid", out)
def test_v_prefix_invalid(self):
"""A 'v' prefix is not part of SemVer."""
rc, out, _ = run_semver(["--check", "v1.2.3"])
self.assertEqual(rc, 1)
self.assertIn("invalid", out)
def test_leading_zero_prerelease_invalid(self):
"""Numeric pre-release identifiers must not have leading zeros."""
rc, _, _ = run_semver(["--check", "1.2.3-alpha.01"])
self.assertEqual(rc, 1)
class TestSemverCheckCompare(unittest.TestCase):
"""--compare applies SemVer precedence."""
def test_compare_lt(self):
"""Lower precedence prints lt."""
rc, out, _ = run_semver(["--compare", "1.2.3", "1.2.4"])
self.assertEqual(rc, 0)
self.assertEqual(out.strip(), "lt")
def test_compare_gt(self):
"""Higher precedence prints gt."""
rc, out, _ = run_semver(["--compare", "2.0.0", "1.9.9"])
self.assertEqual(rc, 0)
self.assertEqual(out.strip(), "gt")
def test_compare_eq(self):
"""Equal precedence prints eq."""
rc, out, _ = run_semver(["--compare", "1.2.3", "1.2.3"])
self.assertEqual(rc, 0)
self.assertEqual(out.strip(), "eq")
def test_prerelease_sorts_below_release(self):
"""1.0.0-alpha < 1.0.0."""
rc, out, _ = run_semver(["--compare", "1.0.0-alpha", "1.0.0"])
self.assertEqual(rc, 0)
self.assertEqual(out.strip(), "lt")
def test_prerelease_numeric_below_alphanumeric(self):
"""Numeric pre-release identifiers sort below alphanumeric ones."""
rc, out, _ = run_semver(["--compare", "1.0.0-1", "1.0.0-alpha"])
self.assertEqual(rc, 0)
self.assertEqual(out.strip(), "lt")
def test_build_metadata_ignored(self):
"""Build metadata does not affect precedence."""
rc, out, _ = run_semver(["--compare", "1.0.0+a", "1.0.0+b"])
self.assertEqual(rc, 0)
self.assertEqual(out.strip(), "eq")
def test_invalid_compare_exit_1(self):
"""An invalid version in --compare is an input error."""
rc, _, err = run_semver(["--compare", "1.2.3", "nope"])
self.assertEqual(rc, 1)
self.assertIn("invalid", err.lower())
class TestSemverCheckSort(unittest.TestCase):
"""--sort orders versions ascending by precedence."""
def test_sort_basic(self):
"""Simple numeric ordering."""
rc, out, _ = run_semver(["--sort", "1.2.3", "1.0.0", "1.2.0"])
self.assertEqual(rc, 0)
lines = out.splitlines()
self.assertEqual(lines, ["1.0.0", "1.2.0", "1.2.3"])
def test_sort_prerelease_before_release(self):
"""Pre-releases sort before the same core without one."""
rc, out, _ = run_semver(
["--sort", "1.0.0", "1.0.0-beta.2", "1.0.0-alpha.1"]
)
self.assertEqual(rc, 0)
lines = out.splitlines()
self.assertEqual(lines, ["1.0.0-alpha.1", "1.0.0-beta.2", "1.0.0"])
def test_sort_build_metadata_keeps_input_order(self):
"""Equal-precedence versions (build only differs) keep input order."""
rc, out, _ = run_semver(["--sort", "1.0.0+b", "1.0.0+a", "1.0.0+c"])
self.assertEqual(rc, 0)
lines = out.splitlines()
self.assertEqual(lines, ["1.0.0+b", "1.0.0+a", "1.0.0+c"])
def test_sort_invalid_exit_1(self):
"""An invalid version in --sort is an input error."""
rc, _, err = run_semver(["--sort", "1.0.0", "2.x.0"])
self.assertEqual(rc, 1)
self.assertIn("invalid", err.lower())
class TestSemverCheckJsonOutput(unittest.TestCase):
"""--json output is machine-parseable."""
def test_check_json_valid(self):
"""Valid --check --json parses with fields."""
rc, out, _ = run_semver(["--check", "1.2.3", "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertTrue(data["valid"])
self.assertEqual(data["version"], "1.2.3")
self.assertEqual(data["parsed"]["major"], 1)
def test_check_json_invalid(self):
"""Invalid --check --json carries a reason."""
rc, out, _ = run_semver(["--check", "01.2.3", "--json"])
self.assertEqual(rc, 1)
data = json.loads(out)
self.assertFalse(data["valid"])
self.assertIsNotNone(data["reason"])
def test_compare_json(self):
"""--compare --json parses with a relation."""
rc, out, _ = run_semver(["--compare", "1.0.0", "1.0.1", "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["relation"], "lt")
def test_sort_json(self):
"""--sort --json parses with a sorted list."""
rc, out, _ = run_semver(["--sort", "2.0.0", "1.0.0", "--json"])
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["sorted"], ["1.0.0", "2.0.0"])
class TestSemverCheckExitCodes(unittest.TestCase):
"""Exit codes and usage errors."""
def test_no_mode_exit_2(self):
"""No mode given is a usage error (exit 2)."""
rc, _, err = run_semver([])
self.assertEqual(rc, 2)
self.assertIn("usage", err.lower())
def test_conflicting_modes_exit_2(self):
"""--check and --compare together is a usage error."""
rc, _, _ = run_semver(["--check", "1.0.0", "--compare", "1.0.0", "2.0.0"])
self.assertEqual(rc, 2)
def test_no_traceback_on_invalid(self):
"""Invalid input never produces a traceback."""
rc, out, err = run_semver(["--check", "1.2"])
self.assertEqual(rc, 1)
self.assertNotIn("Traceback", err)
self.assertNotIn("Traceback", out)
def test_help_exits_0(self):
"""--help exits 0 and describes usage."""
rc, out, _ = run_semver(["--help"])
self.assertEqual(rc, 0)
self.assertIn("usage", out.lower())
self.assertIn("semver_check", out.lower())
if __name__ == "__main__":
unittest.main()
tests/test_version_bump.py
"""Tests for version_bump.py.
Covers: Conventional Commits bump rules (feat/fix/breaking/! and 0.x
handling), pre-release increment semantics, --from-file parsing,
--git-range, --json output, and exit codes.
Discoverable by both pytest and unittest (unittest.TestCase classes).
"""
import json
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "scripts")
VERSION_BUMP = os.path.join(SCRIPTS_DIR, "version_bump.py")
def run_version_bump(args, cwd=None):
"""Run version_bump.py with given args, return (returncode, stdout, stderr)."""
cmd = [sys.executable, VERSION_BUMP] + args
proc = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=30,
cwd=cwd,
)
return proc.returncode, proc.stdout, proc.stderr
def write_temp_file(content, suffix=".txt"):
"""Write content to a temp file and return its path (caller unlinks)."""
with tempfile.NamedTemporaryFile(mode="w", suffix=suffix, delete=False) as tmp:
tmp.write(content)
return tmp.name
class TestVersionBumpBumpRules(unittest.TestCase):
"""Conventional Commits bump rules."""
def test_feat_bumps_minor(self):
"""feat commits bump minor."""
commits = write_temp_file("feat: add login\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("1.3.0", out)
finally:
os.unlink(commits)
def test_fix_bumps_patch(self):
"""fix commits bump patch."""
commits = write_temp_file("fix(api): retry timeout\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("1.2.4", out)
finally:
os.unlink(commits)
def test_breaking_change_footer_bumps_major(self):
"""BREAKING CHANGE footer bumps major."""
commits = write_temp_file("feat: new API\n\nBREAKING CHANGE: endpoint renamed\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("2.0.0", out)
finally:
os.unlink(commits)
def test_bang_type_bumps_major(self):
"""feat! bumps major."""
commits = write_temp_file("feat!: drop legacy endpoint\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("2.0.0", out)
finally:
os.unlink(commits)
def test_bang_scope_bumps_major(self):
"""feat(scope)! bumps major."""
commits = write_temp_file("feat(api)!: rename field\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("2.0.0", out)
finally:
os.unlink(commits)
def test_zero_major_breaking_bumps_minor(self):
"""On a 0.x version, breaking changes bump minor instead of major."""
commits = write_temp_file("feat!: unstable api change\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "0.2.0", "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("0.3.0", out)
finally:
os.unlink(commits)
def test_zero_major_feat_bumps_minor(self):
"""On a 0.x version, feat follows Release Please and bumps minor."""
commits = write_temp_file("feat: add helper\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "0.5.0", "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("0.6.0", out)
finally:
os.unlink(commits)
def test_other_types_bump_patch(self):
"""chore/docs/refactor commits bump patch."""
commits = write_temp_file("chore: bump deps\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("1.2.4", out)
finally:
os.unlink(commits)
def test_breaking_wins_over_feat(self):
"""A breaking change dominates any feat in the batch."""
commits = write_temp_file("feat: add widget\nfix!: remove endpoint\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("2.0.0", out)
finally:
os.unlink(commits)
class TestVersionBumpPrerelease(unittest.TestCase):
"""Pre-release tag semantics."""
def test_new_prerelease_starts_at_one(self):
"""A fresh pre-release starts at tag.1."""
commits = write_temp_file("feat: add login\n")
try:
rc, out, _ = run_version_bump(
[
"--current-version", "1.2.3",
"--commits-file", commits,
"--pre-release", "rc",
]
)
self.assertEqual(rc, 0)
self.assertIn("1.3.0-rc.1", out)
finally:
os.unlink(commits)
def test_same_prerelease_series_increments(self):
"""Same tag and numeric suffix increments without bumping core."""
commits = write_temp_file("feat: add login\n")
try:
rc, out, _ = run_version_bump(
[
"--current-version", "1.2.0-alpha.1",
"--commits-file", commits,
"--pre-release", "alpha",
]
)
self.assertEqual(rc, 0)
self.assertIn("1.2.0-alpha.2", out)
finally:
os.unlink(commits)
def test_different_prerelease_tag_restarts(self):
"""A different tag restarts the series on a fresh core bump."""
commits = write_temp_file("feat: add login\n")
try:
rc, out, _ = run_version_bump(
[
"--current-version", "1.2.0-alpha.1",
"--commits-file", commits,
"--pre-release", "beta",
]
)
self.assertEqual(rc, 0)
self.assertIn("1.3.0-beta.1", out)
finally:
os.unlink(commits)
def test_invalid_pre_release_choice_exit_2(self):
"""An unknown pre-release tag is a usage error (exit 2)."""
commits = write_temp_file("feat: x\n")
try:
rc, _, _ = run_version_bump(
[
"--current-version", "1.2.3",
"--commits-file", commits,
"--pre-release", "gamma",
]
)
self.assertEqual(rc, 2)
finally:
os.unlink(commits)
class TestVersionBumpJsonOutput(unittest.TestCase):
"""--json output is machine-parseable."""
def test_json_parseable(self):
"""--json output parses and has expected fields."""
commits = write_temp_file("feat: add login\nfix: retry\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits, "--json"]
)
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["current_version"], "1.2.3")
self.assertEqual(data["next_version"], "1.3.0")
self.assertEqual(data["bump"], "minor")
self.assertEqual(data["commit_count"], 2)
self.assertEqual(data["breaking_count"], 0)
self.assertEqual(data["source"], "commits-file")
finally:
os.unlink(commits)
def test_json_reports_breaking(self):
"""Breaking counts are surfaced in JSON."""
commits = write_temp_file("feat!: break\n")
try:
rc, out, _ = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits, "--json"]
)
self.assertEqual(rc, 0)
data = json.loads(out)
self.assertEqual(data["breaking_count"], 1)
self.assertEqual(data["bump"], "major")
finally:
os.unlink(commits)
def test_deterministic_output(self):
"""Two runs on the same input produce identical JSON."""
commits = write_temp_file("feat: a\nfix: b\nchore: c\n")
try:
rc1, out1, _ = run_version_bump(
["--current-version", "1.0.0", "--commits-file", commits, "--json"]
)
rc2, out2, _ = run_version_bump(
["--current-version", "1.0.0", "--commits-file", commits, "--json"]
)
self.assertEqual(rc1, 0)
self.assertEqual(rc2, 0)
self.assertEqual(out1, out2)
finally:
os.unlink(commits)
class TestVersionBumpFromFile(unittest.TestCase):
"""--from-file reads package.json / pyproject.toml versions."""
def test_package_json(self):
"""Reads version from package.json."""
commits = write_temp_file("feat: add login\n")
package = write_temp_file('{"name": "x", "version": "2.1.0"}', suffix=".json")
try:
rc, out, _ = run_version_bump(
["--from-file", package, "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("2.2.0", out)
finally:
os.unlink(commits)
os.unlink(package)
def test_pyproject_toml(self):
"""Reads version from a [project] table in pyproject.toml."""
commits = write_temp_file("fix: retry\n")
toml = write_temp_file(
"[project]\nname = \"demo\"\nversion = \"3.1.5\"\n", suffix=".toml"
)
try:
rc, out, _ = run_version_bump(
["--from-file", toml, "--commits-file", commits]
)
self.assertEqual(rc, 0)
self.assertIn("3.1.6", out)
finally:
os.unlink(commits)
os.unlink(toml)
def test_from_file_missing_version_exit_1(self):
"""A package.json without a version field is an input error."""
commits = write_temp_file("feat: x\n")
package = write_temp_file('{"name": "x"}', suffix=".json")
try:
rc, _, err = run_version_bump(
["--from-file", package, "--commits-file", commits]
)
self.assertEqual(rc, 1)
self.assertIn("version", err.lower())
finally:
os.unlink(commits)
os.unlink(package)
class TestVersionBumpGitRange(unittest.TestCase):
"""--git-range reads commits from git log."""
@unittest.skipUnless(shutil.which("git"), "git not available")
def test_git_range_bumps_minor(self):
"""git log subjects drive the bump."""
with tempfile.TemporaryDirectory() as tmp:
subprocess.run(["git", "init", "-q"], cwd=tmp, check=True)
subprocess.run(
["git", "-c", "user.name=t", "-c", "user.email=t@t",
"commit", "--allow-empty", "-q", "-m", "chore: init"],
cwd=tmp, check=True,
)
first_sha = subprocess.run(
["git", "rev-parse", "HEAD"], cwd=tmp, check=True,
capture_output=True, text=True,
).stdout.strip()
subprocess.run(
["git", "-c", "user.name=t", "-c", "user.email=t@t",
"commit", "--allow-empty", "-q", "-m", "feat: widget"],
cwd=tmp, check=True,
)
rc, out, _ = run_version_bump(
["--current-version", "1.0.0", "--git-range", "{}..HEAD".format(first_sha)],
cwd=tmp,
)
self.assertEqual(rc, 0)
self.assertIn("1.1.0", out)
@unittest.skipUnless(shutil.which("git"), "git not available")
def test_git_range_failure_exit_1(self):
"""An invalid git range is an input error."""
rc, _, err = run_version_bump(
["--current-version", "1.0.0", "--git-range", "nope..HEAD"]
)
self.assertEqual(rc, 1)
self.assertIn("error", err.lower())
class TestVersionBumpExitCodes(unittest.TestCase):
"""Exit codes and error handling."""
def test_no_args_exit_2(self):
"""Missing required arguments is a usage error (exit 2)."""
rc, _, err = run_version_bump([])
self.assertEqual(rc, 2)
self.assertIn("usage", err.lower())
def test_invalid_current_version_exit_1(self):
"""A non-SemVer current version is an input error."""
commits = write_temp_file("feat: x\n")
try:
rc, _, err = run_version_bump(
["--current-version", "not.a.version", "--commits-file", commits]
)
self.assertEqual(rc, 1)
self.assertIn("invalid", err.lower())
finally:
os.unlink(commits)
def test_no_conventional_commits_exit_1(self):
"""Input with no conventional commits is an input error."""
commits = write_temp_file("merge branch 'main'\n\nsome random text\n")
try:
rc, _, err = run_version_bump(
["--current-version", "1.2.3", "--commits-file", commits]
)
self.assertEqual(rc, 1)
self.assertIn("no conventional commits", err.lower())
finally:
os.unlink(commits)
def test_missing_commits_file_exit_1(self):
"""A missing commits file is an input error."""
rc, _, err = run_version_bump(
["--current-version", "1.2.3", "--commits-file", "/nonexistent/commits.txt"]
)
self.assertEqual(rc, 1)
self.assertIn("error", err.lower())
def test_no_traceback_on_error(self):
"""Errors never produce a Python traceback."""
rc, out, err = run_version_bump(
["--current-version", "1.2.3", "--commits-file", "/nonexistent/x.txt"]
)
self.assertNotEqual(rc, 0)
self.assertNotIn("Traceback", err)
self.assertNotIn("Traceback", out)
def test_help_exits_0(self):
"""--help exits 0 and describes usage."""
rc, out, _ = run_version_bump(["--help"])
self.assertEqual(rc, 0)
self.assertIn("usage", out.lower())
self.assertIn("version_bump", out.lower())
if __name__ == "__main__":
unittest.main()