evals/dependency-bot-choice-and-migration/criteria.json
{
"context": "Tests whether the agent chooses Renovate for the right reasons, migrates without duplicate bots, handles the OpenTofu registry, and is honest about checksum pins.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Renovate justified by coverage gaps",
"max_score": 14,
"description": "Chooses Renovate because Dependabot has no manager for mise.toml or OpenTofu providers, not because of digest handling"
},
{
"name": "One bot per repository",
"max_score": 12,
"description": "Adds renovate.json and deletes dependabot.yml in the same commit so both bots never run together"
},
{
"name": "Security updates preserved",
"max_score": 8,
"description": "States that Dependabot security alerts and security updates remain independent of the choice"
},
{
"name": "Preset and repository split",
"max_score": 12,
"description": "Encodes schedule, release age, grouping, prefixes, and registry in an organization preset and limits the repository file to opt-outs or approvals"
},
{
"name": "Require config file gating",
"max_score": 10,
"description": "Relies on the organization-level require-config setting so unmigrated repositories receive nothing"
},
{
"name": "OpenTofu registry override",
"max_score": 12,
"description": "Sets registryUrls to registry.opentofu.org for provider and module datasources and disables the hashicorp/terraform dependency"
},
{
"name": "Checksum pins stay manual",
"max_score": 14,
"description": "States that neither bot rewrites a sha256 stored beside a version and keeps Ansible checksum pins manual or proposes fetching upstream checksums at install time"
},
{
"name": "Majors via dashboard approval",
"max_score": 8,
"description": "Uses dependencyDashboardApproval rather than disabling deliberate major upgrades outright"
},
{
"name": "Config validation and readback",
"max_score": 10,
"description": "Validates the config before pushing and checks the first run for correct prefixes, grouping, and registry with no PRs from the retired bot"
}
]
}
evals/dependency-bot-choice-and-migration/task.md
# Dependency Bot Choice and Migration
## Problem/Feature Description
A private infrastructure repository pins tool versions in `mise.toml`, pins
OpenTofu providers in `versions.tf`, runs container images by tag and digest
in Compose files, and stores binary versions with sha256 checksums in Ansible
role defaults. It currently runs Dependabot for GitHub Actions and Compose.
The organization has installed the free hosted Renovate app with
"Require config file" enabled and wants a gradual migration across its
repositories without duplicate pull requests.
Decide whether this repository should move to Renovate, and describe the
migration if so.
## Output Specification
Produce `dependency-updates-plan.md` containing:
- the bot choice with the concrete gaps that justify it;
- what changes in one migration commit and what stays untouched;
- how the organization preset and repository config split responsibilities;
- the OpenTofu registry handling;
- which pins remain manual and why;
- the readback that proves the first run behaved.
Do not mutate a live organization.
evals/draft-pr-and-pnpm-bootstrap/criteria.json
{
"context": "Tests whether the agent correctly distinguishes pull-request activity types from draft state and replaces a legacy pnpm 11 bootstrap with the pnpm/setup successor that owns the declared package manager, runtime, install, and cache.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Draft behavior explained",
"max_score": 15,
"description": "The explanation states that pull_request activity types such as opened and synchronize can fire for draft PRs, so including ready_for_review does not itself suppress draft CI"
},
{
"name": "Draft-aware job gate",
"max_score": 20,
"description": "Each pull-request entry job, or a shared root job that gates every dependent verification job, has an `if:` condition requiring `github.event.pull_request.draft == false` while remaining valid for every configured event"
},
{
"name": "Ready event retained",
"max_score": 10,
"description": "The pull_request trigger retains ready_for_review so verification starts when a draft is marked ready"
},
{
"name": "Update events retained",
"max_score": 5,
"description": "The trigger still handles opened, synchronize, and reopened pull-request activity"
},
{
"name": "Current pnpm setup action",
"max_score": 20,
"description": "The workflow replaces both pnpm/action-setup and actions/setup-node with pnpm/setup pinned to full commit `84cb39b217b10273981911c288cd62326dc7c6d2` and the same-line v2.0.2 version comment"
},
{
"name": "Declared runtime and cache retained",
"max_score": 10,
"description": "pnpm/setup resolves pnpm from packageManager and Node from devEngines.runtime without duplicating those versions in the workflow, and enables its pnpm store cache"
},
{
"name": "Frozen install",
"max_score": 5,
"description": "The workflow installs dependencies with pnpm using the frozen lockfile behavior explicitly or through pnpm's documented CI default"
},
{
"name": "Unprivileged PR event",
"max_score": 10,
"description": "The workflow keeps `pull_request` and does not switch to `pull_request_target` or add write permissions or secrets"
},
{
"name": "Minimal correction",
"max_score": 5,
"description": "The solution changes only the draft gate and replaces the obsolete pnpm bootstrap with its direct successor without unrelated workflow redesign"
}
]
}
evals/draft-pr-and-pnpm-bootstrap/task.md
# Keep Draft Pull Requests Out of pnpm CI
## Problem
A repository's `.github/workflows/verify.yml` listens for pull-request activity types `opened`, `synchronize`, `reopened`, and `ready_for_review`. The maintainer expected that list to suppress draft pull requests, but opening or pushing to a draft still starts the verify job.
The repository has moved to pnpm 11 and declares Node through
`devEngines.runtime`, but its workflow still carries the older two-action
bootstrap. `actions/setup-node` asks for the pnpm store before
`pnpm/action-setup` has installed pnpm, so clean runners fail. The maintained
`pnpm/setup` successor can install the declared pnpm and Node runtime, restore
the pnpm store cache, and optionally run the install itself.
Make the smallest safe workflow correction. Draft pull requests must not
execute verification jobs, marking a PR ready must start verification, and the
pnpm 11 bootstrap and cache must work on a clean runner. Replace the legacy
two-action bootstrap with `pnpm/setup` pinned to the full commit for v2.0.2.
Keep the workflow on the unprivileged `pull_request` event.
## Input Files
The following files represent the current repository state. Extract them before beginning.
=============== FILE: .github/workflows/verify.yml ===============
name: Verify
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
permissions:
contents: read
jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 24
cache: pnpm
- uses: pnpm/action-setup@d15e628ca66d93ee5f352c71671a7bc6a97af5c9 # v6.0.8
with:
run_install: false
- run: pnpm install --frozen-lockfile
- run: pnpm test
=============== END FILE ===============
=============== FILE: package.json ===============
{
"private": true,
"packageManager": "pnpm@11.21.0",
"devEngines": {
"runtime": {
"name": "node",
"version": "24",
"onFail": "download"
}
},
"scripts": {
"test": "vitest run"
}
}
=============== END FILE ===============
## Output
Produce the corrected `.github/workflows/verify.yml` and a short explanation of why both fixes are necessary.
evals/github-action-marketplace-release-with-moving-major-tag/criteria.json
{
"context": "Tests a TypeScript Marketplace action release with bundled runtime, App-signed source writeback, and a maintained moving-major-tag handoff.",
"type": "weighted_checklist",
"checklist": [
{ "name": "Bundled entrypoint", "max_score": 10, "description": "action.yml runs dist/index.js or another bundled path, never src/index.ts." },
{ "name": "Build verification", "max_score": 8, "description": "Verification removes dist, rebuilds the TypeScript bundle, and requires an empty scoped porcelain status with untracked files included, proving the checked-in tree has no changed, missing, stale, or new outputs." },
{ "name": "Moving major tag", "max_score": 10, "description": "A maintained plugin or small tested repo-owned action updates v<major>; workflow YAML does not contain inline force-push shell." },
{ "name": "Durable major-tag repair", "max_score": 8, "description": "The updater proves the candidate is the highest eligible stable release in its major line from peeled Git ref OIDs, rejects an unknown or newer pointer, and atomically compare-and-swaps the raw major-tag ref against the observed old OID or expected absence. It rereads exact remote ref parity, does not use Release fields as commit truth, and cannot roll back on stale recovery." },
{ "name": "No npm publish", "max_score": 6, "description": "The configuration does not include @semantic-release/npm." },
{ "name": "Atomic signed writeback", "max_score": 8, "description": "The workflow uses either @jno21/semantic-release-github-commit@1.0.1 under a named concrete external branch lease that blocks merges and direct pushes, or a full-SHA-pinned App-signed API integration with the analyzed SHA as expected head; @semantic-release/github publishes only afterward." },
{ "name": "Prepare and tag order", "max_score": 6, "description": "A deterministic prepare step updates each version manifest before the selected signed writeback, and tagging plus Release publication continue only from the returned signed commit OID." },
{ "name": "Writeback options", "max_score": 6, "description": "The signed writeback includes only existing regular version manifests updated by prepare, uses a message containing [skip ci], and excludes dist/**. If plugin v1.0.1 is selected it uses files and commitMessage, not assets or message." },
{ "name": "App-signed boundary", "max_score": 8, "description": "The App token is scoped to release writeback, no custom identity is supplied, and checkout persists no credentials. Preflight and Actions concurrency alone do not qualify as a lock; plugin v1.0.1 additionally requires immediate if: always() origin cleanup." },
{ "name": "skip ci guards", "max_score": 6, "description": "Verify and release both skip [skip ci] writeback commits." },
{ "name": "Release concurrency", "max_score": 4, "description": "Release has a job-level non-cancellable concurrency group." },
{ "name": "Full history", "max_score": 4, "description": "Verify and release checkouts use fetch-depth: 0." },
{ "name": "Pinned release action", "max_score": 6, "description": "cycjimmy/semantic-release-action is pinned to a full commit SHA with an exact version comment." },
{ "name": "Immutable completion proof", "max_score": 10, "description": "The workflow reads back immutable: true, runs gh release verify, proves the exact release tag, verified writeback, bundled runtime, and moving major tag agree, and implements exact-tag backfill plus parity reread for a missing metadata-only Release without another bump or published-release mutation." }
]
}
evals/github-action-marketplace-release-with-moving-major-tag/task.md
# Publish a TypeScript GitHub Action to the Marketplace with Automated Releases
## Problem/Feature Description
Apex Platform has built `notify-on-failure`, a TypeScript GitHub Action that sends Slack notifications when a workflow job fails. The action is used internally across dozens of repositories, and several external teams have requested access. The team wants to publish it to the GitHub Actions Marketplace and set up automated releases using semantic-release so that every `feat:` or `fix:` commit to `main` automatically creates a new GitHub Release and advances the version.
The big challenge is distribution: users of GitHub Actions typically pin to a major version tag like `uses: apex-platform/notify-on-failure@v2` and expect that tag to always point to the latest stable release in that major line. If the team just creates `v2.1.0` but never updates the `v2` tag, all consumers are stuck on whatever version was current when they set up their workflow.
The action is written in TypeScript, but GitHub runs JavaScript. The compiled output must be the action's runtime entrypoint. CI must verify both the TypeScript source and the published JavaScript.
The organization requires verified commits on `main`. Pull requests must build
the checked-in `dist/` bundle and fail when rebuilding changes it. The release
Environment provides `RELEASE_APP_CLIENT_ID` and
`RELEASE_APP_PRIVATE_KEY` for an installed GitHub App. Release-time writeback
is limited to existing regular version manifests that a deterministic prepare
step actually updates, through GitHub's App-signed commit path and without
custom author/committer fields; do not pass `dist/**` to a plugin that cannot
preserve deletions and Git modes. A preflight head check and Actions concurrency
are not an atomic branch lock. Use plugin v1.0.1 only if a concrete external
branch lease blocks every merge and direct push from before semantic-release
starts release analysis through the plugin's API ref update. Otherwise use a
full-SHA-pinned App-signed API integration with the analyzed SHA as its expected
head. If plugin v1.0.1 is selected, restore a credential-free `origin`
immediately afterward in an `if: always()` step.
Bundle verification must remove `dist/`, rebuild it, and require an empty
`git status --porcelain=v1 --untracked-files=all -- dist` result so changed,
missing, stale, and newly generated outputs all fail the gate.
The organization enforces immutable GitHub Releases. Because the compiled
bundle is committed before tagging and no asset is appended after publication,
semantic-release may publish this metadata-only release directly. The workflow
must read back `immutable: true`, run `gh release verify`, and prove the exact
release tag, signed default-branch writeback, bundled runtime, and moving major
tag all resolve to the intended release. A retry must inspect existing state and
must not mutate the published release. A later recovery run must also backfill
a missing metadata-only GitHub Release or repair the moving major tag from the
existing trusted tag even though semantic-release no longer reports a new
release. Before moving `v<major>`, prove the candidate is the highest eligible
published stable SemVer in that major line, reject an unknown or newer current
pointer, and resolve the candidate's peeled Git ref commit OID. Observe the raw
`refs/tags/v<major>` OID, update it with an atomic expected-old-OID
compare-and-swap (including an expected-absence precondition), then reread the
remote ref and require it to equal the candidate commit OID. GitHub Release
fields are not the commit source of truth.
## Output Specification
Produce the following files:
- `.github/workflows/ci.yml`: GitHub Actions workflow with verify and release jobs
- `.releaserc.json`: semantic-release configuration suitable for a marketplace action
- `action.yml`: the action manifest (you may adapt or complete the partial version provided below)
## Input Files
The following files are provided. Extract them before beginning.
=============== FILE: action.yml ===============
name: "Notify on Failure"
description: "Sends a Slack notification when a workflow job fails"
author: "Apex Platform"
inputs:
slack-webhook-url:
description: "Slack incoming webhook URL"
required: true
message:
description: "Custom message to include in the notification"
required: false
default: "A workflow job failed"
runs:
using: "node24"
main: "src/index.ts"
=============== END FILE ===============
=============== FILE: package.json ===============
{
"name": "notify-on-failure",
"version": "2.0.0",
"description": "Sends a Slack notification when a workflow job fails",
"scripts": {
"build": "esbuild src/index.ts --bundle --platform=node --outfile=dist/index.js",
"test": "vitest run",
"lint": "eslint src",
"typecheck": "tsc --noEmit"
},
"devDependencies": {
"esbuild": "^0.21.0",
"typescript": "^5.4.0",
"vitest": "^1.6.0",
"eslint": "^8.57.0",
"@actions/core": "^1.10.1"
}
}
=============== END FILE ===============
=============== FILE: .node-version ===============
24
=============== END FILE ===============
evals/go-goreleaser-homebrew-tap-pipeline/criteria.json
{
"context": "Tests a tag-only semantic-release plus GoReleaser pipeline with draft-first immutable publication, attestation, and a native GitHub App-signed Homebrew cask update.",
"type": "weighted_checklist",
"checklist": [
{ "name": "Tag-only semantic-release", "max_score": 8, "description": "Plugins include commit-analyzer, release-notes-generator, and github, but no npm or source-writeback plugin." },
{ "name": "Draft-first release", "max_score": 6, "description": "@semantic-release/github creates a draft and GoReleaser is configured to reuse that draft without publishing early." },
{ "name": "Exact release-state recovery", "max_score": 8, "description": "The workflow requires one stable-format tag at HEAD and uses a tested repo-owned helper plus the source-repository workflow token to resolve and peel the remote Git ref to that commit while checkout credentials stay unpersisted. It uses an authenticated exact-tag lookup that sees drafts and fails closed when an expected Release lookup fails. Explicit recovery treats only an unambiguous not-found response as absence, backfills that Release, and rereads it; authentication, authorization, rate-limit, network, and server failures remain errors. One non-prerelease draft is resumed, one stable published immutable Release skips mutations, and API failures, create conflicts, or duplicate exact-tag state never become a green no-op. A visibility retry is bounded and preserves the terminal error only when transient lookup lag is demonstrated." },
{ "name": "GoReleaser exact-tag build", "max_score": 6, "description": "Every GoReleaser invocation runs release --clean from a full-SHA-pinned goreleaser-action and receives the uniquely validated tag as GORELEASER_CURRENT_TAG." },
{ "name": "Tap naming", "max_score": 6, "description": "The cross-repository tap uses a homebrew- prefixed repository name." },
{ "name": "Scoped App token", "max_score": 6, "description": "Source Release operations use the source-repository workflow token; a separate short-lived App token names only the tap repository with Contents write. No combined source/tap write token or broad PAT is introduced." },
{ "name": "Attestation permissions", "max_score": 6, "description": "The release job grants id-token: write and attestations: write for file attestations, and does not grant artifact-metadata: write unless it creates an artifact storage record." },
{ "name": "Attest before publish", "max_score": 8, "description": "Before attestation or publication, the draft's complete asset name and SHA-256 set must equal the current build manifest; release assets are then attested while the release is still draft." },
{ "name": "Current cask publisher", "max_score": 8, "description": ".goreleaser.yaml uses homebrew_casks rather than deprecated brews and targets the tap repository." },
{ "name": "Native signed tap commit", "max_score": 12, "description": "The cask sets commit_author.use_github_app_token: true and does not set custom author/committer identity fields." },
{ "name": "No extra tap commit job", "max_score": 6, "description": "GoReleaser owns the cask update; there is no second OS job, formula renderer, local git push, or generic commit action." },
{ "name": "Full history", "max_score": 4, "description": "Verify and release checkouts use fetch-depth: 0." },
{ "name": "Release concurrency", "max_score": 4, "description": "Release has a job-level non-cancellable concurrency group." },
{ "name": "Step-scoped tokens", "max_score": 4, "description": "At GoReleaser step scope, the source workflow token is GITHUB_TOKEN and the tap-only App token is HOMEBREW_TAP_TOKEN." },
{ "name": "Setup contract", "max_score": 2, "description": "Root SETUP.md documents RELEASE_APP_CLIENT_ID, RELEASE_APP_PRIVATE_KEY, tap-only repository scope, and Contents write for the minted App token." },
{ "name": "Publish and prove", "max_score": 6, "description": "A release-tag ruleset blocks update/deletion and restricts creation to the release actor. The workflow rereads the unchanged peeled remote tag OID immediately before explicitly publishing the draft, then proves draft: false, prerelease: false, immutable: true, gh release verify, the same tag OID and exact asset manifest, and the tap commit signature." }
]
}
evals/go-goreleaser-homebrew-tap-pipeline/task.md
# Automate Binary Releases for a Go CLI Tool
## Problem/Feature Description
Redwood Systems ships `vaultctl`, a Go command-line tool for secrets rotation used by their infrastructure teams. The project has grown from an internal tool to one adopted by a handful of partner companies, and the team wants to provide polished distribution: pre-built binaries for Linux/macOS/Windows, a Homebrew cask so Mac users can simply `brew install`, and signed build attestation for supply-chain compliance.
Releases depend on someone remembering to cut one. The process does not maintain a consistent changelog or build binaries, and the Homebrew cask in the separate `redwood-systems` tap repository is months out of date. Conventional commits on `main` should determine the version, create a GitHub Release, build cross-platform binaries, and update the cask without a manual release command.
The cross-repo Homebrew update needs credentials beyond the default GitHub
token, which already covers source-repository Release operations. Mint a
short-lived GitHub App installation token scoped only to the tap and use
GoReleaser's native GitHub App commit-author support so that update is signed
without granting the tap publisher source write access or adding a second
commit job. Both repositories require verified commits. The team also wants
immutable GitHub Releases and build provenance attestation for the assets.
Release recovery must be state-specific: create or resume the mutable draft for
the exact trusted tag when publication is incomplete, but skip every asset
mutation when that Release is already published and immutable. A missing draft
must not make an existing tag unrecoverable, and every recovery rereads parity.
Enumerate every tag pointing at `HEAD`, filter by the configured stable release
tag format, and require exactly one eligible tag; do not let `git describe`
choose among multiple tags. Pass that selected exact tag as
`GORELEASER_CURRENT_TAG` to every GoReleaser invocation so a co-located tag
cannot redirect publication. Resolve and peel the remote tag, require its commit
OID to equal `HEAD` before any backfill or build, and reread the same remote OID
immediately before immutable publication and again at completion. Protect the
release-tag namespace against updates/deletion and restrict creation to the
release actor. Resolve the remote ref through a tested repo-owned helper using
the source-repository workflow token and authenticated GitHub Git Refs/Tags
APIs; the checkout keeps persisted credentials disabled. The separate tap App
token must never be used for source reads or writes.
Use an authenticated exact-tag lookup that can see drafts. When the current run
already created or identified an expected draft, lookup failure must fail closed
rather than reporting absence or success. Add a bounded visibility retry only
if the design demonstrates a transient lookup-lag requirement. In explicit
recovery, accept only an unambiguous not-found response as a missing Release;
authentication, authorization, rate-limit, network, and server failures remain
errors. Backfill a confirmed missing Release from the trusted tag and reread
exact state; create conflict or duplicate exact-tag state must not become a
green no-op.
Reject a prerelease for the stable tag. Before attestation or publication,
require the resumed draft's complete asset names and SHA-256 digests to equal
the current build manifest; missing, extra, or mismatched assets fail closed.
## Output Specification
Produce the following files:
- `.github/workflows/ci.yml`: complete GitHub Actions workflow with verify and release jobs
- `.releaserc.json`: semantic-release configuration
- `.goreleaser.yaml`: GoReleaser configuration including Homebrew cask automation
- `scripts/resolve-remote-tag-oid` and focused tests: authenticated annotated-tag peeling with fail-closed errors
Include a brief `SETUP.md` at the repo root documenting the release Environment App credentials (`RELEASE_APP_CLIENT_ID` / `RELEASE_APP_PRIVATE_KEY`) and the explicit repository scope required for the minted token.
## Input Files
The following files represent the current repository state. Extract them before beginning.
=============== FILE: go.mod ===============
module github.com/redwood-systems/vaultctl
go 1.22
=============== END FILE ===============
=============== FILE: Makefile ===============
.PHONY: verify build
verify:
go vet ./...
go test ./...
golangci-lint run
build:
go build -o bin/vaultctl ./cmd/vaultctl
=============== END FILE ===============
evals/homebrew-signed-api-fallback/criteria.json
{
"context": "Tests the narrow signed-API fallback for a non-Go Homebrew publisher that lacks native GitHub App commit signing.",
"type": "weighted_checklist",
"checklist": [
{ "name": "Durable release gate", "max_score": 10, "description": "The tap job resolves an exact trusted tag, requires a published immutable release that passes gh release verify, and can repair missing tap parity on a later run without relying solely on new_release_published." },
{ "name": "Scoped App token", "max_score": 12, "description": "The source workflow token is explicitly Contents read only; the write token is a separate short-lived App token naming only homebrew-tap with Contents write." },
{ "name": "Safe tap checkout", "max_score": 8, "description": "The tap checkout targets main, uses the App token, and sets persist-credentials: false." },
{ "name": "Narrow deterministic generation", "max_score": 12, "description": "Generation updates only Formula/envctl.rb from the exact published version/assets and does not commit or push." },
{ "name": "Pinned signed commit action", "max_score": 15, "description": "pgaskin/push-signed-commits is pinned to a full commit SHA with an exact version comment and receives the App token." },
{ "name": "Commit path and head boundary", "max_score": 8, "description": "Only Formula/envctl.rb is staged. The signed action uses the observed tap checkout parent as expectedHeadOid on main and rejects concurrent branch advancement." },
{ "name": "No unsigned escape", "max_score": 10, "description": "There is no ordinary git commit/push, custom author/committer identity, or generator-owned final commit." },
{ "name": "Signature readback", "max_score": 10, "description": "The workflow reads the tap default-branch commit, requires verification.verified: true, and proves the formula version/digests match the immutable release." },
{ "name": "Setup contract", "max_score": 8, "description": "SETUP.md documents RELEASE_APP_CLIENT_ID, RELEASE_APP_PRIVATE_KEY, the tap-only token scope, and Contents write." },
{ "name": "No broad credential", "max_score": 7, "description": "No classic PAT or long-lived organization-wide tap token is recommended." }
]
}
evals/homebrew-signed-api-fallback/task.md
# Add a Signed Homebrew Fallback for a Non-Go CLI
## Problem/Feature Description
Acme Tools publishes `envctl`, a non-Go CLI, and needs its release workflow to
update `acme-tools/homebrew-tap`. Its existing formula generator ends with an
ordinary local commit and push. The tap now requires verified commits, so a bot
name and noreply email are insufficient.
The generator has no native GitHub App-signed commit mode. Preserve its useful
formula-generation behavior, but prevent it from committing. After a release is
published, a dependent Linux job reads source state with the source
repository's read-only workflow token, then mints a separate short-lived App
token scoped only to the tap. It deterministically prepares only
`Formula/envctl.rb` and commits that path with the full-SHA-pinned
`pgaskin/push-signed-commits` action. Stage no other path, use the observed tap
checkout parent as the atomic expected head, and fail rather than overwrite if
the tap advances. Read back the resulting tap commit and fail if GitHub does
not report `verification.verified: true`.
The handoff must use durable state. It should run whenever the exact trusted
release tag is published and immutable but tap parity is missing, including a
later recovery run. Do not gate repair solely on semantic-release's
`new_release_published` output.
## Output Specification
Update `.github/workflows/release.yml` and write a short `SETUP.md`. Document
the release Environment's `RELEASE_APP_CLIENT_ID` variable and
`RELEASE_APP_PRIVATE_KEY` secret, the tap-only token scope, and
`contents: write`. Keep the source workflow token at `contents: read`. Do not
add a PAT, custom bot identity, ordinary `git push`, or a manual tap PR.
## Input Files
The current workflow publishes through semantic-release and exposes
`new_release_published` and `new_release_version` as job outputs. It currently
runs a formula generator directly after semantic-release using the default
repository token. Replace only that Homebrew handoff; preserve the existing
release system, but replace the transient output gate with exact release-state
discovery and an idempotent parity check.
evals/lane-aware-change-detection-and-concurrency/criteria.json
{
"context": "Tests whether the agent sets up lane-independent change detection that only builds/deploys affected apps, uses correct concurrency settings (non-cancellable deploy, cancellable verification/e2e, same group key across main.yml and deploy.yml), and uses the correct job condition syntax for deploy gates.",
"type": "weighted_checklist",
"checklist": [
{
"name": "paths-filter for detection",
"max_score": 8,
"description": "Uses dorny/paths-filter pinned to a full commit SHA with the current v4.0.3 version comment (not v2 or v3) to detect which lane changed"
},
{
"name": "Lockfile in filter",
"max_score": 8,
"description": "The paths-filter includes the package lockfile (package-lock.json, pnpm-lock.yaml, yarn.lock, or similar) in at least one lane's filter rules"
},
{
"name": "fetch-depth 0 on changes job",
"max_score": 9,
"description": "The changes/detect job's checkout step includes `fetch-depth: 0`"
},
{
"name": "Lane conditional build",
"max_score": 8,
"description": "Each verification/build job has an `if:` condition that gates it on the corresponding lane being detected as changed (referencing the changes job output)"
},
{
"name": "Deploy non-cancellable",
"max_score": 10,
"description": "The deploy job(s) include `concurrency: { cancel-in-progress: false }` (or equivalent)"
},
{
"name": "Verification/e2e cancellable",
"max_score": 8,
"description": "At least one verification or e2e job includes `concurrency: { cancel-in-progress: true }` (or equivalent)"
},
{
"name": "Concurrency group lane-scoped",
"max_score": 9,
"description": "The deploy concurrency group key is scoped to environment and lane, such as `deploy-production-web`, and NOT to `${{ github.ref }}`"
},
{
"name": "Shared concurrency key",
"max_score": 10,
"description": "The deploy concurrency group key in main.yml and in deploy.yml (or the manual re-deploy section) uses the same string value"
},
{
"name": "Explicit result check",
"max_score": 10,
"description": "Deploy job `if:` condition uses `needs.<job>.result == 'success'` (NOT `success()`) for each upstream dependency"
},
{
"name": "Manual deploy workflow",
"max_score": 8,
"description": "A separate workflow_dispatch workflow exists (deploy.yml or equivalent) for manually re-deploying a verified artifact or image for a validated ref without re-running verification"
},
{
"name": "validated redeploy ref",
"max_score": 8,
"description": "The manual deploy workflow validates the requested ref, passes it through env or outputs, and uses the validated value for checkout or artifact/image lookup rather than interpolating `inputs.ref` directly in shell"
},
{
"name": "Independent lane verification",
"max_score": 4,
"description": "The written explanation or workflow structure shows that a change to only one app does not trigger the build/deploy of the other app (each lane's jobs are gated independently)"
}
]
}
evals/lane-aware-change-detection-and-concurrency/task.md
# GitHub Actions Pipeline for a Two-App Monorepo
## Problem/Feature Description
A platform team runs a monorepo that contains two deployable apps: `apps/dashboard` (a TypeScript React frontend) and `apps/api` (a Node.js Express backend). Both apps share a `packages/` directory of internal libraries and a root `pnpm-lock.yaml`.
The team has two pressing problems. First, every push to `main` triggers a full rebuild and redeploy of both apps even when only one of them changed. This doubles CI time and has caused accidental rollbacks when a clean deploy of one app brought along stale code from the other. Second, when engineers push rapid fixes to `main` during incidents, deploys sometimes race each other and the wrong artifact ends up on the host. At the same time, they need a way for an on-call engineer to manually re-deploy a verified artifact or image for a specific app and validated git ref without re-running all the tests.
Design and write the GitHub Actions workflows that solve both problems. The frontend lane promotes a verified static build artifact to the `production` GitHub Environment. The API lane promotes an immutable image reference to the same `production` Environment. The deploy provider supports OIDC, so the deploy jobs must use `id-token: write` and environment-scoped role/config variables instead of long-lived repository secrets.
## Output Specification
Produce the following files:
- `.github/workflows/main.yml`: push-to-main pipeline with lane-aware detection, verify, e2e, and deploy stages for both apps
- `.github/workflows/deploy.yml`: manual re-deploy workflow (`workflow_dispatch`) for a verified artifact or image on a chosen lane and validated ref
- `.github/workflows/verify.yml`: pull request verification workflow with no deployment
- `pipeline-design.md`: a brief explanation of change detection, rapid pushes, environment credential scope, and how manual re-deploy uses the main pipeline's concurrency
evals/npm-release-pipeline-workflow-structure/criteria.json
{
"context": "Tests an npm semantic-release workflow with OIDC publishing, GitHub App-signed source writeback, bounded credentials, and race-safe verification/release structure.",
"type": "weighted_checklist",
"checklist": [
{ "name": "fetch-depth verify", "max_score": 4, "description": "The verify checkout uses fetch-depth: 0." },
{ "name": "fetch-depth release", "max_score": 4, "description": "The release checkout uses fetch-depth: 0." },
{ "name": "Verify concurrency", "max_score": 5, "description": "Verification has a cancellable concurrency group." },
{ "name": "Release concurrency", "max_score": 5, "description": "Release has a job-level non-cancellable concurrency group." },
{ "name": "skip ci verify", "max_score": 4, "description": "Verification skips commits whose message contains [skip ci]." },
{ "name": "skip ci release", "max_score": 4, "description": "Release skips commits whose message contains [skip ci]." },
{ "name": "Atomic App-signed writeback", "max_score": 9, "description": "The workflow either uses @jno21/semantic-release-github-commit@1.0.1 under a named concrete external branch lease that blocks merges and direct pushes from before release analysis through its ref update, or a full-SHA-pinned App-signed API integration with the analyzed SHA as expected head. Preflight and Actions concurrency alone do not qualify." },
{ "name": "No custom commit identity", "max_score": 4, "description": "The workflow does not set GIT_AUTHOR or GIT_COMMITTER name/email fields; GitHub must sign the App commit." },
{ "name": "Release permissions", "max_score": 6, "description": "Release declares contents: write and id-token: write, without unrelated write scopes unless configured behavior needs them." },
{ "name": "semantic-release action pin", "max_score": 3, "description": "cycjimmy/semantic-release-action is pinned to a full commit SHA with an exact version comment." },
{ "name": "Prepare and publish order", "max_score": 8, "description": "commit analysis and notes precede deterministic changelog and npm preparation; the selected signed writeback commits those outputs, and tagging plus GitHub/registry publication continue only from the returned signed commit OID." },
{ "name": "Matching preset", "max_score": 4, "description": "commit-analyzer and release-notes-generator both use the conventionalcommits preset." },
{ "name": "Writeback configuration", "max_score": 7, "description": "The selected signed writeback includes only the prepared existing package.json and CHANGELOG.md and uses chore(release): ${nextRelease.version} with [skip ci]. If plugin v1.0.1 is selected it uses files and commitMessage, not assets or message." },
{ "name": "Trusted publishing", "max_score": 7, "description": "npm publishing uses Trusted Publishing/OIDC on a GitHub-hosted runner, including for a private source repository, without registry-url or NPM_TOKEN." },
{ "name": "Release gated by verify", "max_score": 6, "description": "The release job depends on successful verification." },
{ "name": "Credential boundary", "max_score": 6, "description": "Checkout does not persist write credentials and the App token is exposed only at the signed writeback boundary. If plugin v1.0.1 is selected, an immediate if: always() cleanup restores a credential-free origin." },
{ "name": "npm metadata", "max_score": 4, "description": "package.json has matching public repository metadata and public publishConfig.access for the scoped package." },
{ "name": "Immutable completion proof", "max_score": 10, "description": "After the metadata-only release, the workflow reads back immutable: true, runs gh release verify, binds the peeled remote tag commit to the verified App writeback, reads package/changelog parity from that commit, and separately proves main contains it. Exact-tag recovery backfills only a missing npm or Release boundary, never mutates an existing immutable Release, and rereads parity; detect-and-stop or a normal semantic-release rerun is insufficient." }
]
}
evals/npm-release-pipeline-workflow-structure/task.md
# Set Up Automated Release Pipeline for npm Library
## Problem/Feature Description
Fieldstone Labs maintains `@fieldstone/form-validator`, a TypeScript library published to npm. A developer currently runs `npm version`, pushes a tag, and publishes by hand. Two releases shipped without a changelog update, and one local publish used stale dependencies.
The team wants to automate this using GitHub Actions and semantic-release, so that every conventional commit pushed to `main` that warrants a release (feat, fix, or breaking change) automatically: runs the test suite, bumps the version, updates the changelog, publishes to npm through npm Trusted Publishing/OIDC, creates a GitHub Release, and commits the version bump back to the repo. They want protection against two releases accidentally racing each other, and they want the version bump commit to never retrigger CI.
The organization requires verified signatures on `main`. The release
Environment provides `RELEASE_APP_CLIENT_ID` and `RELEASE_APP_PRIVATE_KEY` for
an installed GitHub App that can write this repository. The source writeback
must use GitHub's App-signed commit path; a configured bot name or noreply email
is not a signature. A superseded-run preflight and Actions concurrency are not
an atomic branch lock. Use plugin v1.0.1 only if the solution also names and
uses a concrete external branch lease that blocks every merge and direct push
from before semantic-release starts release analysis through the plugin's API
ref update. Otherwise use a full-SHA-pinned App-signed API integration that
sends the analyzed SHA as its expected head and fails closed on mismatch. If
plugin v1.0.1 is selected, immediately restore `origin` to a credential-free
URL in an `if: always()` step.
The organization also enforces immutable GitHub Releases. This package has no
post-publication release assets, so semantic-release may publish the metadata-
only release directly. The workflow must then prove `immutable: true`, run
`gh release verify`, resolve and peel the remote release tag, and require that
commit to be the verified App-signed writeback. Read `package.json` and
`CHANGELOG.md` from that immutable commit for npm parity; check the live default
branch separately only to prove it contains the writeback commit. Retries must
inspect the existing release and registry state instead of
creating another bump or trying to mutate a published release. Include a
validated backfill path for an existing trusted tag when npm or the GitHub
Release was published but the other boundary is missing; a normal
semantic-release rerun is not sufficient recovery.
## Output Specification
Produce the following files in the workspace:
- `.github/workflows/ci.yml`: the complete GitHub Actions workflow with verify and release jobs
- `.releaserc.json`: the semantic-release configuration file
Both files should be ready to commit to the repo root as-is (no placeholders left unfilled). You may create a `package.json` stub if needed to illustrate the configuration, but it is not required.
## Input Files
The following files represent the current state of the repository. Extract them before beginning.
=============== FILE: package.json ===============
{
"name": "@fieldstone/form-validator",
"version": "2.3.1",
"description": "TypeScript form validation library",
"main": "dist/index.js",
"types": "dist/index.d.ts",
"scripts": {
"build": "tsc",
"test": "vitest run",
"lint": "eslint src",
"typecheck": "tsc --noEmit",
"verify": "npm run lint && npm run typecheck && npm run test && npm run build"
},
"devDependencies": {
"typescript": "^5.4.0",
"vitest": "^1.6.0",
"eslint": "^8.57.0"
},
"repository": {
"type": "git",
"url": "git+https://github.com/fieldstone/form-validator.git"
},
"publishConfig": {
"access": "public"
}
}
=============== END FILE ===============
=============== FILE: CHANGELOG.md ===============
# Changelog
## 2.3.1
- Existing release history.
=============== END FILE ===============
=============== FILE: .node-version ===============
24
=============== END FILE ===============
evals/organization-cost-safe-security-baseline/criteria.json
{
"context": "Tests whether the agent recommends a cost-safe organization security configuration, preserves free dependency protections, and catches the public-to-private Advanced Security billing transition.",
"type": "weighted_checklist",
"checklist": [
{
"name": "All current repositories",
"max_score": 8,
"description": "Applies the baseline organization security configuration to all current repositories"
},
{
"name": "Default for new repositories",
"max_score": 8,
"description": "Makes the same configuration the default for all newly created repositories"
},
{
"name": "Enforced organization policy",
"max_score": 8,
"description": "Prevents repository owners from modifying the centrally configured features"
},
{
"name": "Secret Protection disabled",
"max_score": 10,
"description": "Disables the paid Secret Protection bundle in the fleet-wide baseline"
},
{
"name": "Code Security disabled",
"max_score": 10,
"description": "Disables the paid Code Security bundle and does not retain blanket legacy advanced_security enablement"
},
{
"name": "CodeQL default setup disabled",
"max_score": 10,
"description": "Disables CodeQL default setup as an organization-wide default"
},
{
"name": "Free dependency protections retained",
"max_score": 12,
"description": "Enables the dependency graph, Dependabot alerts, and Dependabot security updates"
},
{
"name": "Public capability distinction",
"max_score": 6,
"description": "Explains that disabling paid bundles does not prove every overlapping public-repository security capability is disabled"
},
{
"name": "Visibility transition preflight",
"max_score": 12,
"description": "Checks attached configuration, effective paid features, active-committer usage, and projected billing before and after a public-to-private change"
},
{
"name": "Narrow paid exception path",
"max_score": 8,
"description": "Uses a separate narrow configuration or explicit repository opt-in with accepted cost and trigger design for paid exceptions"
},
{
"name": "Live readback and safe rollout",
"max_score": 8,
"description": "Requires live setting and license-usage readback, uses a bounded rollout, and does not mutate a live organization without authorization"
}
]
}
evals/organization-cost-safe-security-baseline/task.md
# Cost-Safe Organization Security Baseline
## Problem/Feature Description
A GitHub Team organization contains both public and private repositories. Its
current organization security configuration enables `advanced_security` and
CodeQL by default for public repositories only. One public repository may soon
become private, and the organization owner wants to avoid surprise
per-active-committer Advanced Security charges while retaining the useful free
dependency protections available across the fleet.
Design the recommended organization-level default for all current and future
repositories. Distinguish paid Secret Protection and Code Security features
from the dependency graph and Dependabot features. Include the visibility-change
preflight and readback needed to catch billing changes. Paid security features
may be enabled for an exceptional repository only through an explicit,
cost-approved opt-in.
## Output Specification
Produce `github-security-baseline.md` containing:
- the proposed organization security configuration and enforcement scope;
- the paid and free feature choices, including CodeQL default setup;
- a safe rollout and live readback checklist;
- the public-to-private visibility transition preflight;
- the exception path for a repository that genuinely needs a paid feature.
Do not mutate a live GitHub organization or claim that disabling the paid
bundles disables every overlapping security capability on public repositories.
evals/pipeline-topology-and-artifact-integrity/criteria.json
{
"context": "Tests whether the agent follows the verify → e2e → deploy pipeline topology without collapsing stages, passes the same artifact through all stages without rebuilding, and hands off to real monitoring/rollback context instead of a cheap smoke job. Also checks artifact upload options and step summary.",
"type": "weighted_checklist",
"checklist": [
{
"name": "No rebuild in deploy",
"max_score": 10,
"description": "The deploy path consumes the exact payload produced by the verify/build path rather than running a build command itself (no `npm run build`, `pnpm run build`, etc. inside the deploy job)"
},
{
"name": "Exact payload tested in e2e",
"max_score": 9,
"description": "The e2e stage tests the exact deploy payload via same-job filesystem handoff, immutable release/registry/provider ref, or a documented same-run artifact handoff rather than running a fresh build"
},
{
"name": "Payload boundary justified",
"max_score": 7,
"description": "Uses the most durable available payload boundary (same-job static handoff, registry/release asset, image digest, provider-native package) or explicitly documents why a same-run GitHub Actions artifact is acceptable scratch storage"
},
{
"name": "Missing output fails",
"max_score": 7,
"description": "The workflow fails immediately if the build produces no deployable output, either via an explicit output-path assertion or artifact upload settings such as `if-no-files-found: error`"
},
{
"name": "Framework output covered",
"max_score": 6,
"description": "The chosen payload path includes framework output directories such as `.next/` or `.output/` when relevant, rather than accidentally deploying only a partial visible-file tree"
},
{
"name": "Lane-specific payload identity",
"max_score": 5,
"description": "The payload identity is lane-specific (e.g. `web-dist`, image digest output, provider package id, or release asset name) rather than a generic ambiguous name like `dist` or `build`"
},
{
"name": "Separate stages",
"max_score": 7,
"description": "Verify, e2e, and deploy are separate jobs, or the workflow explicitly uses the documented same-job static handoff exception and keeps build, e2e, credential loading, and deploy ordered in one trusted job"
},
{
"name": "Monitoring handoff present",
"max_score": 10,
"description": "The deploy output or follow-up summary links the deployed URL, monitoring dashboard, alert or synthetic-check coverage, deploy marker when available, and rollback runbook"
},
{
"name": "No cheap smoke substitute",
"max_score": 6,
"description": "The workflow does not add a shallow curl/wget/Playwright smoke job as the main post-deploy proof unless it is explicitly wired to the repo's real synthetic monitoring contract"
},
{
"name": "GITHUB_STEP_SUMMARY",
"max_score": 5,
"description": "A step in the deploy job writes to `$GITHUB_STEP_SUMMARY` including what was deployed and where (URL or environment)"
},
{
"name": "Deploy needs both verify and e2e",
"max_score": 5,
"description": "The deploy job declares `needs: [verify-<lane>, e2e-<lane>]` (or equivalent) to depend on both upstream jobs, unless using the documented same-job static handoff where deploy is gated after verify and e2e steps"
},
{
"name": "Post-deploy handoff has no deploy credentials",
"max_score": 3,
"description": "Any monitoring, notification, or incident handoff job is read-only and does not receive OIDC or provider deploy credentials"
},
{
"name": "Timeouts set",
"max_score": 5,
"description": "Non-trivial verify, e2e, deploy, and result jobs set explicit `timeout-minutes` values instead of relying on GitHub's long default"
},
{
"name": "Artifact exception hygiene",
"max_score": 5,
"description": "If Actions artifacts are used as same-run scratch storage, upload steps set `if-no-files-found: error`, `retention-days` between 1 and 3, lane-specific artifact names, and record the artifact digest"
},
{
"name": "Stable no-op result",
"max_score": 6,
"description": "The workflow avoids trigger-level path skips for required checks and uses an internal changes/no-op path plus a stable final result job for branch protection"
},
{
"name": "Matrix controls",
"max_score": 4,
"description": "Matrixed build/e2e jobs use `fail-fast: false` when full lane evidence matters and set `max-parallel` when external capacity or provider limits require it"
}
]
}
evals/pipeline-topology-and-artifact-integrity/task.md
# Reliable GitHub Actions Deploy Pipeline for a React SPA
## Problem/Feature Description
A product team deploys its Vite React SPA by running `npm run build` locally and uploading `dist/` through the provider dashboard. The team has grown to seven engineers, and two untested local builds broke production last week. GitHub Actions must enforce that the tested artifact is the deployed artifact.
The pipeline should build the app once, run end-to-end tests against that output, and then promote it through the `production` GitHub Environment with the provider's OpenID Connect (OIDC) deploy identity. A misconfigured Vite output path produced no files twice this month, so the pipeline must reject an empty build. The framework uses a non-standard output structure. After deployment, on-call engineers need links to the live site's monitoring dashboard, alert policy, synthetic check, deploy marker, and rollback runbook. That handoff currently depends on tribal knowledge.
## Output Specification
Produce a working GitHub Actions workflow at `.github/workflows/main.yml` that triggers on push to `main` and implements the full build -> test -> deploy flow described above. Use a repo-owned provider-thin deploy script or local action that accepts artifact path and environment; do not write a provider cookbook.
The deploy job must declare the `production` GitHub Environment, use `id-token: write`, and keep provider identifiers in environment vars rather than hardcoded workflow values.
Include a brief `deploy-summary.md` explaining each job, the artifacts passed between jobs, the deployment identity boundary, and the rationale.
evals/rust-cargo-dist-immutable-owner/criteria.json
{
"context": "Tests a Rust binary release where semantic-release owns signed Cargo version/tag preparation, cargo-dist alone owns immutable GitHub Release assets, and a custom signed publisher updates Homebrew.",
"type": "weighted_checklist",
"checklist": [
{ "name": "Signed Cargo preparation", "max_score": 10, "description": "semantic-release deterministically prepares existing Cargo.toml and Cargo.lock before an App-signed writeback. The @jno21/semantic-release-github-commit package is pinned to exact v1.0.1; any external GitHub Action is pinned to a full commit SHA with its version comment. A repo-owned [skip release] marker has matching branch-job guards, and recognized skip-ci instructions that suppress tag workflows are not used." },
{ "name": "Verified manifest commit", "max_score": 10, "description": "The workflow either uses plugin v1.0.1 under a named concrete external branch lease blocking merges and direct pushes from before analysis through ref update, or an App-signed API integration with the analyzed SHA as expected head. It uses a scoped App token without custom identity, performs plugin origin cleanup when applicable, and proves the tag points to the returned verified Cargo version commit." },
{ "name": "cargo-dist sole release owner", "max_score": 12, "description": "The semantic-release config omits @semantic-release/github and other asset publishers; cargo-dist's generated tag workflow alone creates the GitHub Release and uploads binaries." },
{ "name": "Immutable release transaction", "max_score": 10, "description": "cargo-dist assembles the complete asset set before publication, then the workflow proves draft: false, immutable: true, gh release verify success, and expected asset names/digests." },
{ "name": "Current cargo-dist workflow", "max_score": 4, "description": "The solution uses dist init/generate and its generated plan, build, host, publish, and announce workflow rather than a stale cargo-dist action or a build-only command." },
{ "name": "Homebrew formula generation", "max_score": 6, "description": "dist configuration enables the Homebrew installer and names a homebrew- prefixed tap." },
{ "name": "No unsigned built-in publisher", "max_score": 8, "description": "The config does not use built-in publish-jobs = [\"homebrew\"] when the tap requires signatures; it selects a custom post-announce reusable job instead." },
{ "name": "Signed custom tap publisher", "max_score": 10, "description": "The custom post-announce workflow first verifies the exact published immutable Release, stages only the generated formula, and commits it through a full-SHA-pinned signed API action using the observed tap checkout parent as expectedHeadOid. It rejects tap drift and has no local git commit/push or custom bot identity." },
{ "name": "Scoped cross-repo token", "max_score": 8, "description": "The tap job reads source Release state with its source workflow token and mints a separate short-lived App token naming only the tap repository with least Contents permissions and no broad PAT." },
{ "name": "Artifact source integrity", "max_score": 6, "description": "The formula derives URLs and checksums from the exact verified immutable release assets, not a rebuild or mutable workflow artifact." },
{ "name": "Live parity readback", "max_score": 8, "description": "The workflow verifies source tag, Cargo versions, immutable release assets, tap formula version/digests, and tap commit signature all agree." },
{ "name": "Durable recovery", "max_score": 4, "description": "The same reusable workflow has a validated manual recovery path keyed by exact tag and durable release/tap state, can repair missing tap parity after publication, and never mutates an immutable release." },
{ "name": "History and action pins", "max_score": 4, "description": "Release checkout uses fetch-depth: 0 and high-trust actions are pinned to full commit SHAs with exact version comments." }
]
}
evals/rust-cargo-dist-immutable-owner/task.md
# Automate an Immutable Rust CLI Release with cargo-dist
## Problem/Feature Description
Northstar Tools ships `trailctl`, a Rust CLI distributed as cross-platform
binaries and a Homebrew formula, but not through crates.io. Conventional
commits on `main` should determine the next version automatically.
The organization enforces immutable GitHub Releases and verified commits on the
source and Homebrew tap default branches. Semantic-release may prepare the
Cargo version and create the tag, but cargo-dist's generated tag workflow must
be the sole GitHub Release and binary-asset owner. Publishing a GitHub Release
from semantic-release first would freeze it before cargo-dist uploads assets.
Cargo-dist may generate the Homebrew formula, but its built-in Homebrew
publisher uses an ordinary local commit. Configure a custom reusable
post-announce job that commits the generated formula through a full-SHA-pinned
signed API action with a short-lived GitHub App token. The recovery path must
reconcile an existing immutable release and missing tap update by exact tag. A
tap write must stage only the generated formula, use the observed tap checkout
parent as the atomic expected head, and fail if the tap advances. A
superseded-run preflight and Actions concurrency are not an atomic branch lock.
Use plugin v1.0.1 only if a concrete external branch lease blocks every merge
and direct push from before semantic-release starts release analysis through
the plugin's API ref update. Otherwise use a full-SHA-pinned App-signed API
integration with the analyzed SHA as its expected head. If plugin v1.0.1 is
selected, restore a credential-free `origin` immediately after semantic-release
in an `if: always()` step.
Use a repo-owned `[skip release]` marker and branch-job guards for the version
commit; GitHub's recognized `[skip ci]` would also suppress cargo-dist's tag
workflow.
## Output Specification
Produce:
- `.github/workflows/ci.yml` for verification and semantic-release version/tag creation
- `.releaserc.json` with deterministic Cargo preparation and signed manifest writeback, but no GitHub Release publisher
- `dist-workspace.toml` and the generated cargo-dist release workflow
- `.github/workflows/publish-homebrew.yml` as the custom post-announce signed
tap publisher, callable by cargo-dist and manual recovery
- `SETUP.md` documenting `RELEASE_APP_CLIENT_ID`,
`RELEASE_APP_PRIVATE_KEY`, tap-only write-token scope, and least permissions
The workflows must prove the tag points to the signed Cargo manifest commit,
cargo-dist publishes the complete immutable release, release assets/digests
verify, and the signed tap formula references those same assets.
## Input Files
=============== FILE: Cargo.toml ===============
[package]
name = "trailctl"
version = "0.4.0"
edition = "2024"
[[bin]]
name = "trailctl"
path = "src/main.rs"
=============== END FILE ===============
=============== FILE: Cargo.lock ===============
# Existing committed lockfile for the CLI.
=============== END FILE ===============
evals/rust-release-plz-cargo-dist-dual/criteria.json
{
"context": "Tests release-plz plus cargo-dist dual distribution with event-triggering App auth, single GitHub Release ownership, crates.io OIDC, signed Homebrew publication, and durable recovery.",
"type": "weighted_checklist",
"checklist": [
{ "name": "Release PR preparation", "max_score": 8, "description": "release-plz maintains a Release PR that deterministically updates Cargo.toml, Cargo.lock, and CHANGELOG.md before merge." },
{ "name": "Event-triggering App token", "max_score": 12, "description": "The default GITHUB_TOKEN has no repository write permissions. Both release-plz jobs mint/pass a short-lived source App token: Contents plus Pull requests write for the PR job, Contents write plus Pull requests read for the release job. They do not use the default GITHUB_TOKEN for events that must trigger PR CI or cargo-dist." },
{ "name": "Verified default branch", "max_score": 8, "description": "The version files reach main through the verified Release PR merge, and release-plz release performs no later default-branch writeback." },
{ "name": "Trusted crates.io publish", "max_score": 8, "description": "The release job grants id-token: write only for crates.io OIDC and uses trusted publishing without granting the default GITHUB_TOKEN repository write or supplying CARGO_REGISTRY_TOKEN." },
{ "name": "release-plz ownership boundary", "max_score": 10, "description": "release-plz.toml keeps git_tag_enable = true but sets git_release_enable = false, so release-plz publishes crates.io and the tag without creating a competing GitHub Release." },
{ "name": "cargo-dist sole release owner", "max_score": 10, "description": "The App-created tag triggers cargo-dist's generated plan/build/host/publish/announce workflow, which alone creates the GitHub Release and complete binary asset set." },
{ "name": "Immutable release proof", "max_score": 10, "description": "cargo-dist publishes the complete release once, then the workflow proves draft: false, immutable: true, gh release verify success, and expected asset names/digests." },
{ "name": "Signed post-announce tap", "max_score": 10, "description": "Homebrew generation is enabled, the unsigned built-in publisher is disabled, and a custom post-announce workflow reads source state with the read-only source token, then uses a separate tap-only App token and full-SHA-pinned signed API action to stage and commit only the formula. It uses the observed tap checkout parent as expectedHeadOid and rejects tap drift." },
{ "name": "State-specific recovery", "max_score": 10, "description": "Validated exact-tag recovery publishes only a genuinely missing boundary and never republishes a completed immutable boundary. An already-immutable Release with an incomplete asset set is inconsistent and must roll forward to a new version; recovery does not depend on rerunning a completed publisher." },
{ "name": "Live parity", "max_score": 8, "description": "Completion rereads Cargo/default branch, verified merge commit, tag target, crates.io version, immutable assets/digests, formula contents, and tap commit signature." },
{ "name": "Workflow guardrails", "max_score": 6, "description": "Trusted checkouts use full history without persisted write credentials, release jobs are non-cancellable, manual recovery validates inputs before secrets, and high-trust actions use full-SHA pins with exact version comments." }
]
}
evals/rust-release-plz-cargo-dist-dual/task.md
# Automate a Dual crates.io and Binary Rust Release
## Problem/Feature Description
Keystone Systems ships `glyph`, a Rust crate that is both a library on crates.io
and a cross-platform CLI distributed through an immutable GitHub Release and a
Homebrew formula. Use release-plz for the Release PR, checked-in Cargo version
and changelog, crates.io publication, and version tag. Use cargo-dist's
generated tag workflow as the sole GitHub Release and binary-asset owner.
The release-plz configuration must disable its GitHub Release publisher while
keeping tag creation enabled. Both release-plz jobs must use a short-lived,
repository-scoped GitHub App token so the automated Release PR receives CI and
the created tag actually triggers cargo-dist; the default repository
`GITHUB_TOKEN` is insufficient for those follow-up workflow events. crates.io
uses trusted publishing/OIDC rather than a long-lived registry token.
The source and Homebrew tap default branches require verified commits.
Cargo-dist generates the formula but must not use its unsigned built-in
Homebrew publisher. A custom post-announce reusable workflow verifies the
published immutable Release and commits only the formula through a pinned
App-signed API action. Recovery reconciles crates.io, tag, GitHub Release,
assets, and tap state without republishing immutable boundaries. If a published
immutable Release has an incomplete asset set, fail closed and roll forward to
a new version rather than attempting asset repair.
The tap write stages only the generated formula, uses the observed tap checkout
parent as the atomic expected head, and fails if the tap advances.
## Output Specification
Produce:
- `.github/workflows/release-plz.yml` for Release PR and crates.io/tag release
- `release-plz.toml`
- `dist-workspace.toml` and cargo-dist's generated tag workflow
- `.github/workflows/publish-homebrew.yml` with workflow-call and validated
manual recovery entrypoints
- `SETUP.md` documenting trusted publishing plus separately minted source-only
release-plz and tap-only Homebrew App tokens
## Input Files
=============== FILE: Cargo.toml ===============
[package]
name = "glyph"
version = "1.3.0"
edition = "2024"
[lib]
path = "src/lib.rs"
[[bin]]
name = "glyph"
path = "src/main.rs"
=============== END FILE ===============
=============== FILE: Cargo.lock ===============
# Existing committed lockfile.
=============== END FILE ===============
=============== FILE: CHANGELOG.md ===============
# Changelog
## 1.3.0
- Existing release history.
=============== END FILE ===============
evals/secrets-management-and-workflow-permissions/criteria.json
{
"context": "Tests whether the agent uses environment-scoped runtime secrets, OIDC for deploy identity (not long-lived credentials), correct secret hygiene in workflow steps, and minimal workflow permissions.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Environment-scoped runtime env",
"max_score": 12,
"description": "Keeps runtime application secrets in the production GitHub Environment or runtime secret store, rather than referencing repository-level `secrets.*` directly from the deploy step"
},
{
"name": "op:// references in template",
"max_score": 8,
"description": "An env template file is committed to the repo containing secret-store references (not plaintext secrets), e.g. `secret://production/api/DATABASE_URL` format"
},
{
"name": "OIDC for cloud credentials",
"max_score": 12,
"description": "Uses an OIDC-backed provider identity mechanism for deploy access, NOT long-lived provider credentials in repository secrets"
},
{
"name": "id-token write at job level",
"max_score": 9,
"description": "The deploy job grants `id-token: write` permission at the job level (not at the workflow root), and the workflow root uses `contents: read` only"
},
{
"name": "Root permissions minimal",
"max_score": 7,
"description": "The workflow-level `permissions:` block grants only `contents: read` (or is absent and the job-level grants are used exclusively)"
},
{
"name": "Production environment declared",
"max_score": 10,
"description": "The secret-bearing deploy job declares the production GitHub Environment before loading deploy credentials or runtime secrets"
},
{
"name": "Post-deploy handoff credential isolation",
"max_score": 10,
"description": "Any monitoring, notification, synthetic-check, or incident handoff job after deploy has read-only permissions and asserts cloud/provider credential variables are absent"
},
{
"name": "No secrets as CLI flags",
"max_score": 7,
"description": "Secret values are passed to commands via environment variables (the `env:` block), NOT as positional or named command-line arguments"
},
{
"name": "No env file dump",
"max_score": 7,
"description": "No workflow step uses `cat`, `echo`, or `print` to output the full contents of a rendered `.env` or secrets file"
},
{
"name": "non-sensitive ids in vars",
"max_score": 7,
"description": "Non-sensitive identifiers like account IDs, project IDs, regions, service names, and environment URLs are referenced from `vars.*` (not `secrets.*`) in the workflow"
},
{
"name": "Fine-grained PAT",
"max_score": 6,
"description": "Cross-repo GitHub writes use a narrowly scoped credential (preferred: short-lived GitHub App installation token; otherwise a purpose-scoped secret name such as `OPS_TRIGGER_TOKEN`) and the explanation does NOT reference a classic `ghp_` PAT or an unscoped token"
},
{
"name": "GitHub secrets only for bootstrap",
"max_score": 5,
"description": "The written explanation or comments describe repository secrets as bootstrap-only and place production deploy/runtime secrets on GitHub Environments or the runtime secret store"
}
]
}
evals/secrets-management-and-workflow-permissions/task.md
# Secure Secrets and Credentials Wiring for a GitHub Actions Deploy Pipeline
## Problem/Feature Description
A platform team is migrating a Node.js API service from manual dashboard deployments to GitHub Actions. The service connects to a PostgreSQL database, calls the Stripe API, and uses several internal service tokens. A previous attempt to automate the deployment stored these connection strings and API keys directly in GitHub repository secrets and passed them to the deploy command via workflow YAML. A security audit flagged this setup: secrets were visible in workflow logs, rotation required updating multiple repositories manually, and there was no clear separation between the credentials the CI system needs to operate and the credentials the running application needs.
The deploy provider supports OIDC federation for CI identity. The application's runtime secret store already contains all runtime credentials under reference paths such as `secret://production/api/DATABASE_URL`. The goal is to redesign the secrets and credentials wiring so that long-lived deploy credentials are eliminated from repository secrets, runtime application secrets stay in the runtime secret store or production GitHub Environment, and nothing sensitive appears in logs or workflow YAML.
## Output Specification
Produce the following files:
- `.github/workflows/main.yml`: the push-to-main deploy workflow with correct permissions, OIDC auth, environment-scoped secret loading, and a read-only monitoring and incident handoff
- `deploy/production.env.example`: the committed env template with plausible but fictional `secret://` references
- `secrets-design.md`: the credential categories, where each lives, and why, including GitHub Environments, provider identity, runtime secret stores, and `vars.*`
Do not include any real credentials or tokens in the files. Any monitoring, notification, synthetic-check, or incident handoff job must not receive deploy-provider credentials and should fail if provider credential env vars are present.
evals/swift-cocoapods-immutable-recovery/criteria.json
{
"context": "Tests a Swift library release with App-signed version files, SwiftPM tag distribution, CocoaPods publication, metadata-only immutable GitHub Release, and state-specific recovery.",
"type": "weighted_checklist",
"checklist": [
{ "name": "Swift verification", "max_score": 6, "description": "The read-only verify job runs the repository's Swift tests plus podspec validation before release." },
{ "name": "Deterministic version prepare", "max_score": 8, "description": "A narrow prepare script validates and updates the existing Package.swift and OrbitKit.podspec to exactly nextRelease.version." },
{ "name": "Prepare and publish order", "max_score": 8, "description": "Commit analysis and notes precede deterministic version preparation; the selected signed writeback commits those files, and the tag plus CocoaPods/GitHub publication continue only from the returned signed commit OID." },
{ "name": "App-signed version commit", "max_score": 10, "description": "The selected mechanism commits only Package.swift and OrbitKit.podspec with a [skip ci] release message, a scoped App token, and no custom identity. The @jno21/semantic-release-github-commit package is pinned to exact v1.0.1; any external GitHub Action is pinned to a full commit SHA with its version comment." },
{ "name": "Head stability", "max_score": 6, "description": "The workflow either uses plugin v1.0.1 under a named concrete external branch lease blocking merges and direct pushes from before analysis through ref update, or an App-signed API integration with the analyzed SHA as expected head. Preflight and Actions concurrency alone do not qualify." },
{ "name": "Credential boundary", "max_score": 8, "description": "The release Environment contains the App credentials and CocoaPods token; secrets are step-scoped and checkout persists no write credential. Plugin v1.0.1 additionally requires immediate if: always() cleanup to a credential-free origin." },
{ "name": "CocoaPods publication", "max_score": 8, "description": "A narrow publish script runs pod trunk push OrbitKit.podspec and verifies the exact version exists on trunk." },
{ "name": "Immutable metadata release", "max_score": 8, "description": "@semantic-release/github is the sole GitHub Release owner, appends no assets after publication, and completion requires immutable: true plus gh release verify." },
{ "name": "Tag and version parity", "max_score": 8, "description": "The tag points to the verified version commit and both checked-in version strings plus the podspec source tag equal the release version." },
{ "name": "State-specific backfill", "max_score": 12, "description": "Validated exact-tag recovery publishes only a missing pod or missing GitHub Release, rejects inconsistent state, never creates another bump, and does not rely on a normal semantic-release rerun." },
{ "name": "Live completion proof", "max_score": 8, "description": "Initial and recovery paths reread the peeled tag target and verified signed commit, version files from that commit, containment in the live default branch, CocoaPods Trunk version, GitHub immutability, release verification, and parity." },
{ "name": "Release guardrails", "max_score": 4, "description": "Release uses full history, skip-ci guards on verify/release, non-cancellable release concurrency, and full-SHA high-trust action pins." },
{ "name": "No unsigned escape", "max_score": 6, "description": "No local git commit/push, GPG workaround, custom bot identity, second writeback job, or destructive tag/release retry is introduced." }
]
}
evals/swift-cocoapods-immutable-recovery/task.md
# Automate a Signed SwiftPM and CocoaPods Release
## Problem/Feature Description
Northwind Mobile publishes `OrbitKit` through both Swift Package Manager and
CocoaPods. Conventional commits on `main` should determine the next version.
The release must deterministically update the checked-in version in
`Package.swift` and `OrbitKit.podspec`, commit those files through GitHub's
App-signed API path, create the tag on that verified commit, push the podspec to
CocoaPods trunk, and publish a metadata-only immutable GitHub Release.
The `release` Environment holds `RELEASE_APP_CLIENT_ID`,
`RELEASE_APP_PRIVATE_KEY`, and `COCOAPODS_TRUNK_TOKEN`. A preflight head check
and Actions concurrency are not an atomic branch lock. Use plugin v1.0.1 only
if a concrete external branch lease blocks every merge and direct push from
before semantic-release starts release analysis through the plugin's API ref
update. Otherwise use a full-SHA-pinned App-signed API integration with the
analyzed SHA as its expected head. The selected writeback may receive only the
existing regular version files and no custom identity. If plugin v1.0.1 is
selected, an `if: always()` step must immediately restore a credential-free
`origin` afterward.
CocoaPods trunk and GitHub Releases are separate immutable boundaries. Provide
a validated exact-tag recovery path for either `tag + pod, no GitHub Release`
or `tag + GitHub Release, no pod`. It must publish only the missing boundary,
reread full parity, and never create another bump or depend on a normal
semantic-release rerun.
## Output Specification
Produce:
- `.github/workflows/ci.yml` with verification, release, and validated recovery
- `.releaserc.json`
- `scripts/prepare-release.sh`
- `scripts/publish-cocoapods.sh`
- `SETUP.md` documenting the Environment credentials and App scope
Use full-SHA pins with exact version comments for high-trust actions.
## Input Files
=============== FILE: Package.swift ===============
// swift-tools-version: 6.0
import PackageDescription
let orbitKitVersion = "1.4.2"
let package = Package(
name: "OrbitKit",
products: [.library(name: "OrbitKit", targets: ["OrbitKit"])],
targets: [.target(name: "OrbitKit"), .testTarget(name: "OrbitKitTests", dependencies: ["OrbitKit"])]
)
=============== END FILE ===============
=============== FILE: OrbitKit.podspec ===============
Pod::Spec.new do |spec|
spec.name = "OrbitKit"
spec.version = "1.4.2"
spec.summary = "Shared mobile primitives"
spec.source = { :git => "https://github.com/northwind-mobile/orbit-kit.git", :tag => "v#{spec.version}" }
spec.source_files = "Sources/OrbitKit/**/*.swift"
spec.ios.deployment_target = "15.0"
end
=============== END FILE ===============
references/actions-security.md
# Actions Security
Use when a workflow executes project code or loads publish, signing, deploy, or
other privileged credentials.
## Trust Boundary
- Never use `pull_request_target` to check out, install, build, test, package,
or otherwise execute pull-request code.
- Fork and pull-request jobs use `pull_request`, read-only permissions, and no
delivery secrets.
- Secret-bearing work runs only on trusted branches, protected tags, or a
validated manual dispatch.
- Manual inputs are validated in a secretless job, refs resolve to one immutable
SHA, and downstream jobs consume only sanitized outputs.
## Permissions and Credentials
Default workflow permissions to `contents: read` or `{}`. Grant write, OIDC,
attestation, or pull-request permissions only to the job that needs them.
Monitoring and notification jobs stay read-only.
Use `persist-credentials: false` through checkout, install, build, pack, and
test in privileged workflows. Add write credentials only at the narrow delivery
boundary. Fetch full history only when tags, history, or affected detection
requires it.
## Dependencies, Caches, and Logic
- Pin high-trust remote Actions to reviewed full SHAs and keep an automated
update path. Repository-level SHA enforcement is useful only after the
current allowlist and updater contract are understood.
- Run `actionlint`, `zizmor`, and appropriate secret scanners. Use supported
configuration instead of shell glue that merely silences them.
- When the owner maintains many repositories, define the scan baseline once as
a reusable workflow in the owner's `.github` repository
(`on: workflow_call`, every image and Action digest-pinned there).
- Give each repository a thin caller job
(`uses: <owner>/.github/.github/workflows/<name>.yml@main`) that owns its
triggers.
- Version and digest bumps then land in one place for every adopter.
- A repository with bespoke scanner needs keeps its own copy deliberately.
- `zizmor`'s blanket pin policy flags the caller's branch ref; adopters allow
first-party refs while keeping hash pins for everything else
(`.github/zizmor.yml`: `unpinned-uses` policies
`"<owner>/.github/*": ref-pin`, `"*": hash-pin`).
- Never share package caches from untrusted pull requests with privileged
publish, signing, release, or deploy jobs.
- Keep workflow YAML orchestration-thin. Prefer maintained Actions and the
repository's existing typed validation/task surfaces. When custom parsing,
ref policy, summaries, provider branching, or security-sensitive logic is
unavoidable, use a tested typed module or local action with explicit inputs
and outputs. Do not grow inline shell or move the same spaghetti into a new
`.sh` file; shell may only dispatch a few already-defined commands.
## Payloads and Artifacts
Actions artifacts are temporary same-run storage, not a durable release or
recovery boundary. A later run or recoverable deploy should consume an
immutable GitHub Release asset, registry version, image digest, provider-native
package, or signed archive with checksum/provenance.
Do not rebuild after verification unless the provider is intentionally the
builder and records equivalent provenance. Keep secret-bearing delivery jobs
non-cancellable and make retries reconcile durable state.
references/dependency-updates.md
# Dependency Updates
Choose one update bot per repository. Running both opens duplicate pull
requests. Dependabot security alerts and security updates are a separate
GitHub feature and stay on under either choice.
## Choose
| Signal | Choice |
| --- | --- |
| Only npm, Go, Cargo, or GitHub Actions manifests | Dependabot is sufficient; Renovate is equivalent |
| Pins in `mise.toml`, OpenTofu or Terraform providers, or annotated version variables | Renovate; Dependabot has no manager for these |
| Container images pinned by digest | Either; both update tag and digest together |
| Organization already runs one bot on most repositories | Match it; one mental model beats a marginal feature |
| Fork or mirror with no owned manifests | Neither |
The hosted Mend Renovate app is free for private repositories on the
Community plan. Its `IAC`, `SAST`, and `SCA` columns are separate paid Mend
scanners, unrelated to dependency updates. Verify the current plan on the
[Mend-hosted overview](https://docs.renovatebot.com/mend-hosted/overview/)
before relying on any limit.
Neither bot rewrites a sha256 checksum stored beside a version. Ansible and
script pins with checksums stay manual, or let the target fetch the upstream
`.sha256` file at install time so only the version needs bumping.
## Renovate
- Extend one organization preset (`github>uinaf/renovate-config`) and keep
repository files to opt-outs and approvals. Encode schedule, release age,
grouping, commit prefixes, and registry overrides once. The preset
repository must be public: the hosted app reads public repositories with
a token that cannot see private presets, and the failure is a
"Cannot find preset's package" issue on every public consumer.
- Automerge with Renovate's own `automerge` and `platformAutomerge: false`.
Renovate then waits for every visible check and skips repositories with
no checks. GitHub platform automerge merges immediately unless branch
rules require status checks, which most single-owner repositories lack.
Keep majors on `dependencyDashboardApproval`.
- Structure checks that require `.github/dependabot.yml` (for example a
workspace-kit `workspace.json` required-files list) must require
`renovate.json` instead, or the migration commit fails its own hook.
- Keep **Require config file** on in the Mend organization settings so
unmigrated repositories receive nothing while they still run Dependabot.
Turn **Create onboarding PRs** off when migrating by commit.
- Migrate a repository in one commit: add `renovate.json`, delete
`.github/dependabot.yml`, update any documentation that names Dependabot.
Renovate skips onboarding when a config already exists on the default
branch.
- OpenTofu repositories set `registryUrls` to `https://registry.opentofu.org`
for the `terraform-provider` and `terraform-module` datasources and disable
the `hashicorp/terraform` dependency, which otherwise tracks Terraform
releases for `required_version`.
- Use `dependencyDashboardApproval` for majors that need a planned migration
instead of `enabled: false`; the update stays visible on the dashboard.
- Validate with `npx --yes --package renovate -- renovate-config-validator`
before pushing. The validator checks option names only; it does not
resolve preset names. A misspelled preset such as `:pinDigests` instead
of `docker:pinDigests` passes locally, then opens an "Action Required"
issue and blocks all pull requests until fixed.
## Dependabot
- Configure only ecosystems and manifests that exist.
- Use monthly or weekly schedules with a cooldown, group patch and minor
updates, and separate majors.
- Prefix commits per ecosystem (`ci` for Actions, `deps` otherwise) so
release tooling classifies them.
- Preserve compatibility constraints with `ignore` rules rather than closing
pull requests repeatedly.
## Readback
After the first run, confirm the bot opened pull requests with the expected
prefix, grouping, and registry, and that the retired bot opened none. For
Renovate, the hosted job log on the Mend developer portal shows why a
repository produced nothing.
references/deploy-environments.md
# Deploy Environments
Use when changing target selection, promotion policy, provider identity, or
manual deploys. Provider commands remain in repo-owned scripts or infrastructure
code.
## Environment Contract
- Declare one GitHub Environment per blast radius: for example `staging`,
`production`, or an isolated preview.
- Keep production-only secrets and variables on that Environment.
- Use protection rules only for intentional human approval or policy gates.
- Publish jobs may suppress deployment records when no protection-rule app
needs a deployment object. Running-service deploys keep records enabled and
publish their target URL.
- Environment branch/tag policy constrains the workflow run ref, not a different
ref checked out later.
## Identity and Isolation
Prefer provider federation/OIDC scoped to repository, Environment, intended
branch or tag, audience, and one deployment role. Separate staging and
production identities and state. Static credentials are an Environment-scoped
fallback with a documented reason.
When a repo uses SST or an equivalent app-plus-infrastructure deployer, map its
stages to GitHub Environments and keep provider state/resource ownership
isolated. The tool remains a thin promotion layer; it does not justify shared
production/staging state, secret CLI flags, or an unverified rebuild.
## Payload Promotion
Promote the exact payload proved by verification: same trusted job output,
release asset, registry version, provider-native package, or immutable image
digest. Record source commit, producing run, reference, and checksum/digest for
manual or older-version promotion. Verify existence and provenance before
loading credentials.
Runtime secrets belong in the provider secret store or selected Environment;
non-sensitive account/project/region names belong in Environment variables.
Pass secrets through environment variables or stdin and never print rendered
configuration.
## Manual Deploys
Treat `workflow_dispatch` as promotion, not a new build:
1. Validate environment, lane, and ref in a secretless job.
2. Resolve the ref to one immutable SHA and prove the corresponding payload.
3. For production, restrict to the default branch/current default SHA or a
protected release tag explicitly supported by the repo contract.
4. Emit sanitized outputs, then load the target Environment and credentials.
5. Reuse the normal deploy concurrency key and downstream proof.
After deploy, hand off to repository-owned monitoring, alerting, synthetic
checks, and rollback. A shallow CI curl is not production evidence.
references/deploy-secrets.md
# Secrets and Credentials
Use this reference to keep deploy credentials short-lived, scoped, and quiet in logs. Prefer OIDC and GitHub Environments over provider-specific static-token recipes.
## Layers
Deploy workflows usually touch three secret classes:
- CI identity: the trust material GitHub Actions uses to authenticate to the deploy provider.
- Deploy configuration: environment names, project IDs, regions, service names, URLs, and role names.
- Runtime secrets: values the running app reads, such as database URLs, payment keys, signing keys, and internal service tokens.
Keep these classes separate. Repo-level secrets are bootstrap-only; production deploy credentials and runtime secrets belong to GitHub Environments or the provider's secret system.
## OIDC First
When the provider supports federation, use GitHub's OIDC token instead of
long-lived credentials. Grant `id-token: write` only to the Environment-scoped
job that uses a maintained provider Action or repository-owned typed identity
client; do not invent a shell credential broker.
The provider trust policy should bind at least:
- repository owner/name
- branch, protected tag, or GitHub Environment
- provider audience
- deployment role or environment
Use one identity per blast radius. Production and staging should not share the same provider role.
## Static Tokens
Use static tokens only when federation is unavailable or the provider API does not support it.
- Store static tokens on the GitHub Environment, not as repository-level secrets.
- Give each token one purpose and one environment.
- Prefer narrowly scoped provider tokens over broad user tokens.
- Rotate static tokens on a schedule and after any runner, dependency, or workflow compromise.
- Document why OIDC was not used.
## Runtime Secrets
Runtime secrets should be resolved by the deploy provider or environment-specific secret store whenever possible. If the workflow must render runtime config:
- render inside the GitHub runner after the environment is selected
- keep the rendered file in `$RUNNER_TEMP`
- transfer only to the deploy target that needs it
- remove it at the end of the job when practical
- log key names or counts only, never values
Do not commit plaintext runtime values. Template files may contain secret-store references when those references are non-sensitive without their corresponding access token.
## Logging Hygiene
GitHub masks declared secrets in logs. It does not reliably mask:
- values rendered to disk
- substrings of secrets concatenated with other text
- secrets passed as command-line arguments
- provider tokens returned by CLI debug output
Rules:
1. Pass secrets through env vars or stdin.
2. Disable verbose CLI logging in secret-bearing jobs.
3. Avoid `set -x` in deploy scripts.
4. Debug rendered env files by logging key counts or key names only.
references/deploy-troubleshooting.md
# Deploy Troubleshooting
Use only after a concrete deploy failure or mismatch.
## Selection and Payload
- **Every lane deployed:** inspect affected-graph outputs. Shared packages,
lockfiles, workflow, and infrastructure changes must fan out to every
consuming lane.
- **Production differs from E2E:** compare source SHA, producing run, payload
reference, digest/checksum, and deployed version. Remove any deploy-time
rebuild or checkout drift; promote the verified payload.
- **Artifact quota blocks deploy:** replace cross-run Actions-artifact handoff
with a durable release asset, registry version, image digest, or
provider-native package. Same-run deploys may keep the verified build in one
trusted job.
- **Unsafe manual ref:** validate and resolve inputs in a secretless job, prove
payload existence, then load the Environment and credentials.
## Identity and Concurrency
- **Environment secret missing:** verify the job declares the intended
Environment, its branch policy permits the run ref, protections completed,
and the secret exists at that scope.
- **OIDC rejected:** compare actual repository, event, ref, Environment,
audience, and provider role claims with the trust policy. Separate staging
and production roles.
- **Older deploy wins:** every push and manual path for the same target/lane
must share one non-cancellable `deploy-<environment>-<lane>` critical section.
- **Post-deploy job has provider credentials:** split monitoring, notification,
and synthetic work into a read-only job with no inherited auth setup.
## False Green
If users fail while the workflow is green, remove shallow endpoint-check
claims. Confirm the deployed URL and payload identity, publish the repository's
real deploy marker, inspect monitoring and alert coverage, and expose the
rollback pointer.
Every deploy summary should name environment, lane, source commit, payload
identity, deployed URL, monitoring/alert evidence, and rollback route. Prefer a
small tested repo-owned summary helper over repeated workflow shell.
references/deploy-workflows.md
# Deploy Workflows
Use for GitHub Actions that verify and promote a running application or
service. Provider mechanics belong in maintained Actions, the repository's
typed task surface, a tested local action, or infrastructure code.
## Shape and Trust
Use the smallest workflow layout the repository can operate:
- pull requests and merge queue: verification only, read-only credentials
- default-branch push: detect affected lanes, verify, build one payload, test
that payload, deploy through a GitHub Environment
- manual promotion: validate a release tag, digest, deployment id, or exact
SHA in a secretless job, then promote the already verified payload
Do not use `pull_request_target` to execute project code. GitHub Environment
branch rules constrain the workflow run ref, not a separately checked-out
manual input.
Keep complex ref resolution, change mapping, summaries, and provider branching
in tested typed repository code. Workflow YAML should orchestrate narrow
commands and maintained actions; a separate shell file does not make inline
shell complexity structured.
## Lane Detection and Required Checks
Use path filters for simple independent surfaces or the repository's dependency
graph for monorepos. Shared packages, lockfiles, workflows, containers, and
infrastructure paths must fan out to every consuming lane.
Do not use trigger-level path filters when branch protection requires the
workflow. Detect no-op lanes inside the workflow and end with one stable result
job that runs under `always()`, fails closed on unexpected skips, and reports
why a lane ran or did not run.
## Verified Payload
The same payload crosses build, e2e, and deploy:
- same-job filesystem output for a simple trusted static deploy
- immutable container digest
- package or GitHub Release asset
- provider-native package or deployment id
Do not rebuild after e2e. GitHub Actions artifacts are acceptable as short
same-run scratch storage only when the repository accepts their quota and
retention coupling; they are not the default production registry.
Manual redeploys identify the source commit, producing run, payload reference,
and digest. Prove payload existence before loading deploy credentials.
## Concurrency, Permissions, and Caches
- Verification may cancel superseded runs.
- Deploys serialize with one non-cancellable key per environment and lane,
shared by automatic and manual paths.
- Workflow permissions start read-only and add OIDC or provider scopes only to
the Environment-scoped deploy job.
- Monitoring and notification follow-ups stay read-only and receive no deploy
credentials.
- Privileged deploys do not consume caches populated by untrusted pull requests.
Read [Environments](deploy-environments.md) for target and identity policy and
[credentials](deploy-secrets.md) for secret ownership and log hygiene.
## Deployment Proof and Handoff
Deploy success is the provider's accepted immutable payload plus the
repository's real monitoring or synthetic evidence, not a shallow curl added to
make CI look complete.
End each run with a concise handoff:
- environment and lane
- source revision and payload identity
- deployed URL or provider deployment id
- monitoring and alert surface
- rollback command or runbook
Keep summary formatting in a repository helper once it exceeds a few lines.
Use [deploy troubleshooting](deploy-troubleshooting.md) only for a concrete
failure or mismatch.
references/implementations.md
# Maintained Implementations
Use only when creating or materially rewriting GitHub workflow code. These are
working public examples, not universal templates. Read each repository's guide,
manifest, scripts, and live GitHub settings; reuse the named contract while
adapting triggers, gates, identities, targets, and current dependency pins.
| Contract | Code to inspect |
| --- | --- |
| Pull-request verification plus npm trusted publishing, App-signed version writeback, and immutable release readback | [`uinaf/workspace-kit` verify](https://github.com/uinaf/workspace-kit/blob/main/.github/workflows/verify.yml), [release](https://github.com/uinaf/workspace-kit/blob/main/.github/workflows/release.yml), and [semantic-release config](https://github.com/uinaf/workspace-kit/blob/main/.releaserc.json) |
| Monorepo verification and a non-cancellable Environment-scoped Cloudflare deploy, separate from package release | [`uinaf/attach` main](https://github.com/uinaf/attach/blob/main/.github/workflows/main.yml), [release](https://github.com/uinaf/attach/blob/main/.github/workflows/release.yml), and [task graph](https://github.com/uinaf/attach/blob/main/package.json) |
| Draft-first binary release, checksums, provenance attestations, immutable publication, and downstream Homebrew update | [`uinaf/tccutil-rs` CI/release](https://github.com/uinaf/tccutil-rs/blob/main/.github/workflows/ci.yml) and [semantic-release config](https://github.com/uinaf/tccutil-rs/blob/main/.releaserc.json) |
| Organization-level collaboration defaults | [`uinaf/.github`](https://github.com/uinaf/.github) |
Before reuse, confirm the linked repository is still public and active, open the
current source rather than relying on this summary, and preserve the target
repository's own lifecycle and policy. If no example matches, implement the
smallest tested module or local action in the repository's primary language
instead of growing inline workflow or shell logic.
references/release-targets.md
# Publish Targets
Load only the selected target. Shared trust, immutable-publication, recovery,
and completion rules live in [release-workflows.md](release-workflows.md).
## Target Matrix
| Target | Version owner | Publish boundary | Required proof |
| --- | --- | --- | --- |
| npm package | semantic-release, changesets, or repo-selected manager | npm trusted publishing or scoped token fallback | packed contents, registry version, provenance, tag/Release parity |
| SwiftPM | Git tag | trusted Git tag | resolved package revision and release-tag parity |
| CocoaPods | prepared podspec plus Trunk | CocoaPods token | Trunk version plus tag, podspec, and Release parity |
| Go binary | semantic-release or repo-selected tag owner plus GoReleaser | draft GitHub Release, then publish | complete assets, checksums, attestation, immutable Release |
| Rust library | release-plz or repo-selected Cargo manager | crates.io trusted publishing | crate version, tag, provenance, changelog or manifest parity |
| Rust binary | cargo-dist or equivalent single asset owner | draft GitHub Release | generated workflow, installers, complete immutable assets |
| GitHub Action | release tag plus moving major pointer | Git refs and Release | committed bundle, immutable version tag, monotonic major tag |
| Homebrew | source publisher plus tap writer | signed tap commit | formula/cask digest, audit, verified tap commit |
Do not combine two version managers or two GitHub Release owners for the same
artifact.
## npm
Use GitHub-hosted Actions for npm trusted publishing, including private
repositories: [npm does not support self-hosted runners for trusted
publishing](https://docs.npmjs.com/trusted-publishers/). Configure the package
for the exact repository, workflow file, and Environment; grant
`id-token: write`; remove `NPM_TOKEN`. Use a granular package-scoped token on
the release Environment only when trusted publishing is unavailable.
Before enabling automation, prove the package already exists or perform the
explicitly authorized one-time bootstrap publication. Verify `npm pack` output,
public scoped-package access, repository metadata, CLI `bin` contents, and the
published registry version.
## SwiftPM and CocoaPods
SwiftPM publishes through the tag. CocoaPods adds a separate immutable Trunk
boundary. Prepare version files deterministically, sign any protected-branch
writeback through the selected API path, then publish the podspec with the
Environment-scoped token.
If only one boundary succeeds, backfill the missing boundary from the exact
trusted tag. Never republish an existing pod version or create a second bump as
generic retry behavior.
## Go and GoReleaser
Use one tool to choose/create the tag and GoReleaser to build the complete
asset set. Keep the GitHub Release as a draft while GoReleaser uploads; compare
the exact draft asset names and digests with the current build manifest before
publishing.
Bind every GoReleaser invocation to the selected exact tag. Resolve and peel
the remote tag, require it to match the intended commit, and reread it before
immutable publication. If GoReleaser updates Homebrew, prefer its native
GitHub-App commit path and omit custom author or committer fields.
## Rust
Choose one version owner:
- Binary without crates.io: a tag/version manager prepares the Cargo version
and tag; cargo-dist alone creates the GitHub Release and assets.
- Library or crates.io distribution: release-plz owns Cargo versions and
publication. For dual binary distribution, disable its GitHub Release
creation and let cargo-dist own assets.
Prefer crates.io trusted publishing when available. Do not mix release-plz
with a semantic-release Cargo writeback. Commit `Cargo.lock` for binaries unless
the repository has an explicit contrary contract.
## Homebrew
Treat the tap as a separate signed-write destination. The source workflow token
cannot write a sibling tap. Prefer an organization-owned GitHub App and mint a
tap-only installation token.
Use the publisher's native GitHub-App commit support when available. Otherwise
generate the formula or cask deterministically and use a narrow API commit that
checks the tap head observed before generation. Read back signature
verification and run the tap's applicable `brew audit` path.
Compute URLs and checksums from the exact immutable source release. A tap update
is downstream reconciliation and must be repairable without mutating that
release.
## GitHub Actions
Build and commit the action bundle during the pull request. Verification should
delete or rebuild the generated surface and fail on any changed, missing,
stale, or untracked output before tagging.
Version tags are immutable; a major tag such as `v1` is an intentionally
mutable compatibility pointer. Update it only to the highest eligible published
stable release in that major, using peeled Git-ref commit identities and an
expected-old-or-absent compare-and-swap. Reread the pointer after mutation.
## Monorepos
Choose coordinated or independent versions deliberately. Independent packages
need collision-free tag formats and per-package working directories. For
coordinated releases, use the repository's established workspace release tool
instead of parallel semantic-release jobs invented during GitHub setup.
references/release-troubleshooting.md
# Release Troubleshooting
Use only after a concrete release failure. Durable tag, release, registry,
default-branch, signature, tap, and deployment state outrank workflow status.
## No or Wrong Version
- **No release:** inspect commits since the last reachable tag and dry-run the
repo-pinned release launcher. Non-release commit types are normally no-ops.
- **Wrong version:** verify full history, reachable tags, analyzer rules, and
matching analyzer/notes presets.
- **Recursive bump:** the writeback and every trigger path must use the same
repository skip convention.
## Writeback or Tag Failure
- Confirm the selected GitHub App is installed for the repo, has the required
content scope, and is allowed by effective branch rules.
- A token used by ordinary Git transport does not make a commit signed. Use the
release tool's App-native signed-writeback path or a reviewed API path that
preserves the required tree semantics.
- If logs name `github-actions[bot]`, find the remaining persisted checkout
credential or legacy local push path.
- Race or dangling-tag failures require one non-cancellable release critical
section plus an atomic expected-head/ref check where the tool mutates live
branch or tag state.
## Partial Publication
If a tag or source bump exists but a registry, asset, release, tap, or deploy
step failed:
1. Read the exact durable state of every target.
2. Do not create another version, delete an immutable release, or assume a
normal rerun resumes after an existing tag.
3. Use the state-specific backfill contract in
[release workflows](release-workflows.md#partial-failure-recovery).
4. Re-prove parity after recovery.
If the exact tag says a Release should already exist, use an authenticated
exact-tag lookup and fail closed on lookup errors. Add a bounded retry only for
demonstrated transient visibility lag. In explicit backfill, distinguish a
confirmed not-found response from authentication, rate-limit, network, and
server failures; only confirmed absence may create the missing Release.
When Actions artifact quota blocks deploy after successful publication, remove
the temporary artifact dependency and promote from the durable release asset,
registry, image digest, or provider-native package.
## Target-Specific Checks
- **npm auth/403:** verify trusted-publisher owner, repository, workflow, and
Environment match the actual job; require OIDC and a supported Node/npm
toolchain. Use a narrow Environment token only when trusted publishing is
unavailable. Public scoped packages need the correct access and repository
metadata.
- **CocoaPods duplicate:** the version may already exist. Reconcile trunk state;
never blindly republish the same version.
- **GoReleaser dirty tree:** find generated files written before release and
keep outputs outside the source tree or use the repo's clean-release mode.
- **Unsigned Homebrew tap update:** use the selected GoReleaser version's
GitHub App-native commit path, omit identity overrides that defeat signing,
and read back signature verification.
- **Marketplace major tag stale:** update the intentional mutable major pointer
only after selecting the highest eligible immutable stable release. Use an
expected-old-OID compare-and-swap and verify the peeled remote target.
- **Immutable asset replacement:** a published release is not a scratch area.
Skip mutation and resume only missing downstream parity, deploy, or smoke
checks.
Exact action inputs and plugin options change. Inspect the repository-pinned
major and its upstream documentation before diagnosing an option-name error.
references/release-workflows.md
# Release Workflows
Use for GitHub Actions that version, tag, sign, publish, or distribute an
immutable artifact.
## Workflow Shape
- Pull requests and default-branch pushes run the repository's verification
contract with read-only credentials.
- Release runs only from a trusted default-branch or validated protected-tag
event after verification.
- Manual release or backfill inputs are validated in a secretless job and
resolve to one immutable SHA or trusted tag before checkout or credentials.
- Release concurrency is non-cancellable and serialized for the publication
boundary. Verification concurrency may remain cancellable.
- Use one release-state owner. Multiple tools must not race to create the same
tag or GitHub Release.
Keep `persist-credentials: false` through checkout, install, build, pack, and
test. Introduce a scoped write identity only at the exact tag, release,
registry, or signed-writeback boundary.
## Permissions and Secrets
Start with `permissions: {}` or `contents: read`. Add only the job scopes the
selected target requires:
- `contents: write` for tags, releases, or source writeback
- `id-token: write` for trusted publishing, provider OIDC, or keyless provenance
- `attestations: write` for file attestations
- issue or pull-request write only when the configured release tool uses it
Use an approval-free `release` Environment when it is only a scoped credential
boundary. Keep deployment records for running-service deploys and custom
deployment-protection apps.
## Immutable Publication
Published GitHub Releases lock their tags and assets. Assemble the complete
transaction before publication:
```text
build -> verify payload -> create or resume draft -> attach complete manifest
-> verify checksums, signatures, and provenance -> publish once
```
If semantic-release chooses the version and notes before another tool uploads
assets, configure it to create a draft. If all assets already exist, one
publisher may create the draft, upload, and publish atomically. Never append or
replace assets after publication.
A recovery reads the exact tag and Release state first. When a tag implies that
a Release should exist, use an authenticated exact-tag lookup that can see
drafts. A failed expected lookup is an error, not a successful no-op. Add a
bounded visibility retry only when the repository has demonstrated transient
lookup lag. Published means mutation is over; continue only missing downstream
reconciliation.
## Signed Writeback
An App token authenticates a write but does not sign a local `git commit`.
Prefer the selected release tool's native GitHub API writeback that lets GitHub
sign the commit. Use a generic API commit only when no native path exists and
it preserves the full tree plus an expected-head compare-and-swap.
Reject superseded runs before release analysis. Workflow concurrency and a
head preflight do not form an atomic branch lease; a source writeback that
cannot compare the expected head needs a real external lease or a different
writeback implementation.
For semantic-release-specific plugin order, version files, and dry-run checks,
read [semantic-release.md](semantic-release.md).
## Completion Proof
One real release for each distinct workflow shape must prove every applicable
boundary:
- published, non-draft, immutable GitHub Release
- release verification and exact asset manifest
- peeled tag resolves to the intended commit
- protected-branch writeback is verified and contained in the live default branch
- version files at the release commit match the published version
- registry, tap, moving action tag, or deploy pointer references the same
version and payload digest
- a retry reaches the same state without mutating published assets
No-release and dry-run paths leave publication and downstream parity unverified.
## Partial-Failure Recovery
Reconcile durable state before choosing a repair:
| Durable state | Repair boundary |
| --- | --- |
| prepared signed commit, no tag | validate parent, tree, version, and signature; create the missing tag and run only missing publishers |
| tag exists, Release missing | create from the trusted tag; use a draft for assets |
| registry exists, Release missing | backfill the Release; never republish the registry version |
| tag exists, registry missing | publish the exact tagged package through a validated backfill |
| draft has partial assets | repair the draft, verify the complete manifest, publish once |
| immutable Release exists, downstream missing | run idempotent downstream reconciliation keyed by the tag |
- Backfills validate inputs before secrets, reread every durable boundary
after repair, and never create another version bump.
- In an explicit recovery path where a missing Release is a valid state,
accept only an unambiguous not-found response as absence.
- Authentication, authorization, rate-limit, network, and server failures
remain errors.
- Backfill from the already-trusted tag, then reread the exact Release.
- A create conflict or duplicate exact-tag state is reconciliation work, not
success.
## Supply-Chain and Handoff
- Secret-bearing jobs install fresh by default and do not consume caches
populated by untrusted pull requests.
- Build, package, sign, and publish from the trusted release commit or tag.
- Prefer registry versions, release assets, provider packages, or image digests
over GitHub Actions artifacts as the release-to-deploy boundary.
- Add finite timeouts and stable final checks for conditional or matrixed paths.
- Keep policy parsing and recovery logic in an existing structured tool or a
tested typed module/local action, not inline workflow blocks or ad-hoc shell.
references/repo-settings.md
# Repository Settings
Use when the task changes live GitHub repository policy. Read effective state
through the API or UI; checked-in files do not prove settings.
## Inspect First
Read default branch, merge methods, protections and rulesets, required checks,
merge queue, conversation resolution, signed-commit and tag rules, allowed
writers, Actions policy, Environments, visibility, security features, and
repository metadata. Preserve current policy unless the request owns it.
Before requiring pull requests or checks, inventory every default-branch
writer: maintainers, release and dependency bots, generated-data jobs, deploy
writebacks, and GitHub Apps. Each must move through a PR or have an explicit,
scoped compatible path.
## Collaboration Policy
- Pull requests are a review mechanism, not a default prerequisite. When direct
updates are authorized, keep the local gate and default-branch CI aligned and
allow verified fast-forward pushes.
- Post-push CI detects a broken commit after the branch moves. Run the local
gate before a direct push and monitor CI; require pre-merge checks when the
default branch cannot tolerate that detection window.
- Prefer squash merge and automatic branch deletion when the repository wants
a linear release-driving mainline; preserve intentional alternatives.
- Block force pushes and deletion on the default branch.
- Require conversation resolution only where pull requests are already the
delivery path; do not use it merely to force pull-request creation.
- Require signed commits when every protected-branch writer can satisfy the
rule. An unavailable or plan-gated API is an unconfirmed gap, not evidence
that no rule exists.
- Merge queue requires required workflows to handle `merge_group`.
- Protect release tag families. Document intentional mutable pointers such as
a marketplace major tag separately from immutable release tags.
Running a check does not enforce it. Require the smallest stable voting surface
that represents the repository's real gate. When matrices, conditional lanes,
or no-op paths make raw job names unstable, use one final `always()` gate that
fails closed on unexpected skips. Keep advisory dependency, release, deploy,
and report jobs non-blocking unless policy explicitly makes them voting.
Use non-strict required checks by default. Require up-to-date branches or merge
queue only when integration risk justifies the extra executions. Roll fleet
policy out to a small verified cohort before enabling an organization-wide
required context.
## Actions and Environments
- Default Actions permissions to read-only; widen per job.
- Allow only intended remote and local Actions. Require full-SHA pins when the
repository has an updater path for them.
- Fork pull requests remain read-only and receive no delivery secrets.
- Use `release` for publish credentials and environment-specific names for
running services. Keep deployment records enabled for service deploys.
- Environment branch policy constrains the workflow run ref, not an arbitrary
ref checked out later by a manual workflow.
## Security Surfaces
Public repositories whose `SECURITY.md` routes reporters to GitHub private
vulnerability reporting must have that setting enabled and read back. Private
repositories do not expose the same public reporting surface; route them to an
existing private maintainer channel.
For an organization that wants useful free defaults without per-active-committer
Advanced Security charges, suggest one enforced organization security
configuration with this baseline:
- apply it to all current repositories and make it the default for all new
repositories;
- do not allow repository owners to modify the configured features;
- disable the paid **Secret Protection** bundle;
- disable the paid **Code Security** bundle and legacy blanket
`advanced_security` enablement;
- disable CodeQL default setup;
- enable the dependency graph, Dependabot alerts, and Dependabot security
updates.
This is a billing-safe baseline, not a claim that every overlapping public-repo
security feature is off. GitHub may provide some secret scanning or other
security capabilities for public repositories without consuming a paid
license. Read back both the effective repository settings and Advanced Security
license usage instead of inferring them from the configuration label.
Treat repository visibility changes as billing-sensitive:
- Before and after a public-to-private transition, read back the attached
security configuration, effective paid features, active-committer license
usage, and projected billing.
- A feature that was free for a public repository can become billable when
that repository becomes private.
- If a repository genuinely needs Secret Protection, Code Security, or CodeQL,
use a separate narrow configuration or explicit repository opt-in with
accepted cost, owner, and trigger design; do not weaken the organization
baseline for the whole fleet.
Do not enable CodeQL default setup as a blanket baseline:
- Its pull-request and scheduled behavior is not a configurable
post-merge-only scan and can block merge state even when not required.
- Prefer the repository's deliberate `actionlint`, `zizmor`, secret scanning,
push protection, dependency alerts, and tested language-specific security
checks.
- Add CodeQL only when explicitly chosen for a repository with an accepted
cost and trigger design; do not make it a fleet-wide required context.
## Metadata and Readback
Keep description, homepage, topics, visibility, and community files aligned
with the repository's actual public surface. After each authorized change,
read back the effective rule or setting rather than trusting a successful write
response.
references/semantic-release.md
# Semantic Release
Use only after the repository has selected semantic-release and Conventional
Commits as its version contract. Inspect the repo-pinned tool and action
versions before changing configuration.
## Version Decision
- `feat` produces a minor, `fix` a patch, and an explicit breaking change a
major unless repository release rules say otherwise.
- Analyzer and notes generator must use the same preset and release rules.
- Squash-merge repositories should validate PR titles; direct-push paths should
validate commit subjects. Local hooks are feedback, not enforcement.
- Fetch full tag history in verification and release jobs.
Dry-run from the intended release branch before the first real publication and
inspect both the computed version and notes. Preserve the repository-owned
launcher and pinned versions rather than introducing `npx ...@latest`.
## Plugin Order
Order plugins by lifecycle:
1. analyze commits
2. generate notes
3. prepare versioned files or changelog
4. publish registries or assets
5. perform any signed source writeback
6. create or finalize the GitHub Release
A source-writeback plugin must run after every file-preparation plugin and
before the release tag is finalized. List writeback files explicitly. Do not
use a tree-limited API plugin for symlinks, executable files, deletions, or
generated trees it cannot faithfully preserve.
For protected branches, prefer the selected release tool's GitHub App-native
signed writeback. An App token used by ordinary `git commit` or `git push` does
not itself produce a verified signature. The App must also be allowed by the
effective branch rules.
## Concurrency and Branches
- Keep one non-cancellable release critical section per release branch.
- Do not assume Actions concurrency is an atomic branch lease. A tool that
writes against the live branch head must reject unexpected head movement or
run behind an external control that prevents it.
- Configure only branches and prerelease channels the repository actually
publishes.
- A release writeback should not recursively trigger verification or another
release; use the repository's established skip mechanism consistently.
## Publication and Recovery
Build and verify all assets before making an immutable GitHub Release public.
If the release tool creates the release early, keep it as a draft until assets
and downstream registries are ready.
Semantic-release is not a transaction across GitHub, registries, taps, or
deployments. After a partial failure, inspect tag, release, registry, default
branch, and signature state before retrying. A rerun may stop at an existing
tag and never invoke the failed publisher. Follow the durable-state recovery
rules in [release workflows](release-workflows.md#partial-failure-recovery).
Completion requires the peeled tag to resolve to the intended signed version
commit, the live default branch to contain it when source writeback is part of
the contract, every registry to expose the same version, and the GitHub Release
to have the intended immutable state and assets.
references/templates.md
# Collaboration Files
Use when introducing or aligning GitHub-facing templates and contributor
policy. First inspect a public `<owner>/.github` defaults repository: local
files override shared defaults, and shared files are not copied into clones or
release archives.
## Ownership
- Use owner-level defaults for policy true across every target repository.
- The pull-request template and `SECURITY.md` live in the owner defaults.
- Delete repository-local copies of them unless a repository genuinely
diverges.
- `CONTRIBUTING.md` stays repository-local because it carries environment
setup and repo-specific workflow.
- Licenses remain repository-local.
- Add a code of conduct only when an actual enforcement and contact owner
exists.
- Shared issue templates are risky because any repository-local issue
configuration disables the shared set.
Changing or creating a public defaults repository is a public policy change;
obtain authorization first.
## Pull Requests and Issues
A good owner default gives the body three headings that are the
problem-first flow itself, with guidance in comments (live example:
[uinaf/.github](https://github.com/uinaf/.github)):
```md
## Problem
<!-- as the requester stated it, not the mechanism -->
## Solution
<!-- a short problem-lead sentence, then labeled bullets; never a
paragraph wall. Name a risk only when there is a real one. -->
## Proof
<!-- only what CI cannot show: a screenshot, before/after numbers.
Delete this section when CI covers everything. -->
```
Prefer the shared owner default over repository-local copies; delete local
overrides unless the repository genuinely needs different fields. Title
the way the repository titles merged work, outcome over mechanism. No
implementation inventories, no ceremonial checklists, and no extra
headings that restate the diff. The comments are the verbosity control:
agents fill templates literally, so guidance written there is the one
place it reliably lands.
Create issue forms only when their fields improve triage. Common distinct
routes are bug, feature, and (only when supported) question. Vulnerabilities
always route to `SECURITY.md`, never a public issue form.
## Security and Contributing
`SECURITY.md` should say not to file public vulnerabilities, point to a private
route that works for the repository visibility, request affected surface,
impact, minimal reproduction, and mitigations, and avoid response promises the
maintainer cannot meet.
- Public repositories may use GitHub private vulnerability reporting only
after the setting is enabled and verified.
- Private repositories route to an existing private maintainer channel; do not
promise the public Security-tab workflow.
`CONTRIBUTING.md` owns setup, canonical validation, and branch/PR expectations.
Link deeper release or deploy runbooks instead of copying them into README,
templates, and agent guidance.
Repository descriptions and topics should help humans route the project using
its real purpose, artifact type, language or framework, and canonical public
URL. Never leak private client, organization, host, or adjacent-repo facts into
public metadata.
SKILL.md
---
name: gh-setup
description: "Set up or align a repository's GitHub collaboration and delivery surface: repo settings, branch or ruleset policy, templates, Dependabot or Renovate, Actions hardening, Environments, releases, publishing, and deploy workflows. Use for GitHub setup, CI/CD policy, protected delivery, package releases, or app deployment. Do not use for product architecture, provider infrastructure internals, application security review, or repository boot/readiness work."
disable-model-invocation: true
---
# GitHub Setup
Make GitHub the enforceable shell around the repository's existing build,
verification, release, and deployment contracts.
## Inspect and Classify
Before changing files or live settings:
1. Read repository guidance, manifests, verification commands, release or deploy
scripts, `.github/`, contributor/security docs, and any repository-owned
delivery runbook.
2. Read live GitHub state: default branch, merge methods, effective branch
rules, Actions policy, Environments, protected tags, security settings, and
every human or automated default-branch writer affected by the change.
3. Record the relevant before-state and rollback path.
4. Classify the delivery shape:
- **Versioned artifact:** read [release workflows](references/release-workflows.md)
and only the matching section of [publish targets](references/release-targets.md).
- **Running app or service:** read [deploy workflows](references/deploy-workflows.md),
then [Environments](references/deploy-environments.md) or
[credentials](references/deploy-secrets.md) when those boundaries change.
- **Both:** publish one immutable payload, then deploy that payload instead
of rebuilding it.
Use repo-local commands as authority. If the repository cannot reproducibly
build, verify, package, observe, or roll back the claimed surface, report that
prerequisite instead of hiding it in workflow YAML.
## Shared Contract
- Pull requests execute untrusted code with read-only credentials.
- Trusted release and deploy jobs load credentials only after verification and
input or ref validation.
- Workflow permissions default to read-only or `{}` and widen per job.
- High-trust remote Actions use reviewed immutable pins with an update path.
- Environment secrets and policy match the release or deployment blast radius.
- Release, publish, signing, promotion, and deploy critical sections are
non-cancellable and reconcilable.
- One verified payload crosses build, test, publish, and deploy boundaries.
- Required checks use a stable final result when matrices, conditional lanes,
or no-op paths make individual jobs unstable.
- A green workflow is not completion until live settings and downstream state
are read back.
Read [Actions security](references/actions-security.md) before workflows execute
project code, load secrets, publish, sign, or deploy.
## Runner Cost
Runner minutes are billed compute. Every trigger, runner size, and rerun is a
cost decision; default to the cheapest shape that still proves the contract.
- Read live repository visibility before choosing runners. Public repositories
use standard GitHub-hosted runners; private repositories may use the smallest
suitable Blacksmith runner. Preserve the required OS and architecture when
migrating (for example, Linux x64 Ubuntu 24.04 to `ubuntu-24.04`). Reusable
workflows choose from the caller's visibility, not the workflow owner's.
- Use Linux for portable checks. macOS and other large runners are reserved for
platform-bound jobs (native apps, Darwin-only APIs, Homebrew taps) and must be
gated behind path filters or restricted to `pull_request` +
`workflow_dispatch`. Runner changes preserve required proof, scan coverage,
triggers, permissions, and Environments.
- Provider requirements still apply to private repositories: npm trusted
publishing requires GitHub-hosted runners; use the [npm publish
contract](references/release-targets.md#npm).
- Secret and history scans trigger on `pull_request`, a weekly `schedule`, and
`workflow_dispatch` — never on `push`. The merge commit's tree was already
scanned in the pull request; the weekly cron covers history and new detector
rules. Call the shared `uinaf/.github/.github/workflows/scan.yml@main`
reusable workflow; do not copy scanner jobs or `docker build` scanner images
per run.
- Every verification workflow declares workflow-level concurrency:
`group: ${{ github.workflow }}-${{ github.ref }}`,
`cancel-in-progress: ${{ github.event_name == 'pull_request' }}`. Release,
publish, and deploy critical sections keep their own non-cancellable keys.
- A workflow triggered on both `push: [main]` and `pull_request` pays twice per
merged change. Keep push-to-main lanes for release/deploy work and for repos
whose policy allows direct pushes; do not add a push trigger to re-verify a
tree a required PR check already verified.
- Jitter cron minutes away from :00/:30; weekly is the default scan cadence.
- Expensive-per-run jobs (simulators, cross-compiles, e2e) sit behind
`dorny/paths-filter` lanes or `workflow_dispatch`, with an `always()` result
job when branch protection needs a stable check.
- Watch failure rates: a workflow that fails half its runs bills full minutes
for red. Fix or gate flaky jobs instead of rerunning them.
When implementing rather than only auditing, read [maintained
implementations](references/implementations.md) and start from the closest
tested shape. Reuse its contract, not its literal versions, identities, or
provider details.
## Repository Policy
Read [repository settings](references/repo-settings.md) for merge methods,
rulesets, required checks, signed commits, tags, Actions policy, Environments,
the cost-safe organization security baseline, CodeQL posture, and repository
metadata.
Preserve existing approval, actor, signed-commit, tag, and status-check rules
unless the requested change owns them. Running a check and enforcing it are
separate operations. Before requiring pull requests or a check, inventory
release bots, dependency bots, generated writebacks, and maintainers who still
write the default branch.
Do not require pull requests by default. When repository policy permits direct
updates and a reproducible local gate is mirrored by default-branch CI, allow
verified fast-forward pushes. Require pull requests only for pre-merge review,
untrusted contributions, merge queues, checks that must pass before the default
branch moves, or an explicit owner policy. Post-push CI detects regressions
after the branch moves, so run the local gate before pushing and monitor CI to
completion.
## Collaboration Files
Read [templates](references/templates.md) when adding or aligning pull-request
templates, issue forms, `SECURITY.md`, `CONTRIBUTING.md`, or shared community
defaults.
- Prefer public owner-level defaults only for policy true across every repo.
- Keep templates short and evidence-oriented; avoid checklist theater.
- Public security guidance needs a working private reporting route. Private
repos use an existing private maintainer channel.
- Read [dependency updates](references/dependency-updates.md) before adding
or migrating Dependabot or Renovate. Run one bot per repository; keep
security updates on under either.
## Release and Deploy Routes
Release work uses:
- [release workflows](references/release-workflows.md) for trust, publication,
signed writeback, immutable releases, recovery, and completion proof
- [publish targets](references/release-targets.md) for npm, Swift/CocoaPods, Go,
Rust, GitHub Actions, Homebrew, and monorepos
- [semantic-release](references/semantic-release.md) only when that tool is selected
- [release troubleshooting](references/release-troubleshooting.md) only after a
concrete failure or inconsistent durable state
Deploy work uses:
- [deploy workflows](references/deploy-workflows.md) for triggers, lane
detection, verified payloads, concurrency, and monitoring handoff
- [Environments](references/deploy-environments.md) when target selection,
protection, OIDC, or provider boundaries change
- [credentials](references/deploy-secrets.md) when secret ownership or logging changes
- [deploy troubleshooting](references/deploy-troubleshooting.md) only after a
concrete failure
## Verify and Finish
Run repository gates plus `actionlint` and `zizmor` when workflows changed.
Perform the narrowest safe live proof of the delivery contract. Dry-runs and
static inspection cannot prove immutable publication, signed writeback,
registry or tap parity, deployment, monitoring, or rollback.
After authorized live changes, read back every setting, Environment, rule,
release, registry, tag, deployment, or downstream pointer in scope. On partial
failure, reconcile durable state before retrying; never create a new version or
mutate an immutable release merely to make a workflow green.
## Output
```text
files: changed GitHub and documentation surfaces
settings: live changes and readback, or not checked
delivery: target and immutable payload boundary
evidence: local, workflow, and live proof actually exercised
risks: remaining authority, recovery, or downstream gaps
```