docker-development

어떤 스킬인가요?

Use when working with ANY Docker task: writing Dockerfiles, configuring docker-compose/compose.yml, multi-stage builds, docker-bake.hcl, container security audits, .dockerignore optimization, or CI/CD container testing. Triggers on: Dockerfile, docker-compose, container, image build, multi-stage, docker bake, compose.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

# Checkpoints for docker-development skill
# Evaluates Docker configuration best practices

version: 1
skill_id: docker-development

# Only run this skill for repos that actually use Docker. Without this gate
# the skill reports Dockerfile/compose errors against TYPO3 extensions,
# libraries, skill repos, and other non-containerised projects.
# Uses chained `||` (not `test -o`, which is obsolescent in POSIX) and matches
# the patterns the existing checkpoints look for.
preconditions:
  - type: command
    pattern: "test -f Dockerfile || test -f Containerfile || test -f docker-compose.yml || test -f docker-compose.yaml || test -f compose.yml || test -f compose.yaml || test -f docker-bake.hcl"

mechanical:
  # === DOCKERFILE/COMPOSE EXISTENCE ===
  - id: DC-01
    type: file_exists
    target: Dockerfile
    scope: application
    severity: error
    desc: "Dockerfile must exist"

  - id: DC-02
    type: file_exists
    target: docker-compose.yml
    scope: application
    severity: warning
    desc: "docker-compose.yml should exist for local development"

  - id: DC-03
    type: file_exists
    target: .dockerignore
    scope: application
    severity: warning
    desc: ".dockerignore should exist to exclude unnecessary files"

  # === MULTI-STAGE BUILD CHECKS ===
  - id: DC-04
    type: regex
    target: Dockerfile
    scope: application
    pattern: "FROM .+ AS .+"
    severity: warning
    desc: "Dockerfile should use multi-stage builds"

  - id: DC-05
    type: regex
    target: Dockerfile
    scope: application
    pattern: "COPY --from="
    severity: warning
    desc: "Dockerfile should copy artifacts from build stages"

  # === SECURITY: NON-ROOT USER ===
  - id: DC-06
    type: regex
    target: Dockerfile
    scope: application
    pattern: "(USER [^r]|useradd|adduser)"
    severity: warning
    desc: "Dockerfile should run as non-root user"

  - id: DC-07
    type: not_contains
    target: Dockerfile
    scope: application
    pattern: "USER root"
    severity: warning
    desc: "Dockerfile should not explicitly run as root"

  # === HEALTH CHECK ===
  - id: DC-08
    type: contains
    target: Dockerfile
    scope: application
    pattern: "HEALTHCHECK"
    severity: warning
    desc: "Dockerfile should define a HEALTHCHECK instruction"

  - id: DC-09
    type: regex
    target: docker-compose.yml
    scope: application
    pattern: "healthcheck:"
    severity: warning
    desc: "docker-compose.yml should define healthcheck for services"

  # === SECURITY: NO SECRETS IN DOCKERFILE ===
  - id: DC-10
    type: not_contains
    target: Dockerfile
    scope: application
    pattern: "ENV.*PASSWORD"
    severity: error
    desc: "Dockerfile must not contain hardcoded passwords in ENV"

  - id: DC-11
    type: not_contains
    target: Dockerfile
    scope: application
    pattern: "ENV.*SECRET"
    severity: error
    desc: "Dockerfile must not contain hardcoded secrets in ENV"

  - id: DC-12
    type: not_contains
    target: Dockerfile
    scope: application
    pattern: "ENV.*API_KEY"
    severity: error
    desc: "Dockerfile must not contain hardcoded API keys in ENV"

  - id: DC-13
    type: not_contains
    target: Dockerfile
    scope: application
    pattern: "ARG.*PASSWORD"
    severity: warning
    desc: "Dockerfile should not pass passwords as build args"

  # === BEST PRACTICES ===
  - id: DC-14
    type: contains
    target: .dockerignore
    scope: application
    pattern: ".git"
    severity: warning
    desc: ".dockerignore should exclude .git directory"

  - id: DC-15
    type: contains
    target: .dockerignore
    scope: application
    pattern: "node_modules"
    severity: info
    desc: ".dockerignore should exclude node_modules if applicable"

  - id: DC-16
    type: regex
    target: Dockerfile
    scope: application
    pattern: "LABEL.*maintainer|MAINTAINER"
    severity: info
    desc: "Dockerfile should have maintainer information"

  - id: DC-17
    type: not_contains
    target: Dockerfile
    scope: application
    pattern: ":latest"
    severity: warning
    desc: "Dockerfile should pin base image versions, not use :latest"

  # === .DOCKERIGNORE SECURITY ===
  - id: DC-18
    type: contains
    target: .dockerignore
    scope: application
    pattern: ".env"
    severity: warning
    desc: ".dockerignore should exclude .env files to prevent secret leakage"

  - id: DC-19
    type: command
    scope: application
    pattern: "test -f .dockerignore && grep -qE '\\*\\.pem|\\*\\.key' .dockerignore || true"
    severity: info
    desc: ".dockerignore should exclude private key files (*.pem, *.key)"

  # === COMPOSE HEALTH CHECK ORDERING ===
  - id: DC-24
    type: command
    scope: application
    pattern: "test -f docker-compose.yml && grep -q 'condition:' docker-compose.yml || test -f compose.yml && grep -q 'condition:' compose.yml || true"
    severity: info
    desc: "Compose depends_on should use condition: service_healthy for startup ordering"

  # === MODERN COMPOSE FILENAME ===
  - id: DC-25
    type: command
    scope: application
    pattern: "test -f docker-compose.yml || test -f compose.yml"
    severity: warning
    desc: "Docker Compose file should exist (docker-compose.yml or compose.yml)"

  # === LAYER OPTIMIZATION: CLEANUP IN RUN ===
  - id: DC-26
    type: command
    scope: application
    pattern: "! grep -q 'apt-get install' Dockerfile 2>/dev/null || grep -q 'rm -rf /var/lib/apt/lists' Dockerfile"
    severity: info
    desc: "Dockerfile should clean apt cache in the same RUN layer as install"

  # === INTERNAL NETWORKS ===
  - id: DC-27
    type: command
    scope: application
    pattern: "test -f docker-compose.yml && grep -q 'internal:' docker-compose.yml || test -f compose.yml && grep -q 'internal:' compose.yml || true"
    severity: info
    desc: "Compose should use internal networks for database isolation"

llm_reviews:
  # === SUBJECTIVE CHECKS (require LLM judgment) ===
  - id: DC-20
    domain: docker-security
    prompt: |
      Review the Dockerfile for security best practices:
      - Base image is from a trusted source (official images, verified publishers)
      - Minimal base image is used (alpine, distroless, slim variants)
      - No sensitive data or credentials are embedded
      - Proper permission handling (chmod, chown)
      - No unnecessary packages installed

      Report any security concerns found.
    severity: warning
    desc: "Review Dockerfile security configuration"

  - id: DC-21
    domain: docker-security
    prompt: |
      Check docker-compose.yml for security issues:
      - No secrets or passwords in plain text
      - Proper use of environment files or secrets management
      - Network isolation between services
      - No privileged containers unless absolutely necessary
      - No host volume mounts that could be exploited

      Report any security concerns found.
    severity: warning
    desc: "Review docker-compose.yml security configuration"

  - id: DC-22
    domain: docker-efficiency
    prompt: |
      Evaluate the Dockerfile for build efficiency:
      - Proper layer ordering (dependencies before application code)
      - Use of .dockerignore to minimize context
      - Combining RUN commands to reduce layers
      - Proper use of build cache
      - Cleanup of temporary files and package caches

      Report optimization opportunities.
    severity: info
    desc: "Review Dockerfile build efficiency"

  - id: DC-23
    domain: docker-development
    prompt: |
      Evaluate docker-compose.yml for development ergonomics:
      - Volume mounts for live code reloading
      - Environment variables for configuration
      - Proper service dependencies with depends_on
      - Reasonable default resource limits
      - Development-specific overrides available

      Report areas for improvement.
    severity: info
    desc: "Review docker-compose.yml development setup"

# Bind-Mount Ownership: Root-Owned Artifacts on the Host

## The Problem

Containers that run as root and write into a bind-mounted project directory
leave **root-owned files on the host**. Typical producers:

```bash
docker compose run --rm app npm install     # node_modules/ now root-owned
docker compose run --rm app composer install
docker compose run --rm app npm run build  # dist/, public/build/ root-owned
```

The host user then hits failures that look unrelated:

```
npm error EACCES: permission denied, rename '.../node_modules/@babel/code-frame' -> ...
rm: cannot remove 'node_modules/...': Permission denied
```

Host-side `npm install`, build-tool cleanup steps (e.g. webpack/Encore
`cleanupOutputBeforeBuild`), and even `git clean -fdx` fail on these files.

## Diagnosis

```bash
find node_modules public/build -maxdepth 2 -user root | head
```

Any hit means a containerized process wrote there as root.

It rarely announces itself that way, though. What you see first is the tool that
runs next, failing for reasons that read like the application's fault:

- a test suite red with `Permission denied` inside a library's file writer, one
  failure per test that writes output — an application bug, until you look at who
  owns the output directory
- `composer install` / `npm ci` aborting on "Could not delete …" for a path the
  host user never created

The common shape: the container run succeeded, and the *following* host command
is the one that fails. Suspect ownership before debugging the failure it reports.

## Cleanup (no sudo required)

Use a throwaway container — root inside the container can act on what root
created, and the mount scopes it to the project.

**Give the files back** when the container wrote something you want to keep, or
touched a tracked file. Deleting a `composer.lock` the container rewrote loses
the state; deleting a test's output directory only postpones the question:

```bash
docker run --rm -v "$PWD:/work" -w /work alpine \
  chown -R "$(id -u):$(id -g)" /work
```

**Delete** when the artifacts are disposable and gitignored:

```bash
docker run --rm -v "$PWD:/work" -w /work alpine \
  sh -c 'rm -rf node_modules public/build dist'
```

Then reinstall/rebuild as the host user. Either way, verify before trusting the
next run — `find . -not -user "$(id -un)"` should come back empty, and a tracked
file the container rewrote wants `git checkout --` on top of the `chown`.

## Prevention

| Approach | How |
|---|---|
| Run as the host user | `docker compose run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp app npm ci` — the arbitrary UID has no writable home in the container, and npm writes its cache to `$HOME`; point `HOME` (or `npm_config_cache`) at a writable path |
| Fix the UID in the image | `adduser -u 1000 ...` + `USER app` matching the typical host UID |
| Compose-wide | `user: "${UID:-1000}:${GID:-1000}"` on dev services — note `UID`/`GID` are **not** exported environment variables in most shells (bash's `UID` is shell-only); set them in the project `.env` file or `export UID GID` before composing |
| Keep artifacts out of the mount | named volume over `node_modules/`, or build inside the image (multi-stage) instead of into the mount |

Rootless Docker / userns-remap avoids the issue entirely but changes
semantics for the whole daemon.

## Related Gotcha: Named Volumes Mask Image Content

A named volume mounted over a path (e.g. `public/`) is populated from the
image **only on first use**. After deploying a new image, the volume still
holds the **old** content — refresh it explicitly (temp container +
`docker cp`/rsync) or recreate the volume as part of the deploy.

# Where a build credential actually leaks

`docker history` is the check everyone runs, and for a multi-stage build it is
the wrong one. A credential passed as `ARG` into a builder stage never reaches
the shipped layers — that stage is not in the final image — so the history is
clean while the credential is public anyway.

buildx attaches an SLSA provenance attestation to every pushed image and records
the build arguments in it **verbatim**:

```bash
docker buildx imagetools inspect <ref> --format '{{json .Provenance}}' \
  | jq -r '.[].SLSA.buildDefinition.externalParameters.request.args | keys[]'
# build-arg:COMPOSER_AUTH   <- the value sits right there, in cleartext
```

Measured case (2026-08-12): a public GHCR package carried a working GitLab
`glpat-` token in the attestation of **1461** published versions. The `ARG` was
in a `composer-builder` stage that is never shipped.

## Verify it, with a check that can fail

Grep for the **credential pattern**, never for the argument name. The
attestation embeds the push's commit messages and the `RUN` command lines, so
the name matches your own commit text and reports a leak that is not there —
that false positive cost a round of "still broken" in the session that found
this.

```bash
for ref in "$OLD_TAG" "$NEW_TAG"; do
  n=$(docker buildx imagetools inspect "$ref" --format '{{json .Provenance}}' \
      | grep -oE 'glpat-[A-Za-z0-9_-]{10,}|ghp_[A-Za-z0-9]{20,}' | sort -u | wc -l)
  echo "$ref: $n credential values"
done
```

A pre-fix tag returning `1` and a post-fix tag returning `0` is the proof. A
single post-fix `0` on its own proves nothing about the check.

## The fix

```dockerfile
RUN --mount=type=secret,id=composer_auth \
    COMPOSER_AUTH="$(cat /run/secrets/composer_auth 2>/dev/null || true)" \
    composer install --no-dev
```

```hcl
target "app" {
  secret = ["type=env,id=composer_auth,env=COMPOSER_AUTH"]
}
```

`type=env` reads the variable the CI already exports, so the calling workflow
usually needs no change at all. Tolerate a missing secret (`|| true`): forks and
local builds have no credential and should still resolve public dependencies
rather than fail.

## After a leak

Rotation is the only remedy that acts on what is already published — the
attestations of existing versions keep their copy of the value forever, or until
someone deletes those package versions. Rotate first, fix the build second.

## The other leak: your own debugging session

A leak does not need buildx provenance to happen. `docker build --progress=plain`
(or any verbose/raw build log) echoes the literal `RUN` command line with every
`ARG`/`ENV` already interpolated — so a credential-bearing build arg shows up in
cleartext the moment that output reaches a terminal, a piped log file, or a
`tail`. This is a transient leak (nothing gets published), but it lands directly
in whatever you're capturing the session with — chat transcript, screen
recording, CI job log — which is exactly the audience `--mount=type=secret` was
meant to keep it from. It happens even when the project's real Dockerfile is
already fixed with `--mount=type=secret`, because the leak comes from an ad-hoc
debug invocation outside that path, not from the build definition.

Treat it the same as a provenance leak once it happens: rotate the credential
first, then clean up (delete the log file, prune the build cache — the layer
with the interpolated value may still be cached locally even though it was
never pushed). Avoid it by not running debug builds with secret-bearing `ARG`s
through `--progress=plain`/verbose output that you then cat, tail, or pipe
into something you'll read — redact the known secret value first if you must.

# CI Testing Patterns for Docker Images

Deeper CI/CD gotchas beyond the basics. For entrypoint bypass, DNS mocking,
compose validation, and secret-scan exclusions, see SKILL.md's Quick
Reference § CI Testing Gotchas.

## Pattern 1: Worker/Sidecar Services That Reuse an App Image

A compose service that reuses the app image (e.g. a queue worker on the
php-fpm+nginx web image) **inherits the image's baked-in `HEALTHCHECK`**.
Three traps, in the order they typically bite in CI:

1. **Inherited check probes a daemon the worker doesn't run** (nginx, php-fpm)
   → the worker is permanently `unhealthy` and breaks
   `docker compose up -d --wait` — and anything else gating on health.
2. **`healthcheck: { disable: true }` is not a fix when `--wait` is used** —
   compose fails with `container ... has no healthcheck configured`
   (explicitly listed services without a check are un-waitable).
   Give the worker a real check instead.
3. **A naive `pgrep -f` check is *always* healthy** — the `CMD-SHELL`
   wrapper's own command line contains the search string, so `pgrep`
   matches the probe shell itself, even with a dead worker.

```yaml
services:
  worker:
    image: myapp:latest   # inherits the web image's HEALTHCHECK
    command: php bin/console messenger:consume async
    healthcheck:
      # WRONG: matches the probe's own shell -- healthy forever
      # test: ["CMD-SHELL", "pgrep -f 'messenger:consume' || exit 1"]
      # RIGHT: [c]haracter-class guard prevents self-match
      test: ["CMD-SHELL", "pgrep -f '[m]essenger:consume' || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
```

Verify any health probe **both ways**: process up → `healthy` AND process
killed → `unhealthy`. The naive `pgrep` pattern passes the positive test
and hides the bug.

Prefer letting a worker exit on its own limits (`exec` the daemon as PID 1,
restart policy with backoff) over in-container `while true ... || true`
loops that mask fatal errors from orchestration.

## Pattern 2: GitLab CI — image entrypoint must be a shell (or be overridden)

Unlike a test-time `--entrypoint` bypass, GitLab **runs every job's `script:` via `sh -c`**. If the image used as a job `image:` has a non-shell `ENTRYPOINT ["mytool"]`, the runner effectively runs `mytool sh -c '…'` → **`No such command 'sh'`**, and the job fails before the script runs.

```yaml
job:
  image:
    name: registry.example.com/mytool:1.0
    entrypoint: [""]   # let the runner's shell execute the script
  script:
    - mytool --help
```

A CLI image also meant for `docker run mytool …` can keep `ENTRYPOINT ["mytool"]`, but **document** that GitLab consumers must set `entrypoint: [""]`. If the image is *primarily* a CI image, prefer no tool entrypoint (use `CMD`).

## Pattern 3: Restricted runner egress — bundle external assets at build time

CI runners (especially internal/self-hosted) often have **no outbound internet**. An image that fetches something at *runtime* (`page.add_script_tag(url="https://cdn…/axe.min.js")`, `curl https://…` in the entrypoint, a remote `pip`/`npm` install) works locally but fails in CI.

Download the asset at **build time** and load it from the image. Use a multi-stage build so the fetch tooling (`curl`, `ca-certificates`) stays out of the final runtime image:

```dockerfile
# Stage 1: fetch external assets
FROM alpine:3.20 AS asset-builder
RUN apk add --no-cache curl
RUN mkdir -p /opt/axe-core \
 && curl -sSfL https://cdnjs.cloudflare.com/ajax/libs/axe-core/4.9.1/axe.min.js \
      -o /opt/axe-core/axe.min.js

# Stage 2: final image carries only the asset
FROM python:3.12-slim
COPY --from=asset-builder /opt/axe-core/axe.min.js /opt/axe-core/axe.min.js
ENV AXE_PATH=/opt/axe-core/axe.min.js
```

…and have the app prefer the local file (CDN as a dev-only fallback).

## Pattern 4: Test the *built image*, not just the editable dev install

A non-editable install in the image (`pip install .`, `npm install <tarball>`) does not behave like the editable/dev checkout your tests ran against. Classic failure: **data files resolved by walking from `__file__`** (`Path(__file__).resolve().parents[2]/"data"/…`) don't exist under `site-packages`, so the tool can't find its catalog/config inside the container even though `pytest` was green.

- Ship data files as **package data** (Python wheel `force-include`/`package_data`; npm `files`), not via filesystem-relative paths.
- Smoke-test the **built image**, not just the source tree:

```yaml
- run: docker build -t app:test .
- run: docker run --rm --entrypoint python app:test -c "import app; app.load_catalog()"
- run: docker run --rm app:test render fixture.json /tmp/out   # real command, end-to-end
```

## Pattern 5: Bake targets must inherit `docker-metadata-action`

### Problem

When a workflow migrates from `docker/build-push-action` to
`docker/bake-action`, the tags computed by `docker/metadata-action`
(semver from release tags, branch tags, `latest`) are **silently dropped**.
There is no CI error — the registry only ever updates the tags hardcoded in
`docker-bake.hcl`, so release and branch tags go stale or missing.

`docker/metadata-action` writes a generated bake definition exposing a
`docker-metadata-action` target that carries the computed `tags`/`labels`.
A bake target only picks them up if it explicitly inherits that target.

### Solution

Declare a stub `docker-metadata-action` target with local defaults and have
the real target inherit it. In CI, the metadata-action's generated bake file
replaces the stub; locally, the defaults apply.

```hcl
# docker-bake.hcl
target "docker-metadata-action" {
  tags = ["myapp:dev"]   # local default; replaced by metadata-action in CI
}

target "app" {
  inherits  = ["docker-metadata-action"]
  platforms = ["linux/amd64", "linux/arm64"]
  # Do NOT set `tags` here: a target's own attributes override inherited
  # ones, so a local `tags` would discard the CI-computed tags.
}
```

Re-add rolling tags such as `latest`/`production` through the
metadata-action config, not the bake file:

```yaml
- uses: docker/metadata-action@v5
  with:
    images: ghcr.io/org/myapp
    tags: |
      type=raw,value=latest,enable={{is_default_branch}}
```

### Verify Both Paths

```bash
# Local: stub defaults apply
docker buildx bake --print

# CI: simulate the generated metadata file replacing the stub
docker buildx bake -f docker-bake.hcl -f /tmp/metadata-bake.json --print
```

## Pattern 6: On-demand image tags — build on ONE trigger, and bake the version explicitly

A prod-like *variant* image (a profiler build, a debug build) is often published under a content-addressed tag like `:profiling-<sha>` so operators can switch to it on demand. Two traps appear when that image also surfaces its own build provenance (commit, ref, version) on a status page.

**Trap A — the tag race.** If the variant builds on *both* `push: main` and `push: tags`, both runs write the SAME `:profiling-<sha>` tag (same commit → same sha), and last-writer-wins decides which run's baked git-ref survives. A release deploy can then read `ref=main` instead of `ref=v1.2.3`. Fix: build the on-demand variant on ONE trigger that carries the right provenance — tags (plus manual dispatch), not `main`:

```yaml
- name: Build and push profiling image
  # Tag/dispatch only: a main-push build would race the tag build for :profiling-<sha>
  if: startsWith(github.ref, 'refs/tags/') || github.event_name == 'workflow_dispatch'
```

**Trap B — no version in a `.git`-less build.** A Docker build has no `.git`, so anything that derives the version from git or the package's own metadata reads a placeholder — e.g. Composer's `InstalledVersions::getPrettyVersion(<root-package>)` returns `1.0.0+no-version-set`. Bake the version in explicitly: pass a build arg before the dependency install (`COMPOSER_ROOT_VERSION=1.2.3`, or the language's equivalent) so the metadata records it, or have the app read a baked env (`APP_BUILD_REF`) that the Dockerfile declares and the workflow sets from `github.ref_name`:

```dockerfile
ARG APP_BUILD_REF
ENV APP_BUILD_REF=$APP_BUILD_REF
```

With Trap A fixed, that ref is deterministically the release tag.

## Local boot-test pitfalls

When smoke/boot-testing an image by hand (not in the CI matrix):

- **Host-port collisions mislead.** If the published port (`-p HOST:CONTAINER`) is already taken by another container, `docker run -d` leaves the new container unstarted (it stays in `Created` state; the CLI typically exits non-zero, often `125`) while your `curl localhost:HOST` is answered by the *other* container — a false pass, or a baffling failure. Use a free/unique host port, or skip `-p` and probe from inside: `docker exec <c> sh -c 'curl -sf localhost:<port>'`.
- **Foreground apps that log to a file leave `docker logs` empty.** E.g. Tomcat started with `-fg` writes to `logs/catalina.out`, not stdout — an empty `docker logs` does *not* mean "nothing happened". Read the in-container log files (`docker exec <c> sh -c 'tail -n 80 .../catalina.out'`), and check the process and state (`docker inspect -f '{{.State.Status}} {{.State.ExitCode}}' <c>`).
- **Minimal/distroless images have no shell.** `docker exec … sh`/`tail`/`pgrep` won't exist on `scratch`/distroless runtimes — probe with host-side `curl` against a published port, `docker inspect` for state, or a debug sidecar (`docker run --rm --pid container:<c> busybox …`).
- **Grep for real failure signals, not benign noise.** After a bundled-dependency swap, scan logs for `NoSuchMethodError|AbstractMethodError|LinkageError|IncompatibleClassChangeError` (binary incompatibility) — not bare `ClassNotFoundException`, which OSGi/plugin frameworks emit normally.

## Pattern 7: A build without `target:` builds whatever stage comes last

Not to be confused with Pattern 5: that `target` is a bake target inheriting
metadata, this one is the Dockerfile stage a build selects. Same word, different
thing, and reading one does not cover the other.

### Problem

A repository publishes one image from a multi-stage Dockerfile. The build step
names no target, because there was only ever one final stage:

```yaml
- uses: docker/build-push-action@…
  with:
    context: .
    tags: ${{ steps.meta.outputs.tags }}   # no target:
```

Later a second image is added — an nginx to front the php-fpm one — as a stage
appended after the existing runtime. Nothing in the original build changed, yet
it now publishes the *new* stage under the *old* name: Docker builds the last
stage in the file when no target is given.

The failure is silent at build time and loud much later. `php-fpm:latest` was an
nginx image for an hour; the first symptom was a sidecar dying on
`exec: "/bin/bash": no such file or directory`.

### Fix

Name the target explicitly in every build, including the one that was there
first:

```yaml
- uses: docker/build-push-action@…
  with:
    context: .
    target: runtime          # not "whatever is last"
```

Then assert what was actually built, in the job that builds it. A smoke test in
a separate job cannot help here: on a pull request the image never leaves the
build runner, so that job only runs on the default branch — after publishing.

```yaml
- name: The image is php-fpm, not the web stage
  if: github.event_name == 'pull_request'
  env:
    TAGS: ${{ steps.meta.outputs.tags }}
  run: |
    set -euo pipefail
    tag="$(printf '%s\n' "$TAGS" | head -n1)"
    docker run --rm --entrypoint sh "$tag" -c \
      'command -v php-fpm >/dev/null && ! command -v nginx >/dev/null'
```

Prove the assertion by running it against *both* images: each must accept its
own and reject the other. An assertion that only ever sees the correct image
has not been tested.

### Also check `.dockerignore`

A stage that copies from the build context needs that path allowed. A
deny-everything file is common and correct:

```
*
!rootfs/
!config/nginx/    # the web stage copies this
```

Adding a stage means revisiting it — otherwise the build fails on
`failed to compute cache key: "/config/nginx": not found`, which reads like a
missing file rather than an excluded one.

## Pattern 8: Verify shell semantics inside the target image, not on the host

### Problem

A CI script compares two scan results and blocks on what a build adds:

```sh
grep -Fxv -f deployed.txt built.txt > added.txt || true
if [ -s added.txt ]; then exit 1; fi
```

The reasoning was: an empty `deployed.txt` matches nothing, so `-v` prints every
line and all findings count as new — the safe direction. Verified on the
developer machine, where it holds.

The image is Alpine-based and ships **busybox grep**, which does the opposite: an
empty pattern file matches everything, `-v` prints nothing, `added.txt` comes out
empty and the gate passes. And the empty baseline is not an edge case — it is the
normal state whenever the reference scan is clean.

### Fix

Handle the case explicitly rather than relying on a semantic that differs
between implementations:

```sh
if [ ! -s deployed.txt ]; then
    cp built.txt added.txt
else
    grep -Fxv -f deployed.txt built.txt > added.txt || true
fi
```

### The general rule

Any assumption about `grep`, `sed`, `awk`, `sort` or `printf` behaviour that a
CI script depends on has to be checked in the image that will run it:

```sh
docker run --rm --entrypoint sh <the-ci-image> -c '<the exact expression>'
```

GNU coreutils on the host and busybox in an Alpine image disagree on more than
this one case. A local check that passes proves the host's semantics, not the
container's — and the difference surfaces as a gate that silently waves things
through, which is the direction nobody notices.

## Pattern 9: hadolint's floating `latest` is a feature — fix findings, don't pin

CI lint jobs typically run `hadolint/hadolint:latest-alpine`. A hadolint
release can turn a previously-quiet rule into a failure overnight, so an
unrelated MR (a Renovate version bump, a docs change) suddenly goes red.

Diagnose before blaming the diff: if the default branch's last green run
predates the failure and re-running it fails identically, the linter moved,
not your change.

Policy: treat the new finding as surfaced debt and fix it in the same MR —
do not pin the hadolint image. A scoped `.hadolint.yaml` ignore is reserved
for rules whose "fix" creates worse breakage — the canonical case is
DL3008/DL3018 (pinning distro packages breaks on the next mirror sync) — and
always carries a comment stating that rationale. Pinning the linter instead
hides every future rule improvement; the debt only grows.

Concrete case (2026-08-13, `docker/node-red`): a hadolint update started
failing `DL3066` (non-numeric user-id) on `USER root` / `USER node-red`.
Fix: use the numeric ids the Dockerfile already establishes —

```dockerfile
USER 0
RUN apk add --no-cache shadow && usermod -u 10458 node-red && apk del shadow
USER 10458
```

No behavior change, and the id survives environments that cannot resolve
container-internal user names. (The `apk add` here stays unpinned under the
DL3018 ignore above — that is the scoped exception in action, not a
contradiction of it.)

## Pattern 10: Validating a change to a custom base image needs the real image, not a stand-in

SKILL.md's "Mock upstream DNS" gotcha (`docker run --add-host backend:127.0.0.1
nginx-image nginx -t`) assumes `nginx-image` really is the image under test.
When the change under test lives in a *shared/base* image maintained
elsewhere — one that compiles in extra modules or ships modified system
files — substituting a generic public image of the same software (e.g.
`nginx:alpine` in place of a custom nginx+PHP-FPM base image) produces
unrelated, misleading errors instead of testing the actual change:

```
$ docker run --rm -v $PWD/rootfs/etc/nginx:/etc/nginx:ro nginx:alpine nginx -t
nginx: [emerg] getpwnam("www-data") failed          # base has no www-data user
...
nginx: [emerg] open() "/etc/nginx/mime.types" failed # base ships mime.types elsewhere
...
nginx: [emerg] unknown directive "brotli"            # real image compiles brotli in, nginx:alpine doesn't
```

None of these are about the change under test — they're artifacts of the
wrong base. Build or pull the actual image first, then run the check against
it:

```bash
docker build --build-arg PHP_VERSION=84 -t custom-image:test .
docker run --rm --add-host phpfpm:127.0.0.1 --entrypoint sh custom-image:test -c "nginx -t"
```

This costs one build (or pull, if the base is already published and registry
auth is already configured), not a rewrite of the stand-in's `/etc/nginx` to
patch in the missing pieces — that arms race never converges once the real
base changes again.

# Docker-in-Docker (DinD) Testing Patterns

Patterns for running Docker inside Docker in CI environments (Molecule, Testcontainers, nested builds).

## The Overlay-on-Overlay Problem

GitHub Actions runners (and most CI platforms) use the `overlay2` filesystem driver for Docker. When you run Docker inside Docker (e.g., Molecule testing Ansible roles, Testcontainers, nested builds), the inner Docker daemon also tries to use `overlay2`. The Linux kernel **cannot stack overlay-on-overlay** — this fails with:

```
mount source: "overlay", fstype: overlay, err: invalid argument
```

This affects any CI job that starts Docker containers from within a Docker container.

## Solution: VFS Storage Driver

Configure the **inner** Docker daemon to use the `vfs` storage driver instead of `overlay2`. VFS is slower (it copies full layers instead of using overlays) but works reliably inside containers.

### Molecule + geerlingguy.docker Role

```yaml
# molecule/default/prepare.yml
- name: Prepare
  hosts: all
  tasks:
    - name: Install Docker with VFS driver
      ansible.builtin.include_role:
        name: geerlingguy.docker
      vars:
        docker_daemon_options:
          storage-driver: vfs
```

### Direct daemon.json Configuration

```json
{
  "storage-driver": "vfs"
}
```

Write this to `/etc/docker/daemon.json` inside the container before starting Docker.

### GitHub Actions Service Container

```yaml
# .github/workflows/test.yml
jobs:
  test:
    runs-on: ubuntu-latest
    services:
      dind:
        image: docker:dind
        env:
          DOCKER_OPTS: "--storage-driver=vfs"
        options: --privileged
```

### GitLab CI

```yaml
test:
  image: docker:latest
  services:
    - name: docker:dind
      variables:
        DOCKER_OPTS: "--storage-driver=vfs"
  variables:
    DOCKER_HOST: tcp://docker:2376
```

## Systemd in Containers

When testing with containers that run systemd (e.g., Molecule testing on systemd-based OS images), additional configuration is required:

```yaml
# molecule/default/molecule.yml
platforms:
  - name: instance
    image: geerlingguy/docker-debian12-ansible:latest
    command: /lib/systemd/systemd
    privileged: true
    cgroupns_mode: host
    tmpfs:
      - /run
      - /run/lock
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:rw
```

### Why These Settings

| Setting | Purpose |
|---------|---------|
| `command: /lib/systemd/systemd` | Starts systemd as PID 1 |
| `privileged: true` | Grants access to host devices and cgroups |
| `cgroupns_mode: host` | Shares host cgroup namespace (required for cgroup v2) |
| `tmpfs: /run, /run/lock` | Provides writable tmpfs for systemd runtime state |
| `volumes: /sys/fs/cgroup` | Mounts cgroup filesystem read-write |

## Privileged Mode

### When It Is Needed

- **Docker-in-Docker**: The inner Docker daemon needs to create network namespaces, mount filesystems, and manage cgroups
- **Systemd containers**: systemd requires cgroup access and device control
- **iptables/networking**: Containers that modify firewall rules or create network bridges

### Security Implications

- `--privileged` disables all security confinements (AppArmor, seccomp, capabilities)
- The container can access **all host devices** and modify the host kernel
- In CI, this is generally acceptable because the runner is ephemeral
- In production, **never** use `--privileged` — use specific `--cap-add` flags instead

```yaml
# Production alternative: grant only needed capabilities
docker run --cap-add SYS_ADMIN --cap-add NET_ADMIN --security-opt apparmor=unconfined myimage
```

## Alternative Approaches

### Docker Socket Mounting

Mount the host's Docker socket instead of running a full inner daemon:

```yaml
# The container uses the HOST's Docker daemon
docker run -v /var/run/docker.sock:/var/run/docker.sock myimage
```

**Pros**: No overlay-on-overlay issue, faster, less resource usage
**Cons**: Containers share the host daemon — no isolation, cleanup is shared, security risk (container can control host Docker)

### Podman Rootless

Podman runs without a daemon and supports rootless nested containers:

```yaml
# GitHub Actions
- name: Test with Podman
  run: |
    podman run --rm --privileged \
      -v ./:/workspace:Z \
      quay.io/podman/stable \
      podman build /workspace
```

### Buildah for Image Builds

If you only need to build images (not run containers), Buildah avoids DinD entirely:

```yaml
- name: Build with Buildah
  run: |
    buildah bud -t myimage:test .
    buildah push myimage:test docker-daemon:myimage:test
```

## Complete CI Example: Molecule with DinD

```yaml
name: Ansible Role CI

on: [push, pull_request]

jobs:
  molecule:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        distro:
          - debian12
          - ubuntu2404
          - rockylinux9

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install molecule molecule-plugins[docker] ansible

      - name: Run Molecule
        run: molecule test
        env:
          MOLECULE_DISTRO: ${{ matrix.distro }}
```

With the corresponding Molecule prepare step using VFS:

```yaml
# molecule/default/prepare.yml
- name: Prepare
  hosts: all
  tasks:
    - name: Install Docker with VFS driver
      ansible.builtin.include_role:
        name: geerlingguy.docker
      vars:
        docker_daemon_options:
          storage-driver: vfs
```

## Local test runs and disk space

Running container-based test suites locally — Molecule with `geerlingguy` systemd images, repeated nested builds, or rebuilding a CI image between iterations — creates a **fresh image and overlay layer per converge or rebuild**. These accumulate fast and can fill the host disk within a handful of runs. When the disk fills, even diagnostic commands fail to write their output, so you lose the very information needed to recover — prevention beats recovery.

**Before launching** a container test suite, check free space on the Docker storage filesystem:

```bash
df -h $(docker info -f '{{.DockerRootDir}}')
```

**After each run**, prune your own run's artifacts immediately — do not defer cleanup to the end of the session, by which point the disk may already be full:

```bash
docker container prune -f && docker image prune -f && docker builder prune -f
```

Pruning per-run keeps the working set small and surfaces a real disk problem early, while diagnostic commands can still write output.

## Troubleshooting

| Error | Cause | Fix |
|-------|-------|-----|
| `mount: overlay: invalid argument` | Overlay-on-overlay | Set `storage-driver: vfs` |
| `Cannot connect to Docker daemon` | Docker not started in container | Ensure `--privileged` and daemon is running |
| `failed to create shim task` | Missing cgroup access | Add `cgroupns_mode: host` and cgroup volume |
| `System has not been booted with systemd` | systemd not PID 1 | Set `command: /lib/systemd/systemd` |
| `OCI runtime error: container_linux.go` | Insufficient permissions | Add `--privileged` or specific `--cap-add` |

# GPG Signature Verification in Image Builds

Patterns for verifying downloaded release tarballs against GPG keys inside
multi-stage builds — and the layer pitfall that breaks the naive approach.
Distilled from building a central release-key image for PHP/nginx/Node builds.

## Pitfall: gpg import bakes a stale keybox lock into the layer

With gnupg 2.4, `gpg --import` in one `RUN` starts keyboxd/gpg-agent and leaves
`public-keys.d/*.lock` behind in the GnuPG home **inside the committed layer**
(`/root/.gnupg` when building as root, else `$GNUPGHOME`/`~/.gnupg`). The
next `RUN` that touches the keyring (e.g. `gpg --verify`) then hangs on the
stale lock and dies:

```text
gpg: Note: database_open ... waiting for lock (held by 9) ...
gpg: keydb_search failed: Operation timed out
gpg: Can't check signature: No public key
```

If you must import, clean up in the **same** `RUN` that imported:

```dockerfile
RUN gpg --no-tty --batch --import /tmp/keys.asc \
 && gpgconf --kill all \
 && rm -f "$(gpgconf --list-dirs homedir)"/public-keys.d/*.lock
```

## Prefer gpgv: verify without any keyring state

`gpgv` reads **binary** keyring files directly — no import, no `~/.gnupg`,
no agent, no locks, and the accepted signer set is exactly the key files you
pass:

```dockerfile
COPY --from=release-keys /keys/php/8.3/ /tmp/gpg-keys/
RUN set -eux; \
    for k in /tmp/gpg-keys/*.gpg; do \
        [ -s "$k" ] || exit 1; \
        set -- "$@" --keyring "$k"; \
    done; \
    gpgv "$@" php.tar.xz.asc php.tar.xz
```

Convert armored keys once with `gpg --dearmor` (or ship binary exports);
`--keyring` may be repeated per key file.

### `gpgv` is not always in the `gnupg` package

Install `gpgv` explicitly for the base you build on:

- **Debian/Ubuntu**: `gpgv` is a **separate package** — `apt-get install gnupg`
  does NOT provide it. A job that verifies with `gpgv` must
  `apt-get install ... gpgv`, or every call dies with a silent
  `gpgv: command not found` (which inside `if gpgv ...; then` reads as a
  verification *failure*, not a missing binary).
- **Alpine**: the `gnupg` package *does* bundle `gpgv`, so `apk add gnupg` is enough.

Testing on your host (which usually has `gpgv`) hides the Debian gap — run the
verification inside the actual build image before trusting it.

## A signature proves the signer, not the version

`gpgv` (and `gpg --verify`) proves *"signed by a trusted release key"* — **not**
*"is the version I asked for"*. The signature travels with the tarball, so a
**validly signed older release** placed under a newer name passes verification
unchanged. This is harmless when you download straight from the authoritative
origin (it owns the path→content binding), but the moment a **mutable store**
sits in front of the origin — a package-registry cache, a mirror, an artifact
proxy — an attacker who can write that store can serve a real, signed,
*known-vulnerable* `foo-1.2.3.tar.gz` under the `1.2.9` coordinate. It passes
`gpgv`, and the build silently compiles the downgraded version.

Bind the artifact to the requested version after the signature checks out —
assert the tarball's sole top-level directory (or the checksum-file entry)
matches the expected `name-<version>`:

```dockerfile
RUN set -eux; \
    gpgv --keyring /tmp/k.gpg php.tar.xz.asc php.tar.xz; \
    got=$(tar tf php.tar.xz | sed 's#/.*##' | sort -u); \
    [ "$got" = "php-${PHP_VERSION}" ] || { echo "version mismatch: $got" >&2; exit 1; }
```

Checksum-list formats (e.g. Node's `SHASUMS256.txt`) get this for free — the
artifact is looked up by exact filename inside the signed list, so a wrong
version yields no matching line. Detached-signature formats (php/nginx) do not —
add the assertion yourself. It also hardens any later `tar --strip-components=1`
that assumes a single predictable top-level directory.

## Ship keys as a scratch image, not from keyservers

Public keyservers (`keyserver.ubuntu.com`, `keys.openpgp.org`) time out under
parallel CI fan-out and break scheduled builds. Keep reviewed public keys in a
minimal image and consume them via `COPY --from`:

```dockerfile
FROM registry.example.com/support/gpg-keys:latest@sha256:<digest> AS release-keys
```

- Final stage `FROM scratch`: nothing to patch or trust beyond the key files.
- Tag every published digest immutably (e.g. commit SHA alongside `latest`) so
  digest pins never reference an untagged manifest.
- Key *material* belongs to authoritative origins (e.g.
  `php.net/distributions/php-keyring.gpg`, `nginx.org/keys/*.key`), never to a
  keyserver fetch at build time.

A robust shape: a central keys image that also ships `verify-release` /
`get-verified-release` helper scripts *in* the image (POSIX sh, executed by the
consumer's shell; a `scratch` image runs nothing itself).

# Multi-stage caching: keep code-independent installs off the code-copy lineage

**Symptom:** a dev/CI image rebuilds heavy tooling (apt, pecl/xdebug, browser installs, `npm ci`) on **every** source change, even though none of it depends on the code.

**Cause:** the tooling stage is `FROM <the code stage>`, and the code stage ends with `COPY . .`. Docker invalidates every layer downstream of that copy on any tracked-file content change — so the tooling, layered on top, re-runs each time.

**Fix — re-parent, don't reorder.** Put the code-independent installs in a *sibling* stage `FROM base` (a stage with **no** code copy), then pull the built tree into the leaf via one `COPY --from=<code-stage>`:

```dockerfile
# composer/npm install + COPY . . + build (code-dependent)
FROM base AS deps
# apt, xdebug, symfony-cli, npm ci, chromium — NO code copy
FROM base AS devtools
FROM devtools AS dev
# the ONE code-dependent layer of dev
COPY --from=deps --chown=app:app /app /app
```

A source-only edit now invalidates `deps` (and dev's copy) but leaves the whole `devtools` lineage CACHED.

## Guardrails

- **Isolation by lineage:** keep xdebug/chromium in the `devtools → dev → e2e` branch only. A `production`/`profiling`/`tools` stage that is `FROM base`/`FROM deps` must not inherit `devtools`, or it gains xdebug (skews profiling timings) and browser bloat. Verify with the actual `FROM` chain, not by hoping.
- **Same-layer cleanup:** a transient `node_modules` needed only to run `npx playwright install` should be `rm -rf`'d in the *same* `RUN` (the leaf's `COPY --from=deps` overwrites it anyway) so it never bloats the layer. The browser binary lives in `~/.cache/ms-playwright`, outside `node_modules`, so it survives.
- **Verify the cache, not just the build:** `docker build --check` + a cold build prove it *builds*; only a **content** change to a source file + rebuild proves the *caching* — BuildKit hashes content, not mtime, so a bare `touch` won't invalidate. Look for `CACHED` on the tooling layers.

# nginx FastCGI keepalive starves php-fpm

A php-fpm child stays bound to its FastCGI connection for as long as that
connection lives. An nginx keepalive pool to php-fpm therefore does not park
idle sockets — it parks **workers**. Size the pool at or above
`pm.max_children` and it can pin every child, leaving arriving requests to wait
for one to come free.

```nginx
upstream php-fpm {
    server 127.0.0.1:9000;
    keepalive 16;          # against pm.max_children = 10
}
location ~ \.php$ {
    fastcgi_keep_conn on;  # <- makes the pool real
}
```

## The signature — all of it at once, or it is something else

- container at **0 % CPU**, memory flat — nothing is working
- php-fpm **slowlog empty** at a low threshold
  (`request_slowlog_timeout = 5s`): the scripts were never slow, they never got
  a worker
- the stalled requests answer **HTTP 200 after ~60 s**, with TTFB equal to the
  total, and the edge proxy's access log reports the same duration, so the wait
  is behind it
- the very same URLs replayed **one at a time** in the same session take
  ~100 ms

The empty slowlog is the decisive one: it separates "slow code" from "never
scheduled", and it is the measurement most likely to be skipped.

## Why it hides from local testing

Only a client that can issue everything at once builds the queue. Over HTTP/2
through a reverse proxy the page's requests share one connection and arrive
together; a direct HTTP/1.1 client opens at most six connections per origin and
never triggers it. Reproduce **with the proxy in the path** — a stack tested by
addressing the application container directly measures a load shape that
production never sees.

## Measured

TYPO3 backend behind Caddy, `pm.max_children = 10`, opening the backend shell:

| nginx | slowest request | over 5 s |
|---|---|---|
| `keepalive 16` + `fastcgi_keep_conn on` | 61005 ms | 5–8 of 24 |
| `keepalive 8` + `fastcgi_keep_conn on` | 19614 ms | 1 of 24 |
| no pool, `fastcgi_keep_conn off` | **107 ms** | none |

The intermediate value still costs 19.6 s, so this is not a tuning question:
remove the pool. `fastcgi_keep_conn off` is the nginx default, and connection
setup to `127.0.0.1` is not worth holding a worker for.

Raising `pm.max_children` instead trades memory for the same failure one load
step further out — and where the container has a memory cap, the headroom for
`keepalive`-many concurrently pinned children is not there to give.

# Registry Catalogue Probing and Pin Rot

Two questions that look settled and are not: *does this registry publish image
X?* and *is a digest pin still the careful choice?*

## A 401 is a transport answer, not an absence

Authenticated registries refuse anonymous requests for **every** repository, so
an anonymous probe cannot distinguish "not published" from "not logged in":

```bash
curl -s -o /dev/null -w '%{http_code}\n' https://dhi.io/v2/mariadb/tags/list      # 401
curl -s -o /dev/null -w '%{http_code}\n' https://dhi.io/v2/phpmyadmin/tags/list   # 401
```

Both 401. Neither says anything about the catalogue. Ask the registry which
realm it wants, then authenticate against it — the credential is already on the
machine if `docker login` has run:

```bash
curl -sI https://dhi.io/v2/mariadb/tags/list | grep -i www-authenticate
# Bearer realm="https://dhi.io/token",service="registry.docker.io",scope="repository:mariadb:pull"

AUTH=$(jq -r '.auths["dhi.io"].auth' ~/.docker/config.json)      # never echo this
tok=$(curl -s -H "Authorization: Basic $AUTH" \
  "https://dhi.io/token?service=registry.docker.io&scope=repository:mariadb:pull" \
  | jq -r '.token')
curl -s -o /dev/null -w '%{http_code}\n' \
  -H "Authorization: Bearer $tok" https://dhi.io/v2/mariadb/tags/list            # 200
```

Now 200 versus 404 discriminates, and the probe has demonstrated it can return
both — which is what makes a 404 evidence. Run the positive control (an image
you know exists) in the same loop as the question you are actually asking; a
run that returns 404 for everything is measuring your credentials.

`.auths[…].auth` is base64 `user:token`, so treat the value as the secret it
is: pass it through a variable, never into displayed output. With a
`credsStore` configured there is no `auth` field — read the credential from the
helper instead.

## A digest pin can rot into the worse choice

Pinning a third-party image to a digest freezes the application *and* the base
underneath it. That is the point while upstream is publishing; it inverts the
moment upstream resumes rebuilding the same release. Measured on one image
during a single afternoon, counting only findings that have a fix:

| reference | with a fix | CRITICAL+HIGH |
|---|---:|---:|
| digest pinned ten months earlier | 1461 | 258 |
| the floating tag it was pinned from | 293 | 59 |

Same application version in both. The tag moved with the rebuild; the pin did
not. Before writing "pinned, therefore safe" — or the opposite, "upstream never
rebuilds, so there is no update path" — read the tag's timestamp:

```bash
curl -s "https://hub.docker.com/v2/repositories/library/<image>/tags?page_size=10&ordering=last_updated" \
  | jq -r '.results[] | "\(.name)\t\(.last_updated)\t\(.digest[0:19])"'
```

Equal digests across `:latest` and `:X.Y.Z` mean the release tag *is* latest, so
waiting for an upstream rebuild is not a plan. A moved `last_updated` means any
claim resting on staleness has expired and needs re-measuring, not repeating.

## Floating tags are correct for images you rebuild

The pinning rule is about trust, not about syntax: an image your own CI rebuilds
daily should float, because a pin pins the fix out too. Two failure modes when
that exemption is written down:

- **Scoping it to a registry host.** Ownership is what earns the exemption, not
  which host serves the bytes. An organisation publishing to both a private
  registry and `ghcr.io/<org>` needs both recognised — compared as path
  components, never as substrings, or `ghcr.io/someone/<org>-lookalike` and
  `evil.com/<your-registry>/x` qualify.
- **Testing the tag against the literal string `latest`.** `latest-rolling`,
  `latest-alpine` and friends float exactly as much and sail through. Treat a
  `latest-*` or `edge-*` prefix as floating.

## A tag that "should" exist can just not

Don't assume a variant tag exists because the pattern is common elsewhere —
check the catalogue the same way as above. `composer:2-alpine` is not a real
tag: the official `composer` image is Alpine-based *at* `composer:2`, so the
`-alpine` suffix some other images use has nothing to pull. A short,
unqualified reference like `composer:2` in a `.env`/build-arg also fails
differently depending on where it resolves: buildah/podman running
non-interactively enforce short-name resolution and refuse to guess a
registry, erroring with "short-name resolution enforced but cannot prompt
without a TTY" instead of defaulting to Docker Hub. Fully qualify it
(`docker.io/library/composer:2`) rather than relying on the short name to
resolve the same way it does interactively.

## A push failing "unauthorized" is a role question, not always a credential one

`unauthorized to access repository: <repo>, action: push` from a private
registry (Harbor and similar) usually reads like a login problem, so the
instinct is to re-check the password and rebuild. Ask the registry's own RBAC
API first instead of spending a full build+push cycle per guess — Harbor
answers this in two read-only calls:

```bash
curl -s -u "$USER:$TOKEN" https://harbor.example.com/api/v2.0/users/current \
  | jq '.username'
curl -s -u "$USER:$TOKEN" "https://harbor.example.com/api/v2.0/projects/<id>/members" \
  | jq '.[] | select(.entity_name=="'"$USER"'")'
```

A valid login with no membership entry (or `role_id: null`) for the target
project means the account has zero role there — no amount of retrying the
push fixes that; someone with project-admin/Maintainer needs to grant a role
(Developer is enough to push).

---
name: docker-development
description: "Use when working with ANY Docker task: writing Dockerfiles, configuring docker-compose/compose.yml, multi-stage builds, docker-bake.hcl, container security audits, .dockerignore optimization, or CI/CD container testing. Triggers on: Dockerfile, docker-compose, container, image build, multi-stage, docker bake, compose."
license: "(MIT AND CC-BY-SA-4.0)"
compatibility: "Requires docker, docker compose."
metadata:
  version: "1.15.1"
  repository: "https://github.com/netresearch/docker-development-skill"
  author: "Netresearch DTT GmbH"
allowed-tools:
  - "Bash(docker:*)"
  - "Bash(grep:*)"
  - "Read"
  - "Write"
  - "Glob"
  - "Grep"
---

# Docker Development

## Core Principles

1. **Minimal** -- Alpine/distroless, multi-stage
2. **Secure** -- Non-root USER, no layer secrets, pin versions
3. **Testable** -- entrypoint bypass, DNS mocking
4. **Cache-efficient** -- deps first, clean in-layer

## Quick Reference

### Multi-Stage Build (Node.js)

```dockerfile
FROM node:24-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .

FROM node:24-alpine
RUN addgroup -g 1001 app && adduser -u 1001 -G app -D app
USER app
COPY --from=builder /app .
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget -qO- http://localhost:3000/health || exit 1
CMD ["node", "server.js"]
```

### Multi-Stage Build (Go -- scratch/distroless)

```dockerfile
FROM golang:1.26-alpine AS builder
WORKDIR /app
COPY go.* ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server .

FROM gcr.io/distroless/static:nonroot
COPY --from=builder /app/server /server
CMD ["/server"]
```

### Layer Optimization

```dockerfile
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*
```

### Build Cache: Copy Dependency Files First

```dockerfile
COPY package*.json ./
RUN npm ci
COPY . .
```

Manifests before source keeps install layers cached.

### BuildKit Secrets

```dockerfile
RUN --mount=type=secret,id=ssh_key,dst=/root/.ssh/id_rsa git clone git@github.com:org/repo.git
```

`ARG` leaks via `docker history` **and** SLSA provenance --
`references/build-secret-leaks.md`

### Docker Bake (Multi-Platform)

```hcl
target "app" {
  platforms = ["linux/amd64", "linux/arm64"]
  cache-from = ["type=gha"]
  cache-to = ["type=gha,mode=max"]
}
```

## Security Anti-Patterns

| Anti-pattern | Fix |
|---|---|
| `FROM image:latest` | Pin version: `image:1.2.3-alpine` |
| No `USER` directive | `adduser` + `USER appuser` |
| `chmod 777` | Use specific permissions: `chmod 550` |
| `privileged: true` in compose | Remove or use specific `cap_add` |
| `volumes: [/:/host]` | Mount only needed paths |
| `ports: ["0.0.0.0:3000:3000"]` | Bind to `127.0.0.1:3000:3000` |
| `ENV DB_PASSWORD=secret` | Use `--mount=type=secret` or compose secrets |

## CI Testing Gotchas

1. **Bypass entrypoint**: `docker run --rm --entrypoint php myimage -v`
2. **Mock upstream DNS**: `docker run --rm --add-host backend:127.0.0.1 nginx-image nginx -t`
3. **Compose validation**: `cp .env.example .env` before `docker compose config`
4. **Secret scanning**: exclude `.env.example`, README, docs
5. **Root-owned artifacts**: bind-mount dirs (`EACCES`) -- `references/bind-mount-ownership.md`

## .dockerignore

Exclude: `.git`, `node_modules`/`vendor`, `.env*`, `*.pem`, `*.key`

## Compose Essentials

- startup ordering: `depends_on.condition: service_healthy` + `healthcheck` `start_period`
- `networks.internal: true` isolates databases
- `profiles: [debug]`: start only with `--profile debug`
- shared image ref: define ONCE per file as top-level extension field + anchor -- `x-app-image: &app-image registry/app:${APP_IMAGE_VERSION:-85}`, services use `image: *app-image`. The field must sit ABOVE `services:` — an alias is only valid after its anchor in document order. `:-` defaults cover unset AND empty vars (a bare omitted tag silently resolves `:latest`). Anchors are file-local: every overlay file needs its own. Verify both paths: `APP_IMAGE_VERSION= docker compose config` and with an override

## References

- `references/ci-testing.md` -- CI testing patterns for Docker images
- `references/dind-testing-patterns.md` -- Docker-in-Docker testing patterns
- `references/bind-mount-ownership.md` -- root-owned bind-mount artifacts
- `references/gpg-verification.md` -- gpgv patterns; stale keybox locks
- `references/registry-catalogue-and-pin-rot.md` -- catalogue probes; pin rot
- `references/build-secret-leaks.md` -- `ARG` in provenance
- `references/php-fpm-worker-starvation.md` -- keepalive pins php-fpm