references/height-processing.md
# Height processing and normal derivation
Contents: [Pipeline](#pipeline) · [Height sources](#height-sources) · [Depth shaping](#depth-shaping) · [Common shaping](#common-shaping) · [Normal derivation](#normal-derivation) · [Conventions](#conventions) · [Validation report](#validation-report) · [Runtime parity](#runtime-parity)
Both scripts run the same chain. Values are floats in 0..1 until the height is quantised to 8 bits; the normal is derived from that quantised height so the written `_height.png` is exactly what produced `_normal.png`.
## Pipeline
```
albedo RGBA
└─ height source ──► luminance | Depth Anything V2 | height file
└─ depth shaping (flatten, high-pass, invert, normalize) [defaults on for depth]
└─ seam heal (offset) [optional]
└─ soften (density-scaled gaussian) [optional]
└─ levels (black, white, gamma) [optional]
└─ quantise to 8 bit ──► <name>_height.png
└─ 3x3 edge kernel ──► <name>_normal.png + report
└─ MER inference (albedo + height + normal curvature) ──► <name>_mer.png + merReport
```
MER inference is documented in [mer-inference.md](mer-inference.md); this file covers the height and normal stages.
## Height sources
**Luminance** (`--height-source luminance`, default). BT.601 luma (`0.299 R + 0.587 G + 0.114 B`) or the channel average (`--luma average`) of the albedo *as stored*, without sRGB linearisation: on painted and pixel-art textures the luminance steps are the relief the artist drew, and linearising crushes the darks. Light reads as high. Use `--invert-height` when the painting convention is the opposite (dark grout that should be recessed already reads low; dark paint meant to be raised needs the flag). Luminance mistakes painted shadows for pits and highlights for bumps; that is acceptable for stylised art and wrong for photographic textures.
**Depth Anything V2** (`--height-source depth`). A monocular depth model; output is relative inverse depth, 1 = nearest. Use it on textures with real structure the model can read as a photo: bricks, planks, cobble, foliage, machinery. It has nothing to say about a flat two-tone checker (it returns a radial vignette). Small textures are lifted first with a nearest-neighbour integer upscale so the model's ~518 px bicubic resize does not smear pixel edges:
```
factor = clamp(ceil(1024 / min(w, h)), 1, floor(4096 / max(w, h)))
```
The prediction (at the upscaled size) is block-averaged back by the same factor (an exact inverse of nearest-neighbour) and lightly unsharpened (`x + 0.5 · (x − box3(x))`, edge-clamped). Every source texel maps to an exact k×k block, so the round trip is alignment-perfect.
**File** (`--height-source file --height-file h.png`). Any grayscale PNG at the albedo's size (red channel is read). Use it for a height painted in Blockbench, produced by another tool, or a previously generated `_height.png` you want to re-shape. Depth shaping is off by default for this source; `--invert-height` flips it.
## Depth shaping
Applied after the source, in this order. Defaults: `--flatten 1`, `--high-pass 0`, no invert, normalize on for the depth source; all off for luminance and file unless requested.
1. **Flatten** (`--flatten 0..1`): least-squares plane fit over centred pixel coordinates, subtracted (mean preserving). Monocular depth almost always carries a foreground-to-background ramp across the frame; on a tile that puts one border high and the opposite one low. The plane removes the linear part.
2. **High-pass** (`--high-pass R` texels): subtract a wrap-around box blur ×3 (≈ Gaussian) of radius R, recentred on 0.5. Keeps relief smaller than R and kills the non-linear leftovers the plane cannot express. Because the blur wraps, the subtraction cannot introduce a seam. Start around a quarter of the texture width for tiles; smaller values keep only fine grain.
3. **Invert** (`--invert-height`): far reads as high, for sources where the receding surface is the raised one.
4. **Normalize**: min–max stretch back to 0..1, because the edits shrink the range. `--no-normalize` keeps absolute values (useful when several textures must share a relief scale).
## Common shaping
**Seam heal** (`--seam offset --seam-amount 0..0.5`). Wrap-and-heal for tiling textures: shift the field by half a tile (the borders become interior, hence wrapping), then blend the unshifted field back over the seam cross in the centre with a smoothstep ramp. The blend weight is height-steered (the taller surface wins the middle of the transition, windowed by `4t(1−t)` so the borders stay exactly pure), which keeps a raised feature on top instead of ghosting it into the surface behind. Amount is the healed margin as a fraction of each axis.
**Soften** (`--soften MULT`). Gaussian blur with `sigma = min(shortSide / 16 / 4 · MULT, 8)` texels, wrapped when `--wrap` is set, clamped otherwise. The sigma scales with texel density so the blur is a constant fraction of a block face: HD relief (128×, 256×) loses the one-texel stair-steps that make lit relief look faceted, while 16× art (sigma 0.25) is effectively untouched. Leave at 0 for pixel art.
**Levels** (`--levels BLACK,WHITE,GAMMA`). Input black/white points then gamma (`((v − black) / (white − black)) ^ (1 / gamma)`). Use it to clip the flat body of a texture to one plateau (raise black), compress relief (lower white), or bias the midtones before the normal kernel.
## Normal derivation
3×3 edge kernel over the quantised height, per texel:
| Operator | corner | edge | Character |
|---|---|---|---|
| sobel (default) | 1 | 2 | balanced; standard for pixel art |
| prewitt | 1 | 1 | softer, less directional emphasis |
| scharr | 3 | 10 | best rotational symmetry; smooth HD height |
```
dx = -c·s00 + c·s02 - e·s10 + e·s12 - c·s20 + c·s22 (rises to the right → positive)
dy = c·s00 + e·s01 + c·s02 - c·s20 - e·s21 - c·s22 (rises upward → positive)
n = normalize(-dx · strength, ±dy · strength, 1)
rgb = n · 0.5 + 0.5
```
Strength multiplies the gradients with z fixed at 1, so the dial reads as labelled (higher = stronger relief). The kernels are unnormalised, so a one-texel step of Δh gives `dx = 4Δh` under Sobel. Typical values: 0.25 subtle, 0.5 default, 1–2 pronounced pixel-art relief; halve it for every doubling of texture density if the relief should look the same in block space.
Sampling: `--wrap` samples across the borders (seamless tiles), otherwise taps are clamped to the edge (atlases, non-tiling art). A neighbour whose albedo alpha is below `--alpha-threshold` (0–255, default 8) is off-island and replaced by the centre height, so relief never bleeds across transparent UV gutters; off-island texels themselves emit the flat normal `(128, 128, 255)`. The normal PNG is written with opaque alpha.
## Conventions
- **DirectX (green-down)**, `--convention directx`, default. Green > 128 means the surface faces down (+V). This is what the Bedrock RTX pipeline and the reference material tooling assume; the Mojang texture-set spec and the NVIDIA texturing guide only specify the neutral colour `(128, 128, 255)`, so verify in the target renderer and regenerate with `--convention opengl` if lit relief looks inverted top-to-bottom.
- **OpenGL (green-up)**, `--convention opengl`. Used by Blender, Unity, Godot, glTF, labPBR (Java shader packs). Converting between the two is a green-channel inversion; X and Z are shared.
- Height: light = high, 8-bit grayscale. Bedrock texture sets accept `heightmap` **or** `normal`, never both; the heightmap layer is meant for 16× pixel-art sets, the normal layer for everything else.
## Validation report
Printed as JSON by both scripts (`report`), decoding every texel to `(x, y, z)` in −1..1:
| Field | Meaning | Healthy |
|---|---|---|
| `classification` | `standard` (unit-length XYZ), `z-zero` (blue holds 0..1 z), `xy-only`, `heightfield` (≥95 % grayscale) | `standard` |
| `normalizedPct` | share of texels with length within ±0.1 of 1 | ≥ 0.98 |
| `flatPct` | share of texels at the neutral colour | informational; high on flat art or big gutters |
| `negativeZCount` | blue < 128 | 0 |
| `xyTooLongCount` | `hypot(x, y) > 1` | 0 |
| `meanXYBias` | mean x and y | within ±0.1 of 0 |
| `verdict` | `ok` / `warn` / `fail` | `ok` |
`heightfield` or `xy-only` means the wrong image was fed in (a height or albedo passed as a normal, or a normal passed as albedo). Large `meanXYBias` on a tile indicates an unflattened ramp in the height.
## Runtime parity
The WGSL kernels and the torch implementation share the maths above. On the same 8-bit height the outputs match to within 2 levels (GPU `normalize` precision); on identical luminance inputs the height PNGs are byte-identical except where a value sits exactly on a rounding boundary. Depth results differ slightly between runtimes because the PyTorch and ONNX conversions of Depth Anything are not bit-identical.
references/mer-inference.md
# MER inference: metalness, emissive, roughness
Contents: [What MER is](#what-mer-is) · [Inputs](#inputs) · [Metalness](#metalness) · [Emissive](#emissive) · [Roughness](#roughness) · [Overrides](#overrides) · [Report and checks](#report-and-checks) · [Parity](#parity)
Both scripts write `<name>_mer.png` unless `--no-mer` is passed. Blockbench and Bedrock read the three channels from one packed RGB image, and Blockbench requires a MER texture next to the color texture for a material's PBR channels to take effect (a color-only material previews flat).
## What MER is
| Channel | Meaning | 0 | 255 |
|---|---|---|---|
| R | metalness | dielectric | metal (albedo becomes the specular tint) |
| G | emissive | no glow | full-strength glow of the albedo colour |
| B | roughness | mirror | fully matte |
Alpha is written as 255. Bedrock's MERS variant carries subsurface thickness in alpha; the scripts do not produce it. Uniform values exist for Blockbench materials (`mer_value=[M, E, R]`), but a texture is needed the moment any channel varies across the surface.
## Inputs
The MER pass reads the albedo, the shaped 8-bit height (with the albedo alpha as island mask) and the derived normal. Sampling is the same as the normal pass: `--wrap` wraps, otherwise taps clamp; off-island neighbours are replaced by the centre value; off-island texels emit `(0, 0, 255)`: no metal, no glow, matte.
## Metalness
HSV heuristic, binary output:
- **Achromatic metals** (steel, iron, silver, chrome): value ≥ `--metal-brightness` (0.6) and saturation ≤ `--metal-saturation` (0.15), each with a short smoothstep ramp.
- **Gold**: hue within `--gold-hue` (0.11–0.17), saturation ≥ 0.3, value ≥ 0.5.
- **Copper**: hue within `--copper-hue` (0.03–0.10), saturation ≥ 0.3, value ≥ 0.4.
A 3×3 majority vote then removes single-texel speckle from the mask (`--no-metal-clean` keeps the raw threshold, for 16× art where one texel is a deliberate rivet).
The heuristic cannot tell polished steel from white wool, bone, snow, quartz or a pale blue backboard: anything bright and desaturated becomes metal. On stylised content that is the common failure, so treat `metalPct` in the report as a question to answer, not a result to accept. `--metal 0` for materials with no metal, `--metal auto` plus palette overrides when some palette colours are metal, `--metal 1` for a fully metallic part.
## Emissive
Bright **and** saturated texels glow: value ≥ `--emissive-brightness` (0.8) and saturation ≥ `--emissive-saturation` (0.5), multiplied smoothsteps, so a pale highlight does not trigger it. `--emissive-overbright T` adds the physically motivated rule that luminance above `T` cannot be a lit dielectric and must glow; it is off by default (`1`) because pure white is ordinary in pixel art. `--emissive-softness 0..1` blends toward the 3×3 neighbourhood mean, which anti-aliases the glow outline without blurring its interior.
False positives are saturated bright colours that are merely bright: a yellow beak, a red flower, neon signage that is painted, not lit. Check `emissivePct`; fix with `--emissive 0` or palette overrides for the glowing colours only.
## Roughness
Weighted composite, clamped to 0..1, then `--rough-scale` and `--rough-bias`:
| Term | Source | Reads as | Weight |
|---|---|---|---|
| variance | 5×5 luminance variance × 10 | grain, noise, dirt | `--rough-weights` V (0.4) |
| edge | Sobel magnitude of albedo luminance × 2 | painted detail and outlines | `--rough-weights` E (0.3) |
| saturation | `1 − 0.5·S` | pure hues glossier, greys rougher | `--rough-weights` S (0.3) |
| curvature | `−divergence(normal XY) × 2`, clamped −1..1 | crevices rougher, convex ridges smoother | `--rough-curvature` (0.25) |
| height | mix toward `1 − height` (or `height` when negative) | recesses collect dirt (positive) or shine (negative) | `--rough-height` (0, range −1..1) |
The curvature term is what the normal map contributes: divergence of the decoded XY (`0.5·(∂Nx/∂x ± ∂Ny/∂y)`, sign following the green convention) is positive on convex ridges and negative in grooves, so a bevelled edge wears glossier and a mortar line roughens without any colour cue. It is signed and can lower roughness; hold it small on 16× art where every texel boundary is a "ridge". Height influence is the coarse version of the same idea and is off by default; set it around 0.3–0.5 for masonry, bark and soil, negative for worn metal where the raised surface is polished.
Typical outcomes: matte painted pixel art lands around 0.5–0.8; flat pale surfaces fall toward the low end because there is neither variance nor edges, which is often too glossy for cloth or paint. Raise them with `--rough-bias 0.2` or set a uniform `--roughness 0.8` when the material is uniformly matte.
## Overrides
Applied on the CPU after inference, in this order:
1. **Uniform channels**: `--metal`, `--emissive`, `--roughness` each accept `auto` or a 0..1 value that replaces the inferred channel on every island texel.
2. **Palette**: `--palette "#RRGGBB=m,e,r;#RRGGBB=-,-,0.9"` sets channels for texels whose albedo colour matches exactly; `-` leaves that channel as inferred (or as the uniform). This is the precise tool for pixel art, where a material is a handful of palette entries: mark the iron greys metal, the lava oranges emissive, the wool whites matte, and leave the rest inferred. The report counts matched texels (`paletteTexels`); zero means the colour was not exact, so read it from the atlas rather than guessing.
The Python script also stores the three channels on the PyPBR material (`metallic`, `emissive`, `roughness`), so `--material-folder` writes them as separate grayscale PNGs alongside the packed file.
## Report and checks
`merReport` in the JSON:
| Field | Meaning |
|---|---|
| `islandTexels` | texels with alpha ≥ threshold (the denominator) |
| `metalPct`, `emissivePct` | share of island texels at or above 128 in R / G |
| `roughnessMean`, `roughnessRange` | B channel statistics, 0..1 |
| `paletteEntries`, `paletteTexels` | override entries parsed and texels they hit |
| `uniform` | the uniform values in force, `null` = inferred |
Compare the percentages with what the material is: a wooden crate should report `metalPct` 0; a lantern should report a small `emissivePct` on the flame only. Then view the channels (split the PNG or inspect the PyPBR folder) before assigning. In Blockbench, `create_pbr_material` or `configure_material` with `mer_texture` plus the color texture; `get_material_info` confirms the channel, and a lit screenshot shows whether metal regions turned dark and reflective and glow regions light up.
## Parity
The WGSL and torch implementations share the maths. On a 16× tile the packed MER bytes are identical; on a 1024² atlas metal is identical, emissive is within one level and roughness within three (it consumes the normal, which itself differs by up to two levels between runtimes, and float summation order differs in the variance term). The report percentages agree to four decimals.
references/node-vgpu.md
# Node.js runtime: vgpu + Dawn WebGPU + transformers.js
Contents: [Install](#install) · [Run](#run) · [How the script uses vgpu](#how-the-script-uses-vgpu) · [Shaders](#shaders) · [Depth Anything in Node](#depth-anything-in-node) · [Troubleshooting](#troubleshooting)
Tested with Node 23.6, vgpu 0.5.0 (`@vgpu/adapter-node` → `webgpu` 0.4.0 Dawn prebuilt), pngjs 7.0.0, @huggingface/transformers 4.2.0 on Windows 11 (D3D12 adapter, RTX 3080 Ti).
## Install
The script is an ES module that imports `vgpu/node` and `pngjs`, so it must resolve them from a `node_modules` on its path. Either run it inside a project that has them, or create a small one next to your textures:
```bash
mkdir pbr-tools && cd pbr-tools && npm init -y
npm install vgpu pngjs # Dawn arrives through @vgpu/adapter-node → webgpu
npm install @huggingface/transformers # only for --height-source depth
npx vgpu doctor # JSON verdict: adapter found, render probe ok
```
`vgpu` resolves a Dawn binary for the host (win32/x64, linux, darwin prebuilds; a downloadable prebuild as fallback). Machines without a GPU need the portable CPU renderer once: `npx vgpu install-software-renderer`, then pass `--gpu software`. `--gpu auto` (default) prefers hardware.
Copy or reference the whole `scripts/` folder; the script loads `shaders/height-map.wgsl` and `shaders/normal-map.wgsl` relative to its own location.
## Run
```bash
node scripts/albedo_to_normal.mjs albedo.png --out ./pbr --wrap --strength 0.5
node scripts/albedo_to_normal.mjs planks.png --height-source depth --depth-model base --high-pass 32 --seam offset --wrap
node scripts/albedo_to_normal.mjs atlas.png --height-source file --height-file atlas_h.png --soften 1
node scripts/albedo_to_normal.mjs albedo.png --height-source depth --depth-device webgpu --depth-dtype fp16
```
`--depth-device cpu` (default) runs ONNX Runtime on the CPU; `webgpu` runs the model on Dawn as well (`fp16` there halves the download). Model ids: `onnx-community/depth-anything-v2-{small,base,large}` (fp32 ≈ 100 MB / 390 MB / 1.3 GB; fp16 half), cached under `~/.cache/huggingface` on first use.
## How the script uses vgpu
```js
import { compute, init, texture } from "vgpu/node";
const gpu = await init({ adapter: "auto" }); // NodeGpu: gpu.gpu is the raw GPUDevice, gpu.adapter = {name, type}
const albedoTex = texture(gpu, { kind: "2d", size: [w, h], format: "rgba8unorm", usage: ["texture_binding", "copy_dst"] });
gpu.gpu.queue.writeTexture({ texture: albedoTex.gpu }, rgba, { bytesPerRow: w * 4, rowsPerImage: h }, [w, h]);
const heightTex = texture(gpu, { kind: "2d", size: [w, h], format: "rgba8unorm", usage: ["storage_binding", "copy_src"] });
const pass = compute(gpu, wgslSource, { label: "height-map", set: { albedoTex, heightTex, params: { mode: 0, invert: 0 } } });
pass.dispatch(Math.ceil(w / 8), Math.ceil(h / 8)); // workgroup_size(8, 8)
const bytes = await heightTex.read({ mipLevel: 0, region: "all" }); // tightly packed RGBA
gpu.dispose(); // stops Dawn polling so the process exits
```
- Bindings are set by their WGSL names; a uniform struct is set as a nested object with its member names. Every declared binding must be set or the dispatch fails with `VGPU-R1-BINDING-NEVER-SET`.
- `texture()` adds no usage flags implicitly: sampled inputs need `texture_binding`, uploads need `copy_dst`, compute outputs need `storage_binding`, readback needs `copy_src`.
- `Texture.read()` submits, copies to a staging buffer, maps it and strips row padding; `readFloats()` exists for float formats.
- `initFromDevice(rawDevice)` wraps a device created elsewhere (for example ONNX Runtime Web's WebGPU device) so vgpu and the model can share buffers zero-copy; the script keeps the two separate for simplicity.
- Uniform structs follow WGSL host-shareable layout; `npx vgpu check shaders/normal-map.wgsl` prints the reflected bindings and validates the source before a run.
## Shaders
Both files are self-contained (no `import`), so `readFileSync` is enough; shaders that import from `@vgpu/wgsl-std` would need `resolveShader()` from `@vgpu/wgsl/runtime` first.
- `height-map.wgsl`: `albedoTex → heightTex`, BT.601 or average luma, optional invert, albedo alpha carried in `.a`.
- `normal-map.wgsl`: `heightTex → normalTex`, Sobel/Prewitt/Scharr with strength, `wrap` or clamp sampling, `alpha_threshold` island clamping, DirectX/OpenGL green. Transparent texels emit `(0.5, 0.5, 1, 1)`.
- `mer-map.wgsl`: `albedoTex + heightTex + normalTex → merTex`, HSV metal heuristic, brightness×saturation emissive with optional softness, roughness from variance, albedo edges, saturation, normal divergence (curvature) and height influence. Transparent texels emit `(0, 0, 1, 1)`.
The CPU shaping (depth flatten/high-pass/normalize, seam, soften, levels) and the MER overrides (uniform values, palette) live in the script; the GPU passes are the luminance extraction, the normal kernel and the MER inference.
## Depth Anything in Node
```js
import { pipeline, RawImage } from "@huggingface/transformers";
const estimator = await pipeline("depth-estimation", "onnx-community/depth-anything-v2-small", { dtype: "fp32", device: "cpu" });
const { depth } = await estimator(new RawImage(rgbaClamped, width, height, 4)); // depth: RawImage, 1 channel, Uint8Array 0–255, 255 = nearest
```
`RawImage` accepts a `Uint8ClampedArray` with 4 channels straight from `pngjs`; the processor drops alpha. Output is bilinearly interpolated back to the input size by the pipeline. On the tested machine a 1024×1024 estimate took about 1 s on CPU (fp32, small) and the 16× texture round trip (upscaled to 1024) about 2.5 s including model load.
## Troubleshooting
- `VGPU-NODE-SOFTWARE-RENDERER-MISSING` → `npx vgpu install-software-renderer` (only when asking for `--gpu software`).
- `Cannot find package 'vgpu'` → the script is outside a project with the dependencies; see Install. ESM ignores `NODE_PATH`.
- Dawn download blocked or old glibc on Linux → `npx vgpu doctor` names the fix (`npx vgpu install-dawn`).
- `--height-source depth needs npm install @huggingface/transformers` → install it in the same project.
- Process hangs after output → `gpu.dispose()` was skipped; the script always calls it in `finally`.
references/python-pypbr.md
# Python runtime: PyPBR + Depth Anything V2
Contents: [Install](#install) · [Run](#run) · [What PyPBR does here](#what-pypbr-does-here) · [PyPBR API notes](#pypbr-api-notes) · [Depth Anything in Python](#depth-anything-in-python) · [Troubleshooting](#troubleshooting)
Tested with Python 3.13, pypbr 0.1.0b5, torch 2.9.1 (CUDA 12.8), torchvision 0.24, Pillow 12, numpy 2.2, transformers 4.57 on Windows 11 with an RTX 3080 Ti.
## Install
```bash
pip install pypbr # pulls torch, torchvision, numpy, Pillow
pip install transformers # only for --height-source depth
```
`pypbr` declares `torch>=1.7`; if a GPU build of torch is already installed, keep it (`pip install --no-deps pypbr` avoids replacing it). Depth Anything runs on CUDA automatically when `torch.cuda.is_available()`; CPU works and takes a few seconds per texture.
## Run
```bash
python scripts/albedo_to_normal.py albedo.png --out ./pbr --wrap --strength 0.5
python scripts/albedo_to_normal.py planks.png --height-source depth --depth-model base --high-pass 32 --seam offset --wrap
python scripts/albedo_to_normal.py atlas.png --height-source file --height-file atlas_painted_height.png --soften 1
python scripts/albedo_to_normal.py albedo.png --material-folder ./pbr/material # + PyPBR folder (albedo, height, normal, metallic, roughness, emissive)
python scripts/albedo_to_normal.py crate.png --metal 0 --rough-bias 0.2 # non-metal, matte-leaning MER
python scripts/albedo_to_normal.py lamp.png --palette "#ffe26a=-,1,-;#8a8a8a=1,-,0.35" # glow and steel by palette colour
```
`--depth-device auto|cpu|cuda|mps` (default auto). Model choices map to `depth-anything/Depth-Anything-V2-{Small,Base,Large}-hf` (≈100 MB / 390 MB / 1.3 GB downloads on first use, cached by Hugging Face Hub).
## What PyPBR does here
The script builds a `BasecolorMetallicMaterial` from the albedo, the shaped 8-bit height, the derived normal and the inferred `metallic`, `emissive` and `roughness` channels, with `normal_convention` set to match `--convention`. The packed `_mer.png` is assembled from the same bytes. From that object the material can be taken further with PyPBR's own operations before or after export (the Cook-Torrance BRDF in `pypbr.models` can render a lit preview of the metallic/roughness result):
```python
from pypbr.materials import BasecolorMetallicMaterial
from pypbr.io import load_material_from_folder, save_material_to_folder
from pypbr.utils import NormalConvention
material = load_material_from_folder("./pbr/material", preferred_workflow="metallic")
material.normal_convention = NormalConvention.DIRECTX
material.adjust_normal_strength(1.5) # rescale XY, renormalise
material.invert_normal() # flip green, toggles the convention flag
material.resize((32, 32)) # resamples every map (antialias on)
material.tile(2) # repeat every map
material.compute_height_from_normal() # Poisson (FFT) integration back to a height
save_material_to_folder(material, "./pbr/material_v2")
maps = material.to_pil() # {"albedo": RGB, "normal": RGB, "height": L, ...}
```
Maps are `(C, H, W)` float tensors in 0..1 (normals stored decoded in −1..1, exposed encoded through `material.normal_rgb`). Assigning a PIL image, numpy array or CPU float tensor to any attribute name creates that map.
## PyPBR API notes
- **`compute_normal_from_height` is not used by the script.** In 0.1.0b5 it zero-pads the borders (a false gradient along all four edges) and its central difference has the X sign inverted relative to its own docstring, so a ramp rising to the right comes out red-bright instead of red-dark. Verify with a ramp before trusting a newer version. The script derives the normal with the shared Sobel/Prewitt/Scharr kernels and hands the result to the material.
- **`to_pil()` truncates** (`tensor * 255 → byte`), so a value of `k/255` is safe but anything in between rounds down. The script rounds to bytes itself and writes the primary PNGs with Pillow; the `--material-folder` copies go through `save_to_folder`/`to_pil` and can differ from the primary files by one level on a fraction of a percent of texels.
- **Normal assignment rule.** A 3-channel tensor with any value below 0 is stored as-is; one with `min() >= 0` is treated as encoded 0..1, decoded and renormalised. A perfectly flat map (all `(0, 0, 1)`) therefore must be assigned as an image, which the script does.
- `material.height` is `(1, H, W)`; 16-bit and float PNGs are honoured by `load_material_from_folder` (modes `I;16`, `F`), and `to_pil(maps_mode={"height": "I;16"})` writes 16-bit height when needed.
- `NormalConvention.DIRECTX` only records the convention on the material; nothing flips automatically except `invert_normal()`.
## Depth Anything in Python
```python
from transformers import pipeline
estimator = pipeline("depth-estimation", model="depth-anything/Depth-Anything-V2-Small-hf", device="cuda")
result = estimator(pil_image) # {"predicted_depth": tensor(H, W), "depth": PIL "L"}
```
`predicted_depth` is float relative inverse depth at the input size; `depth` is the same normalised to 0–255 with 255 = nearest. The script feeds the nearest-neighbour upscaled albedo (see height-processing.md) and uses `depth`, then block-averages back and unsharpens. Depth Anything 3 (`depth-anything/DA3-*`) is not in the transformers pipeline yet and needs its own package; V2 is what both runtimes share.
## Troubleshooting
- `error: pypbr is required` → `pip install pypbr`. `torch.FloatTensor` type checks inside PyPBR only accept CPU float32 tensors; the script keeps everything on the CPU.
- Emoji/console errors on Windows → set `PYTHONIOENCODING=utf-8`.
- Very slow depth on CPU with `large` → use `small` or `base`; quality difference on textures is modest.
- Report verdict `fail` with `classification: heightfield` → the albedo argument was a grayscale image (a height map). Pass it with `--height-source file` instead.
scripts/albedo_to_normal.mjs
#!/usr/bin/env node
/**
* albedo_to_normal.mjs — albedo PNG to height + tangent-space normal PNGs, on the
* GPU through vgpu (Dawn-backed WebGPU in Node).
*
* node albedo_to_normal.mjs <albedo.png> [--out DIR] [--name NAME] [options]
*
* Height sources (`--height-source`):
* luminance BT.601 luma of the albedo, computed by shaders/height-map.wgsl (default)
* depth Depth Anything V2 via @huggingface/transformers (optional dependency)
* file an existing grayscale height PNG (`--height-file`)
*
* The height is shaped on the CPU (depth flatten / high-pass / normalize / seam
* heal, density softening, levels), quantised to 8 bits, written as
* `<name>_height.png`, then turned into `<name>_normal.png` by
* shaders/normal-map.wgsl. shaders/mer-map.wgsl then infers a packed MER
* (`<name>_mer.png`: R metalness, G emissive, B roughness) from the albedo, the
* height and the normal's curvature; uniform values and per-colour palette
* overrides are applied on the CPU. A validation report is printed as JSON.
*
* Dependencies: vgpu (pulls the `webgpu` Dawn binding), pngjs. Optional:
* @huggingface/transformers for `--height-source depth`.
*/
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { basename, dirname, extname, join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { parseArgs } from "node:util";
import { PNG } from "pngjs";
import { compute, init, texture } from "vgpu/node";
const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url));
const OPERATORS = { sobel: 0, prewitt: 1, scharr: 2 };
const CONVENTIONS = { directx: 0, opengl: 1 };
const DEPTH_MODELS = {
small: "onnx-community/depth-anything-v2-small",
base: "onnx-community/depth-anything-v2-base",
large: "onnx-community/depth-anything-v2-large",
};
const DEPTH_MIN_INPUT = 1024;
const DEPTH_MAX_INPUT = 4096;
const DEPTH_DOWNSCALE_SHARPEN = 0.5;
const BASE_FACE_TEXELS = 16;
const MAX_SOFTEN_SIGMA = 8;
// ---------------------------------------------------------------------------
// CLI
// ---------------------------------------------------------------------------
const { values: opts, positionals } = parseArgs({
allowPositionals: true,
options: {
out: { type: "string" },
name: { type: "string" },
"height-source": { type: "string", default: "luminance" },
"height-file": { type: "string" },
luma: { type: "string", default: "bt601" },
"invert-height": { type: "boolean", default: false },
"depth-model": { type: "string", default: "small" },
"depth-device": { type: "string", default: "cpu" },
"depth-dtype": { type: "string", default: "fp32" },
flatten: { type: "string" },
"high-pass": { type: "string", default: "0" },
normalize: { type: "boolean" },
"no-normalize": { type: "boolean", default: false },
seam: { type: "string", default: "off" },
"seam-amount": { type: "string", default: "0.15" },
soften: { type: "string", default: "0" },
levels: { type: "string", default: "0,1,1" },
operator: { type: "string", default: "sobel" },
strength: { type: "string", default: "0.5" },
convention: { type: "string", default: "directx" },
wrap: { type: "boolean", default: false },
"alpha-threshold": { type: "string", default: "8" },
"no-height": { type: "boolean", default: false },
"no-mer": { type: "boolean", default: false },
metal: { type: "string", default: "auto" },
emissive: { type: "string", default: "auto" },
roughness: { type: "string", default: "auto" },
"metal-brightness": { type: "string", default: "0.6" },
"metal-saturation": { type: "string", default: "0.15" },
"no-metal-clean": { type: "boolean", default: false },
"gold-hue": { type: "string", default: "0.11,0.17" },
"copper-hue": { type: "string", default: "0.03,0.10" },
"rough-weights": { type: "string", default: "0.4,0.3,0.3" },
"rough-curvature": { type: "string", default: "0.25" },
"rough-height": { type: "string", default: "0" },
"rough-scale": { type: "string", default: "1" },
"rough-bias": { type: "string", default: "0" },
"emissive-brightness": { type: "string", default: "0.8" },
"emissive-saturation": { type: "string", default: "0.5" },
"emissive-overbright": { type: "string", default: "1" },
"emissive-softness": { type: "string", default: "0" },
palette: { type: "string", default: "" },
gpu: { type: "string", default: "auto" },
help: { type: "boolean", default: false },
},
});
if (opts.help || positionals.length !== 1) {
console.error(`usage: node albedo_to_normal.mjs <albedo.png> [--out DIR] [--name NAME]
--height-source luminance|depth|file --height-file PATH
--luma bt601|average --invert-height
--depth-model small|base|large --depth-device cpu|webgpu --depth-dtype fp32|fp16|q8
--flatten 0..1 (depth default 1) --high-pass TEXELS --normalize|--no-normalize
--seam off|offset --seam-amount 0..0.5
--soften MULT (density-scaled gaussian, 0 = off) --levels BLACK,WHITE,GAMMA
--operator sobel|prewitt|scharr --strength F --convention directx|opengl
--wrap (tiling texture) --alpha-threshold 0..255 --no-height --gpu auto|hardware|software
MER: --no-mer --metal auto|0..1 --emissive auto|0..1 --roughness auto|0..1
--metal-brightness 0.6 --metal-saturation 0.15 --gold-hue 0.11,0.17 --copper-hue 0.03,0.10 --no-metal-clean
--rough-weights VAR,EDGE,SAT --rough-curvature W --rough-height -1..1 --rough-scale F --rough-bias F
--emissive-brightness 0.8 --emissive-saturation 0.5 --emissive-overbright 1 --emissive-softness 0..1
--palette "#RRGGBB=m,e,r;#RRGGBB=-,-,0.9" (per-colour overrides; '-' keeps the inferred value)`);
process.exit(positionals.length === 1 ? 0 : 2);
}
const num = (value, fallback) => {
const parsed = Number(value);
return Number.isFinite(parsed) ? parsed : fallback;
};
const fail = (message) => {
console.error(`error: ${message}`);
process.exit(1);
};
if (!(opts["height-source"] in { luminance: 1, depth: 1, file: 1 })) fail("unknown --height-source");
if (!(opts.operator in OPERATORS)) fail("unknown --operator");
if (!(opts.convention in CONVENTIONS)) fail("unknown --convention");
if (!(opts["depth-model"] in DEPTH_MODELS)) fail("unknown --depth-model");
if (opts["height-source"] === "file" && !opts["height-file"]) fail("--height-file is required with --height-source file");
// ---------------------------------------------------------------------------
// PNG helpers
// ---------------------------------------------------------------------------
/** Decode a PNG into tightly packed RGBA bytes. */
const readPng = (path) => {
const png = PNG.sync.read(readFileSync(path));
return { width: png.width, height: png.height, rgba: new Uint8Array(png.data) };
};
/** Encode tightly packed RGBA bytes as a PNG file. */
const writePng = (path, width, height, rgba) => {
const png = new PNG({ width, height });
png.data.set(rgba);
writeFileSync(path, PNG.sync.write(png));
};
/** Expand a 0..1 float field to opaque grayscale RGBA bytes. */
const grayToRgba = (field, alpha) => {
const out = new Uint8Array(field.length * 4);
field.forEach((value, i) => {
const byte = Math.round(Math.min(1, Math.max(0, value)) * 255);
out.set([byte, byte, byte, alpha ? alpha[i] : 255], i * 4);
});
return out;
};
// ---------------------------------------------------------------------------
// Height shaping (pure CPU, mirrors the Python script)
// ---------------------------------------------------------------------------
const clamp = (value, lo, hi) => Math.min(hi, Math.max(lo, value));
const wrapIndex = (value, size) => ((value % size) + size) % size;
/** Integer nearest-neighbour factor lifting the short side to DEPTH_MIN_INPUT. */
const depthUpscaleFactor = (width, height) => {
const needed = Math.ceil(DEPTH_MIN_INPUT / Math.min(width, height));
const cap = Math.floor(DEPTH_MAX_INPUT / Math.max(width, height));
return Math.max(1, Math.min(needed, cap));
};
/** Nearest-neighbour RGBA upscale by an integer factor. */
const nnUpscaleRgba = (rgba, width, height, factor) => {
const outW = width * factor;
const out = new Uint8ClampedArray(outW * height * factor * 4);
for (let y = 0; y < height * factor; y++) {
const sy = Math.floor(y / factor);
for (let x = 0; x < outW; x++) {
out.set(rgba.subarray((sy * width + Math.floor(x / factor)) * 4, (sy * width + Math.floor(x / factor)) * 4 + 4), (y * outW + x) * 4);
}
}
return out;
};
/** Exact k×k block average of a single channel; `width`/`height` are the SOURCE dims. */
const downsampleArea = (gray, width, height, factor) => {
const outW = width / factor;
const outH = height / factor;
const out = new Float32Array(outW * outH);
for (let oy = 0; oy < outH; oy++) {
for (let ox = 0; ox < outW; ox++) {
let sum = 0;
for (let dy = 0; dy < factor; dy++) {
for (let dx = 0; dx < factor; dx++) sum += gray[(oy * factor + dy) * width + ox * factor + dx];
}
out[oy * outW + ox] = sum / (factor * factor);
}
}
return out;
};
/** Radius-1 box blur, edge-clamped. */
const boxBlurClamped = (field, width, height) => {
const out = new Float32Array(field.length);
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
let sum = 0;
for (let dy = -1; dy <= 1; dy++) {
for (let dx = -1; dx <= 1; dx++) sum += field[clamp(y + dy, 0, height - 1) * width + clamp(x + dx, 0, width - 1)];
}
out[y * width + x] = sum / 9;
}
}
return out;
};
/** Unsharp mask: `field + amount * (field - blur(field))`. */
const sharpen = (field, width, height, amount) => {
if (amount <= 0) return field;
const blurred = boxBlurClamped(field, width, height);
return field.map((value, i) => clamp(value + amount * (value - blurred[i]), 0, 1));
};
/** Least-squares plane slopes over centred pixel coordinates. */
const fitPlane = (field, width, height) => {
const cx = (width - 1) / 2;
const cy = (height - 1) / 2;
let sxv = 0;
let syv = 0;
let sxx = 0;
let syy = 0;
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
const value = field[y * width + x];
sxv += value * (x - cx);
syv += value * (y - cy);
sxx += (x - cx) ** 2;
syy += (y - cy) ** 2;
}
}
return { gx: sxx > 0 ? sxv / sxx : 0, gy: syy > 0 ? syv / syy : 0 };
};
/** One separable box-blur pass with wrapped sampling. */
const boxBlurWrapAxis = (field, width, height, radius, horizontal) => {
const out = new Float32Array(field.length);
const len = horizontal ? width : height;
const lines = horizontal ? height : width;
const window = 2 * radius + 1;
for (let line = 0; line < lines; line++) {
const idx = (i) => (horizontal ? line * width + wrapIndex(i, len) : wrapIndex(i, len) * width + line);
let sum = 0;
for (let i = -radius; i <= radius; i++) sum += field[idx(i)];
for (let i = 0; i < len; i++) {
out[idx(i)] = sum / window;
sum += field[idx(i + radius + 1)] - field[idx(i - radius)];
}
}
return out;
};
/** Wrap-around box blur ×3 (≈ Gaussian), tile-aware on both axes. */
const blurWrap = (field, width, height, radius) => {
const r = Math.max(1, Math.round(radius));
return [0, 1, 2].reduce((acc) => boxBlurWrapAxis(boxBlurWrapAxis(acc, width, height, r, true), width, height, r, false), field);
};
/** Depth edit chain: flatten, high-pass, invert, normalize. */
const editDepth = (field, width, height, { flatten, highPass, invert, normalize }) => {
let out = Float32Array.from(field);
if (flatten > 0) {
const { gx, gy } = fitPlane(out, width, height);
const cx = (width - 1) / 2;
const cy = (height - 1) / 2;
out = out.map((value, i) => value - flatten * (gx * ((i % width) - cx) + gy * (Math.floor(i / width) - cy)));
}
if (highPass > 0) {
const low = blurWrap(out, width, height, highPass);
out = out.map((value, i) => value - low[i] + 0.5);
}
if (invert) out = out.map((value) => 1 - value);
if (normalize) {
const min = out.reduce((a, b) => Math.min(a, b), Infinity);
const max = out.reduce((a, b) => Math.max(a, b), -Infinity);
if (max - min > 1e-6) out = out.map((value) => (value - min) / (max - min));
}
return out.map((value) => clamp(value, 0, 1));
};
/** Blend weight of B in a two-way blend, letting the taller surface win the middle. */
const blendWeight = (t, hA, hB, bias) => {
const window = 4 * t * (1 - t);
if (bias <= 0 || window <= 0) return t;
const floor = Math.max(hA, hB) - 0.1;
const wA = Math.max(hA - floor, 0);
const wB = Math.max(hB - floor, 0);
const picked = wA + wB > 0 ? wB / (wA + wB) : t;
return t + (picked - t) * bias * window;
};
const smoothstep = (t) => {
const c = clamp(t, 0, 1);
return c * c * (3 - 2 * c);
};
/** Wrap-and-heal seam blend: shift by half a tile and blend the original back over the centre seam. */
const sealOffset = (field, width, height, amount) => {
const halfW = width >> 1;
const halfH = height >> 1;
if (halfW < 1 || halfH < 1) return field;
const reach = clamp(amount * 2, 1e-4, 1);
const spanX = Math.max(1, Math.floor((width - 1) / 2));
const spanY = Math.max(1, Math.floor((height - 1) / 2));
const out = new Float32Array(field.length);
for (let y = 0; y < height; y++) {
const ny = clamp(1 - Math.min(y, height - 1 - y) / spanY, 0, 1);
for (let x = 0; x < width; x++) {
const nx = clamp(1 - Math.min(x, width - 1 - x) / spanX, 0, 1);
const here = y * width + x;
const shifted = ((y + halfH) % height) * width + ((x + halfW) % width);
const distance = Math.min(1, Math.hypot(nx, ny));
const t = distance >= reach ? 1 : smoothstep(distance / reach);
const w = blendWeight(t, field[here], field[shifted], 0.5);
out[here] = field[shifted] * w + field[here] * (1 - w);
}
}
return out;
};
/** Softening sigma proportional to texel density (≈ shortSide/16/4 texels), capped. */
const densitySigma = (width, height, softness) => Math.min((Math.min(width, height) / BASE_FACE_TEXELS / 4) * softness, MAX_SOFTEN_SIGMA);
/** Separable Gaussian blur with periodic (wrap) or clamped edges. */
const gaussianBlur = (field, width, height, sigma, wrap) => {
if (sigma <= 0) return field;
const radius = Math.max(1, Math.ceil(sigma * 3));
const kernel = Float64Array.from({ length: radius * 2 + 1 }, (_, i) => Math.exp(-((i - radius) ** 2) / (2 * sigma * sigma)));
const total = kernel.reduce((a, b) => a + b, 0);
const weights = kernel.map((w) => w / total);
const at = (value, size) => (wrap ? wrapIndex(value, size) : clamp(value, 0, size - 1));
const tmp = new Float32Array(field.length);
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
let acc = 0;
for (let k = -radius; k <= radius; k++) acc += field[y * width + at(x + k, width)] * weights[k + radius];
tmp[y * width + x] = acc;
}
}
const out = new Float32Array(field.length);
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
let acc = 0;
for (let k = -radius; k <= radius; k++) acc += tmp[at(y + k, height) * width + x] * weights[k + radius];
out[y * width + x] = acc;
}
}
return out;
};
/** Input levels: black/white points then gamma. */
const applyLevels = (field, black, white, gamma) => {
const range = Math.max(white - black, 0.001);
return field.map((value) => clamp(value - black, 0, range) / range).map((value) => Math.pow(value, 1 / Math.max(gamma, 0.001)));
};
// ---------------------------------------------------------------------------
// Height sources
// ---------------------------------------------------------------------------
/** Depth Anything V2 through transformers.js; returns 0..1 with 1 = nearest, at native size. */
const estimateDepth = async (rgba, width, height) => {
let transformers;
try {
transformers = await import("@huggingface/transformers");
} catch {
fail("--height-source depth needs `npm install @huggingface/transformers`");
}
const { pipeline, RawImage } = transformers;
const estimator = await pipeline("depth-estimation", DEPTH_MODELS[opts["depth-model"]], {
dtype: opts["depth-dtype"],
device: opts["depth-device"],
});
const factor = depthUpscaleFactor(width, height);
const inputW = width * factor;
const inputH = height * factor;
const input = factor > 1 ? nnUpscaleRgba(rgba, width, height, factor) : new Uint8ClampedArray(rgba);
const result = await estimator(new RawImage(input, inputW, inputH, 4));
const { depth } = Array.isArray(result) ? result[0] : result;
const full = Float32Array.from(depth.data, (byte) => byte / 255);
const native = factor > 1 ? sharpen(downsampleArea(full, inputW, inputH, factor), width, height, DEPTH_DOWNSCALE_SHARPEN) : full;
return { field: native, factor, model: DEPTH_MODELS[opts["depth-model"]] };
};
/** Grayscale PNG (red channel) as a 0..1 field; must match the albedo size. */
const loadHeightFile = (path, width, height) => {
const png = readPng(resolve(path));
if (png.width !== width || png.height !== height) fail(`height file is ${png.width}x${png.height}, albedo is ${width}x${height}`);
return Float32Array.from({ length: width * height }, (_, i) => png.rgba[i * 4] / 255);
};
// ---------------------------------------------------------------------------
// MER overrides (CPU): uniform channel values and per-colour palette entries
// ---------------------------------------------------------------------------
/** `auto` → null, otherwise a 0..1 uniform value for the channel. */
const uniformChannel = (value, flag) => {
if (value === "auto") return null;
const parsed = Number(value);
if (!Number.isFinite(parsed) || parsed < 0 || parsed > 1) fail(`--${flag} must be auto or 0..1`);
return parsed;
};
/** Parse `#RRGGBB=m,e,r;...` into [{ rgb: [r, g, b], values: [m|null, e|null, r|null] }]. */
const parsePalette = (text) =>
text
.split(";")
.map((entry) => entry.trim())
.filter(Boolean)
.map((entry) => {
const match = /^#?([0-9a-f]{6})\s*=\s*([^,]+),([^,]+),([^,]+)$/i.exec(entry);
if (!match) fail(`bad --palette entry "${entry}" (expected #RRGGBB=m,e,r)`);
const rgb = [0, 2, 4].map((offset) => parseInt(match[1].slice(offset, offset + 2), 16));
const values = match.slice(2, 5).map((raw) => {
if (raw.trim() === "-") return null;
const parsed = Number(raw);
if (!Number.isFinite(parsed) || parsed < 0 || parsed > 1) fail(`palette value "${raw}" must be - or 0..1`);
return parsed;
});
return { rgb, values };
});
/** Uniform channel values, then exact-colour palette overrides, on inferred MER bytes. */
const applyMerOverrides = (mer, rgba, alpha, threshold, uniforms, palette) => {
const out = Uint8Array.from(mer);
const lookup = new Map(palette.map(({ rgb, values }) => [rgb.join(","), values]));
let overridden = 0;
for (let i = 0; i < alpha.length; i++) {
if (alpha[i] < threshold) continue;
uniforms.forEach((value, channel) => {
if (value !== null) out[i * 4 + channel] = Math.round(value * 255);
});
const entry = lookup.get(`${rgba[i * 4]},${rgba[i * 4 + 1]},${rgba[i * 4 + 2]}`);
if (!entry) continue;
overridden++;
entry.forEach((value, channel) => {
if (value !== null) out[i * 4 + channel] = Math.round(value * 255);
});
}
return { bytes: out, overridden };
};
/** Coverage and range statistics for a packed MER over island texels. */
const merStats = (mer, alpha, threshold) => {
let count = 0;
let metal = 0;
let emissive = 0;
let roughSum = 0;
let roughMin = 255;
let roughMax = 0;
for (let i = 0; i < alpha.length; i++) {
if (alpha[i] < threshold) continue;
count++;
if (mer[i * 4] >= 128) metal++;
if (mer[i * 4 + 1] >= 128) emissive++;
const rough = mer[i * 4 + 2];
roughSum += rough;
roughMin = Math.min(roughMin, rough);
roughMax = Math.max(roughMax, rough);
}
const pct = (n) => (count ? Number((n / count).toFixed(4)) : 0);
return {
islandTexels: count,
metalPct: pct(metal),
emissivePct: pct(emissive),
roughnessMean: count ? Number((roughSum / count / 255).toFixed(4)) : 0,
roughnessRange: count ? [Number((roughMin / 255).toFixed(4)), Number((roughMax / 255).toFixed(4))] : [0, 0],
};
};
// ---------------------------------------------------------------------------
// GPU passes
// ---------------------------------------------------------------------------
const wgsl = (name) => readFileSync(join(SCRIPT_DIR, "shaders", name), "utf8");
const pair = (text, fallback) => text.split(",").map((value, i) => num(value, fallback[i]));
const uploadTexture = (gpu, width, height, rgba, usage) => {
const tex = texture(gpu, { kind: "2d", size: [width, height], format: "rgba8unorm", usage });
gpu.gpu.queue.writeTexture({ texture: tex.gpu }, rgba, { bytesPerRow: width * 4, rowsPerImage: height }, [width, height]);
return tex;
};
/** Run height-map.wgsl over the albedo; returns the 0..1 field. */
const luminanceHeightGpu = async (gpu, rgba, width, height) => {
const albedoTex = uploadTexture(gpu, width, height, rgba, ["texture_binding", "copy_dst"]);
const heightTex = texture(gpu, { kind: "2d", size: [width, height], format: "rgba8unorm", usage: ["storage_binding", "copy_src"] });
const pass = compute(gpu, wgsl("height-map.wgsl"), {
label: "height-map",
set: { albedoTex, heightTex, params: { mode: opts.luma === "average" ? 1 : 0, invert: opts["invert-height"] ? 1 : 0 } },
});
pass.dispatch(Math.ceil(width / 8), Math.ceil(height / 8));
const bytes = await heightTex.read({ mipLevel: 0, region: "all" });
albedoTex.destroy();
heightTex.destroy();
return Float32Array.from({ length: width * height }, (_, i) => bytes[i * 4] / 255);
};
/** Run normal-map.wgsl over the quantised height (+ alpha mask); returns RGBA bytes. */
const normalFromHeightGpu = async (gpu, heightRgba, width, height) => {
const heightTex = uploadTexture(gpu, width, height, heightRgba, ["texture_binding", "copy_dst"]);
const normalTex = texture(gpu, { kind: "2d", size: [width, height], format: "rgba8unorm", usage: ["storage_binding", "copy_src"] });
const pass = compute(gpu, wgsl("normal-map.wgsl"), {
label: "normal-map",
set: {
heightTex,
normalTex,
params: {
strength: num(opts.strength, 0.5),
operator_type: OPERATORS[opts.operator],
wrap: opts.wrap ? 1 : 0,
convention: CONVENTIONS[opts.convention],
alpha_threshold: clamp(num(opts["alpha-threshold"], 8), 0, 255) / 255,
},
},
});
pass.dispatch(Math.ceil(width / 8), Math.ceil(height / 8));
const bytes = await normalTex.read({ mipLevel: 0, region: "all" });
heightTex.destroy();
normalTex.destroy();
return bytes;
};
/** Run mer-map.wgsl over albedo + height (with island alpha) + normal; returns inferred RGBA bytes. */
const merFromMapsGpu = async (gpu, rgba, heightRgba, normalRgba, width, height) => {
const usage = ["texture_binding", "copy_dst"];
const albedoTex = uploadTexture(gpu, width, height, rgba, usage);
const heightTex = uploadTexture(gpu, width, height, heightRgba, usage);
const normalTex = uploadTexture(gpu, width, height, normalRgba, usage);
const merTex = texture(gpu, { kind: "2d", size: [width, height], format: "rgba8unorm", usage: ["storage_binding", "copy_src"] });
const [goldMin, goldMax] = pair(opts["gold-hue"], [0.11, 0.17]);
const [copperMin, copperMax] = pair(opts["copper-hue"], [0.03, 0.1]);
const [roughVariance, roughEdge, roughSaturation] = pair(opts["rough-weights"], [0.4, 0.3, 0.3]);
const pass = compute(gpu, wgsl("mer-map.wgsl"), {
label: "mer-map",
set: {
albedoTex,
heightTex,
normalTex,
merTex,
params: {
metal_brightness: num(opts["metal-brightness"], 0.6),
metal_saturation: num(opts["metal-saturation"], 0.15),
gold_min: goldMin,
gold_max: goldMax,
copper_min: copperMin,
copper_max: copperMax,
rough_variance: roughVariance,
rough_edge: roughEdge,
rough_saturation: roughSaturation,
rough_curvature: num(opts["rough-curvature"], 0.25),
rough_height: clamp(num(opts["rough-height"], 0), -1, 1),
rough_scale: num(opts["rough-scale"], 1),
rough_bias: num(opts["rough-bias"], 0),
emissive_brightness: num(opts["emissive-brightness"], 0.8),
emissive_saturation: num(opts["emissive-saturation"], 0.5),
emissive_overbright: num(opts["emissive-overbright"], 1),
emissive_softness: clamp(num(opts["emissive-softness"], 0), 0, 1),
wrap: opts.wrap ? 1 : 0,
convention: CONVENTIONS[opts.convention],
alpha_threshold: clamp(num(opts["alpha-threshold"], 8), 0, 255) / 255,
metal_clean: opts["no-metal-clean"] ? 0 : 1,
},
},
});
pass.dispatch(Math.ceil(width / 8), Math.ceil(height / 8));
const bytes = await merTex.read({ mipLevel: 0, region: "all" });
[albedoTex, heightTex, normalTex, merTex].forEach((tex) => tex.destroy());
return bytes;
};
// ---------------------------------------------------------------------------
// Validation (port of the normal-map classifier used by the reference tool)
// ---------------------------------------------------------------------------
const validateNormalMap = (bytes) => {
const total = bytes.length / 4;
let normalized = 0;
let zZero = 0;
let xyTooLong = 0;
let negativeZ = 0;
let flat = 0;
let grayscale = 0;
let sumX = 0;
let sumY = 0;
for (let i = 0; i < bytes.length; i += 4) {
const [r, g, b] = [bytes[i], bytes[i + 1], bytes[i + 2]];
if (r === g && g === b) grayscale++;
if (Math.abs(r - 128) <= 1 && Math.abs(g - 128) <= 1 && Math.abs(b - 255) <= 1) flat++;
const x = (r / 255) * 2 - 1;
const y = (g / 255) * 2 - 1;
const z = (b / 255) * 2 - 1;
sumX += x;
sumY += y;
if (Math.hypot(x, y) > 1 + 1 / 255) xyTooLong++;
if (b < 128) negativeZ++;
if (Math.abs(Math.hypot(x, y, z) - 1) <= 0.1) normalized++;
if (Math.abs(Math.hypot(x, y, b / 255) - 1) <= 0.1) zZero++;
}
const normalizedPct = normalized / total;
const classification =
grayscale / total >= 0.95 ? "heightfield" : normalizedPct >= 0.8 ? "standard" : zZero / total >= 0.8 ? "z-zero" : "xy-only";
const meanXYBias = [sumX / total, sumY / total];
const failing = classification !== "standard" || xyTooLong / total > 0.05 || negativeZ / total > 0.05 || normalizedPct < 0.5;
const warning = xyTooLong > 0 || negativeZ > 0 || normalizedPct < 0.98 || Math.abs(meanXYBias[0]) > 0.1 || Math.abs(meanXYBias[1]) > 0.1;
return {
classification,
total,
normalizedPct: Number(normalizedPct.toFixed(4)),
flatPct: Number((flat / total).toFixed(4)),
negativeZCount: negativeZ,
xyTooLongCount: xyTooLong,
meanXYBias: meanXYBias.map((v) => Number(v.toFixed(4))),
verdict: failing ? "fail" : warning ? "warn" : "ok",
};
};
// ---------------------------------------------------------------------------
// Main
// ---------------------------------------------------------------------------
const albedoPath = resolve(positionals[0]);
if (!existsSync(albedoPath)) fail(`albedo not found: ${albedoPath}`);
const { width, height, rgba } = readPng(albedoPath);
const outDir = resolve(opts.out ?? dirname(albedoPath));
const name = opts.name ?? basename(albedoPath, extname(albedoPath));
mkdirSync(outDir, { recursive: true });
const alpha = Uint8Array.from({ length: width * height }, (_, i) => rgba[i * 4 + 3]);
const source = opts["height-source"];
const isDepth = source === "depth";
const flatten = num(opts.flatten, isDepth ? 1 : 0);
const normalize = opts.normalize ?? (isDepth && !opts["no-normalize"]);
const gpu = await init({ adapter: opts.gpu });
try {
let field;
let depthInfo = null;
if (source === "luminance") field = await luminanceHeightGpu(gpu, rgba, width, height);
if (source === "file") field = loadHeightFile(opts["height-file"], width, height);
if (isDepth) {
const estimate = await estimateDepth(rgba, width, height);
field = estimate.field;
depthInfo = { model: estimate.model, upscaleFactor: estimate.factor, device: opts["depth-device"], dtype: opts["depth-dtype"] };
}
field = editDepth(field, width, height, {
flatten,
highPass: num(opts["high-pass"], 0),
invert: isDepth || source === "file" ? opts["invert-height"] : false,
normalize,
});
if (opts.seam === "offset") field = sealOffset(field, width, height, clamp(num(opts["seam-amount"], 0.15), 0, 0.5));
const softenSigma = densitySigma(width, height, num(opts.soften, 0));
if (softenSigma > 0.05) field = gaussianBlur(field, width, height, softenSigma, opts.wrap);
const [black, white, gamma] = opts.levels.split(",").map((v, i) => num(v, [0, 1, 1][i]));
if (black !== 0 || white !== 1 || gamma !== 1) field = applyLevels(field, black, white, gamma);
const heightRgba = grayToRgba(field, alpha);
const heightPath = join(outDir, `${name}_height.png`);
if (!opts["no-height"]) writePng(heightPath, width, height, grayToRgba(field));
const normalRgba = await normalFromHeightGpu(gpu, heightRgba, width, height);
const normalPath = join(outDir, `${name}_normal.png`);
writePng(normalPath, width, height, normalRgba);
let merPath = null;
let merReport = null;
if (!opts["no-mer"]) {
const threshold = clamp(num(opts["alpha-threshold"], 8), 0, 255);
const uniforms = [uniformChannel(opts.metal, "metal"), uniformChannel(opts.emissive, "emissive"), uniformChannel(opts.roughness, "roughness")];
const palette = parsePalette(opts.palette);
const inferred = await merFromMapsGpu(gpu, rgba, heightRgba, normalRgba, width, height);
const { bytes: merRgba, overridden } = applyMerOverrides(inferred, rgba, alpha, threshold, uniforms, palette);
merPath = join(outDir, `${name}_mer.png`);
writePng(merPath, width, height, merRgba);
merReport = {
...merStats(merRgba, alpha, threshold),
paletteEntries: palette.length,
paletteTexels: overridden,
uniform: { metal: uniforms[0], emissive: uniforms[1], roughness: uniforms[2] },
};
}
console.log(
JSON.stringify(
{
albedo: albedoPath,
size: [width, height],
adapter: gpu.adapter,
heightSource: source,
depth: depthInfo,
height: opts["no-height"] ? null : heightPath,
normal: normalPath,
mer: merPath,
settings: {
operator: opts.operator,
strength: num(opts.strength, 0.5),
convention: opts.convention,
wrap: opts.wrap,
flatten,
highPass: num(opts["high-pass"], 0),
normalize,
seam: opts.seam,
softenSigma: Number(softenSigma.toFixed(3)),
levels: [black, white, gamma],
},
report: validateNormalMap(normalRgba),
merReport,
},
null,
2,
),
);
} finally {
gpu.dispose();
}
scripts/albedo_to_normal.py
#!/usr/bin/env python3
"""albedo_to_normal.py — albedo PNG to height + tangent-space normal PNGs with PyPBR.
python albedo_to_normal.py <albedo.png> [--out DIR] [--name NAME] [options]
Height sources (``--height-source``):
luminance BT.601 luma of the albedo (default)
depth Depth Anything V2 through ``transformers`` (optional dependency)
file an existing grayscale height PNG (``--height-file``)
The height is shaped in float space (depth flatten / high-pass / normalize / seam
heal, density softening, levels), quantised to 8 bits and written as
``<name>_height.png``. The normal is derived from that 8-bit height with a
Sobel / Prewitt / Scharr kernel (edge-clamped or wrapped, alpha-aware for atlas
islands), stored on a PyPBR material and exported as ``<name>_normal.png``.
A packed MER (``<name>_mer.png``: R metalness, G emissive, B roughness) is
inferred from the albedo, the height and the normal's curvature, with uniform
values and per-colour palette overrides applied afterwards. ``--material-folder``
additionally writes the PyPBR material folder (``albedo.png``, ``height.png``,
``normal.png``, ``metallic.png``, ``roughness.png``, ``emissive.png``). A
validation report is printed as JSON.
Dependencies: pypbr (pulls torch, torchvision, numpy, Pillow). Optional:
transformers for ``--height-source depth``.
The kernel maths mirrors ``albedo_to_normal.mjs`` / ``shaders/normal-map.wgsl`` so
both runtimes produce the same maps from the same height.
"""
from __future__ import annotations
import argparse
import json
import math
import sys
from pathlib import Path
import numpy as np
import torch
import torch.nn.functional as F
from PIL import Image
try:
from pypbr.materials import BasecolorMetallicMaterial
from pypbr.utils.enums import NormalConvention
except ImportError: # pragma: no cover - dependency guard
sys.exit("error: pypbr is required (`pip install pypbr`)")
OPERATOR_WEIGHTS = {"sobel": (1.0, 2.0), "prewitt": (1.0, 1.0), "scharr": (3.0, 10.0)}
DEPTH_MODELS = {
"small": "depth-anything/Depth-Anything-V2-Small-hf",
"base": "depth-anything/Depth-Anything-V2-Base-hf",
"large": "depth-anything/Depth-Anything-V2-Large-hf",
}
DEPTH_MIN_INPUT = 1024
DEPTH_MAX_INPUT = 4096
DEPTH_DOWNSCALE_SHARPEN = 0.5
BASE_FACE_TEXELS = 16
MAX_SOFTEN_SIGMA = 8.0
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("albedo", help="albedo/base-color PNG")
parser.add_argument("--out", help="output directory (default: next to the albedo)")
parser.add_argument("--name", help="output stem (default: albedo file stem)")
parser.add_argument("--height-source", choices=("luminance", "depth", "file"), default="luminance")
parser.add_argument("--height-file", help="grayscale height PNG for --height-source file")
parser.add_argument("--luma", choices=("bt601", "average"), default="bt601")
parser.add_argument("--invert-height", action="store_true", help="dark reads as high")
parser.add_argument("--depth-model", choices=tuple(DEPTH_MODELS), default="small")
parser.add_argument("--depth-device", default="auto", help="auto|cpu|cuda|mps")
parser.add_argument("--flatten", type=float, help="plane-removal strength 0..1 (depth default 1)")
parser.add_argument("--high-pass", type=float, default=0.0, help="wrap-aware high-pass radius in texels")
parser.add_argument("--normalize", dest="normalize", action="store_true", default=None)
parser.add_argument("--no-normalize", dest="normalize", action="store_false")
parser.add_argument("--seam", choices=("off", "offset"), default="off")
parser.add_argument("--seam-amount", type=float, default=0.15)
parser.add_argument("--soften", type=float, default=0.0, help="density-scaled gaussian multiplier, 0 = off")
parser.add_argument("--levels", default="0,1,1", help="BLACK,WHITE,GAMMA input levels")
parser.add_argument("--operator", choices=tuple(OPERATOR_WEIGHTS), default="sobel")
parser.add_argument("--strength", type=float, default=0.5)
parser.add_argument("--convention", choices=("directx", "opengl"), default="directx")
parser.add_argument("--wrap", action="store_true", help="tiling texture: sample across the border")
parser.add_argument("--alpha-threshold", type=int, default=8, help="0..255; texels below are off-island")
parser.add_argument("--no-height", action="store_true")
parser.add_argument("--material-folder", help="also save the PyPBR material folder here")
mer = parser.add_argument_group("MER inference")
mer.add_argument("--no-mer", action="store_true", help="skip the packed MER")
mer.add_argument("--metal", default="auto", help="auto or a uniform 0..1 value")
mer.add_argument("--emissive", default="auto", help="auto or a uniform 0..1 value")
mer.add_argument("--roughness", default="auto", help="auto or a uniform 0..1 value")
mer.add_argument("--metal-brightness", type=float, default=0.6)
mer.add_argument("--metal-saturation", type=float, default=0.15)
mer.add_argument("--no-metal-clean", action="store_true", help="skip the 3x3 majority vote that removes metal speckle")
mer.add_argument("--gold-hue", default="0.11,0.17")
mer.add_argument("--copper-hue", default="0.03,0.10")
mer.add_argument("--rough-weights", default="0.4,0.3,0.3", help="VARIANCE,EDGE,SATURATION weights")
mer.add_argument("--rough-curvature", type=float, default=0.25, help="crevices rougher / ridges smoother, from the normal")
mer.add_argument("--rough-height", type=float, default=0.0, help="-1..1; positive = recesses rougher")
mer.add_argument("--rough-scale", type=float, default=1.0)
mer.add_argument("--rough-bias", type=float, default=0.0)
mer.add_argument("--emissive-brightness", type=float, default=0.8)
mer.add_argument("--emissive-saturation", type=float, default=0.5)
mer.add_argument("--emissive-overbright", type=float, default=1.0, help="luminance above this glows; >= 1 disables")
mer.add_argument("--emissive-softness", type=float, default=0.0)
mer.add_argument("--palette", default="", help='per-colour overrides "#RRGGBB=m,e,r;..." ("-" keeps inferred)')
args = parser.parse_args()
if args.height_source == "file" and not args.height_file:
parser.error("--height-file is required with --height-source file")
return args
# ---------------------------------------------------------------------------
# Height shaping (float tensors, shape (H, W), values 0..1)
# ---------------------------------------------------------------------------
def luminance_height(rgb: torch.Tensor, mode: str, invert: bool) -> torch.Tensor:
"""Height from the stored 8-bit albedo (no sRGB linearisation), like the GPU pass."""
weights = torch.tensor([0.299, 0.587, 0.114]) if mode == "bt601" else torch.full((3,), 1 / 3)
height = (rgb * weights.view(3, 1, 1)).sum(0)
return 1 - height if invert else height
def depth_upscale_factor(width: int, height: int) -> int:
needed = math.ceil(DEPTH_MIN_INPUT / min(width, height))
cap = DEPTH_MAX_INPUT // max(width, height)
return max(1, min(needed, cap))
def box_blur_clamped(field: torch.Tensor) -> torch.Tensor:
padded = F.pad(field[None, None], (1, 1, 1, 1), mode="replicate")
return F.avg_pool2d(padded, 3, stride=1)[0, 0]
def sharpen(field: torch.Tensor, amount: float) -> torch.Tensor:
if amount <= 0:
return field
return (field + amount * (field - box_blur_clamped(field))).clamp(0, 1)
def estimate_depth(albedo: Image.Image, model_key: str, device: str) -> tuple[torch.Tensor, int, str]:
"""Depth Anything V2; returns (0..1 field with 1 = nearest at native size, factor, model id)."""
try:
from transformers import pipeline
except ImportError:
sys.exit("error: --height-source depth needs `pip install transformers`")
resolved = device
if device == "auto":
resolved = "cuda" if torch.cuda.is_available() else "cpu"
model_id = DEPTH_MODELS[model_key]
estimator = pipeline("depth-estimation", model=model_id, device=resolved)
width, height = albedo.size
factor = depth_upscale_factor(width, height)
rgb = albedo.convert("RGB")
upscaled = rgb.resize((width * factor, height * factor), Image.NEAREST) if factor > 1 else rgb
depth = estimator(upscaled)["depth"] # PIL "L", 255 = nearest, at the input size
full = torch.from_numpy(np.asarray(depth, dtype=np.float32) / 255.0)
if factor == 1:
return full, factor, model_id
native = F.avg_pool2d(full[None, None], factor)[0, 0]
return sharpen(native, DEPTH_DOWNSCALE_SHARPEN), factor, model_id
def fit_plane(field: torch.Tensor) -> tuple[float, float]:
height, width = field.shape
xs = torch.arange(width, dtype=torch.float32) - (width - 1) / 2
ys = torch.arange(height, dtype=torch.float32) - (height - 1) / 2
gx = float((field * xs).sum() / (xs.pow(2).sum() * height)) if width > 1 else 0.0
gy = float((field * ys[:, None]).sum() / (ys.pow(2).sum() * width)) if height > 1 else 0.0
return gx, gy
def blur_wrap(field: torch.Tensor, radius: float) -> torch.Tensor:
"""Wrap-around box blur ×3 (≈ Gaussian), tile-aware on both axes."""
r = max(1, round(radius))
kernel = torch.full((1, 1, 1, 2 * r + 1), 1 / (2 * r + 1))
out = field[None, None]
for _ in range(3):
out = F.conv2d(F.pad(out, (r, r, 0, 0), mode="circular"), kernel)
out = F.conv2d(F.pad(out, (0, 0, r, r), mode="circular"), kernel.transpose(2, 3))
return out[0, 0]
def edit_depth(field: torch.Tensor, flatten: float, high_pass: float, invert: bool, normalize: bool) -> torch.Tensor:
"""Depth edit chain: flatten, high-pass, invert, normalize."""
out = field.clone()
height, width = out.shape
if flatten > 0:
gx, gy = fit_plane(out)
xs = torch.arange(width, dtype=torch.float32) - (width - 1) / 2
ys = torch.arange(height, dtype=torch.float32) - (height - 1) / 2
out = out - flatten * (gx * xs[None, :] + gy * ys[:, None])
if high_pass > 0:
out = out - blur_wrap(out, high_pass) + 0.5
if invert:
out = 1 - out
if normalize:
span = float(out.max() - out.min())
if span > 1e-6:
out = (out - out.min()) / span
return out.clamp(0, 1)
def seal_offset(field: torch.Tensor, amount: float) -> torch.Tensor:
"""Wrap-and-heal: shift by half a tile, blend the original back over the centre seam."""
height, width = field.shape
half_w, half_h = width // 2, height // 2
if half_w < 1 or half_h < 1:
return field
reach = min(1.0, max(1e-4, amount * 2))
span_x = max(1, (width - 1) // 2)
span_y = max(1, (height - 1) // 2)
xs = torch.arange(width, dtype=torch.float32)
ys = torch.arange(height, dtype=torch.float32)
nx = (1 - torch.minimum(xs, width - 1 - xs) / span_x).clamp(0, 1)
ny = (1 - torch.minimum(ys, height - 1 - ys) / span_y).clamp(0, 1)
distance = torch.sqrt(nx[None, :] ** 2 + ny[:, None] ** 2).clamp(max=1)
ramp = (distance / reach).clamp(0, 1)
t = torch.where(distance >= reach, torch.ones_like(ramp), ramp * ramp * (3 - 2 * ramp))
shifted = torch.roll(field, shifts=(half_h, half_w), dims=(0, 1))
# Height-steered blend weight: the taller surface wins the middle of the transition.
window = 4 * t * (1 - t)
floor = torch.maximum(field, shifted) - 0.1
w_a = (field - floor).clamp(min=0)
w_b = (shifted - floor).clamp(min=0)
picked = torch.where(w_a + w_b > 0, w_b / (w_a + w_b).clamp(min=1e-9), t)
w = t + (picked - t) * 0.5 * window
return shifted * w + field * (1 - w)
def density_sigma(width: int, height: int, softness: float) -> float:
return min((min(width, height) / BASE_FACE_TEXELS / 4) * softness, MAX_SOFTEN_SIGMA)
def gaussian_blur(field: torch.Tensor, sigma: float, wrap: bool) -> torch.Tensor:
if sigma <= 0:
return field
radius = max(1, math.ceil(sigma * 3))
taps = torch.arange(-radius, radius + 1, dtype=torch.float32)
weights = torch.exp(-(taps**2) / (2 * sigma * sigma))
kernel = (weights / weights.sum()).view(1, 1, 1, -1)
mode = "circular" if wrap else "replicate"
out = F.conv2d(F.pad(field[None, None], (radius, radius, 0, 0), mode=mode), kernel)
out = F.conv2d(F.pad(out, (0, 0, radius, radius), mode=mode), kernel.transpose(2, 3))
return out[0, 0]
def apply_levels(field: torch.Tensor, black: float, white: float, gamma: float) -> torch.Tensor:
span = max(white - black, 0.001)
return ((field - black).clamp(0, span) / span).pow(1 / max(gamma, 0.001))
# ---------------------------------------------------------------------------
# Normal derivation (mirrors shaders/normal-map.wgsl)
# ---------------------------------------------------------------------------
def shift(field: torch.Tensor, dx: int, dy: int, wrap: bool) -> torch.Tensor:
"""Neighbour lookup: value at (x + dx, y + dy), wrapped or edge-clamped."""
if wrap:
return torch.roll(field, shifts=(-dy, -dx), dims=(0, 1))
pad = max(abs(dx), abs(dy), 1)
padded = F.pad(field[None, None].float(), (pad, pad, pad, pad), mode="replicate")[0, 0]
height, width = field.shape
return padded[pad + dy : pad + dy + height, pad + dx : pad + dx + width]
def height_to_normal(
height8: torch.Tensor,
island: torch.Tensor,
operator: str,
strength: float,
convention: str,
wrap: bool,
) -> torch.Tensor:
"""Tangent-space normal in [-1, 1], shape (3, H, W); off-island texels are flat."""
corner, edge = OPERATOR_WEIGHTS[operator]
def tap(dx: int, dy: int) -> torch.Tensor:
neighbour = shift(height8, dx, dy, wrap)
on_island = shift(island, dx, dy, wrap) > 0.5
return torch.where(on_island, neighbour, height8)
s00, s01, s02 = tap(-1, -1), tap(0, -1), tap(1, -1)
s10, s12 = tap(-1, 0), tap(1, 0)
s20, s21, s22 = tap(-1, 1), tap(0, 1), tap(1, 1)
dx = -corner * s00 + corner * s02 - edge * s10 + edge * s12 - corner * s20 + corner * s22
dy = corner * s00 + edge * s01 + corner * s02 - corner * s20 - edge * s21 - corner * s22
nx = -dx * strength
ny = -dy * strength if convention == "opengl" else dy * strength
normal = F.normalize(torch.stack([nx, ny, torch.ones_like(nx)]), dim=0)
flat = torch.tensor([0.0, 0.0, 1.0]).view(3, 1, 1).expand_as(normal)
return torch.where((island > 0.5)[None], normal, flat)
# ---------------------------------------------------------------------------
# MER inference (mirrors shaders/mer-map.wgsl)
# ---------------------------------------------------------------------------
def rgb_to_hsv(rgb: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""Per-texel hue (0..1), saturation and value from a (3, H, W) tensor."""
r, g, b = rgb[0], rgb[1], rgb[2]
c_max = rgb.max(0).values
c_min = rgb.min(0).values
delta = c_max - c_min
safe = delta.clamp(min=1e-9)
h_r = torch.remainder((g - b) / safe, 6.0)
h_g = (b - r) / safe + 2.0
h_b = (r - g) / safe + 4.0
h = torch.where(c_max == r, h_r, torch.where(c_max == g, h_g, h_b)) / 6.0
h = torch.where(h < 0, h + 1.0, h)
h = torch.where(delta > 1e-5, h, torch.zeros_like(h))
s = torch.where(c_max > 0, delta / c_max.clamp(min=1e-9), torch.zeros_like(c_max))
return h, s, c_max
def smoothstep(edge0: float, edge1: float, x: torch.Tensor) -> torch.Tensor:
t = ((x - edge0) / max(edge1 - edge0, 1e-9)).clamp(0, 1)
return t * t * (3 - 2 * t)
def island_tap(field: torch.Tensor, island: torch.Tensor, dx: int, dy: int, wrap: bool) -> torch.Tensor:
"""Neighbour value, or the centre value when the neighbour is off-island."""
return torch.where(shift(island, dx, dy, wrap) > 0.5, shift(field, dx, dy, wrap), field)
def infer_mer(rgb: torch.Tensor, height8: torch.Tensor, normal: torch.Tensor, island: torch.Tensor, args: argparse.Namespace) -> torch.Tensor:
"""Packed MER in 0..1, shape (3, H, W): metalness, emissive, roughness."""
wrap = args.wrap
h, s, v = rgb_to_hsv(rgb)
luma = luminance_height(rgb, "bt601", False)
gold_min, gold_max = (float(x) for x in args.gold_hue.split(","))
copper_min, copper_max = (float(x) for x in args.copper_hue.split(","))
achromatic = smoothstep(args.metal_brightness - 0.1, args.metal_brightness, v) * (1 - smoothstep(args.metal_saturation - 0.05, args.metal_saturation, s))
gold = smoothstep(gold_min - 0.01, gold_min, h) * (1 - smoothstep(gold_max, gold_max + 0.01, h)) * (s >= 0.3) * (v >= 0.5)
copper = smoothstep(copper_min - 0.01, copper_min, h) * (1 - smoothstep(copper_max, copper_max + 0.01, h)) * (s >= 0.3) * (v >= 0.4)
metal = (torch.maximum(achromatic, torch.maximum(gold, copper)) >= 0.5).float()
if not args.no_metal_clean:
votes = torch.stack([island_tap(metal, island, dx, dy, wrap) for dy in (-1, 0, 1) for dx in (-1, 0, 1)]).sum(0)
metal = (votes >= 5).float()
bright_saturated = smoothstep(args.emissive_brightness - 0.05, args.emissive_brightness, v) * smoothstep(args.emissive_saturation - 0.05, args.emissive_saturation, s)
over = args.emissive_overbright
exceeds = smoothstep(over - 0.05, over, luma) if over < 1.0 else torch.zeros_like(luma)
emissive = torch.maximum(bright_saturated, exceeds)
if args.emissive_softness > 0:
taps = [island_tap(emissive, island, dx, dy, wrap) for dy in (-1, 0, 1) for dx in (-1, 0, 1)]
emissive = torch.lerp(emissive, torch.stack(taps).mean(0), min(1.0, args.emissive_softness))
w_var, w_edge, w_sat = (float(x) for x in args.rough_weights.split(","))
taps = torch.stack([island_tap(luma, island, dx, dy, wrap) for dy in range(-2, 3) for dx in range(-2, 3)])
variance = (taps.pow(2).mean(0) - taps.mean(0).pow(2)).clamp(min=0)
variance_term = (variance * 10).clamp(0, 1)
l = {(dx, dy): island_tap(luma, island, dx, dy, wrap) for dy in (-1, 0, 1) for dx in (-1, 0, 1)}
gx = -l[(-1, -1)] + l[(1, -1)] - 2 * l[(-1, 0)] + 2 * l[(1, 0)] - l[(-1, 1)] + l[(1, 1)]
gy = l[(-1, -1)] + 2 * l[(0, -1)] + l[(1, -1)] - l[(-1, 1)] - 2 * l[(0, 1)] - l[(1, 1)]
edge_term = (torch.sqrt(gx * gx + gy * gy) * 2).clamp(0, 1)
saturation_term = 1 - s * 0.5
nx, ny = normal[0], normal[1]
ysign = -1.0 if args.convention == "opengl" else 1.0
ddx = island_tap(nx, island, 1, 0, wrap) - island_tap(nx, island, -1, 0, wrap)
ddy = island_tap(ny, island, 0, 1, wrap) - island_tap(ny, island, 0, -1, wrap)
curvature_term = (-(0.5 * (ddx + ysign * ddy)) * 2).clamp(-1, 1)
roughness = variance_term * w_var + edge_term * w_edge + saturation_term * w_sat + curvature_term * args.rough_curvature
roughness = (roughness * args.rough_scale + args.rough_bias).clamp(0, 1)
influence = max(-1.0, min(1.0, args.rough_height))
if influence != 0:
field = 1 - height8 if influence > 0 else height8
roughness = torch.lerp(roughness, field, abs(influence))
mer = torch.stack([metal, emissive.clamp(0, 1), roughness])
off = torch.tensor([0.0, 0.0, 1.0]).view(3, 1, 1).expand_as(mer)
return torch.where((island > 0.5)[None], mer, off)
def uniform_channel(value: str, flag: str) -> float | None:
"""'auto' → None, otherwise a validated 0..1 uniform value."""
if value == "auto":
return None
try:
parsed = float(value)
except ValueError:
sys.exit(f"error: --{flag} must be auto or 0..1")
if not 0 <= parsed <= 1:
sys.exit(f"error: --{flag} must be auto or 0..1")
return parsed
def parse_palette(text: str) -> list[tuple[tuple[int, int, int], list[float | None]]]:
"""Parse '#RRGGBB=m,e,r;...' into [((r, g, b), [m|None, e|None, r|None])]."""
entries = []
for raw in (e.strip() for e in text.split(";") if e.strip()):
try:
colour, values = raw.split("=")
colour = colour.strip().lstrip("#")
parts = [p.strip() for p in values.split(",")]
if len(colour) != 6 or len(parts) != 3:
raise ValueError
rgb = tuple(int(colour[i : i + 2], 16) for i in (0, 2, 4))
channels = [None if p == "-" else float(p) for p in parts]
except ValueError:
sys.exit(f'error: bad --palette entry "{raw}" (expected #RRGGBB=m,e,r)')
if any(c is not None and not 0 <= c <= 1 for c in channels):
sys.exit(f'error: palette values in "{raw}" must be - or 0..1')
entries.append((rgb, channels))
return entries
def apply_mer_overrides(
mer8: np.ndarray,
rgb8: np.ndarray,
island: np.ndarray,
uniforms: list[float | None],
palette: list[tuple[tuple[int, int, int], list[float | None]]],
) -> tuple[np.ndarray, int]:
"""Uniform channel values, then exact-colour palette overrides, on (H, W, 3) MER bytes."""
out = mer8.copy()
for channel, value in enumerate(uniforms):
if value is not None:
out[..., channel] = np.where(island, round(value * 255), out[..., channel])
overridden = 0
for rgb, channels in palette:
match = island & np.all(rgb8 == np.array(rgb, dtype=np.uint8), axis=-1)
overridden += int(match.sum())
for channel, value in enumerate(channels):
if value is not None:
out[..., channel] = np.where(match, round(value * 255), out[..., channel])
return out, overridden
def mer_stats(mer8: np.ndarray, island: np.ndarray) -> dict:
"""Coverage and range statistics for a packed MER over island texels."""
count = int(island.sum())
if count == 0:
return {"islandTexels": 0, "metalPct": 0, "emissivePct": 0, "roughnessMean": 0, "roughnessRange": [0, 0]}
metal = int((mer8[..., 0][island] >= 128).sum())
emissive = int((mer8[..., 1][island] >= 128).sum())
rough = mer8[..., 2][island].astype(np.float32) / 255
return {
"islandTexels": count,
"metalPct": round(metal / count, 4),
"emissivePct": round(emissive / count, 4),
"roughnessMean": round(float(rough.mean()), 4),
"roughnessRange": [round(float(rough.min()), 4), round(float(rough.max()), 4)],
}
# ---------------------------------------------------------------------------
# Validation (port of the normal-map classifier used by the reference tool)
# ---------------------------------------------------------------------------
def validate_normal_map(rgb8: np.ndarray) -> dict:
flat_rgb = rgb8.reshape(-1, 3).astype(np.float32)
total = flat_rgb.shape[0]
r, g, b = flat_rgb[:, 0], flat_rgb[:, 1], flat_rgb[:, 2]
grayscale = int(((r == g) & (g == b)).sum())
flat = int(((np.abs(r - 128) <= 1) & (np.abs(g - 128) <= 1) & (np.abs(b - 255) <= 1)).sum())
x, y, z = r / 255 * 2 - 1, g / 255 * 2 - 1, b / 255 * 2 - 1
xy_len = np.hypot(x, y)
xy_too_long = int((xy_len > 1 + 1 / 255).sum())
negative_z = int((b < 128).sum())
normalized = int((np.abs(np.sqrt(x * x + y * y + z * z) - 1) <= 0.1).sum())
z_zero = int((np.abs(np.sqrt(x * x + y * y + (b / 255) ** 2) - 1) <= 0.1).sum())
normalized_pct = normalized / total
if grayscale / total >= 0.95:
classification = "heightfield"
elif normalized_pct >= 0.8:
classification = "standard"
elif z_zero / total >= 0.8:
classification = "z-zero"
else:
classification = "xy-only"
bias = [float(x.mean()), float(y.mean())]
failing = classification != "standard" or xy_too_long / total > 0.05 or negative_z / total > 0.05 or normalized_pct < 0.5
warning = xy_too_long > 0 or negative_z > 0 or normalized_pct < 0.98 or abs(bias[0]) > 0.1 or abs(bias[1]) > 0.1
return {
"classification": classification,
"total": total,
"normalizedPct": round(normalized_pct, 4),
"flatPct": round(flat / total, 4),
"negativeZCount": negative_z,
"xyTooLongCount": xy_too_long,
"meanXYBias": [round(v, 4) for v in bias],
"verdict": "fail" if failing else "warn" if warning else "ok",
}
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main() -> None:
args = parse_args()
albedo_path = Path(args.albedo).resolve()
if not albedo_path.exists():
sys.exit(f"error: albedo not found: {albedo_path}")
out_dir = Path(args.out).resolve() if args.out else albedo_path.parent
out_dir.mkdir(parents=True, exist_ok=True)
name = args.name or albedo_path.stem
albedo = Image.open(albedo_path).convert("RGBA")
width, height = albedo.size
rgba = torch.from_numpy(np.asarray(albedo, dtype=np.float32) / 255.0).permute(2, 0, 1)
rgb, alpha = rgba[:3], rgba[3]
island = (alpha >= args.alpha_threshold / 255).float()
is_depth = args.height_source == "depth"
flatten = args.flatten if args.flatten is not None else (1.0 if is_depth else 0.0)
normalize = args.normalize if args.normalize is not None else is_depth
depth_info = None
if args.height_source == "luminance":
field = luminance_height(rgb, args.luma, args.invert_height)
elif args.height_source == "file":
height_img = Image.open(Path(args.height_file).resolve()).convert("L")
if height_img.size != (width, height):
sys.exit(f"error: height file is {height_img.size[0]}x{height_img.size[1]}, albedo is {width}x{height}")
field = torch.from_numpy(np.asarray(height_img, dtype=np.float32) / 255.0)
else:
field, factor, model_id = estimate_depth(albedo, args.depth_model, args.depth_device)
depth_info = {"model": model_id, "upscaleFactor": factor, "device": args.depth_device}
invert_after = args.invert_height if args.height_source != "luminance" else False
field = edit_depth(field, flatten, args.high_pass, invert_after, normalize)
if args.seam == "offset":
field = seal_offset(field, min(0.5, max(0.0, args.seam_amount)))
soften_sigma = density_sigma(width, height, args.soften)
if soften_sigma > 0.05:
field = gaussian_blur(field, soften_sigma, args.wrap)
black, white, gamma = (float(v) for v in args.levels.split(","))
if (black, white, gamma) != (0.0, 1.0, 1.0):
field = apply_levels(field, black, white, gamma)
# Quantise once; the normal is derived from the same 8-bit height that gets written.
height8 = (field.clamp(0, 1) * 255).round() / 255
normal = height_to_normal(height8, island, args.operator, args.strength, args.convention, args.wrap)
# Round to bytes here (PyPBR's to_pil truncates), and build the PyPBR material from
# those exact bytes so any further PyPBR operation starts from what was written.
height_bytes = (height8 * 255).round().to(torch.uint8).numpy()
normal_bytes = ((normal + 1.0) * 0.5 * 255).round().clamp(0, 255).to(torch.uint8).permute(1, 2, 0).numpy()
height_img = Image.fromarray(height_bytes, mode="L")
normal_img = Image.fromarray(normal_bytes, mode="RGB")
convention = NormalConvention.DIRECTX if args.convention == "directx" else NormalConvention.OPENGL
material = BasecolorMetallicMaterial(albedo=albedo.convert("RGB"), height=height_img, normal_convention=convention)
decoded = torch.from_numpy(normal_bytes.astype(np.float32)).permute(2, 0, 1) / 255 * 2 - 1
# A [-1, 1] tensor is stored verbatim; a fully flat map (min == 0) must go in as an image
# so PyPBR decodes it instead of mistaking it for an already-decoded tensor.
material.normal = decoded if float(decoded.min()) < 0 else normal_img
mer_path = None
mer_report = None
mer_img = None
if not args.no_mer:
uniforms = [uniform_channel(args.metal, "metal"), uniform_channel(args.emissive, "emissive"), uniform_channel(args.roughness, "roughness")]
palette = parse_palette(args.palette)
inferred = infer_mer(rgb, height8, decoded, island, args)
mer8 = (inferred * 255).round().clamp(0, 255).to(torch.uint8).permute(1, 2, 0).numpy()
rgb8 = (rgb * 255).round().to(torch.uint8).permute(1, 2, 0).numpy()
island_np = island.numpy() > 0.5
mer8, overridden = apply_mer_overrides(mer8, rgb8, island_np, uniforms, palette)
mer_img = Image.fromarray(mer8, mode="RGB")
# PyPBR keeps the channels separately; the packed PNG is the Bedrock/Blockbench deliverable.
material.metallic = Image.fromarray(mer8[..., 0], mode="L")
material.emissive = Image.fromarray(mer8[..., 1], mode="L")
material.roughness = Image.fromarray(mer8[..., 2], mode="L")
mer_report = {
**mer_stats(mer8, island_np),
"paletteEntries": len(palette),
"paletteTexels": overridden,
"uniform": {"metal": uniforms[0], "emissive": uniforms[1], "roughness": uniforms[2]},
}
height_path = out_dir / f"{name}_height.png"
normal_path = out_dir / f"{name}_normal.png"
if not args.no_height:
height_img.save(height_path)
normal_img.save(normal_path)
if mer_img is not None:
mer_path = out_dir / f"{name}_mer.png"
mer_img.save(mer_path)
if args.material_folder:
material.save_to_folder(str(Path(args.material_folder).resolve()))
print(
json.dumps(
{
"albedo": str(albedo_path),
"size": [width, height],
"heightSource": args.height_source,
"depth": depth_info,
"height": None if args.no_height else str(height_path),
"normal": str(normal_path),
"mer": str(mer_path) if mer_path else None,
"materialFolder": str(Path(args.material_folder).resolve()) if args.material_folder else None,
"settings": {
"operator": args.operator,
"strength": args.strength,
"convention": args.convention,
"wrap": args.wrap,
"flatten": flatten,
"highPass": args.high_pass,
"normalize": normalize,
"seam": args.seam,
"softenSigma": round(soften_sigma, 3),
"levels": [black, white, gamma],
},
"report": validate_normal_map(normal_bytes),
"merReport": mer_report,
},
indent=2,
)
)
if __name__ == "__main__":
main()
scripts/shaders/height-map.wgsl
// height-map.wgsl — albedo luminance to height (self-contained, no imports).
//
// Height is read straight from the 8-bit albedo as stored (no sRGB linearisation):
// pixel-art luminance steps are the relief the artist painted, and linearising them
// crushes the darks. `mode` picks BT.601 luma (default) or a plain channel average.
// The albedo alpha is carried through in `.a` so the normal pass can treat
// transparent atlas gutters as "outside the island".
struct HeightParams {
mode: u32, // 0 = bt601, 1 = average
invert: u32, // 1 = dark is high
}
@group(0) @binding(0) var albedoTex: texture_2d<f32>;
@group(0) @binding(1) var heightTex: texture_storage_2d<rgba8unorm, write>;
@group(0) @binding(2) var<uniform> params: HeightParams;
fn luminance_bt601(rgb: vec3f) -> f32 {
return dot(rgb, vec3f(0.299, 0.587, 0.114));
}
@compute @workgroup_size(8, 8)
fn main(@builtin(global_invocation_id) gid: vec3u) {
let dims = textureDimensions(albedoTex, 0);
if (gid.x >= dims.x || gid.y >= dims.y) { return; }
let coord = vec2i(gid.xy);
let texel = textureLoad(albedoTex, coord, 0);
var height = select(luminance_bt601(texel.rgb), (texel.r + texel.g + texel.b) / 3.0, params.mode == 1u);
height = select(height, 1.0 - height, params.invert == 1u);
textureStore(heightTex, coord, vec4f(height, height, height, texel.a));
}
scripts/shaders/mer-map.wgsl
// mer-map.wgsl — packed MER (R = metalness, G = emissive, B = roughness) inferred
// from the albedo, the shaped height and the derived normal (self-contained).
//
// Metalness: HSV heuristic. Bright, desaturated texels (steel, iron, silver) and
// two hue bands (gold, copper) with enough saturation and value. Binary output;
// per-colour palette overrides and uniform values are applied on the CPU.
// Emissive: bright AND saturated texels glow (lava, glowstone, neon). An optional
// over-bright fallback flags luminance above `emissive_overbright` regardless of
// saturation (set >= 1 to disable). `emissive_softness` blends toward the 3x3
// mean so the glow outline is anti-aliased instead of stair-stepped.
// Roughness: weighted composite of 5x5 luminance variance (grain), albedo Sobel
// edge magnitude (detail), inverse saturation (pure hues read glossier), a
// curvature term from the normal map's divergence (crevices collect dirt and
// roughen, convex ridges wear smooth) and an optional height influence
// (positive = recesses rougher). Then scale + bias, clamped.
//
// Sampling follows normal-map.wgsl: `wrap` selects periodic edges, and any
// neighbour whose alpha (heightTex.a) is below `alpha_threshold` is off-island
// and replaced by the centre value. Off-island texels emit (0, 0, 1): no metal,
// no glow, fully rough. Alpha is always 1.
struct MerParams {
metal_brightness: f32,
metal_saturation: f32,
gold_min: f32,
gold_max: f32,
copper_min: f32,
copper_max: f32,
rough_variance: f32,
rough_edge: f32,
rough_saturation: f32,
rough_curvature: f32,
rough_height: f32,
rough_scale: f32,
rough_bias: f32,
emissive_brightness: f32,
emissive_saturation: f32,
emissive_overbright: f32,
emissive_softness: f32,
wrap: u32, // 0 = clamp, 1 = wrap
convention: u32, // 0 = directx, 1 = opengl (sign of the normal's Y for curvature)
alpha_threshold: f32, // 0..1; neighbours below this are off-island
metal_clean: u32, // 1 = 3x3 majority vote on the metal mask (removes speckle)
}
@group(0) @binding(0) var albedoTex: texture_2d<f32>;
@group(0) @binding(1) var heightTex: texture_2d<f32>; // r = height, a = island mask
@group(0) @binding(2) var normalTex: texture_2d<f32>;
@group(0) @binding(3) var merTex: texture_storage_2d<rgba8unorm, write>;
@group(0) @binding(4) var<uniform> params: MerParams;
fn luminance_bt601(rgb: vec3f) -> f32 {
return dot(rgb, vec3f(0.299, 0.587, 0.114));
}
fn rgb_to_hsv(rgb: vec3f) -> vec3f {
let c_max = max(rgb.r, max(rgb.g, rgb.b));
let c_min = min(rgb.r, min(rgb.g, rgb.b));
let delta = c_max - c_min;
var h: f32 = 0.0;
if (delta > 0.00001) {
if (c_max == rgb.r) {
h = ((rgb.g - rgb.b) / delta) % 6.0;
} else if (c_max == rgb.g) {
h = (rgb.b - rgb.r) / delta + 2.0;
} else {
h = (rgb.r - rgb.g) / delta + 4.0;
}
h = h / 6.0;
if (h < 0.0) { h += 1.0; }
}
let s = select(0.0, delta / c_max, c_max > 0.0);
return vec3f(h, s, c_max);
}
fn wrap_or_clamp(coord: vec2i, dims: vec2i) -> vec2i {
if (params.wrap == 1u) {
return ((coord % dims) + dims) % dims;
}
return clamp(coord, vec2i(0), dims - vec2i(1));
}
fn on_island(coord: vec2i, dims: vec2i) -> bool {
return textureLoad(heightTex, wrap_or_clamp(coord, dims), 0).a >= params.alpha_threshold;
}
/// Albedo luminance at `coord`, or `centre` when off-island.
fn luma_tap(coord: vec2i, dims: vec2i, centre: f32) -> f32 {
let c = wrap_or_clamp(coord, dims);
let l = luminance_bt601(textureLoad(albedoTex, c, 0).rgb);
return select(centre, l, on_island(coord, dims));
}
/// Decoded normal XY at `coord`, or `centre` when off-island.
fn normal_tap(coord: vec2i, dims: vec2i, centre: vec2f) -> vec2f {
let c = wrap_or_clamp(coord, dims);
let n = textureLoad(normalTex, c, 0).xy * 2.0 - 1.0;
return select(centre, n, on_island(coord, dims));
}
/// Binary metal classification for one texel: bright desaturated, gold or copper.
fn metal_at(coord: vec2i, dims: vec2i) -> f32 {
let c = wrap_or_clamp(coord, dims);
let hsv = rgb_to_hsv(textureLoad(albedoTex, c, 0).rgb);
let achromatic = smoothstep(params.metal_brightness - 0.1, params.metal_brightness, hsv.z)
* (1.0 - smoothstep(params.metal_saturation - 0.05, params.metal_saturation, hsv.y));
let gold = smoothstep(params.gold_min - 0.01, params.gold_min, hsv.x)
* (1.0 - smoothstep(params.gold_max, params.gold_max + 0.01, hsv.x))
* step(0.3, hsv.y) * step(0.5, hsv.z);
let copper = smoothstep(params.copper_min - 0.01, params.copper_min, hsv.x)
* (1.0 - smoothstep(params.copper_max, params.copper_max + 0.01, hsv.x))
* step(0.3, hsv.y) * step(0.4, hsv.z);
return step(0.5, max(achromatic, max(gold, copper)));
}
/// Raw emissive response for one texel (no softness).
fn emissive_at(coord: vec2i, dims: vec2i) -> f32 {
let c = wrap_or_clamp(coord, dims);
let rgb = textureLoad(albedoTex, c, 0).rgb;
let hsv = rgb_to_hsv(rgb);
let bright_saturated = smoothstep(params.emissive_brightness - 0.05, params.emissive_brightness, hsv.z)
* smoothstep(params.emissive_saturation - 0.05, params.emissive_saturation, hsv.y);
let over = params.emissive_overbright;
let exceeds = select(0.0, smoothstep(over - 0.05, over, luminance_bt601(rgb)), over < 1.0);
return max(bright_saturated, exceeds);
}
@compute @workgroup_size(8, 8)
fn main(@builtin(global_invocation_id) gid: vec3u) {
let dims = vec2i(textureDimensions(albedoTex, 0));
if (i32(gid.x) >= dims.x || i32(gid.y) >= dims.y) { return; }
let c = vec2i(gid.xy);
if (!on_island(c, dims)) {
textureStore(merTex, c, vec4f(0.0, 0.0, 1.0, 1.0));
return;
}
let rgb = textureLoad(albedoTex, c, 0).rgb;
let hsv = rgb_to_hsv(rgb);
let centre_luma = luminance_bt601(rgb);
// --- Metalness -----------------------------------------------------------
// Optional 3x3 majority vote: a lone metal texel in a dielectric field (or a
// lone hole in a metal one) is speckle from the threshold, not material.
var metal = metal_at(c, dims);
if (params.metal_clean == 1u) {
var votes = 0.0;
for (var dy = -1; dy <= 1; dy++) {
for (var dx = -1; dx <= 1; dx++) {
let n = c + vec2i(dx, dy);
votes += select(metal, metal_at(n, dims), on_island(n, dims));
}
}
metal = step(5.0, votes);
}
// --- Emissive ------------------------------------------------------------
var emissive = emissive_at(c, dims);
if (params.emissive_softness > 0.0) {
var sum = 0.0;
for (var dy = -1; dy <= 1; dy++) {
for (var dx = -1; dx <= 1; dx++) {
let n = c + vec2i(dx, dy);
sum += select(emissive, emissive_at(n, dims), on_island(n, dims));
}
}
emissive = mix(emissive, sum / 9.0, clamp(params.emissive_softness, 0.0, 1.0));
}
// --- Roughness -----------------------------------------------------------
// 5x5 luminance variance: grain.
var sum = 0.0;
var sum_sq = 0.0;
for (var dy = -2; dy <= 2; dy++) {
for (var dx = -2; dx <= 2; dx++) {
let l = luma_tap(c + vec2i(dx, dy), dims, centre_luma);
sum += l;
sum_sq += l * l;
}
}
let mean = sum / 25.0;
let variance = max(sum_sq / 25.0 - mean * mean, 0.0);
let variance_term = clamp(variance * 10.0, 0.0, 1.0);
// Albedo Sobel edge magnitude: detail.
let s00 = luma_tap(c + vec2i(-1, -1), dims, centre_luma);
let s01 = luma_tap(c + vec2i( 0, -1), dims, centre_luma);
let s02 = luma_tap(c + vec2i( 1, -1), dims, centre_luma);
let s10 = luma_tap(c + vec2i(-1, 0), dims, centre_luma);
let s12 = luma_tap(c + vec2i( 1, 0), dims, centre_luma);
let s20 = luma_tap(c + vec2i(-1, 1), dims, centre_luma);
let s21 = luma_tap(c + vec2i( 0, 1), dims, centre_luma);
let s22 = luma_tap(c + vec2i( 1, 1), dims, centre_luma);
let gx = -s00 + s02 - 2.0 * s10 + 2.0 * s12 - s20 + s22;
let gy = s00 + 2.0 * s01 + s02 - s20 - 2.0 * s21 - s22;
let edge_term = clamp(sqrt(gx * gx + gy * gy) * 2.0, 0.0, 1.0);
// Inverse saturation: pure hues read glossier.
let saturation_term = 1.0 - hsv.y * 0.5;
// Curvature from the normal's divergence. Convex (ridge) is positive with
// DirectX green; OpenGL flips the Y term. Crevices (negative) roughen.
let n_c = textureLoad(normalTex, c, 0).xy * 2.0 - 1.0;
let ysign = select(1.0, -1.0, params.convention == 1u);
let ddx = normal_tap(c + vec2i(1, 0), dims, n_c).x - normal_tap(c + vec2i(-1, 0), dims, n_c).x;
let ddy = normal_tap(c + vec2i(0, 1), dims, n_c).y - normal_tap(c + vec2i(0, -1), dims, n_c).y;
let divergence = 0.5 * (ddx + ysign * ddy);
let curvature_term = clamp(-divergence * 2.0, -1.0, 1.0);
var roughness = variance_term * params.rough_variance
+ edge_term * params.rough_edge
+ saturation_term * params.rough_saturation
+ curvature_term * params.rough_curvature;
roughness = clamp(roughness * params.rough_scale + params.rough_bias, 0.0, 1.0);
// Height influence: positive pulls recesses rough and peaks glossy.
let influence = params.rough_height;
if (influence != 0.0) {
let h = textureLoad(heightTex, c, 0).r;
let field = select(h, 1.0 - h, influence >= 0.0);
roughness = mix(roughness, field, abs(influence));
}
textureStore(merTex, c, vec4f(metal, clamp(emissive, 0.0, 1.0), roughness, 1.0));
}
scripts/shaders/normal-map.wgsl
// normal-map.wgsl — height to tangent-space normal (self-contained, no imports).
//
// 3x3 edge kernel (Sobel / Prewitt / Scharr) over the height texture's red channel.
// `wrap` = 1 samples across the border (seamless tiles); 0 clamps to the edge
// (atlases, non-tiling art). A neighbour whose alpha is below `alpha_threshold`
// is treated as off-island and replaced by the centre height, so relief never
// bleeds across transparent UV gutters.
//
// Encoded green follows `convention`: 0 = DirectX (green-down, what Minecraft
// Bedrock `_normal.png` expects), 1 = OpenGL (green-up). X is shared by both.
// Strength multiplies the gradients with z = 1, so higher = stronger relief.
// Transparent texels emit a flat normal (128, 128, 255) with opaque alpha.
struct NormalParams {
strength: f32,
operator_type: u32, // 0 = sobel, 1 = prewitt, 2 = scharr
wrap: u32, // 0 = clamp, 1 = wrap
convention: u32, // 0 = directx, 1 = opengl
alpha_threshold: f32, // 0..1; neighbours below this are off-island
}
@group(0) @binding(0) var heightTex: texture_2d<f32>;
@group(0) @binding(1) var normalTex: texture_storage_2d<rgba8unorm, write>;
@group(0) @binding(2) var<uniform> params: NormalParams;
fn wrap_or_clamp(coord: vec2i, dims: vec2i) -> vec2i {
if (params.wrap == 1u) {
return ((coord % dims) + dims) % dims;
}
return clamp(coord, vec2i(0), dims - vec2i(1));
}
/// Height at `coord`, or `centre` when that texel is off-island.
fn tap(coord: vec2i, dims: vec2i, centre: f32) -> f32 {
let texel = textureLoad(heightTex, wrap_or_clamp(coord, dims), 0);
return select(centre, texel.r, texel.a >= params.alpha_threshold);
}
@compute @workgroup_size(8, 8)
fn main(@builtin(global_invocation_id) gid: vec3u) {
let dims = vec2i(textureDimensions(heightTex, 0));
if (i32(gid.x) >= dims.x || i32(gid.y) >= dims.y) { return; }
let c = vec2i(gid.xy);
let here = textureLoad(heightTex, c, 0);
if (here.a < params.alpha_threshold) {
textureStore(normalTex, c, vec4f(0.5, 0.5, 1.0, 1.0));
return;
}
let centre = here.r;
let s00 = tap(c + vec2i(-1, -1), dims, centre);
let s01 = tap(c + vec2i( 0, -1), dims, centre);
let s02 = tap(c + vec2i( 1, -1), dims, centre);
let s10 = tap(c + vec2i(-1, 0), dims, centre);
let s12 = tap(c + vec2i( 1, 0), dims, centre);
let s20 = tap(c + vec2i(-1, 1), dims, centre);
let s21 = tap(c + vec2i( 0, 1), dims, centre);
let s22 = tap(c + vec2i( 1, 1), dims, centre);
// (corner, edge) tap weights per operator.
var corner = 1.0;
var edge = 2.0;
if (params.operator_type == 1u) { edge = 1.0; }
if (params.operator_type == 2u) { corner = 3.0; edge = 10.0; }
// dx > 0 when height rises to the right; dy > 0 when it rises upward (toward row 0).
let dx = -corner * s00 + corner * s02 - edge * s10 + edge * s12 - corner * s20 + corner * s22;
let dy = corner * s00 + edge * s01 + corner * s02 - corner * s20 - edge * s21 - corner * s22;
let nx = -dx * params.strength;
let ny_gl = -dy * params.strength;
let ny = select(-ny_gl, ny_gl, params.convention == 1u);
let normal = normalize(vec3f(nx, ny, 1.0));
textureStore(normalTex, c, vec4f(normal * 0.5 + 0.5, 1.0));
}
SKILL.md
---
name: blockbench-albedo-to-pbr
description: Derive height, tangent-space normal and packed MER (metalness, emissive, roughness) maps from an albedo/base-color texture for Blockbench PBR materials, using PyPBR (Python) or vgpu + Dawn WebGPU (Node.js), with optional Depth Anything V2 height estimation and MER inferred from the albedo plus the normal's curvature. Use after a color texture exists (painted, imported, or generated with blockbench-gpt-image-textures) when a material needs normal, heightmap or MER channels, a Bedrock RTX texture_set, or when asked to make a normal map, bump map, height map, roughness, metalness or emissive map from a texture. Triggers on albedo to normal, normal map from texture, height map from albedo, MER map, roughness from texture, metalness map, emissive map, Depth Anything, PyPBR, vgpu, Sobel normal, DirectX vs OpenGL normal, texture_set normal, heightmap or metalness_emissive_roughness layer.
---
# Albedo to Normal and MER for Blockbench
PBR channels are derived, not painted: albedo → height field → tangent-space normal, then metalness, emissive and roughness inferred from the albedo, the height and the normal's curvature and packed into one MER image. Two scripts share one CLI and the same maths; pick by runtime. Load [blockbench-use](../blockbench-use/SKILL.md) first for discovery and verification, and [blockbench-pbr-materials](../blockbench-pbr-materials/SKILL.md) to assign the results. This skill produces the image files only.
| Runtime | Script | Install | Height estimation |
|---|---|---|---|
| Python | `scripts/albedo_to_normal.py` | `pip install pypbr` (+ `transformers` for depth) | Depth Anything V2 through transformers, CUDA when available |
| Node.js | `scripts/albedo_to_normal.mjs` | `npm install vgpu pngjs` (+ `@huggingface/transformers` for depth) | Depth Anything V2 ONNX through transformers.js, CPU or WebGPU |
Both write `<name>_height.png` (grayscale, light = high), `<name>_normal.png` (RGB, opaque) and `<name>_mer.png` (R metalness, G emissive, B roughness) and print a JSON report. Runtime setup, API notes and troubleshooting: [references/python-pypbr.md](references/python-pypbr.md), [references/node-vgpu.md](references/node-vgpu.md). Node needs the script's `scripts/` folder reachable from a project that has the dependencies installed; Python only needs the packages.
## Workflow
### 1. Get the color texture as a file
Run `list_textures` and note the target texture's UUID, `bitmap_size` and `uv_size`; it does not report a file path. If the texture was imported from a file you wrote (for example the downscaled atlas from the GPT image workflow), use that file. Otherwise fetch the image with `get_texture` and save the returned PNG to disk, or have Blockbench save the texture. Work on the atlas at its native texel size, not on an upscaled generation canvas, so the maps share the texel grid with the color map.
Check the atlas before deriving anything: relief scale must be consistent across faces, so apply [UV scale and distortion guidance](../blockbench-texturing/references/uv-scale-and-distortion.md) to the color map first. A normal map inherits every stretch in the UV layout.
### 2. Choose the height source
| Source | Flag | Use for |
|---|---|---|
| Luminance (default) | `--height-source luminance` | Pixel art and flat-shaded painted textures. Light reads as high; add `--invert-height` when dark paint is the raised part (dark grout should stay low). |
| Depth Anything V2 | `--height-source depth` | Photographic or painterly textures with real structure: bricks, planks, cobble, foliage, machinery. Useless on a two-tone checker. Depth is downloaded on first use. |
| Height file | `--height-source file --height-file h.png` | A height painted in Blockbench, from another tool, or a previous `_height.png` to reshape. |
Depth estimation sees a photo, not a tile: it returns a foreground-to-background ramp that breaks tiling. The script keeps `--flatten 1` and normalize on for depth; add `--high-pass <texels>` (about a quarter of the tile width) for non-linear bias and `--seam offset` for seamless tiles. Small textures are nearest-neighbour upscaled to at least 1024 px before the model and block-averaged back, so 16× art keeps its edges. Details: [references/height-processing.md](references/height-processing.md).
### 3. Run
```bash
# tiling 16x block, pronounced pixel relief
python scripts/albedo_to_normal.py block.png --out ./pbr --wrap --strength 1
node scripts/albedo_to_normal.mjs block.png --out ./pbr --wrap --strength 1
# entity/prop atlas with transparent gutters (islands isolated automatically)
python scripts/albedo_to_normal.py goal_atlas.png --out ./pbr --strength 0.5
# HD tile from a painterly texture, depth-estimated, softened for smooth relief
node scripts/albedo_to_normal.mjs planks.png --out ./pbr --height-source depth --high-pass 32 --seam offset --wrap --soften 1
```
| Flag | Default | Effect |
|---|---|---|
| `--wrap` | off | Sample across borders. On for seamless tiles, off for atlases and non-tiling art. |
| `--strength F` | 0.5 | Gradient multiplier. 0.25 subtle, 0.5 default, 1–2 strong pixel relief; halve per doubling of texture density. |
| `--operator` | sobel | `sobel`, `prewitt` (softer), `scharr` (smooth HD height). |
| `--convention` | directx | `directx` green-down (Bedrock RTX assumption) or `opengl` green-up (Blender, glTF, labPBR). |
| `--alpha-threshold` | 8 | Albedo alpha below this is off-island: no relief bleeds across gutters; gutters get the flat normal. |
| `--soften M` | 0 | Density-scaled Gaussian on the height (≈ shortSide/64 · M texels). Use 1 for ≥64× textures; keep 0 for 16×. |
| `--levels B,W,G` | 0,1,1 | Input black/white/gamma on the height before the kernel. |
| `--flatten`, `--high-pass`, `--normalize`/`--no-normalize`, `--invert-height` | depth: 1, 0, on | Depth shaping, also available for `file` and `luminance` sources. |
| `--seam offset --seam-amount A` | off, 0.15 | Wrap-and-heal seam blend on the height for tiles. |
| `--no-height`, `--no-mer` | off | Skip writing `_height.png` / `_mer.png`. |
| `--metal`, `--emissive`, `--roughness` | auto | `auto` infers the channel; a 0..1 value makes it uniform (`--metal 0` for anything that is not metal). |
| `--palette "#RRGGBB=m,e,r;..."` | none | Exact-colour MER overrides; `-` keeps the inferred channel. The precise tool for pixel-art palettes. |
| `--rough-curvature W`, `--rough-height I`, `--rough-weights V,E,S`, `--rough-scale`, `--rough-bias` | 0.25, 0, 0.4/0.3/0.3, 1, 0 | Roughness terms: normal curvature (crevices rougher, ridges smoother), height influence, variance/edge/saturation weights, output remap. |
| `--metal-brightness`, `--metal-saturation`, `--gold-hue`, `--copper-hue`, `--no-metal-clean` | 0.6, 0.15, 0.11–0.17, 0.03–0.10, clean on | Metal heuristic thresholds; the clean-up is a 3×3 majority vote against speckle. |
| `--emissive-brightness`, `--emissive-saturation`, `--emissive-overbright`, `--emissive-softness` | 0.8, 0.5, 1 (off), 0 | Emissive thresholds, over-bright fallback, glow-edge anti-aliasing. |
| Python `--depth-device`, `--material-folder`; Node `--depth-device`, `--depth-dtype`, `--gpu` | | Runtime specifics; see the references. |
MER inference is heuristic. Read [references/mer-inference.md](references/mer-inference.md) before tuning it: bright desaturated colours read as metal and bright saturated colours read as glow, which is wrong for wool, bone, snow, pale skin, yellow beaks and painted signs. Decide the material first, then choose `auto`, a uniform value or palette overrides per channel.
### 4. Read the report
The JSON on stdout ends with `report` and `merReport`. For the normal, require `verdict: "ok"` and `classification: "standard"`; `meanXYBias` near 0 and `negativeZCount` 0. A `heightfield` classification means the albedo argument was a grayscale image; `warn` with a large bias on a tile means an unflattened ramp in the height. `flatPct` is informational (high on flat art and on atlases with big gutters).
For the MER, compare `metalPct` and `emissivePct` with what the material actually is, and `roughnessMean` with how matte it should look. A wooden crate with `metalPct` 0.4 means the pale planks were misread as steel: rerun with `--metal 0` or a palette. `paletteTexels` 0 with a palette means the colour did not match exactly. The report says the encodings are sound, not that the relief direction, scale or material assignments are right; that is step 6.
### 5. Import and assign in Blockbench
```
create_texture: name="<name>_normal", data="<absolute path to <name>_normal.png>"
create_texture: name="<name>_mer", data="<absolute path to <name>_mer.png>"
create_pbr_material: name="<material>", color_texture="<albedo>", normal_texture="<name>_normal", mer_texture="<name>_mer"
# or on an existing material:
assign_texture_channel: material="<material>", texture="<name>_normal", channel="normal"
assign_texture_channel: material="<material>", texture="<name>_mer", channel="mer"
get_material_info: material="<material>"
```
Bedrock texture sets take `normal` **or** `heightmap`, never both. Assign the `_height.png` as `height_texture` instead when the target is 16× vanilla-style pixel art and the pack convention prefers heightmaps; otherwise use the normal. A MER texture requires the color texture on the same material; when the whole surface shares one metal/emissive/roughness value, `mer_value` on the material is simpler than a map (generate with `--no-mer`). Match `uv_width`/`uv_height` on `create_texture` to the color texture in formats with per-texture UV sizes. Channel rules, uniform MER values and `save_material_config` are in [blockbench-pbr-materials](../blockbench-pbr-materials/SKILL.md).
### 6. Verify direction and scale
Look at the lit model with `capture_screenshot` from at least two angles and at the intended viewing distance:
- Raised features must read as raised. Inverted top-to-bottom shading means the wrong green convention: regenerate with the other `--convention`. Everything reading inverted (pits as bumps) means the height sense is wrong: regenerate with `--invert-height`.
- Relief must look the same size on every face. Adjust `--strength` globally; fix uneven results in the UV layout or the color atlas, not by hand-editing the normal.
- Tiling blocks: place several copies and check the seams; `--wrap` and `--seam offset` on the height are the fixes, not painting over the seam.
- No rim around atlas islands. If one appears, the albedo alpha was not clean; raise `--alpha-threshold` or clean the gutters and regenerate.
- Metal regions must be the parts that are metal (they turn dark and reflective under RTX), glow must sit only on light sources, and roughness must not make cloth or paint shiny. Fix with uniform values, palette overrides or the roughness terms, and regenerate the MER rather than painting it.
- The rendered preview only proves Blockbench's shading; confirm in the target game or renderer before delivery.
## What to expect
- Luminance height is an artistic proxy: painted highlights become bumps and painted shadows become pits. It is the right choice for stylised and pixel art, and wrong for photo-sourced textures.
- Depth estimation on textures is relief-plausible, not measured. Expect per-run variation between the PyTorch and ONNX model conversions and treat `--flatten` / `--high-pass` / `--seam` as required cleanup for tiles.
- Both runtimes produce byte-identical heights from the same luminance input, normals within two levels of each other from the same height, and MER within three levels on roughness with identical metal masks.
- MER inference is colour and shape heuristics, not material recognition. Expect false metal on pale desaturated areas and false glow on bright saturated ones; the report makes both visible and the overrides fix them without painting.
- PyPBR's own `compute_normal_from_height` (0.1.0b5) zero-pads borders and flips X, so the Python script derives the normal itself and uses PyPBR for the material object, folder I/O and follow-up operations.
- Node resolves `vgpu`, `pngjs` and the shaders from the script's location; without a GPU, run `npx vgpu install-software-renderer` once and pass `--gpu software`.
## Resources
- [references/height-processing.md](references/height-processing.md): the height and normal pipeline stage by stage, depth shaping, kernel maths, conventions, report fields, runtime parity.
- [references/mer-inference.md](references/mer-inference.md): how metalness, emissive and roughness are inferred, every MER flag, overrides, the MER report and the checks to run.
- [references/python-pypbr.md](references/python-pypbr.md): install, PyPBR material API and quirks, Depth Anything through transformers.
- [references/node-vgpu.md](references/node-vgpu.md): install, vgpu compute/texture API used by the script, WGSL notes, transformers.js depth in Node.
- `scripts/albedo_to_normal.py`: Python pipeline (torch + PyPBR, optional transformers).
- `scripts/albedo_to_normal.mjs`: Node pipeline (vgpu + Dawn, pngjs, optional transformers.js).
- `scripts/shaders/height-map.wgsl`, `normal-map.wgsl`, `mer-map.wgsl`: the GPU passes; also readable as the reference maths.