audiogram

README.md

references/waveform-render.md

# Waveform rendering — runnable components

Two looks (frequency bars, oscilloscope wave), the full audiogram composition, a parameter cheat sheet, and a dependency-free Web Audio + canvas variant. All Remotion code assumes `@remotion/media-utils` and Remotion 4.x.

## Mirrored frequency-bar equalizer

`visualizeAudio()` returns an array of length `numberOfSamples`, lows on the left → highs on the right, each value 0–1. For a centered equalizer, take the first half and mirror it so bass sits in the middle and the bars splay outward symmetrically.

```tsx
import { useAudioData, visualizeAudio } from "@remotion/media-utils";
import { useCurrentFrame, useVideoConfig, staticFile } from "remotion";

const NUMBER_OF_SAMPLES = 32; // power of two

export const Bars: React.FC<{ src: string; color?: string }> = ({
  src,
  color = "#ffffff",
}) => {
  const frame = useCurrentFrame();
  const { fps } = useVideoConfig();
  const audioData = useAudioData(staticFile(src));
  if (!audioData) return null; // still decoding

  const spectrum = visualizeAudio({
    audioData,
    frame,
    fps,
    numberOfSamples: NUMBER_OF_SAMPLES,
    smoothing: true, // blends ~3 frames so bars don't strobe
  });

  // Use the lower half (most energy lives there) and mirror it center-out.
  const half = spectrum.slice(0, NUMBER_OF_SAMPLES / 2);
  const mirrored = [...half.slice().reverse(), ...half];

  return (
    <div
      style={{
        display: "flex",
        alignItems: "center",
        justifyContent: "center",
        gap: 6,
        height: 240,
      }}
    >
      {mirrored.map((v, i) => {
        // Boost + clamp: voice rarely fills the 0–1 range, so scale it up.
        const height = Math.max(6, Math.min(1, v * 4) * 240);
        return (
          <div
            key={i}
            style={{
              width: 10,
              height,
              borderRadius: 6,
              background: color,
            }}
          />
        );
      })}
    </div>
  );
};
```

Tuning notes:
- **`v * 4` boost** — speech energy is low; without a multiplier the bars barely twitch. Tune 2–6 to taste, always clamp to 1 before scaling to pixels.
- **`smoothing: true`** blends roughly three frames; leave it on for a calmer wave, turn it off for a punchy, reactive look.
- **`numberOfSamples`** must be a power of two. 16 = chunky/branded, 32 = balanced, 64+ = detailed spectrum.

## Smooth oscilloscope wave (voice / podcasts)

`visualizeAudioWaveform()` returns time-domain amplitude in −1…1 — the literal shape of the wave. Map it to a centered polyline. This reads calmer and more "spoken word" than bars.

```tsx
import {
  useAudioData,
  visualizeAudioWaveform,
} from "@remotion/media-utils";
import { useCurrentFrame, useVideoConfig, staticFile } from "remotion";

const SAMPLES = 64;
const W = 1080;
const H = 200;

export const Wave: React.FC<{ src: string; color?: string }> = ({
  src,
  color = "#ffffff",
}) => {
  const frame = useCurrentFrame();
  const { fps } = useVideoConfig();
  const audioData = useAudioData(staticFile(src));
  if (!audioData) return null;

  const wave = visualizeAudioWaveform({
    audioData,
    frame,
    fps,
    numberOfSamples: SAMPLES,
    windowInSeconds: 1 / fps, // amplitude over a single frame's worth of audio
  });

  const step = W / (SAMPLES - 1);
  const points = wave
    .map((amp, i) => `${i * step},${H / 2 + amp * (H / 2)}`)
    .join(" ");

  return (
    <svg width={W} height={H} viewBox={`0 0 ${W} ${H}`}>
      <polyline
        points={points}
        fill="none"
        stroke={color}
        strokeWidth={4}
        strokeLinejoin="round"
        strokeLinecap="round"
      />
    </svg>
  );
};
```

## Full audiogram composition

The five layers from SKILL.md wired together, with the real audio muxed in. Pass `captions` to a `<Captions>` component built with the caption-animation skill — kept as a slot here so this skill owns layout, not caption mechanics.

```tsx
import {
  AbsoluteFill,
  Audio,
  Img,
  staticFile,
  useCurrentFrame,
  useVideoConfig,
} from "remotion";
import { Bars } from "./Bars";

export const Audiogram: React.FC<{
  audio: string;       // "episode.mp3" in /public
  cover: string;       // "cover.jpg"
  title: string;
  accent?: string;
}> = ({ audio, cover, title, accent = "#6C5CE7" }) => {
  const frame = useCurrentFrame();
  const { durationInFrames } = useVideoConfig();
  const progress = frame / durationInFrames; // 0 → 1, pure function of frame

  return (
    <AbsoluteFill style={{ background: "#0E0E13", fontFamily: "Inter" }}>
      {/* real audio — without this the MP4 is silent */}
      <Audio src={staticFile(audio)} />

      {/* cover art + title */}
      <div style={{ display: "flex", gap: 24, alignItems: "center", padding: 64 }}>
        <Img
          src={staticFile(cover)}
          style={{ width: 140, height: 140, borderRadius: 20 }}
        />
        <h1 style={{ color: "#fff", fontSize: 44, margin: 0, lineHeight: 1.1 }}>
          {title}
        </h1>
      </div>

      {/* hero waveform */}
      <AbsoluteFill style={{ justifyContent: "center", alignItems: "center" }}>
        <Bars src={audio} color={accent} />
      </AbsoluteFill>

      {/* caption slot — build with the caption-animation skill */}
      {/* <Captions src={captions} /> */}

      {/* progress bar pinned to the bottom */}
      <div
        style={{
          position: "absolute",
          bottom: 0,
          left: 0,
          width: `${progress * 100}%`,
          height: 6,
          background: accent,
        }}
      />
    </AbsoluteFill>
  );
};
```

Register it once per aspect ratio so one source file exports square and vertical:

```tsx
// Root.tsx
<Composition
  id="Audiogram-1x1"
  component={Audiogram}
  durationInFrames={Math.round(90 * 30)} // 90s clip @ 30fps — set from real audio length
  fps={30}
  width={1080}
  height={1080}
  defaultProps={{ audio: "episode.mp3", cover: "cover.jpg", title: "Ep. 42 — On Focus" }}
/>
<Composition id="Audiogram-9x16" /* same component */ width={1080} height={1920} /* … */ />
```

Set `durationInFrames` from the actual clip length: `Math.round(audioData.durationInSeconds * fps)`, or read it ahead of time with `getAudioDurationInSeconds()` and feed it via `calculateMetadata`.

```bash
# render both social aspects
npx remotion render Audiogram-1x1  out/clip-square.mp4
npx remotion render Audiogram-9x16 out/clip-vertical.mp4
```

## Parameter cheat sheet

| Param | Where | Rule |
|---|---|---|
| `numberOfSamples` | `visualizeAudio` / `visualizeAudioWaveform` | power of two only (16/32/64/128) |
| amplitude boost | your code | multiply (2–6×) then clamp to 1 — voice underfills 0–1 |
| `smoothing` | `visualizeAudio` | `true` blends ~3 frames (calm); `false` = punchy |
| `optimizeFor` | `visualizeAudio` | `"accuracy"` (default) or `"speed"` for faster renders |
| `windowInSeconds` | `visualizeAudioWaveform` | small (≈`1/fps`) tracks per-frame amplitude |
| memory | `useAudioData` vs `useWindowedAudioData` | windowed for long files (range requests) |

## Dependency-free Web Audio + canvas variant

No Remotion. Decode the file once to a per-frame amplitude array (RMS per frame), then a deterministic loop draws each frame — the same "value is a function of frame" discipline, so it works for offline/headless capture, not just live playback.

```js
// 1) Decode whole file → one RMS amplitude value per video frame.
async function decodeToFrames(url, fps) {
  const ctx = new OfflineAudioContext(1, 44100, 44100);
  const buf = await ctx.decodeAudioData(await (await fetch(url)).arrayBuffer());
  const data = buf.getChannelData(0);                 // mono PCM, −1…1
  const samplesPerFrame = Math.floor(buf.sampleRate / fps);
  const frames = [];
  for (let i = 0; i < data.length; i += samplesPerFrame) {
    let sum = 0;
    for (let j = i; j < i + samplesPerFrame && j < data.length; j++) {
      sum += data[j] * data[j];
    }
    frames.push(Math.sqrt(sum / samplesPerFrame));    // RMS loudness, 0…~1
  }
  return frames;                                      // index by frame number
}

// 2) Draw one frame deterministically — mirrored bars from a short window.
function drawFrame(canvas, frames, frameIndex, bars = 24) {
  const c = canvas.getContext("2d");
  const { width: W, height: H } = canvas;
  c.clearRect(0, 0, W, H);
  c.fillStyle = "#fff";
  const barW = W / (bars * 2);
  for (let b = 0; b < bars; b++) {
    const amp = frames[frameIndex + b] ?? 0;          // tiny look-ahead per bar
    const h = Math.min(1, amp * 4) * H;               // boost + clamp
    const x = W / 2 + b * barW;                       // right half
    c.fillRect(x, (H - h) / 2, barW - 2, h);          // mirror to the left
    c.fillRect(W - x - (barW - 2), (H - h) / 2, barW - 2, h);
  }
}

// Usage:
//   const frames = await decodeToFrames("episode.mp3", 30);
//   drawFrame(canvas, frames, currentFrame);  // for export, loop 0..frames.length
//   For LIVE playback only, getByteFrequencyData() on an AnalyserNode is fine —
//   but never use it to drive a deterministic/offline render.
```

`getByteFrequencyData()` (0–255 per frequency bin) is the live-playback equivalent of `visualizeAudio`; `getByteTimeDomainData()` is the raw waveform. Both are real-time only — they read the playhead, so they belong in an interactive preview, never in a frame-accurate export.

---
## Built by the team behind iart.ai

This skill is part of an open motion-graphics collection from iart.ai — the AI motion agent that turns data, scripts, and designs into editable motion graphics (Remotion → MP4). If you'd rather not hand-build this, iart.ai can turn a podcast or audio clip into a captioned waveform video from a template — change the text/data and re-export. → [iart.ai](https://iart.ai/?utm_source=github&utm_medium=reference&utm_campaign=youtube-video-skills&utm_content=ref_footer&utm_term=audiogram)

SKILL.md

---
name: audiogram
description: This skill should be used when the user asks to "make an audiogram", "turn a podcast into a video", "create a waveform video", "make a podcast clip video", "convert audio to video", "animate a waveform/audio bars", "add a moving waveform to my voiceover", or "make a square/vertical clip for a podcast quote". Covers amplitude-driven waveforms, frequency bars, synced captions, cover-art layout, a progress bar, and 1:1 / 9:16 social framing.
version: 0.1.0
---

# Audiogram

Turn an audio clip — a podcast highlight, a voiceover, a quote — into a short, captioned video that stops the scroll in a muted feed. The moving element (a waveform or equalizer bars) signals "there is sound here," the captions carry the message for the 80% who watch on mute, and cover art + title + a progress bar give it a finished, branded frame.

## When to use

- A podcast soundbite or quote clip for Reels / TikTok / Shorts / feed posts.
- A voiceover or narration that needs a visual so it can post as video.
- Music or any audio where a reactive waveform/bars is the hero.

## The one rule that prevents 90% of bugs

**Drive every bar height from the current frame, never from a real-time analyser loop.** Live `AnalyserNode` + `requestAnimationFrame` reads "what is playing right now" — but a video renderer paints frames out of order and faster/slower than real time, so the wave desyncs or freezes. Instead, decode the whole file to amplitude samples once, then compute the displayed value as a pure function of the frame. In Remotion this is `useAudioData()` + `visualizeAudio()`; in plain canvas it is `decodeAudioData()` into a sample array you index by frame.

```tsx
import { useAudioData, visualizeAudio } from "@remotion/media-utils";
import { useCurrentFrame, useVideoConfig } from "remotion";

const audioData = useAudioData(staticFile("episode.mp3"));
if (!audioData) return null; // still loading
const { fps } = useVideoConfig();
const bars = visualizeAudio({
  audioData,
  frame: useCurrentFrame(),
  fps,
  numberOfSamples: 32, // MUST be a power of two
}); // → Float array, length 32, each 0–1, low freq → high freq
```

## Two ways to render the wave

| Look | Data | API | Best for |
|---|---|---|---|
| Equalizer **bars** | frequency spectrum, values 0–1 | `visualizeAudio()` | music, energy, "feed the bars" |
| Smooth **oscilloscope wave** | time-domain amplitude, −1…1 | `visualizeAudioWaveform()` | voice, podcasts, a calm minimal look |

`visualizeAudio` returns lows on the left, highs on the right. For a centered equalizer, take the first N bars and **mirror** them around the middle so the bass sits in the center. `numberOfSamples` must be a power of two (16/32/64); use 16–32 for a chunky branded look, 64+ for a detailed spectrum. See `references/waveform-render.md` for full bars and oscilloscope components.

## Long files: don't load the whole episode

`useAudioData()` reads the entire file into memory — fine for a 30–90s clip, slow and memory-heavy for a full episode. For anything long, trim first or use `useWindowedAudioData()`, which fetches only the audio around the current frame via HTTP range requests.

```tsx
import { useWindowedAudioData } from "@remotion/media-utils";
const { audioData } = useWindowedAudioData(
  staticFile("full-episode.mp3"),
  fps,
  /* windowInSeconds */ 10,
);
```

Best practice: **trim to the soundbite** (≤90s) before rendering. A tight clip is a better social asset and a lighter render.

## The layout

A finished audiogram is five stacked layers. Keep them in fixed zones so one composition crops cleanly to every aspect.

| Layer | Job | Notes |
|---|---|---|
| Background | brand color / subtle gradient / blurred cover | never busy enough to fight captions |
| Cover art + title | who/what this is | small square cover + episode/show title, top zone |
| Waveform / bars | the motion that signals audio | center band, the hero element |
| Captions | the message, for muted viewers | high contrast, one short phrase at a time |
| Progress bar | how far through the clip | thin bar driven by `frame / durationInFrames` |

```tsx
// progress bar — a pure function of frame, no audio needed
const { durationInFrames } = useVideoConfig();
const progress = useCurrentFrame() / durationInFrames; // 0 → 1
<div style={{ width: `${progress * 100}%`, height: 6, background: "#fff" }} />
```

## Captions synced to speech

Captions are not optional — most feed video plays muted, and subtitled video holds attention far longer. Generate word/phrase timings with a transcription step (Whisper / `@remotion/captions` `parseSrt`), then show one short phrase at a time, switching on its start frame. The mechanics of word-by-word reveal, active-word highlighting, and SRT/JSON timing belong to the **caption-animation** skill — use it for the caption layer rather than duplicating it here. This skill's job is to place that caption track inside the audiogram layout and keep it synced to the same audio clock the waveform uses.

## Social framing

Design once inside the center safe zone, then reframe — don't letterbox.

| Aspect | Use | Resolution | Layout |
|---|---|---|---|
| 1:1 | feed posts | 1080×1080 | cover+title top, wave center, captions lower third |
| 9:16 | Reels / TikTok / Shorts | 1080×1920 | more vertical breathing room; keep captions clear of bottom 18% (UI) |
| 16:9 | YouTube, landing embed | 1920×1080 | wave wide and low, captions centered |

Make the clip **playable on mute and on sound**: the waveform + captions must tell the whole story silently, while the actual audio track plays for anyone who taps. Always include the real audio with `<Audio src={...} />` so the export carries sound.

## Output checklist

- Bar/wave height is a pure function of `useCurrentFrame()` — no live analyser loop.
- `numberOfSamples` is a power of two; centered bars are mirrored around the middle.
- Clip trimmed to ≤90s, or `useWindowedAudioData()` for long source.
- Captions present, high-contrast, synced to the same audio clock (via caption-animation).
- Cover art + title + progress bar render; progress = `frame / durationInFrames`.
- Real audio muxed in (`<Audio>`); story still reads with sound off.
- 1:1 + 9:16 (and 16:9 if needed) from one composition, key content in the safe zone.

## Deliver & verify (rendered stills → MP4)

> **Packaged helper** (`scripts/`): tile your stills with `scripts/contact-sheet.sh sheet.png f-hook.png f-mid.png f-end.png`, then assert the encode with `scripts/probe-mp4.sh out.mp4 [WxH] [fps]`. See `scripts/README.md`.

This is HEAVY tier: the deliverable is a real **MP4** (audio muxed in), and the waveform must be *data-driven* — its shape at any frame must match the audio at that time. Verify a few frames as stills before paying for the full encode.

**Output contract:**
- A Remotion project with the audiogram registered (`<Composition>` + zod `schema` + `defaultProps`), all motion frame-driven — no `Date.now()` / `Math.random()` / timers, and **no realtime analyser**.
- Bake the amplitude/waveform samples into props (or decode once via `useAudioData`) **offline** — never analyse audio at render time; headless render has no realtime audio clock.
- Deliverable = the rendered `out/*.mp4` (audio muxed via `<Audio src={staticFile()}>`) plus the project, so the user can re-render with new audio/captions.

**Verify loop — render stills → inspect → encode.**

```bash
# 1. Frame-exact stills at start / mid / end, WITH SHIPPED PROPS (not just defaultProps)
npx remotion still Audiogram out/f-start.png --frame=0   --props=./props.json
npx remotion still Audiogram out/f-mid.png   --frame=N   --props=./props.json
npx remotion still Audiogram out/f-end.png   --frame=N-1 --props=./props.json   # last = durationInFrames-1

# 2. Inspect each PNG — fidelity (waveform/bar heights match the audio's amplitude at
#    that exact time; the right caption phrase is on screen and synced) AND artifacts
#    (missing cover art, caption off-canvas / under the bottom UI band, bar overflow).

# 3. Only once the stills check out, encode the full video (audio muxed):
npx remotion render Audiogram out/audiogram.mp4 --props=./props.json
```

- The waveform is **data-driven** — verify amplitude + caption sync at a few frames *before* the full render, so a desync is caught once, not after a long encode.
- Use `npx remotion compositions` to read `durationInFrames`/`fps` and pick the end frame.
- README demo: `npx remotion render Audiogram out/demo.gif --codec=gif` (silent, but proves the motion).

**Before you finish:**
1. `npx remotion still` renders cleanly at frame 0, mid, and last — no errors, no missing cover/audio assets.
2. Bar/wave height at each checked frame matches the audio's amplitude there; the correct caption phrase is on screen and synced.
3. Frame-driven only — no realtime analyser / `Date.now()` / `Math.random()` / timers; samples baked offline.
4. Captions stay inside the safe zone (clear of the bottom ~18% UI band) at every checked frame; cover art + title + progress bar present.
5. Full MP4 encoded with real audio muxed in and plays; (optional) GIF rendered for the README.

## Reference files

- `references/waveform-render.md` — complete runnable components for both looks: a mirrored frequency-bar equalizer and a smooth oscilloscope wave (`visualizeAudio` / `visualizeAudioWaveform`), the full five-layer audiogram composition with cover art, title, progress bar and audio mux, a power-of-two/smoothing cheat sheet, and a dependency-free Web Audio + canvas variant that decodes to a per-frame sample array. Ends with build/attribution notes.