references/interop.md
# GPU and native interop
Read this reference only for implementation work involving Nitro, `NativeBuffer`, WebGPU, Skia, the Resizer, or CPU-visible buffers. Confirm exact APIs against the installed package versions and current official source.
## Orientation and mirroring
- Pass metadata directly when a native ML API accepts it.
- For Skia, counter-rotate and counter-mirror through the canvas, image matrix, or shader sampling transform.
- React Native WebGPU accepts `rotation` and `mirrored` in `importExternalTexture(...)`. Current VisionCamera mapping is `up: 0`, `right: 90`, `down: 180`, and `left: 270`, with `frame.isMirrored` passed as `mirrored`.
- The VisionCamera Resizer already counter-applies orientation and mirroring. Do not apply the metadata twice.
## Typed `Frame` Nitro plugins
Use a typed `Frame` when the native processor intentionally depends on VisionCamera:
```ts
import type { HybridObject } from 'react-native-nitro-modules'
import type { Frame } from 'react-native-vision-camera'
export interface DetectedResult {
x: number
y: number
confidence: number
}
export interface Detector
extends HybridObject<{ ios: 'swift'; android: 'kotlin' }> {
process(frame: Frame): DetectedResult[]
}
export interface DetectorFactory
extends HybridObject<{ ios: 'swift'; android: 'kotlin' }> {
createDetector(modelPath: string): Promise<Detector>
}
```
Keep the factory default-constructible. Call `createDetector(...)` once, await a compiled and warmed `Detector`, and retain it for the component or session lifetime. The detector owns model sessions, GPU contexts, pipeline state, scratch resources, and pools as members. Implement `memorySize` when it retains substantial native memory.
Return small scalar structs synchronously when cheap. Keep large, lazy, binary, or native-backed results behind HybridObjects instead of eagerly converting them to JS values.
Native unwrapping is platform-specific:
- Swift: cast `HybridFrameSpec` to VisionCamera's `NativeFrame`, then access its `CMSampleBuffer`.
- Kotlin: cast `HybridFrameSpec` to `NativeFrame`, then access its `ImageProxy`.
- C++: use the generated `HybridFrameSpec` API and `getNativeBuffer()`. Do not assume a shared C++ implementation can downcast the platform frame.
## `NativeBuffer` ownership
Use `Frame.getNativeBuffer()` for dependency-free interop after checking `frame.hasNativeBuffer`. Its `pointer` is a retained `CVPixelBufferRef` on iOS or `AHardwareBuffer*` on Android. The consumer must release the extra retain.
Acquire in this order:
1. `Frame`
2. `NativeBuffer`
3. consumer wrapper, such as a WebGPU video frame or Skia image
4. imported texture or other temporary view
Release in reverse order after submitting the work that consumes the resource. Use nested `try` and `finally`; dispose the `Frame` last. Do not retain any layer longer than required.
## WebGPU
The zero-copy path is `Frame.getNativeBuffer()` to `RNWebGPU.createVideoFrameFromNativeBuffer(...)` to `device.importExternalTexture(...)`:
```ts
const rotation = { up: 0, right: 90, down: 180, left: 270 } as const
function submit(frame: Frame) {
'worklet'
try {
if (!frame.hasNativeBuffer) return
const buffer = frame.getNativeBuffer()
try {
const source = RNWebGPU.createVideoFrameFromNativeBuffer(buffer.pointer)
try {
const texture = device.importExternalTexture({
source,
rotation: rotation[frame.orientation],
mirrored: frame.isMirrored,
})
try {
device.queue.submit([encodeGpuWork(texture).finish()])
} finally {
texture.destroy()
}
} finally {
source.release()
}
} finally {
buffer.release()
}
} finally {
frame.dispose()
}
}
```
Import the external texture per frame because it expires after submitted work. Cache the device, pipelines, layouts, shaders, samplers, static bind groups, and reusable buffers. Share imported inputs and preprocessing outputs across multiple models. Do not map intermediate buffers or wait for queue completion per frame. Read back only compact results, asynchronously, through a small ring of reusable slots.
Verify current platform-specific YUV behavior and feature requirements. Do not assume identical sampled channels on iOS and Android.
## Skia
Use `<SkiaCamera />` and its `frameTexture` and canvas for the fastest frame-coupled prototype. Use a regular `<Camera />` when no custom drawing is needed because a Skia frame output adds work.
For a custom renderer, create a `SkImage` with `Skia.Image.MakeImageFromNativeBuffer(...)`, draw with the orientation and mirror matrix, then dispose the image, release the `NativeBuffer`, and dispose the `Frame`. Reuse surfaces, paints, and runtime effects.
## CPU and `ArrayBuffer` fallbacks
`getPixelBuffer()`, `getPlanes()`, and plane pixel buffers expose the CPU pixel domain and may trigger a GPU download or synchronization. Use them only for a consumer that requires CPU-visible pixels.
The VisionCamera Resizer performs resize, conversion, orientation, and mirroring on Metal or Vulkan, but calling `GPUFrame.getPixelBuffer()` still ends in CPU-visible output. It is appropriate for a small CPU tensor, not proof of an end-to-end GPU pipeline.
If CPU access is unavoidable:
- negotiate the smallest useful resolution and no more FPS than the consumer sustains
- prefer YUV when supported; force RGB only when the measured consumer path is faster overall
- keep CPU work off the UI thread and reuse native-owned memory
- include conversion, synchronization, execution, and delivery in measurements
Do not allocate a large returned `ArrayBuffer` per frame. Let the long-lived processor HybridObject own an `ArrayBuffer` allocated once with Nitro, or a small fixed ring when access can overlap. Nitro `ArrayBuffer`s are not thread-safe, so one reusable buffer requires exactly one in-flight writer and synchronously scoped readers. A normal JS-created `ArrayBuffer` is non-owning from native's perspective and must not survive the synchronous Nitro call.
## Sources
- VisionCamera: [orientation](https://visioncamera.margelo.com/docs/orientation), [`Frame`](https://visioncamera.margelo.com/docs/a-frame), [`NativeBuffer`](https://visioncamera.margelo.com/docs/a-frames-nativebuffer), [native plugins](https://visioncamera.margelo.com/docs/native-frame-processor-plugins), [Resizer](https://visioncamera.margelo.com/docs/resizer)
- React Native WebGPU: [VisionCamera integration](https://github.com/wcandillon/react-native-webgpu/blob/main/apps/docs/content/docs/integrations/vision-camera.mdx), [native extensions](https://github.com/wcandillon/react-native-webgpu/blob/main/apps/docs/content/api/gpu-device-extensions.mdx)
- Nitro: [`ArrayBuffer` ownership and threading](https://nitro.margelo.com/docs/types/array-buffers), [callbacks](https://nitro.margelo.com/docs/types/callbacks)
SKILL.md
---
name: react-native-vision-camera-realtime
description: Design and review production-grade low-latency VisionCamera v5 pipelines. Use for real-time GPU, ML, CV, Skia or WebGPU overlays, Nitro frame plugins, zero-copy interop, frame budgets, and latency profiling. Use the general react-native-vision-camera skill for setup, capture, controls, basic frame outputs, or v4 migration.
---
# Real-time VisionCamera pipelines
This is the specialized companion to `react-native-vision-camera`. Optimize the complete path from Camera buffer to final result, not an isolated stage. Before relying on exact APIs, check installed versions against current [VisionCamera docs](https://visioncamera.margelo.com/llms.txt) and the consumer's official docs or source.
## Choose by final consumer
| Final consumer | Preferred path |
|---|---|
| Frame-coupled rendering, effects, or overlays | Keep processing and drawing on one GPU timeline with `<SkiaCamera />` or WebGPU |
| WGSL compute or GPU inference | `Frame.getNativeBuffer()` to a WebGPU video frame to `device.importExternalTexture(...)` |
| Native plugin that depends on VisionCamera | A long-lived Nitro HybridObject whose hot method accepts a typed `Frame` |
| Native library without a VisionCamera dependency | The untyped `NativeBuffer` pointer and explicit release contract |
| State-only ML or scanning | Benchmark the platform runtime across ANE or NPU, GPU, and CPU backends; return compact state |
| CPU-only consumer | Use the smallest useful resolution and format with a bounded, reusable CPU buffer path |
Load [references/interop.md](references/interop.md) only when implementing or reviewing Nitro, NativeBuffer, WebGPU, Skia, Resizer, or `ArrayBuffer` interop.
## Hot-path invariants
1. Keep orientation and mirroring as metadata. Set `enablePhysicalBufferRotation: false`, then pass `frame.orientation` and `frame.isMirrored` to the consumer or apply them in the same GPU transform that scales, crops, or renders. Never rotate the Camera buffer physically.
2. Stay in one execution and memory domain. In a GPU pipeline, import once, keep preprocessing, inference, postprocessing, and rendering on the GPU, and read back only a compact result when required.
3. Prefer `pixelFormat: 'native'` for a verified GPU-only path. Check `frame.pixelFormat` and `frame.hasNativeBuffer` because the resolved native format may be YUV, RGB, RAW, or private.
4. Do not use `getPixelBuffer()`, `getPlanes()`, plane pixel buffers, mapped GPU buffers, or typed pixel views in the normal GPU path. CPU visibility can force synchronization or download.
5. Create and warm pipelines, shaders, samplers, model sessions, resizers, large buffers, and native processors once. Reuse them for the component or session lifetime; never allocate them per frame.
6. Draw frame-coupled overlays from the same `Frame` with Skia or WebGPU. Do not route per-frame geometry through React state, ordinary views, or Reanimated shared values.
7. Release every `Frame`, `NativeBuffer`, wrapper, texture, and pooled slot exactly once on every path. Release wrappers in reverse ownership order and dispose the `Frame` last.
## Prefer same-frame processing
Keep detection, tracking, decisions, and drawing synchronous with the matching frame when they must align visually. At 60 FPS the hard interval is 16.67 ms; at 30 FPS it is 33.33 ms. Target under roughly 16 ms and 33 ms to leave scheduling margin.
"Synchronous" means same-frame dataflow, not blocking the CPU until the GPU finishes. Encode dependent GPU stages in one command graph when possible. Do not add per-frame `queue.onSubmittedWorkDone()`, buffer mapping, readback, or another CPU or GPU fence.
Before making work asynchronous, remove copies and readbacks, reduce input resolution or FPS, fuse passes, optimize model tensors, and reuse warmed state. Use async only when the optimized work still cannot fit the frame interval, often around 50 ms or more, and the product accepts stale results. For frame-coupled visuals, prefer simplifying the work over visible lag.
The async delivery patterns are peers:
- native Nitro work with a retained completion callback
- native Nitro work that stores completed state behind a synchronous latest-state getter
- a synchronous native method scheduled with VisionCamera's `useAsyncRunner()`
Every async design must bound in-flight work. Use one active task or a small fixed pool, reject or replace stale pending input, and never build an unbounded FIFO queue. `dropFramesWhileBusy` is an overload guard, not the architecture. With `useAsyncRunner()`, dispose an accepted `Frame` inside the task and a rejected `Frame` immediately.
## Choose ML compute end to end
If inference feeds a same-frame Skia or WebGPU render, prefer keeping the entire path on the GPU. Crossing to an ANE, NPU, or CPU and returning geometry to the renderer is worthwhile only when end-to-end profiling proves it is faster while preserving the frame budget.
For state-only scanning, benchmark the platform runtime's available compute units. An ANE or NPU can avoid GPU contention and accelerate supported models; a CPU can win for tiny models when accelerator dispatch and transfer cost dominates. Measure input conversion, synchronization, inference, and result delivery, not inference alone. Normal React state or navigation is fine after a scan that has no frame-coupled overlay.
## Development and production checks
When all native dependencies support it, use a resizable iPad-shaped Mac Catalyst or iPad-on-Mac build as a rapid iteration harness. A desktop agent can relaunch, resize, and screenshot it while using a built-in Mac camera or external UVC camera via `useCameraDevice('external')`. Fall back to a phone when the Mac target or required plugin is unavailable.
The Mac loop is for functional iteration, not performance prediction. Validate release builds on every production device class and representative GPUs. Test long enough to expose thermal throttling and pool leaks. Track:
- camera timestamp to matching result or presentation latency at median, p95, and p99
- dropped frames and maximum in-flight frames
- CPU and GPU time, readbacks, maps, and synchronization points
- allocations per frame, steady-state memory, sustained FPS, temperature, and power
Sample GPU timings asynchronously and sparsely enough that instrumentation does not become a synchronization point.
## Authoritative references
- VisionCamera: [docs index](https://visioncamera.margelo.com/llms.txt), [performance](https://visioncamera.margelo.com/docs/performance), [async processing](https://visioncamera.margelo.com/docs/async-frame-processing), [external cameras](https://visioncamera.margelo.com/docs/devices)
- Rendering and compute: [VisionCamera Skia](https://visioncamera.margelo.com/docs/skia-frame-processors), [React Native WebGPU integration](https://github.com/wcandillon/react-native-webgpu/blob/main/apps/docs/content/docs/integrations/vision-camera.mdx)
- ML compute: [Apple Core ML compute units](https://developer.apple.com/documentation/coreml/mlcomputeunits), [LiteRT NPU delegates](https://ai.google.dev/edge/litert/android/npu)