evals/evals.json
{
"skill_name": "vision-framework",
"evals": [
{
"id": 1,
"prompt": "Review this iOS 26 Vision helper: it uses `DetectBarcodesRequest` with `[VNBarcodeSymbology]`, stores modern Vision bounding boxes as `CGRect`, converts them with `VNImageRectForNormalizedRect`, starts object tracking with `TrackObjectRequest(observation:)`, sets `trackingLevel`, and calls `generateMask(forInstances:)` on `InstanceMaskObservation`. Provide corrected Swift snippets and explain the API generation boundary.",
"expected_output": "A concise correction that uses modern Swift Vision APIs for iOS 18+ while preserving legacy VNRequest guidance only where needed.",
"files": [],
"assertions": [
"Uses `BarcodeSymbology` with `DetectBarcodesRequest` and does not use `VNBarcodeSymbology` for modern barcode detection.",
"Treats modern observation bounds as `NormalizedRect` and uses `toImageCoordinates(_:origin:)` or `.cgRect` only when conversion is needed.",
"Uses `TrackObjectRequest(detectedObject:)` with `DetectedObjectObservation(boundingBox:)` and does not set `trackingLevel` on modern `TrackObjectRequest`.",
"Calls `InstanceMaskObservation.generateMask(for:)` for modern person-instance masks.",
"Keeps legacy `VNImageRectForNormalizedRect`, `VNBarcodeSymbology`, and `VNTrackObjectRequest.trackingLevel` scoped to legacy VN APIs or VisionKit where applicable."
]
},
{
"id": 2,
"prompt": "Build a SwiftUI live scanner for text and QR codes using VisionKit. Include permission and availability handling, the wrapper shape, barcode symbology types, and when scanning should start.",
"expected_output": "A SwiftUI DataScannerViewController integration that is safe to present and uses VisionKit-specific types correctly.",
"files": [],
"assertions": [
"Includes `NSCameraUsageDescription` and checks camera authorization before or around presentation.",
"Checks both `DataScannerViewController.isSupported` and `DataScannerViewController.isAvailable` before presenting.",
"Uses `DataScannerViewController.RecognizedDataType.barcode(symbologies:)` with VisionKit-supported `VNBarcodeSymbology` values, not modern Vision `BarcodeSymbology`.",
"Wraps `DataScannerViewController` in `UIViewControllerRepresentable` and starts scanning after presentation or on the main actor.",
"Mentions A12 Bionic or later support and a fallback for unavailable devices or visionOS."
]
},
{
"id": 3,
"prompt": "Implement an image-analysis feature that reads text, detects barcodes, estimates horizon angle, finds saliency regions, and optionally runs a custom Core ML classifier through Vision. Keep Core ML conversion and deployment out of scope.",
"expected_output": "A Vision-focused design that uses the correct result shapes for modern requests, treats any Core ML classifier as already prepared, and keeps Core ML conversion and deployment outside the Vision implementation.",
"files": [],
"assertions": [
"Uses `RecognizeTextRequest`, `DetectBarcodesRequest`, `DetectHorizonRequest`, and saliency requests with `perform(on:)` and async/await.",
"Handles `DetectHorizonRequest` as returning one `HorizonObservation` with `angle: Measurement<UnitAngle>`.",
"Handles attention/objectness saliency requests as returning one `SaliencyImageObservation`, not an array of observations.",
"Uses `CoreMLRequest` only for an already-prepared classifier model and handles classifier output as `ClassificationObservation`.",
"Keeps Core ML conversion and deployment out of scope instead of expanding model lifecycle guidance in the Vision answer."
]
}
]
}
references/vision-requests.md
# Vision Request Patterns
Complete implementation patterns for Vision framework requests covering text
recognition, face detection, barcode scanning, segmentation, classification,
and video processing. All patterns target iOS 26+ with Swift 6.3 unless noted.
## Contents
- Complete Text Recognition Pipeline
- Face Detection with Landmarks
- Barcode Detection with All Symbologies
- Person Segmentation with Mask Application
- Instance Segmentation (iOS 18+)
- Image Classification
- Saliency Detection
- Rectangle Detection
- Horizon Detection
- Batch Processing Multiple Requests
- Video Frame Processing with CMSampleBuffer
- Object Tracking Across Video Frames
- Coordinate Normalization Utilities
- Performance Considerations
## Complete Text Recognition Pipeline
Full pipeline from image loading through text extraction with coordinate mapping.
```swift
import Vision
import UIKit
@MainActor
final class TextRecognizer {
func recognizeText(in image: UIImage) async throws -> [RecognizedTextBlock] {
guard let cgImage = image.cgImage else {
throw TextRecognitionError.invalidImage
}
var request = RecognizeTextRequest()
request.recognitionLevel = .accurate
request.recognitionLanguages = [
Locale.Language(identifier: "en-US"),
]
request.usesLanguageCorrection = true
let observations = try await request.perform(on: cgImage)
let imageSize = CGSize(
width: cgImage.width,
height: cgImage.height
)
return observations.compactMap { observation in
guard let candidate = observation.topCandidates(1).first else { return nil }
let imageRect = observation.boundingBox.toImageCoordinates(
imageSize,
origin: .upperLeft
)
return RecognizedTextBlock(
text: candidate.string,
confidence: candidate.confidence,
boundingBox: imageRect
)
}
}
}
struct RecognizedTextBlock: Sendable {
let text: String
let confidence: Float
let boundingBox: CGRect
}
enum TextRecognitionError: Error {
case invalidImage
}
```
### Text Recognition with Language Hints
```swift
func recognizeMultilingualText(in cgImage: CGImage) async throws -> [String] {
var request = RecognizeTextRequest()
request.recognitionLevel = .accurate
request.recognitionLanguages = [
Locale.Language(identifier: "en-US"),
Locale.Language(identifier: "fr-FR"),
Locale.Language(identifier: "de-DE"),
]
request.usesLanguageCorrection = true
request.customWords = ["iOS", "SwiftUI", "Xcode"]
let observations = try await request.perform(on: cgImage)
return observations.compactMap { $0.topCandidates(1).first?.string }
}
```
### Fast Text Recognition for Live Video
```swift
func recognizeTextFast(in sampleBuffer: CMSampleBuffer) async throws -> [String] {
var request = RecognizeTextRequest()
request.recognitionLevel = .fast
request.recognitionLanguages = [Locale.Language(identifier: "en-US")]
let observations = try await request.perform(on: sampleBuffer)
return observations.compactMap { $0.topCandidates(1).first?.string }
}
```
### Legacy Text Recognition (Pre-iOS 18)
```swift
import Vision
func recognizeTextLegacy(
in cgImage: CGImage,
completion: @escaping ([String]) -> Void
) {
let request = VNRecognizeTextRequest { request, error in
guard error == nil,
let observations = request.results as? [VNRecognizedTextObservation]
else {
completion([])
return
}
let strings = observations.compactMap {
$0.topCandidates(1).first?.string
}
completion(strings)
}
request.recognitionLevel = .accurate
request.recognitionLanguages = ["en-US"]
request.usesLanguageCorrection = true
let handler = VNImageRequestHandler(cgImage: cgImage)
DispatchQueue.global(qos: .userInitiated).async {
try? handler.perform([request])
}
}
```
## Face Detection with Landmarks
```swift
import Vision
struct DetectedFace: Sendable {
let boundingBox: NormalizedRect
let landmarks: FaceLandmarkPoints?
let roll: Measurement<UnitAngle>
let yaw: Measurement<UnitAngle>
let captureQuality: FaceObservation.CaptureQuality?
}
struct FaceLandmarkPoints: Sendable {
let leftEye: [NormalizedPoint]
let rightEye: [NormalizedPoint]
let nose: [NormalizedPoint]
let outerLips: [NormalizedPoint]
let faceContour: [NormalizedPoint]
}
func detectFaces(in cgImage: CGImage) async throws -> [DetectedFace] {
// Detect face rectangles
let rectRequest = DetectFaceRectanglesRequest()
let faces = try await rectRequest.perform(on: cgImage)
// Detect landmarks for detailed features
let landmarkRequest = DetectFaceLandmarksRequest()
let landmarkFaces = try await landmarkRequest.perform(on: cgImage)
// Detect capture quality for photo selection
let qualityRequest = DetectFaceCaptureQualityRequest()
let qualityFaces = try await qualityRequest.perform(on: cgImage)
return faces.enumerated().map { index, face in
let landmarks: FaceLandmarkPoints?
if index < landmarkFaces.count,
let lm = landmarkFaces[index].landmarks {
landmarks = FaceLandmarkPoints(
leftEye: lm.leftEye.points,
rightEye: lm.rightEye.points,
nose: lm.nose.points,
outerLips: lm.outerLips.points,
faceContour: lm.faceContour.points
)
} else {
landmarks = nil
}
let quality: FaceObservation.CaptureQuality?
if index < qualityFaces.count {
quality = qualityFaces[index].captureQuality
} else {
quality = nil
}
return DetectedFace(
boundingBox: face.boundingBox,
landmarks: landmarks,
roll: face.roll,
yaw: face.yaw,
captureQuality: quality
)
}
}
```
## Barcode Detection with All Symbologies
```swift
import Vision
struct DetectedBarcode: Sendable {
let payload: String?
let symbology: BarcodeSymbology
let boundingBox: NormalizedRect
}
func detectBarcodes(
in cgImage: CGImage,
symbologies: [BarcodeSymbology] = [.qr, .ean13, .code128]
) async throws -> [DetectedBarcode] {
var request = DetectBarcodesRequest()
request.symbologies = symbologies
let observations = try await request.perform(on: cgImage)
return observations.map { barcode in
DetectedBarcode(
payload: barcode.payloadString,
symbology: barcode.symbology,
boundingBox: barcode.boundingBox
)
}
}
// Detect only QR codes with URL content
func detectQRCodes(in cgImage: CGImage) async throws -> [URL] {
var request = DetectBarcodesRequest()
request.symbologies = [.qr]
let observations = try await request.perform(on: cgImage)
return observations.compactMap { barcode in
guard let payload = barcode.payloadString else { return nil }
return URL(string: payload)
}
}
```
### Supported Symbologies Reference
```swift
// 1D barcodes
let linearSymbologies: [BarcodeSymbology] = [
.codabar, .code39, .code39Checksum, .code39FullASCII,
.code39FullASCIIChecksum, .code93, .code93i, .code128,
.ean8, .ean13, .gs1DataBar, .gs1DataBarExpanded,
.gs1DataBarLimited, .i2of5, .i2of5Checksum, .itf14,
.msiPlessey, .upce,
]
// 2D barcodes
let matrixSymbologies: [BarcodeSymbology] = [
.qr, .aztec, .dataMatrix, .pdf417, .microPDF417, .microQR,
]
```
## Person Segmentation with Mask Application
### Modern API (iOS 18+)
```swift
import Vision
import CoreImage
import CoreImage.CIFilterBuiltins
func segmentPerson(in cgImage: CGImage) async throws -> CIImage {
var request = GeneratePersonSegmentationRequest()
request.qualityLevel = .accurate // .balanced, .fast
let observation = try await request.perform(on: cgImage)
let maskBuffer = observation.pixelBuffer
let originalImage = CIImage(cgImage: cgImage)
let maskImage = CIImage(cvPixelBuffer: maskBuffer)
// Scale mask to match original image size
let scaleX = originalImage.extent.width / maskImage.extent.width
let scaleY = originalImage.extent.height / maskImage.extent.height
let scaledMask = maskImage.transformed(by: CGAffineTransform(
scaleX: scaleX, y: scaleY
))
return scaledMask
}
// Apply background blur using person mask
func blurBackground(of cgImage: CGImage, blurRadius: Double = 20.0) async throws -> CIImage {
let mask = try await segmentPerson(in: cgImage)
let original = CIImage(cgImage: cgImage)
let blurFilter = CIFilter.gaussianBlur()
blurFilter.inputImage = original
blurFilter.radius = Float(blurRadius)
guard let blurredImage = blurFilter.outputImage else {
throw SegmentationError.noMask
}
let blendFilter = CIFilter.blendWithMask()
blendFilter.inputImage = original // foreground (person)
blendFilter.backgroundImage = blurredImage // blurred background
blendFilter.maskImage = mask
guard let result = blendFilter.outputImage else {
throw SegmentationError.noMask
}
return result
}
enum SegmentationError: Error {
case noMask
}
```
### Legacy API (Pre-iOS 18)
```swift
func segmentPersonLegacy(in cgImage: CGImage) throws -> CVPixelBuffer {
let request = VNGeneratePersonSegmentationRequest()
request.qualityLevel = .accurate
request.outputPixelFormat = kCVPixelFormatType_OneComponent8
let handler = VNImageRequestHandler(cgImage: cgImage)
try handler.perform([request])
guard let maskBuffer = request.results?.first?.pixelBuffer else {
throw SegmentationError.noMask
}
return maskBuffer
}
```
### Instance Segmentation (iOS 18+)
Separate masks per person for individual effects.
```swift
// Modern API (iOS 18+)
func segmentIndividualPeople(in cgImage: CGImage) async throws -> [CVPixelBuffer] {
let request = GeneratePersonInstanceMaskRequest()
let observation = try await request.perform(on: cgImage)
let indices = observation.allInstances
return try indices.map { index in
try observation.generateMask(for: IndexSet(integer: index))
}
}
```
```swift
// Legacy API (iOS 17+)
func segmentIndividualPeopleLegacy(in cgImage: CGImage) throws -> [CVPixelBuffer] {
let request = VNGeneratePersonInstanceMaskRequest()
let handler = VNImageRequestHandler(cgImage: cgImage)
try handler.perform([request])
guard let result = request.results?.first else { return [] }
let indices = result.allInstances
return try indices.map { index in
try result.generateMask(forInstances: IndexSet(integer: index))
}
}
```
## Image Classification
```swift
import Vision
func classifyImage(_ cgImage: CGImage, maxResults: Int = 5) async throws -> [(String, Float)] {
let request = ClassifyImageRequest()
let observations = try await request.perform(on: cgImage)
return observations.prefix(maxResults).map { observation in
(observation.identifier, observation.confidence)
}
}
```
## Saliency Detection
Identify the most visually important or attention-grabbing regions.
```swift
// Attention-based saliency (what humans would look at)
func detectAttentionSaliency(in cgImage: CGImage) async throws -> [NormalizedRect] {
let request = GenerateAttentionBasedSaliencyImageRequest()
let saliency: SaliencyImageObservation = try await request.perform(on: cgImage)
return saliency.salientObjects?.map(\.boundingBox) ?? []
}
// Objectness-based saliency (distinct objects)
func detectObjectSaliency(in cgImage: CGImage) async throws -> [NormalizedRect] {
let request = GenerateObjectnessBasedSaliencyImageRequest()
let saliency: SaliencyImageObservation = try await request.perform(on: cgImage)
return saliency.salientObjects?.map(\.boundingBox) ?? []
}
```
## Rectangle Detection
Detect rectangular shapes for document edges, business cards, etc.
```swift
func detectRectangles(in cgImage: CGImage) async throws -> [NormalizedRect] {
var request = DetectRectanglesRequest()
request.minimumAspectRatio = 0.3
request.maximumAspectRatio = 1.0
request.minimumSize = 0.1
request.maximumObservations = 5
let observations = try await request.perform(on: cgImage)
return observations.map(\.boundingBox)
}
```
## Horizon Detection
Detect the horizon angle for auto-straightening photos.
```swift
func detectHorizon(in cgImage: CGImage) async throws -> Measurement<UnitAngle> {
let request = DetectHorizonRequest()
let observation = try await request.perform(on: cgImage)
return observation.angle
}
```
## Batch Processing Multiple Requests
Run multiple requests on the same image simultaneously for efficiency.
```swift
func analyzeImage(_ cgImage: CGImage) async throws -> ImageAnalysisResult {
async let textResults = {
var req = RecognizeTextRequest()
req.recognitionLevel = .accurate
return try await req.perform(on: cgImage)
}()
async let faceResults = {
let req = DetectFaceRectanglesRequest()
return try await req.perform(on: cgImage)
}()
async let barcodeResults = {
var req = DetectBarcodesRequest()
req.symbologies = [.qr, .ean13]
return try await req.perform(on: cgImage)
}()
let text = try await textResults
let faces = try await faceResults
let barcodes = try await barcodeResults
return ImageAnalysisResult(
recognizedText: text.compactMap { $0.topCandidates(1).first?.string },
faceCount: faces.count,
barcodePayloads: barcodes.compactMap(\.payloadString)
)
}
struct ImageAnalysisResult: Sendable {
let recognizedText: [String]
let faceCount: Int
let barcodePayloads: [String]
}
```
### Legacy Batch Processing
With the legacy API, pass multiple requests to a single handler call.
```swift
func analyzeImageLegacy(_ cgImage: CGImage) throws {
let textRequest = VNRecognizeTextRequest { request, error in
// Handle text results
}
let faceRequest = VNDetectFaceRectanglesRequest { request, error in
// Handle face results
}
let barcodeRequest = VNDetectBarcodesRequest { request, error in
// Handle barcode results
}
let handler = VNImageRequestHandler(cgImage: cgImage)
try handler.perform([textRequest, faceRequest, barcodeRequest])
}
```
## Video Frame Processing with CMSampleBuffer
Process live camera frames from AVCaptureSession.
```swift
import AVFoundation
import Vision
final class VisionVideoProcessor: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate, Sendable {
private let processingQueue = DispatchQueue(label: "vision.processing", qos: .userInitiated)
func setupCapture(session: AVCaptureSession) {
let output = AVCaptureVideoDataOutput()
output.setSampleBufferDelegate(self, queue: processingQueue)
output.alwaysDiscardsLateVideoFrames = true
if session.canAddOutput(output) {
session.addOutput(output)
}
}
func captureOutput(
_ output: AVCaptureOutput,
didOutput sampleBuffer: CMSampleBuffer,
from connection: AVCaptureConnection
) {
Task {
do {
var request = RecognizeTextRequest()
request.recognitionLevel = .fast
let observations = try await request.perform(on: sampleBuffer)
let strings = observations.compactMap {
$0.topCandidates(1).first?.string
}
// Dispatch results to main actor for UI update
await MainActor.run {
// Update UI with recognized strings
}
} catch {
// Handle error
}
}
}
}
```
### Object Tracking Across Video Frames
#### Modern API (iOS 18+)
`TrackObjectRequest` is a stateful request that maintains tracking context
internally. No need for a separate sequence handler.
```swift
import Vision
final class ObjectTracker {
private var request: TrackObjectRequest?
/// Initialize tracking with a bounding box in normalized coordinates
func startTracking(boundingBox: NormalizedRect) {
let observation = DetectedObjectObservation(boundingBox: boundingBox)
request = TrackObjectRequest(detectedObject: observation)
}
/// Track object in next video frame
func track(in pixelBuffer: CVPixelBuffer) async throws -> NormalizedRect? {
guard let request else { return nil }
let results = try await request.perform(on: pixelBuffer)
guard let tracked = results.first else {
request = nil
return nil
}
return tracked.boundingBox
}
func stopTracking() {
request = nil
}
}
```
#### Legacy API
```swift
final class LegacyObjectTracker {
private var sequenceHandler = VNSequenceRequestHandler()
private var currentObservation: VNDetectedObjectObservation?
func startTracking(boundingBox: CGRect) {
currentObservation = VNDetectedObjectObservation(boundingBox: boundingBox)
}
func track(in pixelBuffer: CVPixelBuffer) throws -> CGRect? {
guard let observation = currentObservation else { return nil }
let trackRequest = VNTrackObjectRequest(detectedObjectObservation: observation)
trackRequest.trackingLevel = .accurate
try sequenceHandler.perform([trackRequest], on: pixelBuffer)
guard let result = trackRequest.results?.first as? VNDetectedObjectObservation,
result.confidence > 0.3 else {
currentObservation = nil
return nil
}
currentObservation = result
return result.boundingBox
}
func stopTracking() {
currentObservation = nil
}
}
```
## Coordinate Normalization Utilities
Vision uses normalized coordinates (0...1) with bottom-left origin. These
utilities convert to UIKit/SwiftUI coordinate systems.
```swift
import Vision
import UIKit
enum VisionCoordinateConverter {
/// Convert modern Vision NormalizedRect to image-pixel coordinates
static func toImageCoordinates(
_ normalizedRect: NormalizedRect,
imageSize: CGSize
) -> CGRect {
normalizedRect.toImageCoordinates(imageSize, origin: .upperLeft)
}
/// Convert legacy normalized Vision rect to image-pixel coordinates
static func toImageCoordinates(
_ normalizedRect: CGRect,
imageWidth: Int,
imageHeight: Int
) -> CGRect {
VNImageRectForNormalizedRect(normalizedRect, imageWidth, imageHeight)
}
/// Convert legacy normalized Vision point to image-pixel coordinates
static func toImageCoordinates(
_ normalizedPoint: CGPoint,
imageWidth: Int,
imageHeight: Int
) -> CGPoint {
VNImagePointForNormalizedPoint(normalizedPoint, imageWidth, imageHeight)
}
/// Convert modern Vision rect directly to UIKit/image coordinates
static func toUIKitCoordinates(
_ normalizedRect: NormalizedRect,
viewSize: CGSize
) -> CGRect {
normalizedRect.toImageCoordinates(viewSize, origin: .upperLeft)
}
/// Convert an array of modern normalized points to UIKit points
static func toUIKitPoints(
_ normalizedPoints: [NormalizedPoint],
viewSize: CGSize
) -> [CGPoint] {
normalizedPoints.map {
$0.toImageCoordinates(viewSize, origin: .upperLeft)
}
}
/// Convert an array of legacy normalized points to UIKit points
static func toUIKitPoints(
_ normalizedPoints: [CGPoint],
viewSize: CGSize
) -> [CGPoint] {
normalizedPoints.map { point in
CGPoint(
x: point.x * viewSize.width,
y: (1.0 - point.y) * viewSize.height // flip Y
)
}
}
}
```
## Performance Considerations
### Recognition Level Selection
| Use Case | Level | Typical Latency |
|---|---|---|
| Live camera preview | `.fast` | ~30ms per frame |
| Photo library scan | `.accurate` | ~200-500ms per image |
| Batch document OCR | `.accurate` | ~200-500ms per page |
| Barcode scanner | `.fast` or `.balanced` | ~15-50ms per frame |
### Memory Management
- Reuse `VNSequenceRequestHandler` across video frames (do not recreate per frame)
- For batch processing, process one image at a time to avoid memory spikes
- Release `CVPixelBuffer` references promptly after processing
- Use `autoreleasepool` in tight loops processing many images
```swift
func batchProcess(images: [CGImage]) async throws -> [[String]] {
var allResults: [[String]] = []
for image in images {
var request = RecognizeTextRequest()
request.recognitionLevel = .accurate
let obs = try await request.perform(on: image)
let result = obs.compactMap { $0.topCandidates(1).first?.string }
allResults.append(result)
}
return allResults
}
```
### Threading
- Modern API (`perform(on:)`) is async and safe to call from any context
- Legacy API: create `VNImageRequestHandler` and call `perform` on a background queue
- Never block the main thread with Vision requests
- `VNSequenceRequestHandler` is not thread-safe -- use from a single serial queue
### Request Reuse
Most modern stateless request structs are cheap to create. Use a fresh request
for independent still-image work, and keep stateful final-class requests such as
`TrackObjectRequest` only for the frame sequence that needs their state.
For the legacy API, `VNImageRequestHandler` is tied to a single image. Create a
new handler for each image you process. `VNSequenceRequestHandler` can be reused
across frames in a sequence.
references/visionkit-scanner.md
# VisionKit Scanner Patterns
Complete implementation patterns for DataScannerViewController and
VNDocumentCameraViewController covering availability checking, configuration,
SwiftUI integration, delegate handling, custom overlays, and camera permissions.
All patterns target iOS 26+ with Swift 6.3 unless noted.
## Contents
- Camera Permission Setup
- DataScannerViewController
- Delegate Methods
- SwiftUI Integration
- Custom Overlay UI
- VNDocumentCameraViewController
## Camera Permission Setup
Add the camera usage description to Info.plist before using any scanner:
```xml
<key>NSCameraUsageDescription</key>
<string>Camera access is needed to scan text and barcodes.</string>
```
Request permission before presenting the scanner. The canonical order is:
Info.plist usage string, explicit camera access request, `isSupported` and
`isAvailable` checks, then present the scanner and call `startScanning()` after
presentation on the main actor.
```swift
import AVFoundation
func requestCameraAccess() async -> Bool {
let status = AVCaptureDevice.authorizationStatus(for: .video)
switch status {
case .authorized:
return true
case .notDetermined:
return await AVCaptureDevice.requestAccess(for: .video)
case .denied, .restricted:
return false
@unknown default:
return false
}
}
```
## DataScannerViewController
`DataScannerViewController` provides a full-screen live camera scanner for text
and barcodes with built-in highlighting and interaction. Available on devices
with an A12 Bionic chip or later (iOS 16+), but unsupported for apps running in
visionOS.
### Availability Checking
Always check both hardware support and runtime availability before presenting.
```swift
import Vision
import VisionKit
func canUseDataScanner() -> Bool {
// Hardware check: requires A12 Bionic or later
guard DataScannerViewController.isSupported else {
return false
}
// Runtime check: camera authorized and not restricted
guard DataScannerViewController.isAvailable else {
return false
}
return true
}
```
`isSupported` checks hardware and platform capability (A12+ and not visionOS).
`isAvailable` checks that the camera is authorized and not restricted by Screen
Time or device management. Both must be true.
For barcode scanner configuration, VisionKit uses `VNBarcodeSymbology` values in
`DataScannerViewController.RecognizedDataType.barcode(symbologies:)`. Do not
substitute modern Vision's `BarcodeSymbology` there.
### Configuration and Initialization
```swift
import VisionKit
func createTextScanner() -> DataScannerViewController {
DataScannerViewController(
recognizedDataTypes: [
.text(languages: ["en"]),
],
qualityLevel: .balanced,
recognizesMultipleItems: true,
isHighFrameRateTrackingEnabled: true,
isPinchToZoomEnabled: true,
isGuidanceEnabled: true,
isHighlightingEnabled: true
)
}
func createBarcodeScanner() -> DataScannerViewController {
let barcodeSymbologies: [VNBarcodeSymbology] = [.qr, .ean13, .code128]
DataScannerViewController(
recognizedDataTypes: [
.barcode(symbologies: barcodeSymbologies),
],
qualityLevel: .fast,
recognizesMultipleItems: false,
isHighFrameRateTrackingEnabled: false,
isPinchToZoomEnabled: false,
isGuidanceEnabled: true,
isHighlightingEnabled: true
)
}
func createMixedScanner() -> DataScannerViewController {
let barcodeSymbologies: [VNBarcodeSymbology] = [.qr, .ean13]
DataScannerViewController(
recognizedDataTypes: [
.text(languages: ["en"]),
.barcode(symbologies: barcodeSymbologies),
],
qualityLevel: .balanced,
recognizesMultipleItems: true,
isHighFrameRateTrackingEnabled: true,
isPinchToZoomEnabled: true,
isGuidanceEnabled: true,
isHighlightingEnabled: true
)
}
```
### Recognized Data Types
```swift
// Text with language hints
let textType: DataScannerViewController.RecognizedDataType =
.text(languages: ["en", "fr", "de"])
// Text filtered by content type
let emailType: DataScannerViewController.RecognizedDataType =
.text(textContentType: .emailAddress)
let urlType: DataScannerViewController.RecognizedDataType =
.text(textContentType: .URL)
let phoneType: DataScannerViewController.RecognizedDataType =
.text(textContentType: .telephoneNumber)
let addressType: DataScannerViewController.RecognizedDataType =
.text(textContentType: .fullAddress)
let flightType: DataScannerViewController.RecognizedDataType =
.text(textContentType: .flightNumber)
let trackingType: DataScannerViewController.RecognizedDataType =
.text(textContentType: .shipmentTrackingNumber)
// Barcode with specific symbologies
let qrOnly: DataScannerViewController.RecognizedDataType =
.barcode(symbologies: [.qr])
let retailBarcodes: DataScannerViewController.RecognizedDataType =
.barcode(symbologies: [.ean8, .ean13, .upce, .code128])
```
### Quality Levels
| Level | Use Case | Notes |
|---|---|---|
| `.fast` | Barcode scanning, quick text grab | Lowest latency |
| `.balanced` | General purpose text + barcode | Default choice |
| `.accurate` | Detailed OCR, small text | Higher latency |
### Starting and Stopping
```swift
func presentScanner(_ scanner: DataScannerViewController,
from presenter: UIViewController) {
scanner.delegate = presenter as? DataScannerViewControllerDelegate
presenter.present(scanner, animated: true) {
try? scanner.startScanning()
}
}
func dismissScanner(_ scanner: DataScannerViewController) {
scanner.stopScanning()
scanner.dismiss(animated: true)
}
```
## Delegate Methods
Implement `DataScannerViewControllerDelegate` to handle recognized items and
scanner lifecycle events.
```swift
import VisionKit
final class ScannerCoordinator: NSObject, DataScannerViewControllerDelegate {
var hasStartedScanning = false
var onTextRecognized: ((String) -> Void)?
var onBarcodeRecognized: ((String, VNBarcodeSymbology) -> Void)?
// Called when the user taps on a recognized item
func dataScanner(
_ scanner: DataScannerViewController,
didTapOn item: RecognizedItem
) {
switch item {
case .text(let text):
onTextRecognized?(text.transcript)
case .barcode(let barcode):
if let payload = barcode.payloadStringValue {
onBarcodeRecognized?(payload, barcode.observation.symbology)
}
@unknown default:
break
}
}
// Called when new items appear in the camera view
func dataScanner(
_ scanner: DataScannerViewController,
didAdd addedItems: [RecognizedItem],
allItems: [RecognizedItem]
) {
for item in addedItems {
switch item {
case .text(let text):
print("New text: \(text.transcript)")
case .barcode(let barcode):
print("New barcode: \(barcode.payloadStringValue ?? "nil")")
@unknown default:
break
}
}
}
// Called when items are updated (position or content changes)
func dataScanner(
_ scanner: DataScannerViewController,
didUpdate updatedItems: [RecognizedItem],
allItems: [RecognizedItem]
) {
// Handle position or content updates
}
// Called when items leave the camera view
func dataScanner(
_ scanner: DataScannerViewController,
didRemove removedItems: [RecognizedItem],
allItems: [RecognizedItem]
) {
// Clean up UI for removed items
}
// Called when the scanner becomes unavailable (e.g., camera revoked)
func dataScannerDidChangeUnavailabilityReasons(
_ scanner: DataScannerViewController
) {
// Handle unavailability -- dismiss or show fallback
}
}
```
### Async Sequence for Recognized Items
Use `recognizedItems` for a reactive stream of all currently visible items:
```swift
func observeRecognizedItems(_ scanner: DataScannerViewController) async {
for await items in scanner.recognizedItems {
let texts = items.compactMap { item -> String? in
guard case .text(let text) = item else { return nil }
return text.transcript
}
let barcodes = items.compactMap { item -> String? in
guard case .barcode(let barcode) = item else { return nil }
return barcode.payloadStringValue
}
await MainActor.run {
// Update UI with current texts and barcodes
}
}
}
```
### Capturing a Photo
Capture a still image from the scanner for further processing:
```swift
func captureAndProcess(_ scanner: DataScannerViewController) async throws {
let photo = try await scanner.capturePhoto()
// photo is a UIImage -- process with Vision or save
}
```
## SwiftUI Integration
Wrap `DataScannerViewController` in `UIViewControllerRepresentable` for use
in SwiftUI views.
### Full DataScanner Representable
```swift
import SwiftUI
import AVFoundation
import Vision
import VisionKit
struct DataScannerRepresentable: UIViewControllerRepresentable {
let recognizedDataTypes: Set<DataScannerViewController.RecognizedDataType>
let qualityLevel: DataScannerViewController.QualityLevel
let recognizesMultipleItems: Bool
@Binding var recognizedText: [String]
@Binding var recognizedBarcodes: [String]
func makeUIViewController(context: Context) -> DataScannerViewController {
let scanner = DataScannerViewController(
recognizedDataTypes: recognizedDataTypes,
qualityLevel: qualityLevel,
recognizesMultipleItems: recognizesMultipleItems,
isHighFrameRateTrackingEnabled: true,
isPinchToZoomEnabled: true,
isGuidanceEnabled: true,
isHighlightingEnabled: true
)
scanner.delegate = context.coordinator
return scanner
}
func updateUIViewController(
_ controller: DataScannerViewController,
context: Context
) {
guard !context.coordinator.hasStartedScanning else { return }
context.coordinator.hasStartedScanning = true
Task { @MainActor in
// SwiftUI has inserted the controller by the time update runs.
try? controller.startScanning()
}
}
func makeCoordinator() -> Coordinator {
Coordinator(parent: self)
}
static func dismantleUIViewController(
_ controller: DataScannerViewController,
coordinator: Coordinator
) {
controller.stopScanning()
}
@MainActor
final class Coordinator: NSObject, DataScannerViewControllerDelegate {
let parent: DataScannerRepresentable
var hasStartedScanning = false
init(parent: DataScannerRepresentable) {
self.parent = parent
}
func dataScanner(
_ scanner: DataScannerViewController,
didTapOn item: RecognizedItem
) {
switch item {
case .text(let text):
parent.recognizedText.append(text.transcript)
case .barcode(let barcode):
if let payload = barcode.payloadStringValue {
parent.recognizedBarcodes.append(payload)
}
@unknown default:
break
}
}
func dataScanner(
_ scanner: DataScannerViewController,
didAdd addedItems: [RecognizedItem],
allItems: [RecognizedItem]
) {
// Handle newly recognized items
}
func dataScanner(
_ scanner: DataScannerViewController,
didUpdate updatedItems: [RecognizedItem],
allItems: [RecognizedItem]
) {
// Handle item updates
}
func dataScanner(
_ scanner: DataScannerViewController,
didRemove removedItems: [RecognizedItem],
allItems: [RecognizedItem]
) {
// Handle removed items
}
}
}
```
### SwiftUI Scanner View
```swift
import SwiftUI
import AVFoundation
import VisionKit
struct ScannerView: View {
@State private var recognizedText: [String] = []
@State private var recognizedBarcodes: [String] = []
@State private var isShowingScanner = false
@State private var scannerUnavailable = false
var body: some View {
VStack {
if DataScannerViewController.isSupported {
Button("Scan") {
Task { @MainActor in
guard await requestCameraAccess(),
DataScannerViewController.isAvailable else {
scannerUnavailable = true
return
}
scannerUnavailable = false
isShowingScanner = true
}
}
.fullScreenCover(isPresented: $isShowingScanner) {
let barcodeSymbologies: [VNBarcodeSymbology] = [.qr]
NavigationStack {
DataScannerRepresentable(
recognizedDataTypes: [
.text(languages: ["en"]),
.barcode(symbologies: barcodeSymbologies),
],
qualityLevel: .balanced,
recognizesMultipleItems: true,
recognizedText: $recognizedText,
recognizedBarcodes: $recognizedBarcodes
)
.ignoresSafeArea()
.toolbar {
ToolbarItem(placement: .cancellationAction) {
Button("Done") {
isShowingScanner = false
}
}
}
}
}
} else {
ContentUnavailableView(
"Scanner Not Available",
systemImage: "camera.fill",
description: Text("This device does not support scanning.")
)
}
if scannerUnavailable {
ContentUnavailableView(
"Scanner Not Available",
systemImage: "camera.fill",
description: Text("Camera access is required to scan.")
)
}
List {
Section("Text") {
ForEach(recognizedText, id: \.self) { text in
Text(text)
}
}
Section("Barcodes") {
ForEach(recognizedBarcodes, id: \.self) { barcode in
Text(barcode)
}
}
}
}
}
}
```
### Starting the Scanner After Presentation
The scanner must be started after the view controller is fully presented.
Use `onAppear` with a coordinator flag or start in the completion handler:
```swift
struct AutoStartScannerRepresentable: UIViewControllerRepresentable {
func makeUIViewController(context: Context) -> DataScannerViewController {
let scanner = DataScannerViewController(
recognizedDataTypes: [.text(languages: ["en"])],
qualityLevel: .balanced,
recognizesMultipleItems: false,
isHighFrameRateTrackingEnabled: true,
isHighlightingEnabled: true
)
scanner.delegate = context.coordinator
return scanner
}
func updateUIViewController(
_ controller: DataScannerViewController,
context: Context
) {
guard !context.coordinator.hasStartedScanning else { return }
context.coordinator.hasStartedScanning = true
Task { @MainActor in
// updateUIViewController runs after SwiftUI has inserted the controller.
try? controller.startScanning()
}
}
func makeCoordinator() -> ScannerCoordinator {
ScannerCoordinator()
}
static func dismantleUIViewController(
_ controller: DataScannerViewController,
coordinator: ScannerCoordinator
) {
controller.stopScanning()
}
}
```
## Custom Overlay UI
Add custom views on top of the scanner for region-of-interest indicators,
instructions, or result display.
### Overlay with Region of Interest
```swift
struct ScannerWithOverlay: View {
@State private var isShowingScanner = false
@State private var lastScannedText = ""
var body: some View {
ZStack {
AutoStartScannerRepresentable()
.ignoresSafeArea()
VStack {
// Top instruction bar
Text("Point camera at text or barcode")
.font(.subheadline)
.padding(.horizontal)
.padding(.vertical)
.background(.ultraThinMaterial, in: Capsule())
.padding(.top)
Spacer()
// Scan region indicator
RoundedRectangle(cornerRadius: 12)
.strokeBorder(.white.opacity(0.6), lineWidth: 2)
.frame(width: 280, height: 180)
Spacer()
// Result display
if !lastScannedText.isEmpty {
Text(lastScannedText)
.font(.body)
.padding()
.frame(maxWidth: .infinity)
.background(.ultraThinMaterial)
.clipShape(.rect(cornerRadius: 12))
.padding()
}
}
}
}
}
```
## VNDocumentCameraViewController
`VNDocumentCameraViewController` provides a full-screen document camera with
auto-capture, perspective correction, and multi-page scanning. Available on
all devices running iOS 13+.
### UIKit Presentation
```swift
import VisionKit
final class DocumentScannerPresenter: NSObject,
VNDocumentCameraViewControllerDelegate
{
weak var presenter: UIViewController?
func showDocumentScanner() {
let scanner = VNDocumentCameraViewController()
scanner.delegate = self
presenter?.present(scanner, animated: true)
}
func documentCameraViewController(
_ controller: VNDocumentCameraViewController,
didFinishWith scan: VNDocumentCameraScan
) {
controller.dismiss(animated: true)
for pageIndex in 0..<scan.pageCount {
let pageImage = scan.imageOfPage(at: pageIndex)
// Process each scanned page image
}
}
func documentCameraViewControllerDidCancel(
_ controller: VNDocumentCameraViewController
) {
controller.dismiss(animated: true)
}
func documentCameraViewController(
_ controller: VNDocumentCameraViewController,
didFailWithError error: Error
) {
controller.dismiss(animated: true)
// Handle scanning error
}
}
```
### SwiftUI Document Scanner
```swift
import SwiftUI
import VisionKit
struct DocumentScannerRepresentable: UIViewControllerRepresentable {
@Binding var scannedImages: [UIImage]
@Environment(\.dismiss) private var dismiss
func makeUIViewController(context: Context) -> VNDocumentCameraViewController {
let scanner = VNDocumentCameraViewController()
scanner.delegate = context.coordinator
return scanner
}
func updateUIViewController(
_ controller: VNDocumentCameraViewController,
context: Context
) {}
func makeCoordinator() -> Coordinator {
Coordinator(parent: self)
}
@MainActor
final class Coordinator: NSObject, VNDocumentCameraViewControllerDelegate {
let parent: DocumentScannerRepresentable
init(parent: DocumentScannerRepresentable) {
self.parent = parent
}
func documentCameraViewController(
_ controller: VNDocumentCameraViewController,
didFinishWith scan: VNDocumentCameraScan
) {
parent.scannedImages = (0..<scan.pageCount).map { scan.imageOfPage(at: $0) }
parent.dismiss()
}
func documentCameraViewControllerDidCancel(
_ controller: VNDocumentCameraViewController
) {
parent.dismiss()
}
func documentCameraViewController(
_ controller: VNDocumentCameraViewController,
didFailWithError error: Error
) {
parent.dismiss()
}
}
}
```
### Document Scanner with OCR Pipeline
Combine document scanning with Vision text recognition for a complete OCR flow:
```swift
import SwiftUI
import VisionKit
import Vision
@MainActor
@Observable
final class DocumentOCRModel {
var scannedPages: [UIImage] = []
var extractedText: [String] = []
var isProcessing = false
func processScannedPages() async {
isProcessing = true
defer { isProcessing = false }
extractedText = []
for page in scannedPages {
guard let cgImage = page.cgImage else { continue }
do {
var request = RecognizeTextRequest()
request.recognitionLevel = .accurate
request.recognitionLanguages = [Locale.Language(identifier: "en-US")]
request.usesLanguageCorrection = true
let observations = try await request.perform(on: cgImage)
let pageText = observations
.compactMap { $0.topCandidates(1).first?.string }
.joined(separator: "\n")
extractedText.append(pageText)
} catch {
extractedText.append("[Recognition failed]")
}
}
}
}
struct DocumentOCRView: View {
@State private var model = DocumentOCRModel()
@State private var isShowingScanner = false
var body: some View {
NavigationStack {
List {
if model.isProcessing {
ProgressView("Recognizing text...")
}
ForEach(Array(model.extractedText.enumerated()), id: \.offset) { index, text in
Section("Page \(index + 1)") {
Text(text)
.font(.body)
.textSelection(.enabled)
}
}
}
.navigationTitle("Document OCR")
.toolbar {
Button("Scan") {
isShowingScanner = true
}
}
.fullScreenCover(isPresented: $isShowingScanner) {
DocumentScannerRepresentable(scannedImages: $model.scannedPages)
}
.onChange(of: model.scannedPages) {
Task { await model.processScannedPages() }
}
}
}
}
```
## Performance Considerations
### DataScannerViewController
- Use `.fast` quality for barcode-only scanning
- Set `recognizesMultipleItems = false` when only one result is needed
- Disable `isHighFrameRateTrackingEnabled` for barcode scanning to save power
- Limit `recognizedDataTypes` to only what you need
- Stop scanning when processing results to avoid wasted CPU cycles
### VNDocumentCameraViewController
- Pages are returned as `UIImage` at full resolution -- resize before
processing if memory is a concern
- Process pages sequentially to avoid memory spikes
- Use `autoreleasepool` when processing many pages in a loop
SKILL.md
---
name: vision-framework
description: "Implement computer vision features including text recognition (OCR), face detection, barcode scanning, image segmentation, object tracking, and document scanning in iOS apps. Covers both the modern Swift-native Vision API (iOS 18+) and legacy VNRequest patterns, VisionKit DataScannerViewController for live camera scanning, and CoreMLRequest/VNCoreMLRequest for custom model inference. Use when adding OCR, barcode scanning, face detection, or custom Core ML model inference with Vision."
---
# Vision Framework
Detect text, faces, barcodes, objects, and body poses in images and video using
on-device computer vision. Prefer the modern iOS 18+ request APIs and load the legacy reference only when the deployment target requires it.
See [references/vision-requests.md](references/vision-requests.md) for complete code patterns and
[references/visionkit-scanner.md](references/visionkit-scanner.md) for DataScannerViewController integration.
## Contents
- [Two API Generations](#two-api-generations)
- [Request Pattern (Modern API)](#request-pattern-modern-api)
- [Text Recognition (OCR)](#text-recognition-ocr)
- [Face Detection](#face-detection)
- [Barcode Detection](#barcode-detection)
- [Document Scanning (iOS 26+)](#document-scanning-ios-26)
- [Image Segmentation](#image-segmentation)
- [Object Tracking](#object-tracking)
- [Other Request Types](#other-request-types)
- [Core ML Integration](#core-ml-integration)
- [VisionKit: DataScannerViewController](#visionkit-datascannerviewcontroller)
- [Common Mistakes](#common-mistakes)
- [Review Checklist](#review-checklist)
- [References](#references)
## Two API Generations
Vision has two distinct API layers. Prefer the modern API for new code:
Swift-native request types plus `try await request.perform(on:)`. Keep `VN*`,
`VNImageRequestHandler`, `VNSequenceRequestHandler`, completion handlers, and
legacy `CGRect` helpers inside explicit legacy fallback sections or files.
| Aspect | Modern (iOS 18+) | Legacy |
|---|---|---|
| Pattern | `let result = try await request.perform(on: image)` | `VNImageRequestHandler` + completion handler |
| Request types | Swift types — structs and classes (`RecognizeTextRequest`, `DetectFaceRectanglesRequest`) | ObjC classes (`VNRecognizeTextRequest`, `VNDetectFaceRectanglesRequest`) |
| Concurrency | Native async/await | Completion handlers or synchronous `perform` |
| Observations | Typed return values | Cast `results` from `[Any]` |
| Availability | iOS 18+ / macOS 15+ | iOS 11+ |
The modern API uses the `ImageProcessingRequest` protocol. Each request type
has a `perform(on:orientation:)` method that accepts `CGImage`, `CIImage`,
`CVPixelBuffer`, `CMSampleBuffer`, `Data`, or `URL`. Most requests are
structs; stateful requests such as `GeneratePersonSegmentationRequest`,
`TrackObjectRequest`, `TrackRectangleRequest`, and `DetectTrajectoriesRequest`
are final classes.
## Request Pattern (Modern API)
All modern Vision requests follow the same pattern: create a request, call
`perform(on:)`, and handle the typed result.
```swift
import Vision
func recognizeText(in image: CGImage) async throws -> [String] {
var request = RecognizeTextRequest()
request.recognitionLevel = .accurate
request.recognitionLanguages = [Locale.Language(identifier: "en-US")]
let observations = try await request.perform(on: image)
return observations.compactMap { observation in
observation.topCandidates(1).first?.string
}
}
```
### Legacy Pattern (Pre-iOS 18)
For pre-iOS 18 targets, use the corresponding `VNRequest` with `VNImageRequestHandler` or `VNSequenceRequestHandler`. Load [references/vision-requests.md](references/vision-requests.md) for complete legacy request and handler patterns.
## Text Recognition (OCR)
### Modern: RecognizeTextRequest (iOS 18+)
```swift
var request = RecognizeTextRequest()
request.recognitionLevel = .accurate // .fast for real-time
request.recognitionLanguages = [
Locale.Language(identifier: "en-US"),
Locale.Language(identifier: "fr-FR"),
]
request.usesLanguageCorrection = true
request.customWords = ["SwiftUI", "Xcode"] // domain-specific terms
let observations = try await request.perform(on: cgImage)
for observation in observations {
guard let candidate = observation.topCandidates(1).first else { continue }
let text = candidate.string
let confidence = candidate.confidence // 0.0 ... 1.0
let bounds = observation.boundingBox // NormalizedRect
}
```
### Legacy: VNRecognizeTextRequest
The legacy request uses string language identifiers and the handler pattern in the reference; both generations support accurate and fast recognition levels.
## Face Detection
Detect face rectangles, landmarks (eyes, nose, mouth), and capture quality.
```swift
// Modern API
let faceRequest = DetectFaceRectanglesRequest()
let faces = try await faceRequest.perform(on: cgImage)
for face in faces {
let boundingBox = face.boundingBox // NormalizedRect
let roll = face.roll // Measurement<UnitAngle>
let yaw = face.yaw // Measurement<UnitAngle>
}
// Landmarks (eyes, nose, mouth contours)
var landmarkRequest = DetectFaceLandmarksRequest()
let landmarkFaces = try await landmarkRequest.perform(on: cgImage)
for face in landmarkFaces {
let landmarks = face.landmarks
let leftEye = landmarks?.leftEye.points
let nose = landmarks?.nose.points
}
```
### Coordinate System
Vision uses a normalized coordinate system with origin at the bottom-left.
Convert to UIKit (top-left origin) before display:
```swift
import Vision
func imageRectForDisplay(_ rect: NormalizedRect, imageSize: CGSize) -> CGRect {
rect.toImageCoordinates(imageSize, origin: .upperLeft)
}
```
## Barcode Detection
Detect 1D and 2D barcodes including QR codes.
```swift
var request = DetectBarcodesRequest()
let symbologies: [BarcodeSymbology] = [.qr, .ean13, .code128, .pdf417]
request.symbologies = symbologies
let barcodes = try await request.perform(on: cgImage)
for barcode in barcodes {
let payload = barcode.payloadString // decoded content
let symbology = barcode.symbology // .qr, .ean13, etc.
let bounds = barcode.boundingBox // NormalizedRect
}
```
Type annotate local values first, then assign request properties separately.
## Document Scanning (iOS 26+)
`RecognizeDocumentsRequest` provides structured document reading with layout
understanding beyond basic OCR. Returns `DocumentObservation` objects with a
nested `Container` structure for paragraphs, tables, lists, and barcodes.
Currently, Vision returns one document observation for each image.
```swift
var request = RecognizeDocumentsRequest()
let documents = try await request.perform(on: cgImage)
for observation in documents {
let container = observation.document
// Full text content
let fullText = container.text
// Structured access to paragraphs
for paragraph in container.paragraphs {
let paragraphText = paragraph.text
}
// Tables and lists
for table in container.tables { /* structured table data */ }
for list in container.lists { /* structured list data */ }
// Embedded barcodes detected within the document
for barcode in container.barcodes { /* barcode data */ }
// Document title if detected
if let title = container.title { print(title) }
}
```
For simpler document camera scanning, use VisionKit's
`VNDocumentCameraViewController` which provides a full-screen camera UI with
auto-capture, perspective correction, and multi-page scanning.
## Image Segmentation
### Modern: GeneratePersonSegmentationRequest (iOS 18+)
```swift
var request = GeneratePersonSegmentationRequest()
request.qualityLevel = .accurate // .balanced, .fast
let mask = try await request.perform(on: cgImage)
// mask is a PixelBufferObservation with a pixelBuffer property
let maskBuffer = mask.pixelBuffer
// Apply mask using Core Image: CIFilter.blendWithMask()
```
### Legacy: VNGeneratePersonSegmentationRequest
For older targets, `VNGeneratePersonSegmentationRequest` exposes its mask through the first pixel-buffer observation; use the reference's handler and mask-composition recipe.
Quality levels:
- `.accurate` -- best quality, slowest (~1s), full resolution
- `.balanced` -- good quality, moderate speed (~100ms), 960x540
- `.fast` -- lowest quality, fastest (~10ms), 256x144, suitable for real-time
### Instance Segmentation (iOS 18+)
Separate masks per person for individual effects.
```swift
// Modern API (iOS 18+)
let request = GeneratePersonInstanceMaskRequest()
let observation = try await request.perform(on: cgImage)
let indices = observation.allInstances
for index in indices {
let mask = try observation.generateMask(for: IndexSet(integer: index))
// mask is a CVPixelBuffer with only this person visible
}
```
```swift
// Legacy API (iOS 17+)
let request = VNGeneratePersonInstanceMaskRequest()
let handler = VNImageRequestHandler(cgImage: cgImage)
try handler.perform([request])
guard let result = request.results?.first else { return }
let indices = result.allInstances
for index in indices {
let instanceMask = try result.generateMaskedImage(
ofInstances: IndexSet(integer: index),
from: handler,
croppedToInstancesExtent: false
)
}
```
See [references/vision-requests.md](references/vision-requests.md) for mask composition and Core Image filter
integration patterns.
## Object Tracking
### Modern: TrackObjectRequest (iOS 18+)
`TrackObjectRequest` is a stateful request that maintains tracking context
across frames.
```swift
// Initialize with a detected object's bounding box
let initialObservation = DetectedObjectObservation(boundingBox: detectedBox)
let request = TrackObjectRequest(detectedObject: initialObservation)
for pixelBuffer in framePixelBuffers {
let results = try await request.perform(on: pixelBuffer)
if let tracked = results.first {
let updatedBounds = tracked.boundingBox // NormalizedRect
}
}
```
Modern `TrackObjectRequest` has no `trackingLevel` or `qualityLevel`.
### Legacy: VNTrackObjectRequest
For older targets, use `VNTrackObjectRequest` with one retained `VNSequenceRequestHandler` and feed each result back as the next input observation. The reference contains the complete loop.
## Other Request Types
Vision provides additional requests covered in [references/vision-requests.md](references/vision-requests.md):
| Request | Purpose |
|---|---|
| `ClassifyImageRequest` | Classify scene content (outdoor, food, animal, etc.) |
| `GenerateAttentionBasedSaliencyImageRequest` | Single `SaliencyImageObservation` for where viewers focus attention |
| `GenerateObjectnessBasedSaliencyImageRequest` | Single `SaliencyImageObservation` for object-like regions |
| `GenerateForegroundInstanceMaskRequest` | Foreground object segmentation (not person-specific) |
| `DetectRectanglesRequest` | Detect rectangular shapes (documents, cards, screens) |
| `DetectHorizonRequest` | Detect horizon angle for auto-leveling photos |
| `DetectHumanBodyPoseRequest` | Detect body joints (shoulders, elbows, knees) |
| `DetectHumanBodyPose3DRequest` | 3D human body pose estimation |
| `DetectHumanHandPoseRequest` | Detect hand joints and finger positions |
| `DetectAnimalBodyPoseRequest` | Detect animal body joint positions |
| `DetectFaceCaptureQualityRequest` | Face capture quality scoring (0–1) for photo selection |
| `TrackRectangleRequest` | Track rectangular objects across video frames |
| `TrackOpticalFlowRequest` | Optical flow between video frames |
| `DetectTrajectoriesRequest` | Detect object trajectories in video |
All modern request types above are iOS 18+ / macOS 15+.
## Core ML Integration
Run custom Core ML models through Vision for automatic image preprocessing.
Vision runs already-prepared models with `CoreMLRequest` or `VNCoreMLRequest`;
hand conversion, profiling, packaging, and lifecycle decisions to `coreml`.
```swift
import CoreML
import Vision
// Modern API (iOS 18+): CoreMLRequest takes a CoreMLModelContainer.
let model = try MLModel(contentsOf: modelURL)
let container = try CoreMLModelContainer(model: model, featureProvider: nil)
let request = CoreMLRequest(model: container)
let results = try await request.perform(on: cgImage)
// Classification model
if let classification = results.first as? ClassificationObservation {
let label = classification.identifier
let confidence = classification.confidence
}
```
`CoreMLModelContainer` is the public iOS 18+ Vision container for
`CoreMLRequest`: load an `MLModel`, wrap it with
`CoreMLModelContainer(model:featureProvider:)`, then pass that container to
`CoreMLRequest(model:)`. State result mapping when reviewing Core ML through
Vision: classifiers produce `ClassificationObservation`, image outputs produce
`PixelBufferObservation`, and general predictors produce `CoreMLFeatureValueObservation`.
```swift
// Legacy API
let vnModel = try VNCoreMLModel(for: model)
let request = VNCoreMLRequest(model: vnModel) { request, error in
guard let results = request.results as? [VNClassificationObservation] else { return }
let topResult = results.first
}
let handler = VNImageRequestHandler(cgImage: cgImage)
try handler.perform([request])
```
## VisionKit: DataScannerViewController
`DataScannerViewController` provides a live camera scanner for text and
barcodes; see [references/visionkit-scanner.md](references/visionkit-scanner.md). VisionKit uses
`VNBarcodeSymbology`; modern `DetectBarcodesRequest` uses `BarcodeSymbology`.
### Quick Start
```swift
import AVFoundation
import Vision
import VisionKit
@MainActor
func presentScanner() async {
// Add NSCameraUsageDescription before requesting camera access.
guard await AVCaptureDevice.requestAccess(for: .video) else { return }
guard DataScannerViewController.isSupported,
DataScannerViewController.isAvailable else { return }
let scannerSymbologies: [VNBarcodeSymbology] = [.qr, .ean13]
let scanner = DataScannerViewController(
recognizedDataTypes: [
.text(languages: ["en"]),
.barcode(symbologies: scannerSymbologies)
],
qualityLevel: .balanced,
recognizesMultipleItems: true,
isHighFrameRateTrackingEnabled: true,
isHighlightingEnabled: true
)
scanner.delegate = self
present(scanner, animated: true) {
// Start scanning after presentation, on the main actor.
try? scanner.startScanning()
}
}
```
### SwiftUI Integration
Wrap `DataScannerViewController` in `UIViewControllerRepresentable` and start in
`updateUIViewController` with `Task { @MainActor in try? controller.startScanning() }`; see [references/visionkit-scanner.md](references/visionkit-scanner.md).
## Common Mistakes
**DON'T:** Use the legacy `VNImageRequestHandler` API for new iOS 18+ projects.
**DO:** Use modern Swift-native requests with `perform(on:)` and async/await.
**Why:** Modern API provides type safety, better Swift concurrency support, and cleaner error handling.
**DON'T:** Forget to convert normalized coordinates before drawing bounding boxes.
**DO:** Use `NormalizedRect.toImageCoordinates(_:origin:)` for modern observations, or `VNImageRectForNormalizedRect(_:_:_:)` for legacy `CGRect` observations.
**Why:** Vision uses normalized coordinates (0...1) with bottom-left origin; UIKit uses points with top-left origin.
**DON'T:** Run Vision requests on the main thread.
**DO:** Perform requests on a background thread or use async/await from a detached task.
**Why:** Image analysis is CPU/GPU-intensive and blocks the UI if run on the main actor.
**DON'T:** Use `.accurate` recognition level for real-time camera feeds.
**DO:** Use `.fast` for live video, `.accurate` for still images or offline processing.
**Why:** Accurate recognition is too slow for 30fps video; fast recognition trades quality for speed.
**DON'T:** Treat every Vision observation as having the same properties.
**DO:** Check each observation type for its bounding box, confidence, payload, mask, or angle fields before writing shared helpers.
**Why:** Modern Vision returns strongly typed observations, and result shapes vary by request.
**DON'T:** Recreate stateful tracking requests for each video frame.
**DO:** Keep the same modern `TrackObjectRequest` instance, or use `VNSequenceRequestHandler` with legacy tracking requests.
**Why:** Tracking relies on temporal context across frames.
**DON'T:** Request all barcode symbologies when you only need QR codes.
**DO:** Specify only the symbologies you need in the request.
**Why:** Fewer symbologies means faster detection and fewer false positives.
**DON'T:** Assume `DataScannerViewController` is available on all devices.
**DO:** Check both `isSupported` (hardware) and `isAvailable` (user permissions) before presenting.
**Why:** Requires A12+ chip; `isAvailable` also checks camera access authorization.
## Review Checklist
- [ ] Uses modern Vision API (iOS 18+) unless targeting older deployments
- [ ] Vision requests run off the main thread (async/await or background queue)
- [ ] Normalized coordinates converted before UI display
- [ ] Confidence threshold applied to filter low-quality observations
- [ ] Recognition level matches use case (`.fast` for video, `.accurate` for stills)
- [ ] Language hints set for text recognition when input language is known
- [ ] Barcode symbologies limited to only those needed
- [ ] `DataScannerViewController` availability checked before presentation
- [ ] Camera usage description (`NSCameraUsageDescription`) in Info.plist for VisionKit
- [ ] VisionKit camera access requested before presentation and scanning started after presentation
- [ ] Person segmentation quality level appropriate for use case
- [ ] Stateful tracking request or `VNSequenceRequestHandler` preserved across video frames
- [ ] Error handling covers request failures and empty results
## References
- Vision request patterns: [references/vision-requests.md](references/vision-requests.md)
- VisionKit scanner integration: [references/visionkit-scanner.md](references/visionkit-scanner.md)
- Apple docs: [Vision](https://sosumi.ai/documentation/vision) |
[VisionKit](https://sosumi.ai/documentation/visionkit) |
[RecognizeTextRequest](https://sosumi.ai/documentation/vision/recognizetextrequest) |
[DataScannerViewController](https://sosumi.ai/documentation/visionkit/datascannerviewcontroller) |
[CoreMLRequest](https://sosumi.ai/documentation/vision/coremlrequest) |
[CoreMLModelContainer](https://sosumi.ai/documentation/vision/coremlmodelcontainer)