references/flutter_setup.md
# Flutter Setup for Firebase AI Logic
This guide covers how to integrate Firebase AI Logic (Gemini API) into your
Flutter applications.
> [!IMPORTANT] **Foundational Workflows & CLI-First Approach:**
>
> 1. **Review Foundation:** Before implementing platform-specific code, ALWAYS
> review the foundational `firebase-basics` skill to ensure familiarity with
> core workflows.
> 1. **Backend Provisioning via CLI:** Use the Firebase CLI for backend setup.
> Running `npx firebase-tools init ailogic` is MANDATORY to provision the
> service. `flutterfire configure` does NOT enable the AI service and will
> result in `PERMISSION_DENIED` if skipped.
> 1. **Client Configuration:** Use `flutterfire configure` strictly for
> generating `firebase_options.dart`. Avoid manual Console configuration.
> [!NOTE] `firebase_vertexai` has been replaced by `firebase_ai`. Always use
> `firebase_ai` for new projects.
## Installation
Add the necessary Firebase dependencies to your `pubspec.yaml`:
```yaml
dependencies:
flutter:
sdk: flutter
firebase_core: ^4.0.0
firebase_auth: ^6.0.0
firebase_ai: ^3.0.0
```
Run `flutter pub get` to install the packages.
## Initialization
Initialize Firebase and sign in (anonymously or via authenticated user) before
using AI Logic.
```dart
import 'package:firebase_core/firebase_core.dart';
import 'package:firebase_auth/firebase_auth.dart';
import 'package:firebase_ai/firebase_ai.dart';
void main() async {
WidgetsFlutterBinding.ensureInitialized();
await Firebase.initializeApp();
await FirebaseAuth.instance.signInAnonymously();
runApp(const MyApp());
}
```
## Usage
Use `FirebaseAI.googleAI` for the **Gemini Developer API**.
> [!IMPORTANT] **Model Selection:** Refer to
> [Firebase AI Logic Models](https://firebase.google.com/docs/ai-logic/models.md.txt)
> to find the latest supported model. Do not use any Gemini 1.x or Gemini 2.x
> models (such as `gemini-1.5-flash`, `gemini-2.0-flash`, `gemini-2.5-flash`,
> or `gemini-2.5-flash-lite`).
> [!IMPORTANT] **Choose the Right API Provider:** Always use
> `FirebaseAI.googleAI` (Gemini Developer API) as the default for prototyping
> and standard use. Avoid using the Agent Platform Gemini API (formerly branded
> Vertex AI) unless the app and business use case specifically require
> enterprise-grade scalability or data residency requirements. Note that the
> Gemini Developer API *usually does not* require the Firebase project to be on
> the pay-as-you-go Blaze pricing plan; however, the Agent Platform Gemini API
> does require the Blaze plan.
### Text Generation
```dart
import 'package:firebase_ai/firebase_ai.dart';
import 'package:firebase_auth/firebase_auth.dart';
Future<String> generateText(String prompt) async {
final googleAI = FirebaseAI.googleAI(auth: FirebaseAuth.instance);
// [AGENT] Replace '<latest_supported_model>' with the latest model from https://firebase.google.com/docs/ai-logic/models.md.txt
final model = googleAI.generativeModel(model: '<latest_supported_model>');
final response = await model.generateContent([Content.text(prompt)]);
return response.text ?? 'No response';
}
```
### Chat Session
```dart
final chat = model.startChat(history: [
Content.text('Hello, I am a user.'),
Content.model([TextPart('Hello! How can I help you today?')]),
]);
final response = await chat.sendMessage(Content.text('What is CBT?'));
```
## App Check Debug Provider
For App Check debug tokens during local development and CI/CD, add the plugin
with `flutter pub add firebase_app_check`, then activate the debug providers
right after `Firebase.initializeApp()`:
```dart
import 'package:firebase_app_check/firebase_app_check.dart';
import 'package:flutter/foundation.dart';
// In main(), after `await Firebase.initializeApp();`
if (kDebugMode) {
await FirebaseAppCheck.instance.activate(
providerAndroid: const AndroidDebugProvider(),
providerApple: const AppleDebugProvider(),
providerWeb: WebDebugProvider(),
);
}
```
- Use `providerAndroid`, `providerApple`, and `providerWeb`. The older
`androidProvider` and `appleProvider` parameters (`AndroidProvider.debug`,
`AppleProvider.debug`) still appear in many examples but are deprecated.
- Only use the debug providers in debug builds, as gated by `kDebugMode` above.
Release builds must activate the production providers (Play Integrity, App
Attest or DeviceCheck, and reCAPTCHA Enterprise) instead; see
[App Check for Firebase AI Logic](https://firebase.google.com/docs/ai-logic/app-check.md.txt).
- **CI/CD**: Pass the pre-provisioned token to all three debug providers, e.g.
`debugToken: const String.fromEnvironment('APP_CHECK_DEBUG_TOKEN')`, and build
with `--dart-define=APP_CHECK_DEBUG_TOKEN=<token>`.
- **Flutter web**: `providerWeb` is required on web; without it, `activate()`
throws. `WebDebugProvider` turns on the web debug token itself, so
`web/index.html` doesn't need `self.FIREBASE_APPCHECK_DEBUG_TOKEN`.
- If the app uses `firebase_ai` 3.11.0 or lower, also set the `appCheck`
parameter of `FirebaseAI.googleAI(...)` to `FirebaseAppCheck.instance`. Newer
versions don't need this.
references/ios_setup.md
# Firebase AI Logic iOS Setup Guide
## 1. Import and Initialize
Ensure you have installed the `FirebaseAILogic` SDK via Swift Package Manager.
Use the `xcode-project-setup` skill to automate adding the SPM dependency to the
Xcode project.
```swift
import FirebaseAILogic
// To enable replay protection for generative or preview models, pass
// useLimitedUseAppCheckTokens: true with your backend:
let ai = FirebaseAI.firebaseAI(
backend: .googleAI(),
useLimitedUseAppCheckTokens: true
)
// [AGENT] Replace "<latest_supported_model>" with the latest model from https://firebase.google.com/docs/ai-logic/models.md.txt (do NOT use Gemini 1.x or 2.x models)
let model = ai.generativeModel(modelName: "<latest_supported_model>")
```
### App Check Replay Protection
Generative and preview models enforce replay protection with 5-minute
limited-use App Check tokens. If you call a protected model without enabling
limited-use tokens, the request fails with:
```text
HTTP 403: "To access this model, you must enforce Firebase App Check"
```
To resolve this error, initialize `FirebaseAI` with
`useLimitedUseAppCheckTokens: true`:
```swift
let ai = FirebaseAI.firebaseAI(
backend: .googleAI(),
useLimitedUseAppCheckTokens: true
)
```
This ensures the SDK fetches a fresh, short-lived limited-use token for each
request rather than reusing a standard cached App Check token.
## 2. SwiftUI Integration (Best Practices)
Use the `@Observable` pattern to manage AI state and provide a smooth UX with
loading indicators and error handling.
> **⛔️ CRITICAL WARNING:** Do NOT initialize the model inline as a class
> property if there's any chance the view model is instantiated before
> `FirebaseApp.configure()` executes in the app root. To be safe, initialize the
> model lazily or pass it in from a point in the hierarchy where Firebase is
> guaranteed to be configured.
```swift
import SwiftUI
import FirebaseAILogic
@MainActor
@Observable
final class AIViewModel {
// [AGENT] Replace with the latest model from https://firebase.google.com/docs/ai-logic/models.md.txt
private lazy var model = FirebaseAI.firebaseAI(
backend: .googleAI(),
useLimitedUseAppCheckTokens: true
).generativeModel(modelName: "<latest_supported_model>")
var responseText: String = ""
var isFetching: Bool = false
var errorMessage: String?
func generate(prompt: String) async {
isFetching = true
errorMessage = nil
defer { isFetching = false }
do {
let response = try await model.generateContent(prompt)
self.responseText = response.text ?? "No response"
} catch {
self.errorMessage = error.localizedDescription
}
}
}
struct AIView: View {
@State private var viewModel = AIViewModel()
@State private var prompt = "Write a story about a magic backpack."
var body: some View {
VStack {
TextField("Enter prompt", text: $prompt)
Button("Generate") {
Task { await viewModel.generate(prompt: prompt) }
}
.disabled(viewModel.isFetching)
if viewModel.isFetching {
ProgressView()
} else if let error = viewModel.errorMessage {
Text(error).foregroundStyle(.red)
} else {
ScrollView {
Text(viewModel.responseText)
}
}
}
.padding()
}
}
```
## 3. Safety Settings
You can configure safety thresholds to prevent the model from generating harmful
content.
```swift
let safetySettings = [
SafetySetting(harmCategory: .harassment, threshold: .blockLowAndAbove),
SafetySetting(harmCategory: .hateSpeech, threshold: .blockMediumAndAbove)
]
let model = FirebaseAI.firebaseAI().generativeModel(
modelName: "<latest_supported_model>", // [AGENT] Replace with the latest model from https://firebase.google.com/docs/ai-logic/models.md.txt
safetySettings: safetySettings
)
```
## 4. App Check Debug Provider
For App Check debug tokens during local development and CI/CD, add the
`FirebaseAppCheck` product to the app target (with the `xcode-project-setup`
skill), then set the debug provider factory in debug builds *before*
`FirebaseApp.configure()`. Otherwise the debug provider isn't used.
```swift
import FirebaseAppCheck
import FirebaseCore
// In your app's init(), before configuring Firebase:
#if DEBUG
AppCheck.setAppCheckProviderFactory(AppCheckDebugProviderFactory())
#endif
FirebaseApp.configure()
```
For CI/CD, add an `AppCheckDebugToken` environment variable to the Xcode test
scheme with a value like `$(APP_CHECK_DEBUG_TOKEN)`, then pass the token from
the CI test command:
```bash
xcodebuild test -scheme <scheme> -workspace <project>.xcworkspace \
APP_CHECK_DEBUG_TOKEN=<token>
```
### Debug Token Persistence
When running on simulators or during development, the Firebase iOS SDK generates
a new debug token UUID whenever `NSUserDefaults` is cleared (e.g., simulator
reset or fresh install). To avoid invalidating tokens registered in the Firebase
Console and prevent token churn, persist a stable token by setting the
`AppCheckDebugToken` environment variable, which the SDK reads automatically at
runtime:
> [!WARNING] **CRITICAL: Never Hardcode or Commit Debug Tokens** Debug tokens
> grant access to backend resources without device attestation. Never commit
> debug tokens to version control or hardcode token strings in source code.
- **In Xcode Scheme (Recommended):** Edit Scheme -> Run -> Arguments ->
Environment Variables -> Add `AppCheckDebugToken = <YOUR_DEBUG_TOKEN>`. Keep
user schemes (`xcuserdata/`) unshared and gitignored.
- **In Code (Safe Dynamic Loading Only):** If setting the environment variable
in code before configuring `AppCheckDebugProviderFactory`, load the token
dynamically from a gitignored local file or environment rather than hardcoding
the token literal:
```swift
#if DEBUG
// ✅ SAFE: Load from gitignored local file or process environment
// Note: loadGitIgnoredDebugToken() is a placeholder for your custom helper (e.g., reading from a gitignored plist)
if let debugToken = loadGitIgnoredDebugToken() {
setenv("AppCheckDebugToken", debugToken, 0)
}
let providerFactory = AppCheckDebugProviderFactory()
AppCheck.setAppCheckProviderFactory(providerFactory)
#endif
FirebaseApp.configure() // Configure Firebase AFTER setting the App Check provider factory
```
# Advanced Features
### Chat Session (Multi-turn)
Chat sessions persist state across multiple interactions, which is essential for
ongoing conversations or when using tools like function calling.
```swift
let chat = model.startChat()
Task {
do {
let response1 = try await chat.sendMessage("Hello! I have two dogs in my house.")
print(response1.text ?? "")
let response2 = try await chat.sendMessage("How many paws are in my house?")
print(response2.text ?? "")
} catch {
print("Error in chat: \(error)")
}
}
```
### Function Calling (Tools)
Define functions that the model can request to execute to interact with external
systems. *Note: Advanced workflows like function calling generally require a
multi-turn Chat Session to handle the back-and-forth execution.*
```swift
let getStockPriceTool = Tool.functionDeclarations([
FunctionDeclaration(
name: "getStockPrice",
description: "Get the current stock price for a given symbol.",
parameters: [
"symbol": .string(
description: "The stock symbol, e.g. AAPL"
)
]
)
])
let model = FirebaseAI.firebaseAI().generativeModel(
modelName: "<latest_supported_model>", // [AGENT] Replace with the latest model from https://firebase.google.com/docs/ai-logic/models.md.txt
tools: [getStockPriceTool]
)
// In your task (using a chat session):
let chat = model.startChat()
let response = try await chat.sendMessage("What is the stock price of Apple?")
if let functionCall = response.functionCalls.first {
// Handle the function call (e.g. call a local API and send the result back)
print("Model requested function: \(functionCall.name) with args: \(functionCall.args)")
}
```
references/tts_generation.md
# Client-Side Text-to-Speech (TTS) Generation with Gemini
Firebase AI Logic enables client-side Text-to-Speech (TTS) generation directly
from your mobile and web applications without maintaining custom backend speech
services. Using specialized Gemini TTS models, apps can synthesize natural,
expressive audio with custom voice personas, multi-speaker dialogues, and
in-flight audio streaming.
______________________________________________________________________
## Supported Models
Always specify a dedicated Gemini TTS model when generating audio:
| Model ID | Description |
| :----------------------------- | :--------------------------------------------------------------------------------- |
| `gemini-3.1-flash-tts-preview` | Low-latency speech synthesis (preview); supports single- and multi-speaker output. |
> [!WARNING] Always verify currently supported model availability in the
> [Firebase AI Logic Models documentation](https://firebase.google.com/docs/ai-logic/models.md.txt).
______________________________________________________________________
## Core Concepts & Configurations
### 1. Response Modality
To instruct Gemini to synthesize and return audio, configure
`responseModalities` in the generation configuration to include audio:
- **Swift**: `generationConfig = GenerationConfig(responseModalities: [.audio])`
- **Kotlin**:
`generationConfig { responseModalities = listOf(ResponseModality.AUDIO) }`
- **Web (JS/TS)**: `generationConfig: { responseModalities: ["AUDIO"] }`
### 2. Single-Speaker `SpeechConfig`
Configure voice persona and optional language code (`languageCode`, e.g.
`"en-US"`, `"es-ES"`, `"ja-JP"`) using `SpeechConfig`. Voice names map to Google
prebuilt neural voices (for example: `Puck`, `Charon`, `Kore`, `Fenrir`,
`Aoede`):
- **Swift**:
```swift
let speechConfig = SpeechConfig(
voiceConfig: VoiceConfig(
prebuiltVoiceConfig: PrebuiltVoiceConfig(voiceName: "Puck")
),
languageCode: "en-US"
)
```
- **Kotlin**:
```kotlin
val speechConfig = speechConfig {
voiceConfig = voiceConfig {
prebuiltVoiceConfig = prebuiltVoiceConfig {
voiceName = "Puck"
}
}
languageCode = "en-US"
}
```
- **Web**:
```javascript
speechConfig: {
voiceConfig: {
prebuiltVoiceConfig: {
voiceName: "Puck"
}
},
languageCode: "en-US"
}
```
### 3. Multi-Speaker `MultiSpeakerVoiceConfig` (2-Speaker Dialogues)
For scripts with two distinct characters or roles (e.g., host and guest,
narrator and character), configure `MultiSpeakerVoiceConfig` with
`speakerVoiceConfigs` linking speaker names to specific voices:
- **Swift**:
```swift
let multiSpeakerConfig = SpeechConfig(
multiSpeakerVoiceConfig: MultiSpeakerVoiceConfig(
speakerVoiceConfigs: [
SpeakerVoiceConfig(
speaker: "Host",
voiceConfig: VoiceConfig(
prebuiltVoiceConfig: PrebuiltVoiceConfig(voiceName: "Puck")
)
),
SpeakerVoiceConfig(
speaker: "Guest",
voiceConfig: VoiceConfig(
prebuiltVoiceConfig: PrebuiltVoiceConfig(voiceName: "Aoede")
)
)
]
)
)
```
- **Kotlin**:
```kotlin
val multiSpeakerConfig = speechConfig {
multiSpeakerVoiceConfig = multiSpeakerVoiceConfig {
speakerVoiceConfigs = listOf(
speakerVoiceConfig {
speaker = "Host"
voiceConfig = voiceConfig {
prebuiltVoiceConfig = prebuiltVoiceConfig { voiceName = "Puck" }
}
},
speakerVoiceConfig {
speaker = "Guest"
voiceConfig = voiceConfig {
prebuiltVoiceConfig = prebuiltVoiceConfig { voiceName = "Aoede" }
}
}
)
}
}
```
- **Web**:
```javascript
speechConfig: {
multiSpeakerVoiceConfig: {
speakerVoiceConfigs: [
{
speaker: "Host",
voiceConfig: { prebuiltVoiceConfig: { voiceName: "Puck" } }
},
{
speaker: "Guest",
voiceConfig: { prebuiltVoiceConfig: { voiceName: "Aoede" } }
}
]
}
}
```
______________________________________________________________________
## Audio Directives and Emotional Tags
Gemini TTS models accept natural-language directives and inline tags in prompts
to control tone, cadence, pacing, and emotional expression.
### Directives
Include directives at the start of your prompt to set the overall tone and
scene:
- `[Audio Profile: Warm, conversational podcast co-host with energetic delivery]`
- `[Scene: Quiet bedtime story in a calm room]`
- `[Director's Note: Speak deliberately, pausing for reflection after major questions]`
### Emotional Tags & Delivery Cues
Embed delivery markers inline within dialogue or narration:
- `[whispers]` — Soft, confidential whisper.
- `[laughs]` / `[chuckles]` — Lighthearted laughter during delivery.
- `[sighs]` — Expressive exhale.
- `[slowly]` — Reduced tempo for emphasis or dramatic effect.
- `[excited]` — Higher energy and faster pacing.
- `[pause]` — Deliberate silence.
### Multi-Speaker Prompt Example
```text
[Scene: A lively coffee-shop tech discussion]
[Director's Note: Host is upbeat and curious; Guest is thoughtful and explanatory]
Host: Welcome back! [excited] Today we are diving into client-side audio.
Guest: [laughs] It's about time! No servers in between means virtually zero latency.
Host: [whispers] Tell us the secret... how does it stream?
Guest: [slowly] Pure linear PCM, chunk by chunk, directly to your device's audio engine.
```
______________________________________________________________________
## Audio Decoding and Playback
### Format Specifications
Gemini TTS audio format depends on whether the request is unary or streaming:
- **Unary Requests (`generateContent`)**: Returns complete **WAV (`audio/wav`)**
audio with a standard 44-byte RIFF header included by default (24 kHz, mono,
16-bit signed little-endian PCM). These bytes can be passed directly to
standard media players (`AVAudioPlayer`, `MediaPlayer`, `<audio>`).
- **Streaming Requests (`generateContentStream`)**: Returns headerless raw
**Linear PCM (`audio/l16; rate=24000; channels=1`)** chunks (24 kHz, mono,
16-bit signed little-endian PCM) so chunks can be streamed or concatenated
continuously without per-chunk container headers.
- **Audio Parameters**:
- **Sample Rate**: 24,000 Hz (24 kHz)
- **Channels**: 1 channel (mono)
- **Bit Depth**: 16-bit linear PCM (little-endian signed integer)
- **MIME Types**: `audio/l16` (streaming chunks) or `audio/wav` /
`audio/x-wav` (unary)
### Playback Approaches
1. **Direct Low-Latency PCM Streaming**:
- Stream raw PCM chunks directly into low-level audio renderers without
waiting for the full response:
- **iOS**: `AVAudioEngine` + `AVAudioPlayerNode` scheduling
`AVAudioPCMBuffer`.
- **Android**: `AudioTrack` configured with `ENCODING_PCM_16BIT` and
`CHANNEL_OUT_MONO`.
- **Web**: Web Audio API (`AudioContext`) scheduling PCM chunks into
`AudioBufferSourceNode`.
1. **WAV Container Format**:
- Standard media players (`AVAudioPlayer` on iOS, `MediaPlayer` on Android,
`<audio>` on Web) expect a container header (RIFF/WAV).
- For unary responses, the RIFF header is already present. For assembled
streaming PCM chunks, prepend a standard **44-byte RIFF header** to make
the complete audio immediately playable or saveable as `.wav`.
#### Standard 44-Byte WAV (RIFF) Header Construction
```
Offset Size Field Value
0 4 ChunkID "RIFF"
4 4 ChunkSize 36 + Subchunk2Size (file size - 8)
8 4 Format "WAVE"
12 4 Subchunk1ID "fmt "
16 4 Subchunk1Size 16 (for PCM)
20 2 AudioFormat 1 (PCM linear)
22 2 NumChannels 1 (Mono)
24 4 SampleRate 24000
28 4 ByteRate 48000 (SampleRate * NumChannels * BitsPerSample / 8)
32 2 BlockAlign 2 (NumChannels * BitsPerSample / 8)
34 2 BitsPerSample 16
36 4 Subchunk2ID "data"
40 4 Subchunk2Size Number of PCM bytes
44+ N Data Raw PCM bytes
```
______________________________________________________________________
## Platform Code Snippets
### Swift (iOS)
#### 1. Streaming Playback with `AVAudioEngine`
```swift
import AVFoundation
import FirebaseAILogic
@MainActor
final class SpeechStreamingManager {
private let audioEngine = AVAudioEngine()
private let playerNode = AVAudioPlayerNode()
// 24 kHz, 16-bit Mono PCM
private let audioFormat = AVAudioFormat(
commonFormat: .pcmFormatInt16,
sampleRate: 24000,
channels: 1,
interleaved: false
)!
init() {
audioEngine.attach(playerNode)
audioEngine.connect(playerNode, to: audioEngine.mainMixerNode, format: audioFormat)
}
func streamSpeech(prompt: String) async throws {
let ai = FirebaseAI.firebaseAI()
let speechConfig = SpeechConfig(
voiceConfig: VoiceConfig(
prebuiltVoiceConfig: PrebuiltVoiceConfig(voiceName: "Puck")
)
)
let genConfig = GenerationConfig(
responseModalities: [.audio],
speechConfig: speechConfig
)
let model = ai.generativeModel(
modelName: "gemini-3.1-flash-tts-preview",
generationConfig: genConfig
)
if !audioEngine.isRunning {
try audioEngine.start()
}
playerNode.play()
let stream = model.generateContentStream(prompt)
for try await chunk in stream {
for part in chunk.candidates.first?.content.parts ?? [] {
if let inlineData = part as? InlineDataPart,
inlineData.mimeType.contains("audio/pcm") || inlineData.mimeType.contains("audio/l16") {
schedulePCMBuffer(data: inlineData.data)
}
}
}
}
private func schedulePCMBuffer(data: Data) {
let frameCount = UInt32(data.count) / audioFormat.streamDescription.pointee.mBytesPerFrame
guard frameCount > 0,
let buffer = AVAudioPCMBuffer(pcmFormat: audioFormat, frameCapacity: frameCount) else {
return
}
buffer.frameLength = frameCount
data.withUnsafeBytes { rawBufferPointer in
guard let src = rawBufferPointer.baseAddress else { return }
if let dest = buffer.int16ChannelData?[0] {
UnsafeMutableRawPointer(dest).copyMemory(from: src, byteCount: data.count)
}
}
playerNode.scheduleBuffer(buffer)
}
func stop() {
playerNode.stop()
audioEngine.stop()
}
}
```
#### 2. WAV Converter for `AVAudioPlayer`
```swift
func addWavHeader(to pcmData: Data, sampleRate: Int = 24000, channels: Int = 1, bitDepth: Int = 16) -> Data {
var header = Data()
let byteRate = sampleRate * channels * bitDepth / 8
let blockAlign = channels * bitDepth / 8
let totalDataLen = Int32(pcmData.count)
let totalLength = totalDataLen + 36
header.append(contentsOf: "RIFF".utf8)
header.append(Data(from: totalLength.littleEndian))
header.append(contentsOf: "WAVE".utf8)
header.append(contentsOf: "fmt ".utf8)
header.append(Data(from: Int32(16).littleEndian)) // Subchunk1Size for PCM
header.append(Data(from: Int16(1).littleEndian)) // AudioFormat 1 = PCM
header.append(Data(from: Int16(channels).littleEndian))
header.append(Data(from: Int32(sampleRate).littleEndian))
header.append(Data(from: Int32(byteRate).littleEndian))
header.append(Data(from: Int16(blockAlign).littleEndian))
header.append(Data(from: Int16(bitDepth).littleEndian))
header.append(contentsOf: "data".utf8)
header.append(Data(from: totalDataLen.littleEndian))
return header + pcmData
}
private extension Data {
init<T>(from value: T) {
var val = value
self = Swift.withUnsafeBytes(of: &val) { Data($0) }
}
}
```
______________________________________________________________________
### Kotlin (Android)
#### Streaming Playback with `AudioTrack`
```kotlin
import android.media.AudioAttributes
import android.media.AudioFormat
import android.media.AudioTrack
import com.google.firebase.ai.FirebaseAI
import com.google.firebase.ai.type.ResponseModality
import com.google.firebase.ai.type.generationConfig
import com.google.firebase.ai.type.prebuiltVoiceConfig
import com.google.firebase.ai.type.speechConfig
import com.google.firebase.ai.type.voiceConfig
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
class SpeechStreamingManager {
private val sampleRate = 24000
private val minBufferSize = AudioTrack.getMinBufferSize(
sampleRate,
AudioFormat.CHANNEL_OUT_MONO,
AudioFormat.ENCODING_PCM_16BIT
)
private val audioTrack = AudioTrack.Builder()
.setAudioAttributes(
AudioAttributes.Builder()
.setContentType(AudioAttributes.CONTENT_TYPE_SPEECH)
.setUsage(AudioAttributes.USAGE_MEDIA)
.build()
)
.setAudioFormat(
AudioFormat.Builder()
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
.setSampleRate(sampleRate)
.setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
.build()
)
.setBufferSizeInBytes(minBufferSize * 2)
.setTransferMode(AudioTrack.MODE_STREAM)
.build()
suspend fun streamSpeech(prompt: String) = withContext(Dispatchers.IO) {
val config = generationConfig {
responseModalities = listOf(ResponseModality.AUDIO)
speechConfig = speechConfig {
voiceConfig = voiceConfig {
prebuiltVoiceConfig = prebuiltVoiceConfig {
voiceName = "Puck"
}
}
}
}
val generativeModel = FirebaseAI.getInstance().getGenerativeModel(
modelName = "gemini-3.1-flash-tts-preview",
generationConfig = config
)
audioTrack.play()
try {
val responseStream = generativeModel.generateContentStream(prompt)
responseStream.collect { chunk ->
chunk.candidates.firstOrNull()?.content?.parts?.forEach { part ->
if (part is com.google.firebase.ai.type.InlineDataPart) {
val audioBytes = part.data
audioTrack.write(audioBytes, 0, audioBytes.size)
}
}
}
} finally {
audioTrack.stop()
}
}
fun release() {
audioTrack.release()
}
}
```
#### 2. Convert PCM to WAV for `MediaPlayer`
```kotlin
import android.content.Context
import android.media.MediaPlayer
import java.io.File
import java.io.FileOutputStream
import java.nio.ByteBuffer
import java.nio.ByteOrder
fun addWavHeader(
pcmBytes: ByteArray,
sampleRate: Int = 24000,
channels: Short = 1,
bitDepth: Short = 16
): ByteArray {
val totalDataLen = pcmBytes.size
val totalLength = totalDataLen + 36
val byteRate = sampleRate * channels * bitDepth / 8
val blockAlign = (channels * bitDepth / 8).toShort()
val header = ByteBuffer.allocate(44).order(ByteOrder.LITTLE_ENDIAN).apply {
put("RIFF".toByteArray())
putInt(totalLength)
put("WAVE".toByteArray())
put("fmt ".toByteArray())
putInt(16) // Subchunk1Size
putShort(1) // AudioFormat 1 = PCM
putShort(channels)
putInt(sampleRate)
putInt(byteRate)
putShort(blockAlign)
putShort(bitDepth)
put("data".toByteArray())
putInt(totalDataLen)
}.array()
return header + pcmBytes
}
fun playWavAudio(context: Context, wavBytes: ByteArray) {
val tempFile = File.createTempFile("tts_", ".wav", context.cacheDir).apply {
deleteOnExit()
FileOutputStream(this).use { it.write(wavBytes) }
}
val mediaPlayer = MediaPlayer()
try {
mediaPlayer.setDataSource(tempFile.absolutePath)
mediaPlayer.prepare()
mediaPlayer.start()
mediaPlayer.setOnCompletionListener {
it.release()
tempFile.delete()
}
} catch (e: Exception) {
mediaPlayer.release()
tempFile.delete()
throw e
}
}
```
______________________________________________________________________
### Web (JavaScript/TypeScript)
#### 1. Streaming Playback with Web Audio API
```typescript
import { initializeApp } from "firebase/app";
import { getAI, getGenerativeModel, GoogleAIBackend } from "firebase/ai";
const firebaseConfig = {
// your firebase configuration
};
const app = initializeApp(firebaseConfig);
const ai = getAI(app, { backend: new GoogleAIBackend() });
const model = getGenerativeModel(ai, {
model: "gemini-3.1-flash-tts-preview",
generationConfig: {
responseModalities: ["AUDIO"],
speechConfig: {
voiceConfig: {
prebuiltVoiceConfig: {
voiceName: "Puck",
},
},
},
},
});
export class WebSpeechPlayer {
private audioCtx: AudioContext | null = null;
private nextStartTime: number = 0;
private getAudioContext(): AudioContext {
if (!this.audioCtx) {
if (typeof window === "undefined") {
throw new Error("AudioContext is only available in browser environments.");
}
const AudioContextClass =
window.AudioContext ||
(window as unknown as { webkitAudioContext: typeof AudioContext })
.webkitAudioContext;
this.audioCtx = new AudioContextClass({ sampleRate: 24000 });
}
return this.audioCtx;
}
async playSpeechStream(prompt: string) {
const ctx = this.getAudioContext();
if (ctx.state === "suspended") {
await ctx.resume();
}
this.nextStartTime = ctx.currentTime;
const responseStream = await model.generateContentStream(prompt);
for await (const chunk of responseStream.stream) {
const candidates = chunk.candidates || [];
for (const candidate of candidates) {
for (const part of candidate.content?.parts || []) {
if ("inlineData" in part && part.inlineData?.data) {
const rawPcm = this.base64ToArrayBuffer(part.inlineData.data);
this.queuePcmChunk(rawPcm, ctx);
}
}
}
}
}
private queuePcmChunk(pcmData: ArrayBuffer, ctx: AudioContext) {
const int16Array = new Int16Array(pcmData);
const float32Array = new Float32Array(int16Array.length);
// Convert 16-bit PCM integer samples to -1.0 .. 1.0 float samples
for (let i = 0; i < int16Array.length; i++) {
float32Array[i] = int16Array[i] / 32768.0;
}
const audioBuffer = ctx.createBuffer(1, float32Array.length, 24000);
audioBuffer.copyToChannel(float32Array, 0);
const source = ctx.createBufferSource();
source.buffer = audioBuffer;
source.connect(ctx.destination);
const startTime = Math.max(this.nextStartTime, ctx.currentTime);
source.start(startTime);
this.nextStartTime = startTime + audioBuffer.duration;
}
private base64ToArrayBuffer(base64: string): ArrayBuffer {
const binaryString = window.atob(base64);
const len = binaryString.length;
const bytes = new Uint8Array(len);
for (let i = 0; i < len; i++) {
bytes[i] = binaryString.charCodeAt(i);
}
return bytes.buffer;
}
}
```
#### 2. Convert PCM Base64 to Playable WAV Blob for `<audio>` Elements
```typescript
export function pcmToWavBlob(pcmBytes: Uint8Array, sampleRate = 24000): Blob {
const header = new ArrayBuffer(44);
const view = new DataView(header);
const numChannels = 1;
const bitsPerSample = 16;
const byteRate = (sampleRate * numChannels * bitsPerSample) / 8;
const blockAlign = (numChannels * bitsPerSample) / 8;
const dataSize = pcmBytes.byteLength;
// RIFF identifier
writeString(view, 0, "RIFF");
view.setUint32(4, 36 + dataSize, true);
writeString(view, 8, "WAVE");
// fmt subchunk
writeString(view, 12, "fmt ");
view.setUint32(16, 16, true); // Subchunk1Size
view.setUint16(20, 1, true); // AudioFormat (PCM = 1)
view.setUint16(22, numChannels, true);
view.setUint32(24, sampleRate, true);
view.setUint32(28, byteRate, true);
view.setUint16(32, blockAlign, true);
view.setUint16(34, bitsPerSample, true);
// data subchunk
writeString(view, 36, "data");
view.setUint32(40, dataSize, true);
return new Blob([header, pcmBytes], { type: "audio/wav" });
}
function writeString(view: DataView, offset: number, string: string) {
for (let i = 0; i < string.length; i++) {
view.setUint8(offset + i, string.charCodeAt(i));
}
}
```
______________________________________________________________________
## Best Practices & Troubleshooting
- **Audio Session & Permissions**: On iOS and Android, make sure the app's audio
session/category is configured to allow playback (e.g.,
`AVAudioSessionCategoryPlayback`).
- **Buffering & Jitter Prevention**: When streaming audio chunks via
`generateContentStream`, queue buffers sequentially with exact timestamps or
frame offsets to avoid audio stutter or gaps.
- **Model Compatibility**: TTS features require Gemini models with dedicated
audio synthesis capabilities (`gemini-3.1-flash-tts-preview`). Do not use
standard text-only model IDs.
- **App Check Protection**: Generating audio consumes quota. Protect your API
endpoints by enforcing App Check on every client build.
references/usage_patterns_android.md
# Firebase AI Logic on Android (Kotlin)
First, ensure you have initialized the Firebase App (see `firebase-basics`
skill). Then, initialize the AI Logic service as below
### 0. Enable Firebase AI Logic via CLI
Before adding dependencies in your app, make sure you enable the AI Logic
service in your Firebase Project using the Firebase CLI:
```bash
npx -y firebase-tools@latest init
# When prompted, select 'AI logic' to enable the Gemini API in your project.
```
______________________________________________________________________
### 1. Add Dependencies
In your module-level `build.gradle.kts` (usually `app/build.gradle.kts`), add
the dependency for Firebase AI:
```kotlin
dependencies {
// [AGENT] Fetch the latest available BoM version from https://firebase.google.com/support/release-notes/android before adding this
implementation(platform("com.google.firebase:firebase-bom:<latest_bom_version>"))
// Add the dependency for the Firebase AI library
implementation("com.google.firebase:firebase-ai")
}
```
______________________________________________________________________
### 2. Initialize and Generate Content
In your Activity or Fragment, initialize the `FirebaseAI` service and generate
content using a Gemini model:
```kotlin
import com.google.firebase.Firebase
import com.google.firebase.ai.ai
import com.google.firebase.ai.type.GenerativeBackend
class MainActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
// Initialize with replay protection using limited-use App Check tokens:
val ai = Firebase.ai(
backend = GenerativeBackend.googleAI(),
useLimitedUseAppCheckTokens = true
)
// [AGENT] Replace "<latest_supported_model>" with the latest model from https://firebase.google.com/docs/ai-logic/models.md.txt (do NOT use Gemini 1.x or 2.x models)
val model = ai.generativeModel("<latest_supported_model>")
// Generate content
lifecycleScope.launch {
try {
val response = model.generateContent("Write a story about a magic backpack.")
Log.d(TAG, "Response: ${response.text}")
} catch (e: Exception) {
Log.e(TAG, "Error generating content", e)
}
}
}
}
```
#### App Check Replay Protection on Android
Generative and preview models enforce replay protection using short-lived
(5-minute) limited-use App Check tokens to prevent replay attacks. If you call a
model enforcing replay protection without limited-use tokens, the request is
rejected with:
```text
HTTP 403: "To access this model, you must enforce Firebase App Check"
```
To resolve this on Android, pass `useLimitedUseAppCheckTokens = true` when
calling `Firebase.ai`:
```kotlin
val ai = Firebase.ai(
backend = GenerativeBackend.googleAI(),
useLimitedUseAppCheckTokens = true
)
```
This ensures the SDK requests fresh limited-use tokens (via Play Integrity or
the debug provider) for each request instead of reusing cached tokens.
#### Jetpack Compose (Modern)
Initialize inside a `ComponentActivity` and use `setContent`:
```kotlin
import android.os.Bundle
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.lifecycle.lifecycleScope
import com.google.firebase.Firebase
import com.google.firebase.ai.ai
import kotlinx.coroutines.launch
class MainActivity : ComponentActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
val ai = Firebase.ai
// [AGENT] Replace with the latest model from https://firebase.google.com/docs/ai-logic/models.md.txt
val model = ai.generativeModel("<latest_supported_model>")
lifecycleScope.launch {
val response = model.generateContent("Hello Gemini!")
setContent {
MaterialTheme {
Text("AI Response: ${response.text}")
}
}
}
}
}
```
______________________________________________________________________
### 3. Multimodal Input (Text and Images)
Pass bitmap data along with text prompts:
```kotlin
val image1: Bitmap = ... // Load your bitmap
val image2: Bitmap = ...
val response = model.generateContent(
content {
image(image1)
image(image2)
text("Analyze these images for me. Compare these two items.")
}
)
Log.d(TAG, response.text)
```
______________________________________________________________________
### 4. Chat Session (Multi-turn)
Maintain chat history automatically:
```kotlin
val chat = model.startChat(
history = listOf(
content("user") { text("Hello, I am a software engineer.") },
content("model") { text("Hello! How can I help you today?") }
)
)
lifecycleScope.launch {
val response = chat.sendMessage("What should I learn next?")
Log.d(TAG, response.text)
}
```
______________________________________________________________________
### 5. Streaming Responses
For faster display, stream the response:
```kotlin
lifecycleScope.launch {
model.generateContentStream("Tell me a long story.")
.collect { chunk ->
print(chunk.text) // Update UI incrementally
}
}
```
______________________________________________________________________
### 6. App Check Debug Provider
For App Check debug tokens during local development and CI/CD, install
`DebugAppCheckProviderFactory.getInstance()` as the App Check provider factory
in debug builds:
```kotlin
if (BuildConfig.DEBUG) {
Firebase.appCheck.installAppCheckProviderFactory(
DebugAppCheckProviderFactory.getInstance()
)
}
```
#### Debug Token Persistence
Emulator resets or clearing app storage erase `SharedPreferences`, causing
`DebugAppCheckProviderFactory` to print a new debug token in Logcat. To persist
a stable debug token across emulator resets and fresh installs without
hardcoding secrets:
> [!WARNING] **CRITICAL: Never Hardcode or Commit Debug Tokens** Never hardcode
> debug token strings in `build.gradle.kts` or Kotlin source files. Store the
> token in gitignored `local.properties` and inject it dynamically.
1. In gitignored `local.properties`:
```properties
APP_CHECK_DEBUG_TOKEN=<YOUR_DEBUG_TOKEN>
```
1. In `app/build.gradle.kts`:
```kotlin
val localProperties = java.util.Properties().apply {
val localPropertiesFile = rootProject.file("local.properties")
if (localPropertiesFile.exists()) {
load(localPropertiesFile.inputStream())
}
}
val appCheckDebugToken = localProperties.getProperty("APP_CHECK_DEBUG_TOKEN")
?: System.getenv("APP_CHECK_DEBUG_TOKEN") ?: ""
android {
defaultConfig {
// For normal emulator runs (injects into AndroidManifest)
manifestPlaceholders["firebaseAppCheckDebugSecret"] = appCheckDebugToken
// For instrumentation tests
if (appCheckDebugToken.isNotEmpty()) {
testInstrumentationRunnerArguments["firebaseAppCheckDebugSecret"] = appCheckDebugToken
}
}
}
```
references/usage_patterns_web.md
# Firebase AI Logic on Web (JavaScript)
## Installation
The library is part of the standard Firebase Web SDK:
```bash
npm install firebase@latest
```
## Initialization Pattern
You must initialize the ai-logic service after the main Firebase App.
```JavaScript
import { initializeApp } from "firebase/app";
import { getAI, getGenerativeModel, GoogleAIBackend } from "firebase/ai";
// If running in Firebase App Hosting, you can skip Firebase Config and instead use:
// const app = initializeApp();
const firebaseConfig = {
// ... your firebase config
};
const app = initializeApp(firebaseConfig);
// Initialize the AI Logic service (defaults to Gemini Developer API)
// Pass useLimitedUseAppCheckTokens: true to enable replay protection:
const ai = getAI(app, {
backend: new GoogleAIBackend(),
useLimitedUseAppCheckTokens: true,
});
const generationConfig = {
candidateCount: 1,
maxOutputTokens: 2048,
stopSequences: [],
temperature: 0.7, // Balanced: creative but focused
topP: 0.95, // Standard: allows a wide range of probable tokens
topK: 40, // Standard: considers the top 40 tokens
};
// Specify the config as part of creating the `GenerativeModel` instance
// [AGENT] Replace "<latest_supported_model>" with the latest model from https://firebase.google.com/docs/ai-logic/models.md.txt (do NOT use Gemini 1.x or 2.x models like gemini-1.5-flash, gemini-2.0-flash, or gemini-2.5-flash)
const model = getGenerativeModel(ai, { model: "<latest_supported_model>", generationConfig });
```
### App Check Replay Protection
Generative and preview models enforce replay protection with 5-minute
limited-use App Check tokens. If you call a protected model without enabling
limited-use tokens, the request fails with:
```text
HTTP 403: "To access this model, you must enforce Firebase App Check"
```
To resolve this error on Web, initialize `getAI` with
`useLimitedUseAppCheckTokens: true`:
```javascript
const ai = getAI(app, {
backend: new GoogleAIBackend(),
useLimitedUseAppCheckTokens: true,
});
```
This instructs the SDK to fetch fresh, short-lived limited-use tokens (such as
via reCAPTCHA Enterprise or the debug provider) for requests rather than reusing
standard cached App Check tokens.
## Core Capabilities
Text-Only Generation
```JavaScript
async function generateText(prompt) {
const result = await model.generateContent(prompt);
const response = await result.response;
return response.text();
}
```
## Multimodal (Text + Images/Audio/Video/PDF input)
Firebase AI Logic accepts Base64 encoded data or specific file references.
```JavaScript
// Helper to convert file to base64 generic object
async function fileToGenerativePart(file) {
const base64EncodedDataPromise = new Promise((resolve) => {
const reader = new FileReader();
reader.onloadend = () => resolve(reader.result.split(',')[1]);
reader.readAsDataURL(file);
});
return {
inlineData: {
data: await base64EncodedDataPromise,
mimeType: file.type,
},
};
}
async function analyzeImage(prompt, imageFile) {
const imagePart = await fileToGenerativePart(imageFile);
const result = await model.generateContent([prompt, imagePart]);
return result.response.text();
}
```
## Chat Session (Multi-turn)
Maintain history automatically using startChat.
```JavaScript
const chat = model.startChat({
history: [
{
role: "user",
parts: [{ text: "Hello, I am a developer." }],
},
{
role: "model",
parts: [{ text: "Great to meet you. How can I help with code?" }],
},
],
});
async function sendMessage(msg) {
const result = await chat.sendMessage(msg);
return result.response.text();
}
```
## Streaming Responses
For real-time UI updates (like a typing effect).
```JavaScript
async function streamResponse(prompt) {
const result = await model.generateContentStream(prompt);
for await (const chunk of result.stream) {
const chunkText = chunk.text();
console.log("Stream chunk:", chunkText);
// Update UI here
}
}
```
Generate Images with Nano Banana
```Javascript
import { initializeApp } from "firebase/app";
import { getAI, getGenerativeModel, GoogleAIBackend, ResponseModality } from "firebase/ai";
// Initialize FirebaseApp
const firebaseApp = initializeApp(firebaseConfig);
// Initialize the Gemini Developer API backend service
const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() });
// Create a `GenerativeModel` instance with a model that supports your use case
const model = getGenerativeModel(ai, {
model: "<latest_supported_image_model>", // [AGENT] Replace with the latest image model from https://firebase.google.com/docs/ai-logic/models.md.txt
// Configure the model to respond with text and images (required)
generationConfig: {
responseModalities: [ResponseModality.TEXT, ResponseModality.IMAGE],
},
});
// Provide a text prompt instructing the model to generate an image
const prompt = 'Generate an image of the Eiffel Tower with fireworks in the background.';
// To generate an image, call `generateContent` with the text input
const result = await model.generateContent(prompt);
// Handle the generated image
try {
const inlineDataParts = result.response.inlineDataParts();
if (inlineDataParts?.[0]) {
const image = inlineDataParts[0].inlineData;
console.log(image.mimeType, image.data);
}
} catch (err) {
console.error('Prompt or candidate was blocked:', err);
}
```
## Advanced Features
Structured Output (JSON) Enforce a specific JSON schema for the response.
```JavaScript
import { getGenerativeModel, Schema } from "firebase/ai";
const jsonModel = getGenerativeModel(ai, {
model: "<latest_supported_model>", // [AGENT] Replace with the latest model from https://firebase.google.com/docs/ai-logic/models.md.txt
generationConfig: {
responseMimeType: "application/json",
// Optional: Define a schema
responseSchema: Schema.object({ ... })
}
});
async function getJsonData(prompt) {
const result = await jsonModel.generateContent(prompt);
return JSON.parse(result.response.text());
}
```
On-Device AI (Hybrid) Automatically switch between local Gemini Nano and cloud
models based on device capability.
The Firebase JavaScript SDK checks for Gemini Nano's availability (after
installation) and switches between on-device or cloud-hosted prompt execution.
This requires specific steps to enable model usage in the Chrome browser, more
info in the
[hybrid inference documentation for Web](https://firebase.google.com/docs/ai-logic/hybrid/web/get-started.md.txt).
```JavaScript
import {getGenerativeModel, InferenceMode } from "firebase/ai";
const hybridModel = getGenerativeModel(ai, { mode: InferenceMode.PREFER_ON_DEVICE });
```
## App Check
See
[App Check with reCAPTCHA Enterprise](https://firebase.google.com/docs/app-check/web/recaptcha-enterprise-provider.md.txt)
for setup instructions.
App Check debug tokens for local development and CI/CD:
- **Local development**: Set `self.FIREBASE_APPCHECK_DEBUG_TOKEN = true;` before
initializing App Check (generates a token in the browser console; do not
hardcode secret strings in client code).
- **CI/CD**: Pass the pre-provisioned token from the environment instead (e.g.
`self.FIREBASE_APPCHECK_DEBUG_TOKEN = process.env.APP_CHECK_DEBUG_TOKEN`).
### Debug Token Persistence
When testing AI Logic on Web with App Check, clearing site data, using
incognito/private windows, or switching browsers generates a new debug token
when `self.FIREBASE_APPCHECK_DEBUG_TOKEN = true` is used. To avoid this debug
token churn without leaking secrets into client bundles, load a stable token
from a gitignored local environment file:
> [!WARNING] **CRITICAL: Never Hardcode or Commit Debug Tokens** Never hardcode
> debug token strings in web source code. Always load them from a local,
> gitignored environment file.
```javascript
import { initializeAppCheck, ReCaptchaEnterpriseProvider } from "firebase/app-check";
if (typeof self !== "undefined") {
if (typeof process !== "undefined" && process.env.NODE_ENV === "development") {
// ✅ SAFE: Load dynamically from Next.js environment variable; fallback to true to auto-generate
self.FIREBASE_APPCHECK_DEBUG_TOKEN =
process.env.NEXT_PUBLIC_APP_CHECK_DEBUG_TOKEN || true;
} else if (typeof import.meta !== "undefined" && import.meta.env?.DEV) {
// ✅ SAFE: Load dynamically from Vite environment variable; fallback to true to auto-generate
self.FIREBASE_APPCHECK_DEBUG_TOKEN =
import.meta.env.VITE_APPCHECK_DEBUG_TOKEN || true;
}
}
const appCheck = initializeAppCheck(app, {
provider: new ReCaptchaEnterpriseProvider("/* reCAPTCHA site key */"),
isTokenAutoRefreshEnabled: true,
});
```
SKILL.md
---
name: firebase-ai-logic-basics
description: >-
Integrates Firebase AI Logic (formerly Vertex AI for Firebase) into mobile and web apps (Android, iOS, Web, and Flutter) to call Gemini models from client code. Use when provisioning or initializing Firebase AI Logic, or adding Gemini features to an app: text generation, multimodal input, chat, streaming, structured output, hybrid on-device inference, App Check protection, or switching models with Remote Config.
version: 1.0.1
metadata:
author: Google LLC
category: AiAndMachineLearning
---
# Firebase AI Logic Basics
## Overview
Firebase AI Logic is a product of Firebase that allows developers to add gen AI
to their mobile and web apps using client-side SDKs. The app can directly call
Gemini models via Firebase AI Logic without the developer needing to manage a
dedicated backend. Firebase AI Logic was previously known as "Vertex AI for
Firebase".
It supports the two Gemini API providers:
- **Gemini Developer API**: It has a free tier ideal for prototyping, and
pay-as-you-go for production
- **Agent Platform Gemini API** (formerly branded Vertex AI): Ideal for scale
with enterprise-grade production readiness, requires Blaze pricing plan
Use the Gemini Developer API as a default, and only Agent Platform Gemini API
(formerly branded Vertex AI) if the application requires it.
______________________________________________________________________
## Setup & initialization
### Prerequisites
- Identify the platform the user is interested in building on prior to starting:
Android, iOS, Web, or Flutter.
- If their platform is unsupported, Direct the user to Firebase Docs to learn
how to set up AI Logic for their application (share this link with the user
https://firebase.google.com/docs/ai-logic/get-started)
- The Firebase CLI commands below run through `npx` on every platform, so they
need **Node.js 20+** and npm. Use the `firebase-basics` skill to check for
them and to log in to the Firebase CLI. If Node.js is missing or older than
version 20, ask the user to install it as that skill describes. Don't install
system packages yourself, for example with `sudo apt-get`, because that
changes the user's machine without their consent.
### 1. Provision Firebase AI Logic (ALL platforms)
Use the **Firebase CLI** to provision Firebase AI Logic for the Firebase
Project:
1. Make sure that you're in a firebase directory (with a `firebase.json`).
1. Verify that you're running Firebase CLI commands against the intended
Firebase Project. When you run this command, the active project is marked
with "current":
```bash
npx -y firebase-tools@latest projects:list
```
1. Make sure there's at least one Firebase App for the target platform
registered with the current Firebase Project:
```bash
npx -y firebase-tools@latest apps:list
```
1. Provision Firebase AI Logic for the Firebase Project:
```bash
npx -y firebase-tools@latest init ailogic
```
This command will enable the Gemini Developer API in the Firebase project.
> [!WARNING] **CRITICAL: Provisioning Firebase AI Logic is Required** For all
> platforms (Android, iOS, Web, or Flutter), even if the app already uses
> Firebase, you MUST run `npx -y firebase-tools@latest init ailogic` to
> provision the service. The configuration files of `GoogleService-Info.plist`,
> `google-services.json`, and `firebase_options.dart` ONLY handle client
> configuration and do NOT enable the Firebase AI Logic service, leading to
> `PERMISSION_DENIED` errors.
If you can't run `init ailogic` yourself (for example because Node.js is missing
or the Firebase CLI isn't logged in) do NOT skip it. The app's requests to
Gemini models will fail until the Firebase AI Logic service is provisioned. Tell
the user to run this command in their project directory, and include it in your
final summary:
```bash
npx -y firebase-tools@latest init ailogic
```
### 2. Add the SDK and initialize the service in the app
Adding the SDK dependency and initializing the service are platform-specific.
Before writing code, read the reference for the target platform listed under
[Initialization Code References](#initialization-code-references).
More info in
[Firebase AI Logic Getting Started](https://firebase.google.com/docs/ai-logic/get-started.md.txt)
______________________________________________________________________
## Core Capabilities
> [!WARNING] **CRITICAL: Use current model names:** Always check the
> [Firebase AI Logic Models documentation](https://firebase.google.com/docs/ai-logic/models.md.txt)
> for the currently supported model names. Do NOT use any Gemini 1.x or Gemini
> 2.x models (such as `gemini-1.5-flash`, `gemini-1.5-pro`, `gemini-2.0-flash`,
> `gemini-2.0-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`, or
> `gemini-2.5-pro`), as older models are deprecated or shut down. Always use a
> currently supported Gemini 3+ model; if URL access is blocked in your
> environment, proceed directly without searching local files.
### Text-Only Generation
### Multimodal (Text + Images/Audio/Video/PDF input)
Firebase AI Logic allows Gemini models to analyze image files directly from your
app. This enables features like creating captions, answering questions about
images, detecting objects, and categorizing images. Beyond images, Gemini can
analyze other media types like audio, video, and PDFs by passing them as inline
data with their MIME type. For files larger than 20 megabytes (which can cause
HTTP 413 errors as inline data), store them in Cloud Storage for Firebase and
pass their URLs to the Gemini Developer API.
### Chat Session (Multi-turn)
Maintain history automatically using `startChat`.
### Streaming Responses
To improve the user experience by showing partial results as they arrive (like a
typing effect), use `generateContentStream` instead of `generateContent` for
faster display of results.
### Text-to-Speech (TTS) Generation
Generate spoken audio directly on client devices without a custom speech
backend. Firebase AI Logic supports speech synthesis using dedicated Gemini TTS
models:
- **Supported Models**: `gemini-3.1-flash-tts-preview`
- **Capabilities**:
- Single-speaker voice persona selection (`voiceName`) and multi-speaker
dialogues (up to 2 distinct speakers) via `SpeechConfig` /
`MultiSpeakerVoiceConfig`
- Direct audio responses by setting `responseModalities` to audio
- Streaming speech responses with `generateContentStream` for low-latency
playback
- Audio directives (`[Audio Profile: ...]`, `[Scene: ...]`,
`[Director's Note: ...]`) and emotional tags (`[whispers]`, `[laughs]`,
`[slowly]`)
- Client-side decoding and playback of 24 kHz 16-bit linear PCM (`audio/l16`)
or WAV container format (`audio/x-wav`)
See the [Text-to-Speech (TTS) Generation Guide](references/tts_generation.md)
for full implementation details, streaming patterns, and copy-paste code
snippets for iOS (Swift), Android (Kotlin), and Web.
### Generate Images with Nano Banana
> [!WARNING] **Use current Image model names:** Always check the
> [Firebase AI Logic Models documentation](https://firebase.google.com/docs/ai-logic/models.md.txt)
> for the currently supported image generation (Nano Banana) model names.
- Requires an upgraded Blaze pay-as-you-go billing plan.
### Search Grounding with the built in googleSearch tool
______________________________________________________________________
## Supported Platforms and Frameworks
Supported Platforms and Frameworks include Kotlin and Java for Android, Swift
for iOS, JavaScript for web apps, Dart for Flutter, and C Sharp for Unity.
## Advanced Features
### Structured Output (JSON)
Enforce a specific JSON schema for the response.
### On-Device AI (Hybrid)
Hybrid inference runs prompts with an on-device model when one is available and
falls back to a cloud-hosted model otherwise (or vice versa). Availability and
setup steps differ by platform, more info in the
[hybrid and on-device documentation](https://firebase.google.com/docs/ai-logic/hybrid.md.txt),
which links to each platform's guide.
______________________________________________________________________
## Security & Production
### Firebase App Check
> [!WARNING] **Critical Safety Requirement:** In order to use AI Logic safely,
> you MUST set up App Check on your app. This prevents unauthorized clients from
> using your API quota and accessing your backend resources.
See
[App Check for Firebase AI Logic](https://firebase.google.com/docs/ai-logic/app-check.md.txt)
for setup instructions. Each platform uses its own attestation provider, so
follow the instructions for the target platform.
#### App Check Debug Tokens for Local Development & CI/CD
Because App Check attestation providers (like Play Integrity or DeviceCheck)
reject emulators, simulators, or CI environments, you must use **App Check Debug
Tokens** during development and testing to bypass standard attestation.
> [!WARNING] **CRITICAL: Never Hardcode or Commit Debug Tokens** App Check debug
> tokens allow clients to bypass attestation and access backend resources
> without a genuine device. Treat them as private secrets. **Never commit debug
> tokens to version control or hardcode raw token strings in client code** (Web,
> Android, iOS, or Flutter). Always inject them through local environment
> variables, gitignored local configurations, or CI secrets. If a token is
> compromised, revoke it immediately in the Firebase Console.
##### Local Development (Auto-Generated)
1. Configure your code's App Check provider to use the debug factory. The API is
platform-specific; see the App Check section of the platform reference or the
App Check documentation linked above.
1. Run your app in the emulator/localhost.
1. Look at your runtime debugger console / Logcat logs for the generated UUID:
- *Example:* `AppCheck debug token: "123a4567-b89c-12d3-e456-789012345678"`
1. Register this token in the Firebase Console under **Security > App Check >
Apps > Manage debug tokens**.
> **💡 Tip (Prevent Debug Token Churn):** Simulator or emulator resets, fresh
> installs, and clearing browser data can make the SDK generate a new debug
> token that you must register again. To keep a stable debug token without
> hardcoding it, see the App Check section of the platform reference.
##### CI/CD Pipelines (Pre-Provisioned)
1. Generate and register a new debug token in the Firebase Console under
**Security > App Check > Apps > Manage debug tokens**.
1. Add this token string as an encrypted secret in your CI system (e.g.
`APP_CHECK_DEBUG_TOKEN`).
1. Configure your build to pass this secret as an environment variable to the
SDK during test execution.
### Firebase Remote Config
Consider that you do not need to hardcode model names (e.g., a specific model
version string). Use Firebase Remote Config to update model versions dynamically
without deploying new client code. See
[Changing model names remotely](https://firebase.google.com/docs/ai-logic/change-model-name-remotely.md.txt)
______________________________________________________________________
## Initialization Code References
- **Android (Kotlin)**
- Gemini API Provider: Gemini Developer API
- Reference: [usage_patterns_android.md](references/usage_patterns_android.md)
- **iOS (Swift)**
- Gemini API Provider: Gemini Developer API
- Reference: [ios_setup.md](references/ios_setup.md)
- **Web Modular API**
- Gemini API Provider: Gemini Developer API
- Reference: [usage_patterns_web.md](references/usage_patterns_web.md)
- **Flutter (Dart)**
- Gemini API Provider: Gemini Developer API
- Reference: [flutter_setup.md](references/flutter_setup.md)
> [!WARNING] **CRITICAL: Use current model names:** Always check the
> [Firebase AI Logic Models documentation](https://firebase.google.com/docs/ai-logic/models.md.txt)
> for the currently supported model names. Do NOT use any Gemini 1.x or Gemini
> 2.x models (such as `gemini-1.5-flash`, `gemini-2.0-flash`,
> `gemini-2.5-flash`, or `gemini-2.5-flash-lite`), as older models are
> deprecated or shut down.
## References
[Android (Kotlin) SDK usage patterns](references/usage_patterns_android.md)
[iOS SDK code examples and usage patterns](references/ios_setup.md)
[Web SDK setup, code examples, and usage patterns](references/usage_patterns_web.md)
[Flutter SDK code examples and usage patterns](references/flutter_setup.md)
[Client-side Text-to-Speech (TTS) generation](references/tts_generation.md)