Realtime API

The Realtime API enables you to transform live video streams with minimal latency using WebRTC. Perfect for building iOS camera effects, video conferencing filters, AR applications, and interactive live streaming.

Quick Start

import DecartSDK
import WebRTC

let model = Models.realtime(.mirage_v2)

// Create client
let config = DecartConfiguration(apiKey: "your-api-key-here")
let client = DecartClient(decartConfiguration: config)

// Create realtime manager
let manager = try client.createRealtimeManager(
    options: RealtimeConfiguration(
        model: model,
        initialPrompt: DecartPrompt(text: "Anime style", enrich: true)
    )
)

// Set up camera capture
let videoSource = manager.createVideoSource()
let videoTrack = manager.createVideoTrack(source: videoSource, trackId: "camera-video")
let capture = RealtimeCapture(
    model: model,
    videoSource: videoSource,
    orientation: .portrait
)
try await capture.startCapture()

// Connect and get transformed stream
let localStream = RealtimeMediaStream(videoTrack: videoTrack, id: .localStream)
let remoteStream = try await manager.connect(localStream: localStream)

// Change style on the fly
manager.setPrompt(DecartPrompt(text: "Cyberpunk city", enrich: true))

// Disconnect when done
await manager.disconnect()
await capture.stopCapture()

Client-Side Authentication

For iOS and macOS applications, use ephemeral keys instead of embedding your permanent API key in the app bundle. Ephemeral keys are short-lived tokens safe to include in client applications.

Learn more about client tokens and why they’re important for security.

Fetching an Ephemeral Key

Your app should fetch an ephemeral key from your backend server before connecting:

import Foundation

struct EphemeralKeyResponse: Codable {
    let apiKey: String
    let expiresAt: String
}

func fetchEphemeralKey() async throws -> String {
    // Replace with your backend URL
    let url = URL(string: "https://your-backend.com/api/realtime-token")!

    var request = URLRequest(url: url)
    request.httpMethod = "POST"
    // Add any auth headers your backend requires
    // request.setValue("Bearer \(userToken)", forHTTPHeaderField: "Authorization")

    let (data, response) = try await URLSession.shared.data(for: request)

    guard let httpResponse = response as? HTTPURLResponse,
          httpResponse.statusCode == 200 else {
        throw DecartError.invalidAPIKey
    }

    let keyResponse = try JSONDecoder().decode(EphemeralKeyResponse.self, from: data)
    return keyResponse.apiKey
}

Connecting with an Ephemeral Key

import DecartSDK

func connectToRealtime() async throws -> DecartRealtimeManager {
    // 1. Fetch ephemeral key from your backend
    let ephemeralKey = try await fetchEphemeralKey()

    // 2. Create client with ephemeral key
    let config = DecartConfiguration(apiKey: ephemeralKey)
    let client = DecartClient(decartConfiguration: config)

    // 3. Set up manager and camera, then connect
    let model = Models.realtime(.mirage_v2)

    let manager = try client.createRealtimeManager(
        options: RealtimeConfiguration(
            model: model,
            initialPrompt: DecartPrompt(text: "Anime style", enrich: true)
        )
    )

    let videoSource = manager.createVideoSource()
    let videoTrack = manager.createVideoTrack(source: videoSource, trackId: "camera-video")
    let capture = RealtimeCapture(
        model: model,
        videoSource: videoSource,
        orientation: .portrait
    )
    try await capture.startCapture()

    let localStream = RealtimeMediaStream(videoTrack: videoTrack, id: .localStream)
    _ = try await manager.connect(localStream: localStream)

    return manager
}

Never hardcode your permanent API key in iOS apps. App bundles can be decompiled, exposing embedded secrets.

Camera Capture

The SDK provides RealtimeCapture for managing camera capture on both iOS and macOS. It handles device selection, format negotiation, and frame rate configuration automatically.

Setting Up Capture

import DecartSDK
import WebRTC

let model = Models.realtime(.mirage_v2)

// Create manager first, then use it to create video source and track
let manager = try client.createRealtimeManager(
    options: RealtimeConfiguration(model: model)
)
let videoSource = manager.createVideoSource()
let videoTrack = manager.createVideoTrack(source: videoSource, trackId: "camera-video")

// Create capture with orientation and initial camera position
let capture = RealtimeCapture(
    model: model,
    videoSource: videoSource,
    orientation: .portrait,        // .portrait or .landscape
    initialPosition: .front        // .front or .back
)

// Start capturing
try await capture.startCapture()

Parameters:

model (required) - Model definition that determines target resolution and FPS
videoSource (required) - WebRTC video source to feed frames into
orientation (optional) - .portrait or .landscape (default: .portrait). In portrait mode, width and height are swapped automatically.
initialPosition (optional) - .front or .back camera (default: .front)

Switching Cameras

// Toggle between front and back cameras
try await capture.switchCamera()

// Check current camera position
let position = capture.position  // .front or .back

On macOS, switchCamera() cycles through all available cameras rather than toggling front/back.

Stopping Capture

await capture.stopCapture()

Use the model’s fps, width, and height properties to ensure optimal performance. RealtimeCapture handles this automatically.

Camera capture requires a real iOS device. The simulator does not support camera access for WebRTC.

Connecting

Create a DecartRealtimeManager and connect with your local media stream:

let manager = try client.createRealtimeManager(
    options: RealtimeConfiguration(
        model: Models.realtime(.mirage_v2),
        initialPrompt: DecartPrompt(
            text: "Lego World",
            enrich: true  // Let Decart enhance the prompt (recommended)
        ),
        connection: .init(
            connectionTimeout: 15  // seconds, default
        ),
        media: .init(
            video: .init(
                maxBitrate: 2_500_000,  // default
                preferredCodec: "VP8"   // default
            )
        )
    )
)

let localStream = RealtimeMediaStream(videoTrack: videoTrack, id: .localStream)
let remoteStream = try await manager.connect(localStream: localStream)

RealtimeConfiguration parameters:

model (required) - Realtime model from Models.realtime()
initialPrompt (optional) - Initial transformation prompt
connection (optional) - Connection configuration
- iceServers - STUN/TURN server URLs (default: Google STUN)
- connectionTimeout - Connection timeout in seconds (default: 15)
- rtcConfiguration - Custom RTCConfiguration for advanced WebRTC tuning
media (optional) - Media configuration
- video.maxBitrate - Max bitrate in bps (default: 2,500,000)
- video.minBitrate - Min bitrate in bps (default: 300,000)
- video.maxFramerate - Max framerate (default: 26)
- video.preferredCodec - Preferred video codec (default: “VP8”)

Returns: RealtimeMediaStream — the transformed remote stream containing an optional videoTrack you can render

Managing Prompts

Change the transformation style dynamically without reconnecting:

// Simple prompt with automatic enhancement
manager.setPrompt(DecartPrompt(text: "Anime style", enrich: true))

// Custom detailed prompt without enhancement
manager.setPrompt(
    DecartPrompt(
        text: "A detailed artistic style with vibrant colors and dramatic lighting",
        enrich: false
    )
)

// With reference image (for lucy_2_rt model)
// On iOS: let referenceData = UIImage(named: "reference")!.jpegData(compressionQuality: 0.8)!
// On macOS: let referenceData = NSImage(named: "reference")!.tiffRepresentation!
let referenceData = try Data(contentsOf: Bundle.main.url(forResource: "reference", withExtension: "jpg")!)
manager.setPrompt(
    DecartPrompt(
        text: "Match this character style",
        referenceImageData: referenceData,
        enrich: true
    )
)

DecartPrompt parameters:

text (required) - Text description of desired style
referenceImageData (optional) - Reference image data (used with lucy_2_rt)
enrich (optional) - Whether to enhance the prompt (default: false)

Prompt enhancement uses Decart’s AI to expand simple prompts for better results. Only the lucy_2_rt model supports reference images.

Connection State

Monitor connection state, service status, generation ticks, and session ID using the events AsyncStream:

// Observe state changes
Task {
    for await state in manager.events {
        switch state.connectionState {
        case .connecting:
            showLoadingIndicator()
        case .connected:
            hideLoadingIndicator()
        case .generating:
            showGeneratingIndicator()
        case .reconnecting:
            showReconnectingIndicator()
        case .disconnected:
            showReconnectButton()
        case .idle:
            break
        case .error:
            showError()
        }

        // Track generation progress
        if let tick = state.generationTick {
            showGenerationTime("\(tick)s")
        }

        // Track session ID
        if let sessionId = state.sessionId {
            print("Session: \(sessionId)")
        }

        // Track queue position
        if let position = state.queuePosition, let size = state.queueSize {
            showQueueStatus("Position \(position) of \(size)")
        }

        // Track service status
        switch state.serviceStatus {
        case .enteringQueue:
            showQueueMessage()
        case .ready:
            hideQueueMessage()
        case .unknown:
            break
        }
    }
}

DecartRealtimeState properties:

connectionState — .idle, .connecting, .connected, .generating, .reconnecting, .disconnected, .error
serviceStatus — .unknown, .enteringQueue, .ready
queuePosition — Current position in queue (nil if not queued)
queueSize — Total queue size (nil if not queued)
generationTick — Seconds elapsed during generation (nil when not generating)
sessionId — Current session identifier (nil before session established)

DecartRealtimeConnectionState helpers:

.isConnected — true when connected or generating
.isInSession — true when connected, connecting, generating, or reconnecting

Auto-Reconnect

The SDK automatically reconnects when an unexpected disconnection occurs (e.g., network interruption). During auto-reconnect, the connection state transitions to .reconnecting while the SDK retries with exponential backoff (up to 5 attempts, max 10s delay). When auto-reconnect succeeds, a new RealtimeMediaStream is emitted via remoteStreamUpdates. You must rebind your UI to the new stream’s video track:

Task {
    for await newRemoteStream in manager.remoteStreamUpdates {
        // Update your UI with the new video track
        self.remoteVideoTrack = newRemoteStream.videoTrack
    }
}

Auto-reconnect is not triggered on user-initiated disconnect(), permanent errors (401/403, invalid key, expired session), or after all retries are exhausted. If all retries fail, the state moves to .error.

Track Management

Replace the video track during an active session (e.g., after switching cameras):

// Create a new video source and track
let newSource = manager.createVideoSource()
let newTrack = manager.createVideoTrack(source: newSource, trackId: "new-video")

// Replace the active track
manager.replaceVideoTrack(with: newTrack)

You can also create audio sources and tracks:

let audioSource = manager.createAudioSource()
let audioTrack = manager.createAudioTrack(source: audioSource, trackId: "audio")

Error Handling

Errors are thrown from async methods and can also arrive through the events stream when the connection state becomes .error:

do {
    let remoteStream = try await manager.connect(localStream: localStream)
} catch let error as DecartError {
    switch error {
    case .invalidAPIKey:
        showError("Invalid API key. Please check your credentials.")
    case .webRTCError(let message):
        showError("Connection error: \(message)")
    case .websocketError(let message):
        showError("WebSocket error: \(message)")
    case .connectionTimeout:
        showError("Connection timed out. Please try again.")
    case .serverError(let message):
        showError("Server error: \(message)")
    default:
        showError(error.localizedDescription)
    }
    print("Error code: \(error.errorCode)")
}

Error Cases:

.invalidAPIKey - API key is invalid or missing
.invalidBaseURL(String?) - Base URL is malformed
.webRTCError(String) - WebRTC connection failed
.websocketError(String) - WebSocket connection error
.connectionTimeout - Connection timed out
.serverError(String) - Server returned an error
.processingError(String) - Processing failed
.invalidInput(String) - Invalid input parameters
.modelNotFound(String) - Specified model doesn’t exist
.networkError(Error) - Network request failed
.queueError(String) - Queue operation failed

Cleanup

Always disconnect and stop capture when done to free up resources:

// Disconnect from the service
await manager.disconnect()

// Stop camera capture
await capture.stopCapture()

Failing to disconnect can leave WebRTC connections open and waste resources.

Complete SwiftUI Example

Here’s a full SwiftUI application using the SDK’s built-in RTCMLVideoViewWrapper:

import SwiftUI
import DecartSDK
import WebRTC

@main
struct RealtimeApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
    }
}

struct ContentView: View {
    @StateObject private var viewModel = RealtimeViewModel()

    var body: some View {
        ZStack {
            // Remote video background
            RTCMLVideoViewWrapper(track: viewModel.remoteVideoTrack)
                .ignoresSafeArea()

            VStack(spacing: 16) {
                // Status bar
                HStack {
                    VStack(alignment: .leading, spacing: 4) {
                        Text("Decart Realtime")
                            .font(.headline)
                            .foregroundColor(.white)
                        Text(viewModel.statusText)
                            .font(.caption)
                            .foregroundColor(
                                viewModel.isConnected ? .green : .white
                            )
                    }
                    Spacer()
                }
                .padding()
                .background(Color.black.opacity(0.6))

                Spacer()

                // Local video preview
                if viewModel.isConnected {
                    HStack {
                        Spacer()
                        RTCMLVideoViewWrapper(
                            track: viewModel.localVideoTrack,
                            mirror: true
                        )
                        .frame(width: 120, height: 160)
                        .clipShape(RoundedRectangle(cornerRadius: 12))
                        .overlay(
                            RoundedRectangle(cornerRadius: 12)
                                .stroke(Color.white, lineWidth: 2)
                        )
                        .padding()
                    }
                }

                // Controls
                VStack(spacing: 12) {
                    if let error = viewModel.lastError {
                        Text(error)
                            .foregroundColor(.red)
                            .font(.caption)
                            .padding(8)
                            .background(Color.black.opacity(0.8))
                            .clipShape(RoundedRectangle(cornerRadius: 8))
                    }

                    HStack(spacing: 12) {
                        TextField("Enter style prompt", text: $viewModel.promptText)
                            .textFieldStyle(.roundedBorder)

                        Button {
                            viewModel.updatePrompt()
                        } label: {
                            Image(systemName: "paperplane.fill")
                                .foregroundColor(.white)
                                .padding(12)
                                .background(
                                    viewModel.isConnected ? Color.blue : Color.gray
                                )
                                .clipShape(RoundedRectangle(cornerRadius: 8))
                        }
                        .disabled(!viewModel.isConnected)
                    }

                    HStack(spacing: 12) {
                        Button {
                            Task { try? await viewModel.switchCamera() }
                        } label: {
                            Image(systemName: "camera.rotate")
                                .foregroundColor(.white)
                                .padding(12)
                                .background(Color.gray.opacity(0.8))
                                .clipShape(Circle())
                        }
                        .disabled(!viewModel.isConnected)

                        Button {
                            Task { await viewModel.toggleConnection() }
                        } label: {
                            Text(viewModel.isConnected ? "Disconnect" : "Connect")
                                .fontWeight(.semibold)
                                .foregroundColor(.white)
                                .frame(maxWidth: .infinity)
                                .padding()
                                .background(
                                    viewModel.isConnected ? Color.red : Color.green
                                )
                                .clipShape(RoundedRectangle(cornerRadius: 12))
                        }
                    }
                }
                .padding()
                .background(Color.black.opacity(0.8))
                .clipShape(RoundedRectangle(cornerRadius: 16))
                .padding()
            }
        }
    }
}

@MainActor
class RealtimeViewModel: ObservableObject {
    @Published var statusText = "Disconnected"
    @Published var promptText = "Turn into a fantasy figure"
    @Published var lastError: String?
    @Published var isConnected = false
    @Published var localVideoTrack: RTCVideoTrack?
    @Published var remoteVideoTrack: RTCVideoTrack?

    private var manager: DecartRealtimeManager?
    private var capture: RealtimeCapture?
    private var stateTask: Task<Void, Never>?
    private var reconnectTask: Task<Void, Never>?

    func toggleConnection() async {
        if isConnected {
            await disconnect()
        } else {
            await connect()
        }
    }

    func connect() async {
        statusText = "Connecting"
        lastError = nil

        do {
            let config = DecartConfiguration(
                apiKey: ProcessInfo.processInfo.environment["DECART_API_KEY"] ?? ""
            )
            let client = DecartClient(decartConfiguration: config)
            let model = Models.realtime(.mirage_v2)

            // Create manager
            let manager = try client.createRealtimeManager(
                options: RealtimeConfiguration(
                    model: model,
                    initialPrompt: DecartPrompt(text: promptText, enrich: true)
                )
            )
            self.manager = manager

            // Set up camera
            let videoSource = manager.createVideoSource()
            let videoTrack = manager.createVideoTrack(
                source: videoSource,
                trackId: "camera-video"
            )
            let capture = RealtimeCapture(
                model: model,
                videoSource: videoSource,
                orientation: .portrait
            )
            try await capture.startCapture()
            self.capture = capture
            self.localVideoTrack = videoTrack

            // Connect
            let localStream = RealtimeMediaStream(
                videoTrack: videoTrack,
                id: .localStream
            )
            let remoteStream = try await manager.connect(localStream: localStream)
            self.remoteVideoTrack = remoteStream.videoTrack

            // Observe state changes
            stateTask = Task { [weak self] in
                for await state in manager.events {
                    guard let self else { return }
                    self.handleState(state)
                }
            }

            // Handle auto-reconnect track rebinding
            reconnectTask = Task { [weak self] in
                for await newStream in manager.remoteStreamUpdates {
                    guard let self else { return }
                    self.remoteVideoTrack = newStream.videoTrack
                }
            }
        } catch {
            lastError = error.localizedDescription
            statusText = "Disconnected"
        }
    }

    func disconnect() async {
        stateTask?.cancel()
        stateTask = nil
        reconnectTask?.cancel()
        reconnectTask = nil
        await manager?.disconnect()
        await capture?.stopCapture()
        manager = nil
        capture = nil
        localVideoTrack = nil
        remoteVideoTrack = nil
        isConnected = false
        statusText = "Disconnected"
    }

    func updatePrompt() {
        manager?.setPrompt(DecartPrompt(text: promptText, enrich: true))
    }

    func switchCamera() async throws {
        try await capture?.switchCamera()
    }

    private func handleState(_ state: DecartRealtimeState) {
        switch state.connectionState {
        case .connecting:
            statusText = "Connecting"
            isConnected = false
        case .connected:
            statusText = "Connected"
            isConnected = true
        case .generating:
            if let tick = state.generationTick {
                statusText = "Generating (\(String(format: "%.1f", tick))s)"
            } else {
                statusText = "Generating"
            }
            isConnected = true
        case .reconnecting:
            statusText = "Reconnecting..."
            isConnected = false
        case .disconnected:
            statusText = "Disconnected"
            isConnected = false
        case .error:
            statusText = "Error"
            isConnected = false
        case .idle:
            break
        }

        if state.serviceStatus == .enteringQueue,
           let position = state.queuePosition {
            statusText = "In queue (position \(position))"
        }
    }
}

Best Practices

Use model properties for video constraints

RealtimeCapture automatically uses the model’s fps, width, and height properties to configure camera capture. Always pass the model when creating a capture instance.

let model = Models.realtime(.mirage_v2)
let capture = RealtimeCapture(model: model, videoSource: videoSource)

Enable prompt enrichment

For best results, set enrich: true to let Decart’s AI enhance your prompts. Only disable it if you need exact prompt control.

Handle auto-reconnect streams

Always observe remoteStreamUpdates to rebind your video track when the SDK auto-reconnects after a network interruption.

Observe state with AsyncStream

Use for await state in manager.events to track connection state, generation ticks, session ID, and queue position in a structured concurrency context.

Clean up properly

Always call manager.disconnect() and capture.stopCapture() when done to avoid memory leaks and unnecessary resource usage.

Test on real devices

Always test camera features on real iOS devices, as the simulator does not support WebRTC camera access.

Request permissions properly

Add camera and microphone usage descriptions to your Info.plist and handle permission denials gracefully in your UI.

API Reference

`DecartClient.createRealtimeManager(options:)`

Creates a realtime manager for a WebRTC session. Parameters:

options: RealtimeConfiguration - Configuration for the realtime session
- model: ModelDefinition - Realtime model from Models.realtime()
- initialPrompt: DecartPrompt - Initial transformation prompt (default: empty)
- connection: ConnectionConfig - Connection settings (default: standard)
- media: MediaConfig - Media settings (default: standard)

Returns: DecartRealtimeManager Throws: DecartError if the signaling URL cannot be constructed

`DecartRealtimeManager.connect(localStream:)`

Connects to the realtime transformation service. Parameters:

localStream: RealtimeMediaStream - Local media stream with camera video track

Returns: RealtimeMediaStream — the transformed remote stream Throws: DecartError if connection fails or times out

`DecartRealtimeManager.setPrompt(_:)`

Changes the transformation style. Parameters:

prompt: DecartPrompt - Prompt with text, optional reference image, and enrich flag

`DecartRealtimeManager.disconnect()`

Closes the connection and cleans up WebRTC resources.

`DecartRealtimeManager.events`

An AsyncStream<DecartRealtimeState> that emits state changes.

`DecartRealtimeManager.remoteStreamUpdates`

An AsyncStream<RealtimeMediaStream> that emits new remote streams after auto-reconnect.

`RealtimeCapture.startCapture()`

Starts camera capture using the model’s target resolution and FPS. Throws: CameraError if no camera is available or format selection fails

`RealtimeCapture.switchCamera()`

Toggles between front and back cameras (iOS) or cycles through available cameras (macOS).

`RealtimeCapture.stopCapture()`

Stops camera capture and releases the capture session.

Next Steps

SDK Overview

Learn about installation, setup, and Swift SDK fundamentals

GitHub

Browse the SDK source code and contribute

Getting started

Realtime

Non-realtime

Provider Integrations

Libraries & SDKs

Frameworks

Resources

Quick Start

Client-Side Authentication

Fetching an Ephemeral Key

Connecting with an Ephemeral Key

Camera Capture

Setting Up Capture

Switching Cameras

Stopping Capture

Connecting

Managing Prompts

Connection State

Auto-Reconnect

Track Management

Error Handling

Cleanup

Complete SwiftUI Example

Best Practices

API Reference

`DecartClient.createRealtimeManager(options:)`

`DecartRealtimeManager.connect(localStream:)`

`DecartRealtimeManager.setPrompt(_:)`

`DecartRealtimeManager.disconnect()`

`DecartRealtimeManager.events`

`DecartRealtimeManager.remoteStreamUpdates`

`RealtimeCapture.startCapture()`

`RealtimeCapture.switchCamera()`

`RealtimeCapture.stopCapture()`

Next Steps

SDK Overview

GitHub

Getting started

Realtime

Non-realtime

Provider Integrations

Libraries & SDKs

Frameworks

Resources

​Quick Start

​Client-Side Authentication

​Fetching an Ephemeral Key

​Connecting with an Ephemeral Key

​Camera Capture

​Setting Up Capture

​Switching Cameras

​Stopping Capture

​Connecting

​Managing Prompts

​Connection State

​Auto-Reconnect

​Track Management

​Error Handling

​Cleanup

​Complete SwiftUI Example

​Best Practices

​API Reference

​DecartClient.createRealtimeManager(options:)

​DecartRealtimeManager.connect(localStream:)

​DecartRealtimeManager.setPrompt(_:)

​DecartRealtimeManager.disconnect()

​DecartRealtimeManager.events

​DecartRealtimeManager.remoteStreamUpdates

​RealtimeCapture.startCapture()

​RealtimeCapture.switchCamera()

​RealtimeCapture.stopCapture()

​Next Steps

SDK Overview

GitHub

Quick Start

Client-Side Authentication

Fetching an Ephemeral Key

Connecting with an Ephemeral Key

Camera Capture

Setting Up Capture

Switching Cameras

Stopping Capture

Connecting

Managing Prompts

Connection State

Auto-Reconnect

Track Management

Error Handling

Cleanup

Complete SwiftUI Example

Best Practices

API Reference

`DecartClient.createRealtimeManager(options:)`

`DecartRealtimeManager.connect(localStream:)`

`DecartRealtimeManager.setPrompt(_:)`

`DecartRealtimeManager.disconnect()`

`DecartRealtimeManager.events`

`DecartRealtimeManager.remoteStreamUpdates`

`RealtimeCapture.startCapture()`

`RealtimeCapture.switchCamera()`

`RealtimeCapture.stopCapture()`

Next Steps