Calling a model from an app, safely

The first architectural decision is also the easiest to get wrong: the API key does not go in the app.

Why the key must live on your backend

An APK can be unzipped and its strings read; obfuscation only raises the effort. A leaked key is someone else's bill. Routing through your own endpoint also lets you rate-limit per user, swap models without an app release, cache common prompts and log what is actually being asked.

// app -> your backend -> model provider
POST /v1/ai/summarise   { "noteId": "123" }

A direct client SDK is acceptable only for a prototype, or with a provider feature built for untrusted clients that issues short-lived, scoped tokens.

Stream, always

A six-second answer that starts appearing in 400 ms feels fast. The same answer delivered at once feels broken. Collect tokens as a Flow and append to state.

fun summarise(noteId: String) = viewModelScope.launch {
    _state.update { it.copy(isStreaming = true, text = "") }
    ai.streamSummary(noteId)
        .onEach { chunk -> _state.update { it.copy(text = it.text + chunk) } }
        .catch { e -> _state.update { it.copy(error = "Could not generate a summary") } }
        .onCompletion { _state.update { it.copy(isStreaming = false) } }
        .collect()
}

Because it is a coroutine, cancellation is free: a Stop button is job.cancel(), and leaving the screen cancels it automatically.

Cost and latency levers

• Send less context. A rolling summary of a conversation instead of every message.
• Cache server-side. Identical prompts — a daily tip, a product summary — are generated once for everyone.
• Pick the small model first. Most app features do not need the largest model; measure quality on your own cases before paying for it.
• Cap output length. Both for cost and because nobody reads 800 words on a phone.

The failure cases to design for

Timeouts, rate limits, a refusal, and an empty response. An AI feature must never block a screen: show the deterministic content, and let the generated part fail quietly with a retry.