On-device vs cloud
Lesson 3 of 42 min read
What actually runs on a phone today, and the honest trade-offs.
On-device inference is genuinely useful now — for the right features. Pick with four questions.
The four questions
- Is the data sensitive? Health notes, photos, messages. On-device means it never leaves the phone, and that is a feature you can put on the store listing.
- Must it work offline? A metro tunnel, a village with no signal. On-device wins outright.
- How good must the output be? A small on-device model is noticeably weaker than a large cloud one. For classification, extraction and short rewrites that is fine; for a long reasoned answer it is not.
- What is the volume? Thousands of calls per user per month is a real bill in the cloud and free on-device.
What is available
- ML Kit — ready-made on-device features: text recognition, translation, barcode, face and pose, smart reply. If one of these solves your problem, use it; it is small, fast and free.
- Gemini Nano via ML Kit GenAI — on-device summarisation, proofreading and rewriting on supported devices. Check availability at runtime; it is not on every phone.
- LiteRT / TensorFlow Lite — your own model, quantised, for classification and vision.
- Apple Foundation Models and Core ML on iOS, the same shape of decision.
The costs people forget
- App or download size. A model is megabytes to hundreds of megabytes. Ship it as a download, not in the APK.
- Device variability. A flagship runs it in 300 ms; a 2 GB budget phone may not run it at all. Always have a path for "not supported here".
- Battery and heat on long or repeated inference.
- No instant upgrade. Improving a cloud model is a config change; improving an on-device one is an app release.
The pattern that usually wins
Hybrid: on-device for the fast, private, common case, cloud for the hard case, with the UI making the fallback invisible. Design the feature so a device that cannot run the model still has something sensible to show.