Shipping A Model In A Flutter App

---
name: shipping-a-model-in-a-flutter-app
description: >-
  Put a local language model inside a Flutter app and get it generating on
  device. Covers llamadart, which runs GGUF through llama.cpp and .litertlm
  through LiteRT-LM under one API on Android, iOS, macOS, Windows, Linux and
  Flutter web, plus LoRA adapter loading, flutter_gemma as the MediaPipe-lineage
  alternative, background isolate and threading behaviour, and whether to
  bundle the model or download it on first run. Use when the project is
  Flutter or Dart and someone wants on-device or offline AI, local inference,
  or a model running without an API. Not for React Native or Expo projects,
  not for iOS native Swift projects, not for Android native Kotlin projects,
  not for JavaScript browser apps using transformers.js or WebLLM, and not for
  models running on a server.
license: Apache-2.0
metadata:
  version: "0.1.0"
  author: "Edward Xi Yang, Ertas AI"
  stage: "ship"
  previous_skill: "inspecting-a-model-bundle"
---

# Shipping a model in a Flutter app

Flutter is the simplest of the four targets in this suite, for one reason:
one package, `llamadart`, covers both mobile-relevant artifact shapes and
every platform Flutter builds for. React Native needs `llama.rn` for GGUF
and `react-native-executorch` for `.pte`, two native modules with two build
paths. Native Android needs a MediaPipe/LiteRT stack for one shape and a
llama.cpp NDK integration for the other. Flutter needs one dependency, one
`LlamaEngine` class, for both.

**Check these floors before writing any integration code, as of 2026-07-27:**
Dart `>= 3.10.7`, Flutter `>= 3.38.0`, iOS deployment target `16.4` or newer,
macOS deployment target `14.0` or newer (source: `llamadart` 0.8.17's own
installation guide, see the table at the end of this section). `llamadart` is
a young, fast-moving 0.x package, so treat every version number in this skill
as an "as of" fact to re-check, not a permanent one. Flutter 3.38.0 and iOS
16.4 are both recent enough to exclude some existing projects; finding that
out after wiring up integration code is a wasted afternoon, not a five-minute
fix.

## Which artifact shape this needs

`llamadart` 0.8.17 (pub.dev, MIT license, published 2026-07-22) eats both
shapes behind the same API. Its own pub.dev description, verbatim: "Dart and
Flutter local LLM inference with llama.cpp GGUF and LiteRT-LM across native
platforms and web."

| You are holding | `llamadart` routes it through | What it eats |
|---|---|---|
| A single `.gguf` file | llama.cpp | GGUF, self-contained |
| A `.litertlm` file | LiteRT-LM | A LiteRT LM file, produced by `litert-torch` |

`LlamaBackend()`, the router passed into `LlamaEngine`, picks the underlying
runtime from the model file's extension: `.gguf` and unknown extensions go
through llama.cpp, `.litertlm` goes through LiteRT-LM (source: `llamadart`'s
own Platform & Backend Matrix doc, "Model format routing"). Both routes share
`LlamaEngine`, `ModelSource`, `ChatSession`, and the download and cache APIs;
only the load-time parameters that are backend-specific get rejected when
used against the wrong format, rather than silently ignored.

**If what you are holding is neither of these,** a merged Hugging Face
checkpoint (`config.json` + `model*.safetensors`) or a PEFT adapter directory
(`adapter_config.json` + `adapter_model.safetensors`), it is not shippable
into a Flutter app as is. Run **inspecting-a-model-bundle** first to confirm
which shape you actually have, then convert:

- To GGUF: `convert_hf_to_gguf.py` then `llama-quantize`, covered in
  `references/llamadart-path.md`.
- To `.litertlm`: `litert-torch export_hf`, also covered there.

A PEFT adapter directory converts to neither format directly. Merge it into
its base model first with `peft`'s `merge_and_unload()`, then export from the
merged checkpoint. The GGUF path has a second option: convert the adapter
itself with `convert_lora_to_gguf.py` and load it as a runtime LoRA against a
base GGUF, covered in the LoRA section of `references/llamadart-path.md`.

**If it is genuinely unclear which of the two to pick,** GGUF is the safer
default: it is the higher-coverage artifact across the open-model ecosystem,
`llamadart` documents it as the format with the broader feature surface
(embeddings, dynamic LoRA, grammar-constrained decoding, KV-cache state
persistence, multimodal projectors), and quantisation tooling for it is
mature. Reach for `.litertlm` when the model is already distributed as a
LiteRT-LM bundle, or the target is specifically the Gemma family on
Android's GPU or NPU delegate path.

**Two other Flutter packages exist and are worth knowing about, not
necessarily reaching for.** `flutter_gemma` 1.4.0 (pub.dev, published
2026-07-26) is the MediaPipe/LiteRT lineage alternative: "Run Gemma and other
LLMs on-device in Flutter (Android, iOS, Web, Desktop)," eating the same
`.task` / `.litertlm` shapes covered in **shipping-a-model-in-an-android-app**.
`cactus` 1.3.0 (pub.dev) is GGUF-based, but its last release was December
2025, seven months stale as of this writing; treat that gap as a caution
before depending on it for anything shipping soon, not a recommendation
either way.

**A word on the web target, since `llamadart` also reaches it.** `llamadart`
compiles into Flutter Web builds (a llama.cpp WebGPU/CPU bridge for GGUF, the
`@litert-lm/core` JavaScript runtime for `.litertlm`), and that is still
Flutter and Dart code, in scope for this skill. A plain JavaScript or React
browser app reaching for `transformers.js` or `WebLLM` directly is a
different stack entirely, and no skill in this suite covers it; that stack
is out of scope here and the browser runtimes' own documentation is the
place to start.

## Install and wiring

```yaml
# pubspec.yaml
dependencies:
  llamadart: ^0.8.17
```

For Flutter iOS and macOS builds that should link Apple XCFrameworks through
Swift Package Manager, also add the runtime companion packages for whichever
backend the app needs:

```yaml
dependencies:
  llamadart: ^0.8.17
  llamadart_llama_cpp_flutter: ^0.0.11 # GGUF / llama.cpp
  llamadart_litert_lm_flutter: ^0.0.7  # .litertlm / LiteRT-LM on iOS
```

Then `flutter pub get`. On the first `flutter run` or `dart run` for a
native target, `llamadart`'s native-assets hook detects the platform and
architecture, resolves matching prebuilt runtime bundles from
`leehack/llamadart-native` and `leehack/litert-lm-native`, and wires them
into the app; no local C++ toolchain setup is needed on the development
machine. That first resolution step does reach the network, on the build
machine, to fetch the runtime bundle. This is separate from the app's own
runtime behaviour: once built, a shipped app loading a model from a local
file path does not need network access to generate, which is the property
this skill's minimal example is written to demonstrate.

Set the Apple deployment target before running an iOS or macOS build. In
Xcode, set `IPHONEOS_DEPLOYMENT_TARGET = 16.4` or
`MACOSX_DEPLOYMENT_TARGET = 14.0` on the relevant Runner configurations; if
the iOS project still uses CocoaPods, set the Podfile platform too:

```ruby
platform :ios, '16.4'
```

Companion packages, native-assets tag overrides, and per-platform backend
trimming (dropping the LiteRT-LM runtime entirely to save package size, for
example) are covered in `references/llamadart-path.md`.

## A minimal working example

This adapts `llamadart`'s own quickstart. The package's published quickstart
loads from a Hugging Face source (`ModelSource.parse('hf://...')`) so a
first-time reader can copy and paste it without inventing a local path first.
That is a fine way to smoke-test the package, but it is a network dependency
this skill's own happy path should not carry: a model already trained,
exported, and sitting on device should load from that local file directly,
with `ModelSource.path(...)`, the local-filesystem constructor confirmed in
`llamadart`'s own `ModelSource` source.

```dart
import 'package:llamadart/llamadart.dart';

Future<void> main() async {
  final engine = LlamaEngine(LlamaBackend());

  try {
    await engine.loadModelSource(
      ModelSource.path('/path/to/model.gguf'),
      modelParams: const ModelParams(contextSize: 4096, gpuLayers: 0),
    );

    final messages = [
      LlamaChatMessage.fromText(
        role: LlamaChatRole.user,
        text: 'What is the meaning of life?',
      ),
    ];

    // Streams one chunk per generated delta; each chunk's
    // choices.first.delta.content is the newly generated text fragment,
    // or null on chunks that carry no new text (for example a final
    // finish-reason chunk).
    await for (final chunk in engine.create(
      messages,
      params: const GenerationParams(maxTokens: 256, temp: 0.7),
    )) {
      final text = chunk.choices.first.delta.content;
      if (text != null) {
        // append text to whatever UI state renders the growing response
      }
    }
  } finally {
    await engine.dispose();
  }
}
```

`engine.create(...)` is the stateless, chat-template-aware entry point: the
model's chat template gets applied, but nothing is remembered between calls,
so a follow-up turn means appending both sides of the exchange to `messages`
by hand before calling it again. For a multi-turn chat UI, `ChatSession`
wraps the same engine and keeps history automatically; the trade-off between
the two, and `ChatSession`'s own example, are in
`references/llamadart-path.md`. A single, non-streaming full response is the
same call awaited to completion rather than iterated chunk by chunk, useful
for a backend job that only needs the final text: collect each chunk's
`choices.first.delta.content` into a buffer instead of rendering each piece
as it arrives.

`gpuLayers: 0` above forces CPU inference, the safest default for a first
run on an unknown device. Raise it, or set `ModelParams.maxGpuLayers`, once
GPU acceleration has been confirmed stable for the target hardware; see the
platform gotchas below.

## The delivery decision

Once the model runs, the next question is how it reaches the device: bundled
into the app, or downloaded on first run. Full detail, the app store size
ceilings for both platforms, and the general first-run download experience
are in `references/model-delivery-and-size-budgets.md`.

`llamadart` does not document a package-level bundling cap of its own, the
way `react-native-executorch`'s `require()` route caps at 512 MB. The app
store ceilings and install-conversion guidance in the shared reference are
what actually bound bundling here, not anything specific to this package.
Where `llamadart` does add something the other three targets in this suite
do not have out of the box: `loadModelSource(...)`, backed by
`DefaultModelDownloadManager` and the higher-level `ModelDownloadController`,
is a first-run-download implementation already built into the package,
rather than something the app has to assemble from scratch. It reports
download progress through the same `onProgress` callback shown above, resumes
a partial download when the server exposes a validator (`ETag` or
`Last-Modified`) or the caller supplies a SHA-256 checksum, and offers cache
policies (`preferCached`, `refresh`, `cacheOnly`, `noCache`) instead of an
app writing that state machine by hand. Of the three things the shared
reference says a first-run download needs to get right, real progress,
resumability, and something for the user to do while it waits, the first two
are largely handled once wired to `ModelDownloadController`; the third is
still a UI decision for the app to make. See `references/llamadart-path.md`
for the download manager's API and its platform-specific default cache
directories.

## Platform gotchas

- **The SDK floors bite before anything else does.** Dart `>= 3.10.7`,
  Flutter `>= 3.38.0`, iOS `16.4`+, macOS `14.0`+, restated from the top of
  this skill because it is the single most common way to lose time here: an
  older project fails to build against `llamadart` at all, not with a
  runtime error that points at the cause.
- **The first build reaches the network; the shipped app's inference does
  not.** The native-assets hook downloads the platform's runtime bundle from
  GitHub on a developer's or CI machine the first time a native target
  builds. That is a build-time dependency on the machine doing the building,
  separate from what a shipped app needs at runtime to load a local model
  file and generate. Conflating the two is an easy way to misdescribe this
  skill's own no-network claim; keep them distinct when explaining it to
  someone else.
- **Runtime LoRA is a GGUF-only feature.** `setLora(...)`, `removeLora(...)`,
  and adapter stacking are documented as supported on native llama.cpp/GGUF
  flows. Native LiteRT-LM accepts exactly one default-scale text LoRA
  adapter at model load, through `ModelParams.loras`, with no runtime
  updates, stacking, or custom scaling; LiteRT-LM web does not expose LoRA
  at all, and WebGPU and LiteRT-LM web both throw an explicit
  unsupported-operation error for the runtime LoRA API rather than silently
  no-opping. See `references/llamadart-path.md` for the full LoRA guide and
  code.
- **Heavy inference already runs off the UI thread, but streaming
  granularity is still a tuning knob.** Native backend operations run in
  background Dart Isolates by the package's own design, specifically so a
  generation call does not freeze the Flutter UI. `GenerationParams`
  exposes `streamBatchTokenThreshold` and `streamBatchByteThreshold`; lower
  values give more granular, more frequent UI updates per token, higher
  values reduce isolate message-passing overhead at the cost of chunkier
  streaming. Default values are reasonable; only change them after measuring
  a real device.
- **Vulkan cooperative-matrix crashes are a driver bug, not a package bug.**
  Some Vulkan drivers on Windows and Linux advertise cooperative matrix
  support and then crash inside `ggml-vulkan`'s property queries. The
  documented workaround is setting `GGML_VK_DISABLE_COOPMAT=1` and
  `GGML_VK_DISABLE_COOPMAT2=1` in the process environment before launching;
  it trades some Vulkan performance for stability and should only be reached
  for when a Vulkan backend is actually crashing or reporting device loss.
- **Package size is configurable, not fixed to "everything included."**
  Native builds bundle every available runtime family by default, so one
  build can load both GGUF and `.litertlm` models. An app that only ever
  ships one format can trim the other out with
  `llamadart_native_runtimes: [llama_cpp]` (or `[litert_lm]`) under
  `hooks.user_defines.llamadart` in `pubspec.yaml`, per platform if needed.
  This is a `llamadart`-specific knob; none of this suite's other three
  shipping targets expose an equivalent.
- **Quantisation floor for GGUF on mobile.** Below roughly 3B parameters,
  quantising past `Q4_K_M` is a known quality cliff, not a bug to debug
  later. Start at `Q4_K_M` and only go lower after confirming quality still
  holds.
- **Nothing here solves per-device model versioning.** If the fine-tune
  changes, a bundled asset and a server-hosted download source both need an
  explicit update path; `llamadart`'s cache metadata tracks whether a given
  cached file matches its source and checksum, not whether a newer model
  version exists that the app should switch to.

Full depth on `ModelSource` variants, the download manager and controller,
`ChatSession`, LoRA, native-assets configuration, and the web build's
differences (narrower LiteRT-LM feature surface, `crossOriginIsolated` and
`navigator.gpu` checks) is in `references/llamadart-path.md`.

## Hand off to

- The artifact shape is not yet confirmed, or it is a merged checkpoint or a
  PEFT adapter that needs converting first: **inspecting-a-model-bundle**
- The model loads but generates badly, repeats, never stops, or the adapter
  will not apply: **debugging-a-bad-fine-tune**
- It is not yet known whether the fine-tune is actually better than the base
  model it started from: **evaluating-a-tuned-model**
- The question is whether running this on device is actually cheaper than an
  API at the expected usage level: **costing-a-model-vs-an-api**
- The target platform is React Native or Expo, native iOS, or native Android
  instead of Flutter: **shipping-a-model-in-a-react-native-app**,
  **shipping-a-model-in-an-ios-app**, or
  **shipping-a-model-in-an-android-app**