On-Device Models with MLC LLM

By Michael Samuel Naeem · December 16, 2025 · 5 min read

Last quarter I shipped a note-taking app with on-device summarization. The product requirement was simple: summarize a meeting transcript without sending it to our API. The engineering requirement was harder — run a 3B model on a three-year-old Android phone without draining the battery in ten minutes. We landed on MLC LLM after burning a week on a generic ONNX runtime that couldn't hold 30 tokens per second on anything but a Pixel 8.

MLC LLM (Machine Learning Compilation for LLM) takes a Hugging Face checkpoint, quantizes it, and compiles device-specific kernels through Apache TVM. The output is a mlc-chat bundle you embed in iOS, Android, or even a browser via WebGPU. The compile step is painful; the runtime is fast.

The compile pipeline

You don't drop a .safetensors file into your app. You run mlc_llm package (or the Python mlc_llm compile flow) against a chosen quantization scheme:

# Example: compile Llama-3.2-3B for iOS Metal
mlc_llm compile \
  HF://meta-llama/Llama-3.2-3B-Instruct \
  --quantization q4f16_1 \
  --device iphone \
  -o ./dist/llama-3.2-3b-iphone

The quantization flag matters more than the model name. q4f16_1 is the usual starting point — 4-bit weights with fp16 activations. Drop to q3f16 if you're memory-bound; quality degrades noticeably on reasoning tasks but stays fine for summarization.

Compile once per target device family. iOS Metal, Android Vulkan, and macOS are separate artifacts. CI should cache compiled bundles keyed by (model_id, quant, device) because recompiling Llama-3B takes 15–30 minutes.

Memory budgeting on mobile

On-device inference fails in predictable ways:

Size the KV cache explicitly. A 4096-token context on a 3B model at 4-bit can add 200–400 MB beyond weights. For a summarization feature, cap context at 2048 tokens and chunk long documents server-side or with a sliding window — don't pretend the phone is a datacenter.

// Android: initialize MLC engine with explicit limits
val engine = MLCEngine(
    modelPath = cacheDir.resolve("llama-3.2-3b-q4").absolutePath,
    maxSeqLen = 2048,
    prefillChunkSize = 512,  // reduces peak memory during prompt ingestion
)

Integration patterns that survive App Review

Download-on-first-use. Ship a 50 MB binary; fetch the 1.8 GB model after consent. Show storage requirements upfront. Pin SHA-256 of the weight file.

Background inference is a trap. iOS will suspend you mid-generation. Run inference only in foreground with a visible progress indicator, or use BGProcessingTask for non-interactive batch work with tight time limits.

Fallback to cloud. If MLCEngine.init() fails (low memory, corrupt cache), degrade gracefully to your API. Log the failure reason — low-end device clusters show up quickly in analytics.

Streaming UI. Token streaming keeps users patient during 3 tok/s generation. Buffer tokens and flush to the TextView every 50 ms to avoid layout thrash.

Performance tuning beyond defaults

Prefill (processing the prompt) and decode (generating tokens) have different bottlenecks. Long prompts bottleneck on compute; long outputs bottleneck on memory bandwidth.

• Reduce prefillChunkSize if you see OOM spikes when the user pastes a wall of text.
• Use speculative decoding only if you have a draft model small enough to fit alongside the main model — rare on phones.
• Warm up the engine on app launch with a one-token generation. First inference after cold start can be 3× slower.

On iPhone 15 Pro we saw ~25 tok/s prefill and ~18 tok/s decode on Llama-3.2-3B q4f16. On a Galaxy A54, the same model ran ~8 tok/s decode — still usable for a two-paragraph summary.

When MLC is the wrong choice

Skip MLC if you need frequent model updates (compile latency kills iteration), if your feature requires 70B-class reasoning, or if your team has no one willing to own the TVM compile toolchain. For a single fixed model on iOS and Android with latency requirements under 5 seconds for short outputs, it's the best open-source path I've used.

If you only target Apple and want less compile ceremony, Apple's MLX Swift bindings are worth evaluating. MLC wins when you need the same model artifact story on both platforms.

Common production mistakes

Teams get on device llm mlc mobile wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Production implementations of on device llm mlc mobile fail when staging mirrors production topology poorly, rollback is untested, and on-call runbooks describe the happy path only.

Debugging and triage workflow

When on device llm mlc mobile misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• MLC LLM GitHub repository
• MLC LLM iOS/Android deployment docs
• Apache TVM documentation
• Llama model cards on Hugging Face
• Apple Metal Performance Shaders overview