Running Local LLMs On-Device: llama.cpp, Ollama, Quantization

By Michael Samuel Naeem · January 15, 2026 · 5 min read

Running an LLM on your own hardware is no longer exotic. A capable 7–8B model, quantized to 4-bit, runs on a laptop with 16GB of RAM and answers in real time. The two tools that made this normal are llama.cpp, the C/C++ inference engine, and Ollama, which wraps it in a friendly CLI and model registry. If you want private, offline, zero-marginal-cost inference, this is the stack.

The catch is that "which model fits" is a memory-math question, and getting quantization wrong means either an out-of-memory crash or needlessly degraded output. I'll cover the fast path with Ollama, the control path with llama.cpp, how to read GGUF quantization labels, and how to size a model to your machine so it actually runs.

The fast path: Ollama

Ollama is the shortest distance to a working local model. Install it, pull a model, and you have an interactive session and a local HTTP API:

ollama run llama3.1:8b
# or just start the server and hit the API
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain quantization in one paragraph."
}'

Ollama pulls a sensibly quantized GGUF by default (typically a 4-bit variant), manages the model cache, and exposes an OpenAI-compatible endpoint at /v1 so existing client code often works with a base-URL swap. For most people building a local prototype or a privacy-sensitive feature, this is all you need. It's also the easiest way to test several models — ollama pull a few and compare.

The control path: llama.cpp

When you need more — a specific quantization, custom build flags for your GPU, or embedding inference directly into an app — go to llama.cpp. You build it with the acceleration backend for your hardware:

# Apple Silicon (Metal is on by default)
make

# NVIDIA (CUDA)
cmake -B build -DGGML_CUDA=ON && cmake --build build

# run a GGUF model
./llama-cli -m models/llama-3.1-8b-Q4_K_M.gguf -p "Hello" -n 128

llama.cpp is what actually runs the model under Ollama, LM Studio, and most local tools. Going direct gives you the llama-server for an OpenAI-compatible API, control over context length and GPU layer offload (-ngl), and the ability to quantize models yourself. It's the right layer when local inference is a product feature, not a convenience.

Reading GGUF quantization labels

GGUF filenames carry the quantization scheme, and decoding them saves a lot of trial and error. A label like Q4_K_M means 4-bit, K-quant, medium. Here's the practical guide:

Q4_K_M is the default for a reason — it's where memory savings stop being nearly free. Below 4-bit, quality degrades fast; above it, you pay a lot of RAM for small gains. Start at Q4_K_M, move up to Q5 or Q6 if you have headroom and notice quality issues, and only drop to Q3 if you must fit a bigger model into limited memory.

Sizing a model to your machine

The memory math is straightforward: you need enough RAM (or VRAM on GPU) for the quantized weights plus the context (KV cache), which grows with context length. A rough rule:

required_memory ≈ model_file_size + kv_cache
kv_cache grows with context length and model size

Practical guidance by memory:

• 8 GB — 3–4B models at Q4, or a 7B at Q3 with short context. Usable, tight.
• 16 GB — 7–8B at Q4_K_M comfortably; the mainstream local target.
• 32 GB — 13–14B at Q4/Q5, or 8B at Q8 with long context.
• 64 GB+ — 30–70B class models at Q4, slowly on CPU, faster with a GPU.

On Apple Silicon, unified memory means the GPU shares system RAM, so a 32GB Mac punches above its weight for local inference. On PCs, VRAM is the binding constraint — offload as many layers to the GPU as fit (-ngl) and let the rest run on CPU.

Where local inference makes sense

Local LLMs win on three axes: privacy (data never leaves the device), cost (no per-token bill), and offline operation. They lose on raw capability — a local 8B model is not GPT-class — and on throughput under concurrent load. So the fit is: on-device assistants, private document analysis, offline features, and cost-sensitive high-volume tasks where a smaller model is good enough.

This is exactly the trend reshaping mobile too, where small language models on mobile and platform runtimes like Gemini Nano on Android bring the same idea to phones. For privacy-first products, running inference on the user's hardware isn't just a cost optimization — it's an architecture for privacy. Start with Ollama to prove the model is good enough, then drop to llama.cpp when you're ready to ship it into something real.

Resources

• llama.cpp — GitHub
• Ollama — official site
• Hugging Face — GGUF models and format docs
• LM Studio — desktop local LLM app
• llama.cpp quantization discussion
• Hugging Face — open LLM leaderboard