Running AI locally on a Raspberry Pi means no monthly API bills, no internet connection needed and your data never leaves the device. It also means working within the limits of a small, low-power computer. Here's what's realistic.
Which Pi?
For AI, use a Raspberry Pi 5, the fastest model by a wide margin. Memory matters most: language models must fit in RAM, so choose the 8 GB or 16 GB version if you want to run them. Use the official power supply and an active cooler, because AI work keeps the processor busy and hot for long periods.
Note that a Raspberry Pi Pico is a microcontroller, not a computer, so it can only run tiny models. See Arduino vs Raspberry Pi for the difference.
Three kinds of AI you can run
1. Vision: recognising objects in camera images
Image classification and object detection run well on a Pi, especially with an accelerator. This is great for wildlife cameras, counting things, or a robot that reacts to what it sees.
2. Speech: listening and talking
Speech-to-text models such as the smaller versions of OpenAI's open-source Whisper, and lightweight text-to-speech voices, can run on a Pi 5. That's enough for an offline voice assistant that understands simple commands.
3. Small language models
A Pi 5 can run small open language models with around 1 to 3 billion parameters using tools like Ollama or llama.cpp. Expect replies at reading speed or slower, and noticeably simpler answers than large cloud models give. They're fine for short, focused jobs like rewording a message or answering from a small set of notes, not for complex reasoning.
Getting started with a language model
Ollama is the simplest route. On Raspberry Pi OS (64-bit), install it with the command from Ollama's official download page, then download and chat with a small model, choosing one from Ollama's model library:
ollama run <model-name>
Ollama also runs a local web API, so your Python programs can use the model too. Start with the smallest version of a model and only try bigger ones if your Pi has the memory to spare.
AI accelerators: the AI HAT+ boards
Raspberry Pi sells add-on boards that do AI maths much faster than the Pi's own processor, using Hailo neural network chips:
| Board | Chip | Speed | Best for |
|---|---|---|---|
| AI HAT+ (13 TOPS) | Hailo-8L | 13 TOPS | Vision models |
| AI HAT+ (26 TOPS) | Hailo-8 | 26 TOPS | Faster or multiple vision models |
| AI HAT+ 2 | Hailo-10H with 8 GB of its own memory | 40 TOPS | Vision plus small generative models, including language models |
TOPS means trillions of operations per second, a rough measure of AI speed. The accelerators only run models converted for their chips, so stick to supported models and Raspberry Pi's own examples to begin with.
Local or cloud?
- Run it on the Pi for privacy, offline use, no running costs, and quick reactions to camera or sensor data.
- Use a cloud API when you need the quality of a large model, long documents or complex reasoning. The Pi can still be the "body", sending requests and acting on the replies. See how to call an AI API from Python.
Many good projects mix both: a local model handles a wake word or spots motion, and a cloud model is called only when something needs real thought, which keeps API costs low.
Going deeper
Will the model fit? Quantisation
A model's weights must fit in memory. At full 16-bit precision, each parameter takes 2 bytes, so a 3-billion-parameter model needs about 6 GB for the weights alone. Quantisation stores weights at lower precision, commonly about 4 bits, cutting that to roughly 1.5 GB with a small loss of quality. Leave room for the operating system and the context (KV cache) as well.
def weights_gb(params_billion, bits_per_weight):
return params_billion * 1e9 * bits_per_weight / 8 / 1e9
for params in (1, 3, 8):
print(f"{params}B params: 16-bit {weights_gb(params, 16):.1f} GB, 4-bit {weights_gb(params, 4):.1f} GB")
1B params: 16-bit 2.0 GB, 4-bit 0.5 GB
3B params: 16-bit 6.0 GB, 4-bit 1.5 GB
8B params: 16-bit 16.0 GB, 4-bit 4.0 GB
Real quantised files are a little larger than this, because some parts stay at higher precision.
Why speed depends on memory bandwidth
Generating each token requires reading essentially all the model's weights from memory. So a rough upper limit on speed is memory bandwidth divided by model size. The Raspberry Pi 5's 32-bit LPDDR4X-4267 memory has a theoretical peak of about 17 GB/s, so a 1.5 GB model can't exceed roughly 11 tokens per second, and real speeds are lower. Smaller models are faster, not just smaller.
Using a local model from Python
Ollama serves a local web API on port 11434. This sends a prompt and prints the reply:
import json, urllib.request
payload = {"model": "your-model-name", "prompt": "Name three uses for a capacitor.", "stream": False}
req = urllib.request.Request(
"http://localhost:11434/api/generate",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(req) as resp:
print(json.loads(resp.read())["response"])
Replace your-model-name with a model you've downloaded. Ollama also offers an OpenAI-compatible endpoint, so many existing libraries work with it.
Faster vision with the AI HAT+
The Hailo accelerators run models compiled into Hailo's own format. Raspberry Pi's camera software integrates with them, and Raspberry Pi and Hailo provide example pipelines for detection, pose estimation and segmentation. Start from those, then retrain a supported model on your own images if you need to detect something specific.
Keeping it cool and stable
Sustained AI work keeps the CPU at full load. Without cooling, a Pi 5 throttles its clock speed to stay safe, which slows inference. Use the active cooler, check temperatures with vcgencmd measure_temp, and use a good power supply, as undervoltage also causes throttling.
Sources
The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.