Home / Learn / Code for makers

How to run AI on a Raspberry Pi

New to this? Read from the top. Already know the basics? Jump to Going deeper for intermediate and advanced detail.

A Raspberry Pi running a small AI model locally, with a chat reply on screen
Beginner

Running AI locally on a Raspberry Pi means no monthly API bills, no internet connection needed and your data never leaves the device. It also means working within the limits of a small, low-power computer. Here's what's realistic.

Which Pi?

For AI, use a Raspberry Pi 5, the fastest model by a wide margin. Memory matters most: language models must fit in RAM, so choose the 8 GB or 16 GB version if you want to run them. Use the official power supply and an active cooler, because AI work keeps the processor busy and hot for long periods.

Note that a Raspberry Pi Pico is a microcontroller, not a computer, so it can only run tiny models. See Arduino vs Raspberry Pi for the difference.

Three kinds of AI you can run

1. Vision: recognising objects in camera images

Image classification and object detection run well on a Pi, especially with an accelerator. This is great for wildlife cameras, counting things, or a robot that reacts to what it sees.

2. Speech: listening and talking

Speech-to-text models such as the smaller versions of OpenAI's open-source Whisper, and lightweight text-to-speech voices, can run on a Pi 5. That's enough for an offline voice assistant that understands simple commands.

3. Small language models

A Pi 5 can run small open language models with around 1 to 3 billion parameters using tools like Ollama or llama.cpp. Expect replies at reading speed or slower, and noticeably simpler answers than large cloud models give. They're fine for short, focused jobs like rewording a message or answering from a small set of notes, not for complex reasoning.

Getting started with a language model

Ollama is the simplest route. On Raspberry Pi OS (64-bit), install it with the command from Ollama's official download page, then download and chat with a small model, choosing one from Ollama's model library:

ollama run <model-name>

Ollama also runs a local web API, so your Python programs can use the model too. Start with the smallest version of a model and only try bigger ones if your Pi has the memory to spare.

AI accelerators: the AI HAT+ boards

Raspberry Pi sells add-on boards that do AI maths much faster than the Pi's own processor, using Hailo neural network chips:

BoardChipSpeedBest for
AI HAT+ (13 TOPS)Hailo-8L13 TOPSVision models
AI HAT+ (26 TOPS)Hailo-826 TOPSFaster or multiple vision models
AI HAT+ 2Hailo-10H with 8 GB of its own memory40 TOPSVision plus small generative models, including language models

TOPS means trillions of operations per second, a rough measure of AI speed. The accelerators only run models converted for their chips, so stick to supported models and Raspberry Pi's own examples to begin with.

Local or cloud?

  • Run it on the Pi for privacy, offline use, no running costs, and quick reactions to camera or sensor data.
  • Use a cloud API when you need the quality of a large model, long documents or complex reasoning. The Pi can still be the "body", sending requests and acting on the replies. See how to call an AI API from Python.

Many good projects mix both: a local model handles a wake word or spots motion, and a cloud model is called only when something needs real thought, which keeps API costs low.

Intermediate and advanced

Going deeper

Will the model fit? Quantisation

A model's weights must fit in memory. At full 16-bit precision, each parameter takes 2 bytes, so a 3-billion-parameter model needs about 6 GB for the weights alone. Quantisation stores weights at lower precision, commonly about 4 bits, cutting that to roughly 1.5 GB with a small loss of quality. Leave room for the operating system and the context (KV cache) as well.

def weights_gb(params_billion, bits_per_weight):
    return params_billion * 1e9 * bits_per_weight / 8 / 1e9

for params in (1, 3, 8):
    print(f"{params}B params: 16-bit {weights_gb(params, 16):.1f} GB, 4-bit {weights_gb(params, 4):.1f} GB")
1B params: 16-bit 2.0 GB, 4-bit 0.5 GB
3B params: 16-bit 6.0 GB, 4-bit 1.5 GB
8B params: 16-bit 16.0 GB, 4-bit 4.0 GB

Real quantised files are a little larger than this, because some parts stay at higher precision.

Why speed depends on memory bandwidth

Generating each token requires reading essentially all the model's weights from memory. So a rough upper limit on speed is memory bandwidth divided by model size. The Raspberry Pi 5's 32-bit LPDDR4X-4267 memory has a theoretical peak of about 17 GB/s, so a 1.5 GB model can't exceed roughly 11 tokens per second, and real speeds are lower. Smaller models are faster, not just smaller.

Using a local model from Python

Ollama serves a local web API on port 11434. This sends a prompt and prints the reply:

import json, urllib.request

payload = {"model": "your-model-name", "prompt": "Name three uses for a capacitor.", "stream": False}
req = urllib.request.Request(
    "http://localhost:11434/api/generate",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(req) as resp:
    print(json.loads(resp.read())["response"])

Replace your-model-name with a model you've downloaded. Ollama also offers an OpenAI-compatible endpoint, so many existing libraries work with it.

Faster vision with the AI HAT+

The Hailo accelerators run models compiled into Hailo's own format. Raspberry Pi's camera software integrates with them, and Raspberry Pi and Hailo provide example pipelines for detection, pose estimation and segmentation. Start from those, then retrain a supported model on your own images if you need to detect something specific.

Keeping it cool and stable

Sustained AI work keeps the CPU at full load. Without cooling, a Pi 5 throttles its clock speed to stay safe, which slows inference. Use the active cooler, check temperatures with vcgencmd measure_temp, and use a good power supply, as undervoltage also causes throttling.

Sources

The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.