Home / Learn / AI, explained simply

Why AI makes things up, and how to stop it

A chatbot confidently giving a wrong figure, with a datasheet check showing the correct value
Beginner

AI models sometimes state things that are simply untrue: a quote nobody said, a function that doesn't exist in a library, a datasheet figure that's off by a factor of ten. This is usually called hallucination. It happens because of how the models work, and while it can't be switched off completely, you can make it much less likely.

Why it happens

A language model is trained to predict likely next words, as our guide to how language models work explains. It doesn't look facts up in a database. It has learned patterns from its training text, and it generates the text that best fits those patterns. Most of the time, the most likely text is also true. But when the model's knowledge is thin, the most likely-sounding text and the true text come apart, and the model produces something that has the right shape: a plausible part number, a convincing reference, a sensible-looking pin assignment.

Training also plays a part. A 2025 paper by researchers at OpenAI and Georgia Tech, Why Language Models Hallucinate, argues that the way models are scored encourages guessing. On most tests a confident guess sometimes scores, while "I don't know" never does, so models learn that guessing pays. The authors suggest scoring that penalises confident errors more than admitting uncertainty.

When to be most careful

  • Precise details: exact numbers, part numbers, pinouts, dates, quotes, legal references and citations.
  • Obscure topics that appeared rarely in the training data, such as a niche sensor or a small company's product.
  • Recent events after the model's training cut-off, unless it can search the web.
  • Questions with a false premise, such as "Which pin on the Pico is the 5 V ADC input?" A model may play along instead of saying there isn't one.
  • Long, multi-step answers, where one early mistake carries through the rest.

How to reduce it when you're chatting

  • Give it the source. Paste the datasheet section or documentation and ask it to answer only from that. Models are far more reliable at reading than at remembering.
  • Allow "I don't know". Say something like "If the text doesn't say, tell me you don't know." It sounds too simple, but model makers recommend it because it works.
  • Ask for quotes. Ask the model to quote the exact sentence its answer rests on. Then check the quote really is in the source.
  • Turn on web search for recent or niche facts, and follow the links it gives.
  • Check anything that matters. For electronics, that means the datasheet before you wire anything up. A wrong voltage is cheap to check and expensive to get wrong.

How to reduce it in your own apps

  • Ground the model with retrieval. RAG finds the relevant passages from your documents and puts them in the prompt, so the model answers from them rather than from memory.
  • Ask for citations to specific passages, and check them in code. A quote that doesn't appear in the source is a red flag.
  • Use structured output with an explicit "unknown" option, so the model has an honest choice instead of having to fill in a field.
  • Use tools for facts and maths. Let the model call a calculator, a database or a search API through tool use instead of working things out in its head.
  • Measure it. Build a set of test questions with known answers, including ones with no answer in your documents, and track how often the app gets them wrong. Our guide to evaluating AI apps shows how.
Temperature isn't the fix. A low temperature makes answers consistent, but a model that's wrong at temperature 0 is consistently wrong. Better context helps far more.

The bottom line

Treat an AI model like a knowledgeable colleague who never wants to say "I'm not sure". It's excellent for explanations, drafts and ideas, and it gets much more reliable when you hand it the source material. For anything safety-related, such as mains wiring, battery charging or current ratings, the datasheet and the regulations are the authority, not the chatbot.

Sources

The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.