Home / Learn / AI, explained simply

How to test and evaluate AI apps

New to this? Read from the top. Already know the basics? Jump to Going deeper for intermediate and advanced detail.

An evaluation table of test questions with pass and fail marks and an overall score
Beginner

Normal software either works or it doesn't. AI features are fuzzier: the same input can give different outputs, and a change that fixes one case can quietly break another. The answer is an evaluation, or "eval": a set of test cases you run every time you change a prompt, model or setting.

Step 1: collect test cases

Gather 20 to 100 realistic inputs. Include:

  • Typical cases that make up most of your traffic.
  • Edge cases: empty input, very long input, other languages, typos.
  • Known failures from real use.
  • Inputs the app should refuse or handle carefully.

Step 2: decide how to score

  • Exact checks when there's a single right answer: a category label, a number, a valid JSON field.
  • Rule checks for properties: under 100 words, mentions the order number, no prices quoted.
  • Model-graded checks for quality: another model scores the answer against a written rubric.
  • Human review for a sample, to make sure the automated scores match what people think.

Step 3: run it automatically

Here's a minimal eval harness. The classify function stands in for your AI call:

tests = [
    {"input": "My parcel never arrived",        "expected": "delivery"},
    {"input": "I was charged twice",            "expected": "billing"},
    {"input": "The LED kit is missing a part",  "expected": "product"},
    {"input": "Where is my order?",             "expected": "delivery"},
]

def classify(text):
    # Replace with a real model call. This stand-in uses keywords.
    t = text.lower()
    if "charged" in t or "refund" in t:
        return "billing"
    if "parcel" in t or "arrived" in t:
        return "delivery"
    return "product"

results = [(t["input"], t["expected"], classify(t["input"])) for t in tests]
passed = sum(e == g for _, e, g in results)
print(f"{passed}/{len(tests)} passed")
for text, expected, got in results:
    if expected != got:
        print(f"FAIL: {text!r} expected {expected}, got {got}")
3/4 passed
FAIL: 'Where is my order?' expected delivery, got product

The failure shows exactly which kind of input needs work. Fix it, re-run everything, and make sure nothing else broke.

Step 4: keep it running

Run the eval before every change goes live, and add every new real-world failure as a test case. Over time the eval becomes your best description of what "working" means.

Intermediate and advanced

Going deeper

Model-graded evaluation

For open-ended outputs, a capable model can grade answers against a rubric. Make the rubric specific ("Does the reply state the refund period? Does it avoid promising anything not in the policy?") and ask for a short reason before the score, which makes grades more consistent and easier to audit. Grade one criterion at a time if possible.

Graders have biases. In pairwise comparisons, models often favour whichever answer appears first, and many prefer longer answers. Swap the order and average both runs, and check the grader against a set of human-labelled examples before trusting it.

Dealing with randomness

Outputs vary between runs, so a single run can mislead, especially on small test sets. Run each case several times and report the pass rate. Treat small differences between two prompts with caution: on 50 cases, one prompt scoring 84% and another 80% may simply be noise. Larger test sets and repeated runs give clearer answers.

Measure what you'll pay for

Record tokens, cost and latency alongside quality for every run. A change that adds two points of accuracy but doubles the cost might not be worth it, and only an eval that tracks both makes that trade-off visible.

Evaluating retrieval and agents

  • RAG: evaluate retrieval (is the right document in the top results?) separately from answer quality. See what is RAG.
  • Agents: score whole tasks. Did it finish, was the final state correct, how many steps and tokens did it use, and did it do anything it shouldn't have?

Avoid overfitting your eval

If you tune a prompt against the same test cases over and over, it can become good at those cases and no better in real use. Keep a separate held-out set that you check only occasionally, and keep adding fresh real-world examples.

Sources

The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.