Home / Learn / AI, explained simply

What are embeddings? AI's way of turning meaning into numbers

New to this? Read from the top. Already know the basics? Jump to Going deeper for intermediate and advanced detail.

Words placed as points in a meaning space: similar words such as resistor and capacitor sit close together, far from unrelated words
Beginner

Search engines that understand what you mean, recommendations that seem to read your mind, and chatbots that answer from your own documents all rely on the same idea: embeddings. Here's what they are and how they work.

Meaning as a position on a map

An embedding is a list of numbers that represents a piece of text, such as a word, sentence or whole document. The numbers are chosen so that texts with similar meanings get similar numbers.

Picture a map where every sentence has a location. "How do I reset my password?" and "I forgot my login details" land close together, even though they share almost no words. "Best pizza in Leeds" lands far away.

Reset my password Forgot my login Can't sign in Best pizza in Leeds Italian restaurants LED keeps burning out Which resistor for an LED?
A simplified 2D picture. Real embeddings have hundreds or thousands of dimensions, but the idea is the same: similar meanings sit close together.

A real embedding isn't 2 numbers, it's usually hundreds or thousands. That gives the model room to capture many shades of meaning at once, such as topic, tone and language.

Where the numbers come from

Embeddings are produced by an embedding model, a neural network trained so that related texts end up near each other and unrelated texts end up far apart. You send it text, and it returns the list of numbers. Many AI providers offer embedding models through their APIs, and there are good free ones you can run yourself.

Measuring similarity

To compare two embeddings, the usual measure is cosine similarity. It looks at the angle between two lists of numbers, treated as arrows. Arrows pointing the same way score close to 1. Unrelated ones score near 0.

Here it is in plain Python, using tiny made-up 4-number embeddings so you can see the idea:

import math

def cosine_similarity(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    size_a = math.sqrt(sum(x * x for x in a))
    size_b = math.sqrt(sum(y * y for y in b))
    return dot / (size_a * size_b)

# Made-up embeddings, just for illustration
reset_password = [0.9, 0.1, 0.0, 0.2]
forgot_login   = [0.8, 0.2, 0.1, 0.3]
pizza_in_leeds = [0.0, 0.9, 0.8, 0.1]

print(round(cosine_similarity(reset_password, forgot_login), 2))    # 0.98
print(round(cosine_similarity(reset_password, pizza_in_leeds), 2))  # 0.1

The two login questions score 0.98, very similar. The pizza search scores just 0.1, barely related. Real embedding models produce numbers like these automatically, just with many more of them.

What embeddings are used for

  • Semantic search: find documents by meaning, not just matching words. Searching "laptop won't turn on" can find an article titled "Troubleshooting power problems".
  • Chatbots that use your documents: often called RAG (retrieval augmented generation). Your documents are split into chunks and embedded. When someone asks a question, the closest chunks are found and given to the language model, so it answers from your information instead of guessing.
  • Recommendations: suggest articles or products close to what someone already liked.
  • Grouping and tidying: cluster customer feedback by theme, or spot duplicate support tickets.

Embeddings and language models

Embeddings are also built into how language models work. The first thing a model does with each token is look up its embedding, a list of numbers describing that token. Everything after that, including the attention layers, works on these numbers rather than on the text itself.

Quick recap

  • An embedding is a list of numbers that captures the meaning of some text.
  • Similar meanings produce similar numbers, so they sit close together.
  • Cosine similarity measures how close two embeddings are.
  • They power semantic search, document chatbots, recommendations and clustering.
Intermediate and advanced

Going deeper

Dimensions and normalisation

Real embeddings have hundreds to a few thousand dimensions. Many embedding models return vectors normalised to length 1. When that's true, cosine similarity equals the simple dot product, which is faster to compute. Here's a search over a small set of vectors using NumPy:

import numpy as np

docs = np.array([[0.9, 0.1, 0.0],    # "reset my password"
                 [0.8, 0.3, 0.1],    # "forgot login details"
                 [0.0, 0.2, 0.9]])   # "pizza near me"
query = np.array([0.85, 0.2, 0.05])  # "can't sign in"

def normalise(v):
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

scores = normalise(docs) @ normalise(query)   # cosine similarity for every doc at once
best = np.argsort(scores)[::-1]
print(np.round(scores, 3), "best match:", best[0])
[0.991 0.99  0.105] best match: 0

Both login-related documents score close to 1, so they're near neighbours of the query, while the pizza document scores far lower. Index 0 ("reset my password") is the top match by a whisker.

Searching millions of vectors

Comparing a query to every stored vector is exact but slow at scale. Vector databases use approximate nearest neighbour indexes, such as HNSW (a layered graph of neighbours), that find almost all of the best matches far faster. You trade a little accuracy for speed, usually controlled by settings in the index.

Practical rules

  • Use the same model for documents and queries. Vectors from different embedding models aren't comparable.
  • Re-embed everything if you change models.
  • Chunk documents before embedding. One vector for a whole book blurs its meaning.
  • Some models expect a different prefix or setting for queries and documents. Check the model's documentation.
  • Similarity scores aren't calibrated probabilities. A "good" score for one model may be poor for another, so set thresholds by testing on your own data.

Beyond text

Images, audio and code can be embedded too. Multimodal models place text and images in the same space, so a text query like "red bicycle" can find matching photos. The same idea powers recommendations, duplicate detection and clustering.

Sources

The main references behind this guide. Numbers we worked out ourselves, such as the code outputs, come from running the code.