Glossary and FAQ

Someone said “we’ll just do ANN over the chunk embeddings” and you nodded. Look it up here. No judgement — everybody nodded at some point.

Glossary

ANN (Approximate Nearest Neighbour) Finding almost the closest vectors instead of definitely the closest, in exchange for enormous speed gains. What every vector database does. You occasionally miss the true best match; you get an answer in milliseconds rather than seconds.

Attention The mechanism that lets each token in your input look at all the others before its vector is finalised. It’s why “river bank” and “savings bank” produce different vectors for the same word. From Attention Is All You Need.

Chunk A piece of a larger document, split out so it can be embedded on its own. Typically a paragraph or section, 200–800 tokens. See chunking.

Cosine similarity A number from -1 to 1 saying how alike two vectors are, based on the angle between them rather than the distance. The standard measure for text. Full explanation.

Cross-encoder A model that reads a query and a document together and scores the pair. Much more accurate than comparing two separate embeddings, and much slower — so it’s used to rerank a shortlist, never to search the whole set.

Dimension One of the numbers in an embedding. text-embedding-3-small produces 1,536 of them, so it’s a 1,536-dimensional vector. Individual dimensions don’t correspond to anything you can name.

Dot product Multiply two vectors element by element and sum the results. When both vectors have length 1, this equals cosine similarity — which is why everyone normalises.

Embedding A list of numbers representing a piece of text (or an image, or audio), where similar meanings produce similar numbers. The thing this whole site is about.

Embedding model The model that turns text into embeddings. Different from a chat model — it produces vectors, not words, and it’s deterministic.

Euclidean distance (L2) Straight-line distance between two points. An alternative to cosine similarity. Smaller means more similar, which is the opposite direction to cosine, and a common source of inverted logic bugs.

HNSW (Hierarchical Navigable Small World) The graph algorithm most vector databases use for ANN search. Multiple layers: enter at a sparse one, hop roughly towards your target, descend into denser layers to refine. Paper.

Hybrid search Running keyword search and vector search together and merging the results. Usually better than either alone, because they fail in different places.

Normalising Scaling a vector so its length is exactly 1, by dividing every element by the original length. Makes cosine similarity a plain dot product.

Pooling Combining the per-token vectors into a single vector for the whole input. Usually a mean average. It’s lossy, which is why long inputs embed poorly.

RAG (Retrieval-Augmented Generation) Before asking an LLM a question, search your own documents for relevant chunks and paste them into the prompt. The “search” step is ordinary embedding search. There is nothing more to the acronym than that.

Reranking Taking the top ~20 results from a fast search and reordering them with a slower, more accurate model. High-value, low-effort quality improvement.

Semantic search Search that matches meaning rather than words. What you build with embeddings.

Token The unit models actually read — somewhere between a character and a word. Roughly 0.75 words each for English. What you’re billed for.

Vector An ordered list of numbers. In this context, interchangeable with “embedding”.

Vector database A database built to store vectors and find nearest neighbours quickly. pgvector, Qdrant, Chroma, Pinecone and friends.

---

FAQ

Do I need a vector database?

Probably not yet. Under about 10,000 vectors, a NumPy array or a plain list in memory is genuinely fine — milliseconds per query. Between 10,000 and a million, pgvector in the Postgres you already run is almost always the right call. Reach for a dedicated vector database above a million vectors, or when you need heavy metadata filtering at high concurrency. Databases that support vectors lists the options.

Adding a database is a real cost: another thing to back up, monitor, secure and hand over. Don’t pay it before you need to.

Why is my search returning rubbish?

Work down this list in order. The first two catch most cases:

1. Are your chunks any good? Print twenty and read them. If they’re cut mid-sentence, or too long, or missing their headings, that’s your answer. Chunking is where retrieval quality is won and lost.
2. Same model for documents and queries? Mixing models produces plausible-looking nonsense with no error message.
3. Are you re-embedding on every request? Works fine in testing, falls over in production.
4. Is your query a different shape from your documents? A three-word query against page-long chunks matches poorly. Consider HyDE — have an LLM write a hypothetical answer, then embed that instead.
5. Do you need keyword search too? If people search by product code or error ID, embeddings alone will never be good enough. Go hybrid — see alternatives to vector embeddings.

How much does this cost?

Embeddings are the cheapest thing in the AI stack by a wide margin — typically a small number of cents per million tokens, versus dollars for chat models. Embedding a million-word corpus costs somewhere around the price of a coffee.

Check OpenAI’s pricing page for current numbers rather than trusting any figure written in a guide, including this one. The bigger cost at scale is usually storage, not the API calls.

Can I use embeddings for images or audio?

Yes. The idea generalises completely. CLIP puts images and text in the same space, so you can search photos with a text query. There are audio and video equivalents. Everything on this site about comparing vectors applies unchanged — only the model differs.

What’s the difference between an embedding model and a chat model?

An embedding model takes text and returns numbers. A chat model takes text and returns text. Embedding models are deterministic — same input, same output, every time — and vastly cheaper. They’re different tools, and RAG uses both: embeddings to find the right documents, a chat model to answer using them.

Do I have to re-embed everything when I change models?

Yes, all of it. Vectors from different models live in unrelated coordinate spaces, and mixing them silently degrades results with no error to alert you. Plan model upgrades as a full re-index — which is a good reason to keep the original text alongside every vector.

Is my data being used to train models?

Depends entirely on the provider and your plan, and it’s worth reading rather than assuming. As of writing, OpenAI states that API data isn’t used for training by default (their policy) — but policies change, and enterprise agreements differ.

If the answer needs to be a certain no, run a model locally. sentence-transformers in Python or Transformers.js in JavaScript never send anything anywhere. Quality is a step below the frontier APIs and often perfectly sufficient.

Can I reverse an embedding back into text?

Not straightforwardly — there’s no decoder, and pooling threw information away. But treating them as anonymised is a mistake: research has shown that substantial portions of the original text can be reconstructed from embeddings alone. Store and protect them as you would the source text, because for privacy purposes that’s closer to what they are.

Why do all my scores sit between 0.1 and 0.4?

Normal for some models, OpenAI’s included. Their vectors aren’t spread evenly across all directions, so scores bunch into a narrow band. It doesn’t mean your search is broken.

This is exactly why you should rank rather than threshold. If you must have a cut-off, derive it from a sample of your own real queries.

Which model should I start with?

text-embedding-3-small if you’re using an API — it’s cheap, fast, and good enough for the overwhelming majority of applications. all-MiniLM-L6-v2 if you want something free and local.

Start there. Upgrade when you have a measurement telling you the model is your bottleneck, which for most projects never happens. The MTEB leaderboard is the reference if you want to compare properly, though benchmark scores don’t always transfer to your own documents.

What are Matryoshka embeddings?

Models trained so the most important information sits at the front of the vector, letting you truncate to a shorter length without much loss. OpenAI’s -3 models support this through a dimensions parameter — you can take text-embedding-3-large down from 3,072 to 256 dimensions and still get strong results at a twelfth of the storage. The paper.

Still stuck?

The single most useful debugging move is to print your chunks and your top ten results with their scores, then read them as a human. Nearly every “the embeddings are bad” report turns out to be “the chunks were bad” or “the query and the documents look nothing alike”, and both are obvious the moment you actually look at the text.

Next

• Choosing an embedding model — which model to actually use
• Databases that support vectors — where to put the vectors
• Evaluating retrieval — how to tell whether any of it is working

References

• OpenAI — Embeddings guide · API reference · Pricing · API data usage policy
• MTEB leaderboard — comparing embedding models
• HNSW paper · Matryoshka Representation Learning · HyDE
• Text Embeddings Reveal (Almost) As Much As Text — embedding inversion, and why you should treat vectors as sensitive
• CLIP — shared image and text embedding space
• sentence-transformers · Transformers.js — running models locally