Chunking and vector databases

The eight-note demo works because eight is a small number. Two things break as you scale, and they break for unrelated reasons. This page covers both.

Problem one: your documents are too long

You want to search a 40-page employee handbook. The naive approach is to embed the whole thing and store one vector.

Two separate reasons that fails:

The hard reason. text-embedding-3-small caps input at 8,191 tokens, roughly 6,000 words. A 40-page handbook is well past that. You get an error, not a truncation.

The interesting reason. Even if it fit, the result would be useless. As covered in how text becomes numbers, the model averages everything down to one vector. Average a document covering holiday policy, fire safety, expenses and the dress code, and you land in a bland spot near the middle of all of them and close to none of them.

And even if the maths were kind, the result is bad product. Someone searches “how many holiday days do I get” and you hand back a 40-page PDF. Technically the right document. Practically useless.

The fix is chunking: split documents into pieces, embed each piece, store each piece as its own row.

How to chunk

The blunt approach is a fixed window of, say, 500 tokens with a 50-token overlap. The overlap exists so a sentence that straddles a boundary still appears whole in one of the two chunks.

It works. It’s also a bit thoughtless, because it cuts wherever it happens to land — mid-sentence, mid-table, mid-thought.

Better: split on structure your document already has. Markdown headings, HTML sections, paragraph breaks, slide boundaries. Documents come pre-divided by their authors into meaningful units; use them.

❌ Fixed 500-token windows
   ...an annual entitlement of 25 days. Section 4 ‖ .2 Sick leave.
   Employees should notify their line...
                                        ↑ cut lands mid-heading

✅ Split on headings
   ┌──────────────────────────────────┐
   │ ## 4.1 Annual leave              │
   │ Full-time employees receive 25…  │
   └──────────────────────────────────┘
   ┌──────────────────────────────────┐
   │ ## 4.2 Sick leave                │
   │ Employees should notify…         │
   └──────────────────────────────────┘

Rules of thumb worth internalising:

• A chunk should answer a question on its own. That’s the test. If a reader needs the previous chunk to make sense of this one, your chunk is too small or cut in the wrong place.
• Aim for 200–800 tokens. A paragraph or a short section. Not a sentence, not a chapter.
• Keep the heading with the body. Prepending ## 4.1 Annual leave to the chunk text gives the embedding valuable context for a few tokens’ cost. Cheap and effective.
• Store metadata alongside. Source document, page, section, URL. You need it to build a link back, and you’ll want to filter on it later.

Don't write your own splitter first

Every major framework ships tested chunkers that already handle Markdown, HTML, PDFs and code. LangChain’s text splitters and LlamaIndex’s node parsers are the usual starting points, and you can use just the splitter without adopting the whole framework.

Write your own once you know exactly why theirs doesn’t fit your documents. That’s a real situation — it’s just not where you start.

Chunking is where retrieval quality is won or lost

Teams spend weeks comparing embedding models and ten minutes on chunking. This is backwards. Bad chunks put a ceiling on quality that no model can lift — if the answer is split across two chunks, no amount of embedding sophistication will retrieve it whole.

If your search is disappointing, look at your chunks before you look at your model. Print twenty of them and read them. It’s unglamorous and it’s usually where the problem is.

Problem two: comparing against everything gets slow

Our demo loops over all eight notes on every query. That’s a brute-force search, and its cost grows linearly:

        8 notes  ×  1,536 dimensions  =        12,288 multiplications
   80,000 chunks ×  1,536 dimensions  =   122,880,000 multiplications
  10,000,000     ×  1,536 dimensions  = 15,360,000,000  per query

The middle row is survivable — NumPy will do it in tens of milliseconds. The bottom row is not, especially if you have concurrent users.

There’s a memory problem too. Ten million vectors at 1,536 dimensions, four bytes per float, is about 61 GB. That’s not fitting in your web server’s heap.

Where the crossover actually is

Don’t reach for infrastructure early. Rough guidance:

• Under ~10,000 vectors — a NumPy array or a plain list is genuinely fine. Milliseconds.
• 10,000 to ~1 million — pgvector in the Postgres you already run. Almost certainly the right answer.
• Over ~1 million, or you need heavy filtering and high concurrency — a dedicated vector database starts to earn its operational cost.

Plenty of production systems live comfortably in the first two rows. Adding a new database to your stack is a real cost — one more thing to back up, monitor, secure and explain to the next person.

What a vector database gives you

Approximate nearest neighbour search (ANN). The key idea, and the reason these things are fast. Instead of comparing against every vector, the index organises them so a query only visits a small fraction of the set. The dominant algorithm is HNSW, which builds a navigable multi-layer graph — you enter at a sparse top layer, hop roughly towards the target, then descend into denser layers to refine.

The word approximate is the trade. You might miss the true best match occasionally in exchange for being orders of magnitude faster. Every implementation exposes knobs for that trade-off (ef_search, nprobe, and similar). For search and recommendations the trade is almost always worth it. For anything where a miss is unacceptable, keep brute force.

Metadata filtering. “Closest chunks, but only from documents this user may read, only from 2024, only in English.” Doing this correctly alongside an ANN index is harder than it sounds and is a large part of what you’re paying for.

Persistence and operations. Vectors survive restarts. You can add and delete without rebuilding the index. Backups exist.

Picking one

Option	Reach for it when
pgvector	You already run Postgres. Vectors become a column, joins work, your existing backups cover it. The default recommendation for most teams.
Chroma	Prototyping. pip install, no server, in-process. Great for getting something working today.
Qdrant	You’ve outgrown pgvector and want strong filtering with predictable performance. Open source, self-hostable.
FAISS	You want a library rather than a database — maximum control, no server, and you handle persistence yourself.
Pinecone	You’d rather pay than operate it. Fully managed.
Elasticsearch / OpenSearch	You already run it for keyword search and want hybrid in one place.

The strongest argument for pgvector

Your vectors can sit in the same row as your data. That means a plain SQL join gets you the chunk text, its document, its author and its permissions in one query — no syncing two systems, no eventual consistency, no orphaned vectors pointing at deleted rows.

That last problem eats more engineering time than people expect. If you can avoid it entirely by keeping one database, do.

Those are the common choices, not the whole field — nearly every mainstream database now stores vectors. Databases that support vectors is the full directory, grouped by what each thing actually is.

What “production” looks like

1. Ingest — a document arrives. Split it on its structure into 200–800 token chunks, keeping headings with bodies.

2. Embed — batch the chunks, embed them in as few API calls as possible, cache by content hash so unchanged chunks cost nothing on re-ingest.

3. Store — write chunk text, vector, and metadata (source, section, URL, permissions, timestamp) as one row.

4. Query — embed the user’s query, ANN search with metadata filters applied, take the top 20 or so.

5. Rerank (optional, high value) — run those 20 through a cross-encoder, which reads query and chunk together rather than comparing two independent vectors. Much more accurate, far too slow to run over everything, perfect for a shortlist. Cohere Rerank and open models both work well. Keep the best 3–5.

6. Use — show them as results, or paste them into an LLM prompt as context. That second one is RAG, and there’s nothing more to it than what you’ve just read.

Hybrid search is usually the honest answer

Embeddings are weak at exact matching — product codes, error identifiers, surnames, version numbers. Keyword search handles those perfectly and always has.

Most strong production systems run both and merge the results, commonly with Reciprocal Rank Fusion, which combines two ranked lists using only their positions. Roughly fifteen lines of code and reliably better than either alone.

If someone tells you embeddings replace keyword search, they haven’t run one in production. They complement it. Alternatives to vector embeddings goes through when each one wins.

Next

• Cosine similarity — pick the right metric when you create that collection
• Glossary and FAQ — every term on this page, defined
• Build it: Python · C# · Java · TypeScript

References

• Efficient and robust approximate nearest neighbor search using HNSW graphs — Malkov & Yashunin, the algorithm behind most vector indexes
• Reciprocal Rank Fusion — Cormack et al., the standard way to merge keyword and vector results
• pgvector · Chroma · Qdrant · FAISS · Pinecone
• LangChain — text splitters · LlamaIndex — node parsers
• Cohere — Rerank overview
• Elasticsearch — kNN search