How text becomes numbers

You called embed("the spare set is in the kitchen drawer") and got 1,536 numbers back. This page is about what happened in between.

Fair warning on scope: I’m not going to teach you to train a model. I’m going to give you the amount of understanding that changes decisions you’ll actually make — why long documents behave badly, why your token bill looks like that, why the same word can produce different vectors.

Step 1: your text is chopped into tokens

Models don’t read characters, and they don’t quite read words either. They read tokens — chunks that sit somewhere in between, worked out from what’s statistically common in the training data.

"The spare set for the front door is in the kitchen drawer."

  ["The", " spare", " set", " for", " the", " front", " door",
   " is", " in", " the", " kitchen", " drawer", "."]

Common words are single tokens. Rarer ones get split up:

"unbelievable"     →  ["un", "bel", "iev", "able"]
"antidisestablish" →  ["ant", "id", "ise", "stab", "lish"]
"🎉"                →  often several tokens on its own

Two consequences you’ll meet in practice:

Your bill is in tokens, not words. For ordinary English, roughly 1 token per 0.75 words — so 1,000 words is about 1,300 tokens. Other languages fare worse. The same sentence in Japanese or Hindi can cost two or three times as many tokens, because the tokenizer was fitted mostly on English text. If you’re building for a non-English market, budget accordingly.

There’s a hard input limit. OpenAI’s text-embedding-3-small accepts 8,191 tokens per input — around 6,000 words. Go over and you get an error, not a truncation. This is the single most common reason people need chunking.

See it for yourself

platform.openai.com/tokenizer shows you exactly how any text splits, with a live count. Genuinely worth thirty seconds — paste in a paragraph of your real data and see what it costs.

Programmatically, tiktoken does the same thing:

import tiktoken

encoder = tiktoken.get_encoding("cl100k_base")
print(len(encoder.encode("The spare set is in the kitchen drawer.")))

Step 2: each token becomes a starting vector

Every token in the model’s vocabulary has a vector attached to it, learned during training. The model looks each one up in a big table.

At this stage the vectors are context-free. The token "bank" gets the same starting vector whether your sentence is about rivers or mortgages. If we stopped here we’d have 2013-era Word2Vec, and “bank” would be permanently confused.

Step 3: the tokens read each other

This is the part that makes modern embeddings work, and it’s where the Transformer architecture earns its reputation.

The model runs the token vectors through a stack of layers. At every layer, each token gets to look at every other token in the input and adjust itself accordingly. That mechanism is called attention, and it means meaning flows between words.

"I sat on the river bank"
                   ↑
        "bank" looks around, notices "river",
        and shifts towards the geography sense

"I withdrew cash from the bank"
                           ↑
        same token, notices "cash" and "withdrew",
        shifts towards the finance sense

Do this across a dozen or so layers and each token’s vector has absorbed a great deal about its surroundings. This is why embeddings handle ambiguity, idiom, and negation with any grace at all — the words genuinely inform each other before anything is finalised.

Grace, not perfection

Negation is still a known soft spot. “The server is running” and “The server is not running” often score as highly similar, because they share almost everything except one small word that flips the entire meaning. If your application turns on that distinction — compliance checks, medical text, incident triage — test it explicitly rather than assuming. Embeddings measure aboutness better than they measure truth.

Step 4: many vectors get squashed into one

After all that, you’ve got one vector per token. But you asked for one vector for the whole sentence. Something has to combine them, and this step is called pooling.

The two common approaches:

• Mean pooling — average all the token vectors together. Simple, and what most sentence-transformer models do.
• CLS pooling — many models prepend a special token whose entire job is to accumulate a summary of the sequence. Take that one and discard the rest.

Either way, everything collapses into one fixed-length vector. Which is what makes the output size independent of the input size — 5 tokens or 5,000, you get the same number of dimensions out.

Why long documents embed badly

Look at mean pooling and you can see the problem coming. Averaging is lossy. Average ten tokens and the result still points somewhere meaningful. Average ten thousand tokens spanning a company handbook — HR policy, fire safety, the Christmas party — and you get a vector sitting in the bland middle of all of it, close to nothing in particular.

This is the argument for chunking, and it’s not about the token limit. Even if the model accepted a million tokens, a whole book in one vector would still be a poor search result. Smaller pieces produce sharper vectors.

Step 5: normalising (usually)

Most providers scale the final vector to length 1 before returning it, which makes cosine similarity a plain dot product. Covered in detail here.

The whole pipeline

  "The spare set is in the kitchen drawer."
                    │
                    ▼
  ┌─────────────────────────────────────────┐
  │ 1. Tokenize    → 10 tokens              │
  │ 2. Look up     → 10 starting vectors    │
  │ 3. Attention   → 10 context-aware ones  │
  │ 4. Pool        → 1 vector               │
  │ 5. Normalize   → length 1               │
  └─────────────────────────────────────────┘
                    │
                    ▼
        [0.021, -0.014, 0.008, ... ]

Milliseconds on a GPU. And every one of those steps is fixed at training time, which is why the same text always gives you the same vector from the same model — embeddings are deterministic, unlike the chat models you may be more used to.

What actually changes your decisions

Three practical takeaways:

Same model, always. Different models produce coordinates in unrelated spaces. Mixing them yields garbage with no error message. Re-embed everything when you upgrade.

Shorter inputs give sharper vectors. Pooling dilutes. Aim for a paragraph or a section, not a document.

Deterministic means cacheable. Same text plus same model equals same vector, forever. Hash your input, cache the result, and stop paying for work you’ve already done.

Choosing a model

You’ll mostly pick along three axes:

	Smaller / faster	Larger / better
Example	text-embedding-3-small (1,536d)	text-embedding-3-large (3,072d)
Cost	Roughly 6× cheaper	Higher per token
Storage	Half the bytes per vector	Double
Quality	Good enough for most work	Measurably better on hard retrieval

Start small. Genuinely — start small. Most teams never need the large model, and the storage difference gets painful at scale. Upgrade when you have a measurement showing retrieval quality is your bottleneck, not before.

You can shorten OpenAI vectors without re-embedding

text-embedding-3-small and -3-large support a dimensions parameter that returns a shorter vector:

response = client.embeddings.create(
    model="text-embedding-3-large",
    input=texts,
    dimensions=256,       # instead of the full 3072
)

This works because the models were trained with Matryoshka representation learning — the important information is packed towards the front, so truncating loses less than you’d expect. Cutting -3-large to 256 dimensions still outperforms some full-size older models, at a twelfth of the storage. Worth benchmarking on your own data before you commit to full-width vectors.

There’s a fuller comparison in choosing an embedding model. For rankings across providers, the MTEB leaderboard is the standard reference. Treat it as a shortlist rather than an answer — benchmark scores on public datasets don’t always survive contact with your particular documents.

Next

• Chunking and vector databases — acting on what we just learned about pooling
• Cosine similarity — comparing the vectors once you have them

References

• OpenAI — Embeddings guide — models, dimensions, token limits, the dimensions parameter
• OpenAI tokenizer and tiktoken — see and count tokens
• Attention Is All You Need — the Transformer paper
• BERT — where contextual embeddings became mainstream
• Matryoshka Representation Learning — why truncating these vectors works
• MTEB leaderboard — cross-provider benchmarks
• SBERT — pooling explained — the pooling step in practice