Going to production

The demo works. Now it has to survive real documents, real traffic and a finance team.

None of what follows is difficult. It’s the set of things that are obvious in hindsight and expensive to retrofit, so it’s worth twenty minutes before you build rather than after.

What this actually costs

Embeddings are the cheapest component in any AI stack, by a wide margin. The surprise is that the API bill usually isn’t the number that matters.

For a corpus of 100,000 chunks at roughly 400 tokens each — about 40 million tokens:

	Rough scale
One-off embedding of the corpus	Cents to low single-digit pounds
Storage at 1,536 dimensions, 4-byte floats	~600 MB of raw vectors
Per-query embedding	Negligible, fractions of a penny
Re-embedding after a model change	The same as the initial run, again

Check current pricing rather than trusting any figure written in a guide, this one included.

The costs that actually bite:

Storage and memory, not tokens. 600 MB of vectors needs somewhere to live, and if you want fast search it wants to be in RAM. Ten million chunks is roughly 61 GB. That’s an infrastructure decision, not a line item.

Re-embedding. Every model upgrade means re-processing everything. Budget for doing it more than once.

The generation step, if you’re doing RAG. Putting five chunks in a prompt on every request costs far more than the embedding did. If your bill looks alarming, that’s almost certainly where it’s coming from.

Shrink the vectors before you scale the hardware

OpenAI’s -3 models accept a dimensions parameter that returns shorter vectors, and they were trained so truncation loses less than you’d expect:

response = client.embeddings.create(
    model="text-embedding-3-large",
    input=texts,
    dimensions=256,        # instead of 3072
)

Cutting -3-large from 3,072 to 256 dimensions is a twelvefold storage reduction, and it still outperforms some full-size older models. Measure it on your own golden set before committing — but measure it before you provision a much larger machine. See Matryoshka embeddings for why this works at all.

Cache. Embeddings are deterministic.

The same text through the same model gives the same vector, every single time. That makes caching trivially correct — there’s no staleness question, because the answer genuinely never changes.

import hashlib

def cache_key(text: str, model: str) -> str:
    """Same text + same model = same vector, forever. Safe to cache indefinitely."""
    return hashlib.sha256(f"{model}\x00{text}".encode()).hexdigest()

Note the model goes into the key. Leave it out and a model upgrade silently serves vectors from the old one — the exact mixed-generation bug that produces confident nonsense with no error message.

Where this pays off:

• Re-ingesting documents. Most chunks are unchanged between runs. Hash first, skip the ones you’ve seen, and a full re-ingest costs almost nothing.
• Repeated queries. Real traffic is enormously repetitive. A short-lived cache on query embeddings removes a network round trip from your hot path.

Batching, and its limits

Send many texts per request. One call for 500 chunks beats 500 calls, by a factor of roughly 500.

But there is a ceiling, and it’s a hard error rather than a truncation: OpenAI’s embeddings endpoint accepts at most 2,048 inputs per request and 300,000 tokens across the whole request. Exceed either and you get an HTTP 400.

def batched(items, size=256):
    for i in range(0, len(items), size):
        yield items[i:i + size]

vectors = []
for batch in batched(chunks, 256):
    vectors.extend(embed(batch))

A few hundred per batch keeps you clear of both limits with room for long chunks. Add retry with backoff around the call; the official SDKs retry some failures for you, so check you aren’t swallowing the exception they eventually raise.

Re-embedding is a migration

This is the operational fact people most often discover too late.

Vectors from different models are not comparable. Upgrading means re-embedding everything, not just new rows. Mixing generations degrades results silently, with no error to alert you.

So treat it like a schema migration, because that’s what it is:

1. Keep the source text. Store the original chunk alongside its vector, always. Without it you cannot re-embed without re-ingesting from scratch, and the original documents may be gone.

2. Record which model produced each vector. A model column. Cheap now, essential later — it’s how you find the stragglers and how you assert that a query and its index agree.

3. Build the new index alongside the old one. Don’t mutate in place. You want the option to compare and to roll back.

4. Compare on your golden set before switching. This is the entire reason to have one. “Newer” is not a synonym for “better on your data”.

5. Switch reads over, then delete the old index once you’re confident.

The silent version of this bug

Someone adds documents while an upgrade is half-finished. Now 90% of your index is model A and 10% is model B. Queries embedded with B match the B chunks slightly better regardless of content, so a small, arbitrary slice of your corpus quietly outranks everything else.

Nothing errors. Results just get subtly, inexplicably worse. The model column is what turns this from a mystery into a query.

Latency

Budget it properly, because a chatbot that takes eight seconds to start answering feels broken regardless of how good the answer is.

  embed the query        30–100 ms   (network round trip)
  vector search           5–50  ms   (ANN index)
  rerank the shortlist   50–200 ms   (cross-encoder, if used)
  ────────────────────────────────
  retrieval total        ~100–350 ms
  generation             1–10 s      ← everything else is noise

Two consequences. First, optimising retrieval below ~100 ms is usually wasted effort when generation takes seconds. Second, stream the answer. Perceived latency is dominated by time-to-first-token, and streaming changes the experience far more than any retrieval optimisation will.

What to monitor

Most teams log the answer and nothing else, which makes every complaint unfixable.

• The retrieved chunk ids, per query. The single most useful thing you can log. When someone reports a bad answer you need to know what the model was given.
• Top similarity score, per query. A drifting distribution is an early warning that something changed — a bad ingest, a partial migration, a shift in what users ask.
• Queries where the top score is low. Your best free source of golden-set additions and content gaps.
• Embedding API errors and latency. Rate limits and timeouts, separated from everything else.
• Index size and freshness. How many documents, and how stale is the oldest.

Log retrieval separately from generation

When quality drops, the first question is always “did retrieval find it, or did generation ignore it?” If both are in one log line you can’t answer that. Two records, joined by a request id, and the question answers itself.

A pre-launch checklist

• Source text stored alongside every vector
• Model name recorded per vector
• Content-hash caching on the ingest path
• Batching with retry and backoff
• A golden set in the repo, running in CI
• Retrieved chunk ids logged per query
• Deleting a document also deletes its vectors
• Access control applied at query time, not after (security and privacy)
• A documented plan for re-embedding

Next

Security and privacyPermissions, PII, multi-tenancy and prompt injection.

Evaluating retrievalThe golden set that half this page depends on.

Chunking and vector databasesChoosing where the vectors live.

References

• OpenAI — Pricing — current embedding and generation costs
• OpenAI — Embeddings guide — batch limits and the dimensions parameter
• Matryoshka Representation Learning — why truncating these vectors works
• pgvector — indexing — HNSW and IVFFlat tuning, and their memory cost
• Qdrant — optimizing performance — quantization and memory-vs-accuracy trade-offs
• openai-python — retries and timeouts — what the SDK already handles for you