Why Swapping Embeddings Is Harder Than Swapping Models
You can replace an LLM by pointing your code at a different API endpoint. Replacing an embedding model is a different job: you have to regenerate every vector you have already stored.
Vectors from different models don't mix: Each model builds its own semantic space. If you search document vectors from model B with a query vector from model A, the results are close to random. Queries and documents must always be embedded by the same model.Dimensions change: Moving from 768 to 1,536 or 3,072 dimensions means building a new index. At float32, one 1,024-dimension vector takes 4KB, so one million chunks need about 4GB for the raw vectors alone.Normalization and distance functions change: If a model outputs normalized vectors, dot product and cosine similarity give the same ranking. Use a dot-product index with a model that doesn't normalize, and the ranking gets distorted. Similarity thresholds, such as "only use results above 0.75," also have to be set again for each model.Deciding Whether to Switch
A higher score on a public benchmark is not a good enough reason to switch. You need evidence that the new model works better on your own data.
Build an Evaluation Query Set
Pull retrieval failures from production logs: unhelpful answers, repeated questions, and escalations to human agents.Pair each query with the document that answers it, and aim for 200–500 pairs. You can start comparing models with as few as 100.Compare with Retrieval Metrics
Recall@k: the share of queries whose correct document appears in the top k results. RAG pipelines pass the top 5–10 chunks to the LLM, so Recall@5 and Recall@10 matter most.MRR: the average of the reciprocal rank of the first correct result. A correct result at rank 1 scores 1.0, and one at rank 3 scores 0.33.Break Results Down by Segment
An average can hide weak spots. Split the results into Korean vs. English queries, queries with internal acronyms or product codes, and short FAQs vs. long manuals. A model often improves the average by 3 points and still does worse on domain terminology.
Designing Zero-Downtime Re-Indexing
Blue-Green Build
Keep serving traffic from the existing index (blue) while you build a new index (green) with the new model. Validate green against the evaluation set, then switch the query embedding model and the target index together. If only one of them changes, search breaks right away, so ship the two settings as a single deployment.
Estimate Cost and Time
One million chunks at 500 tokens each comes to 500 million tokens. At an assumed $0.10 per million tokens, that costs about $50.At 500 chunks per second, the job takes about 2,000 seconds, or roughly 33 minutes. API rate limits usually make it take longer, so set batch sizes and a retry policy before you start.Record progress per chunk ID so you can rerun only the batches that failed.Dual-Write During the Transition
Documents keep getting added and edited while re-indexing runs. During this period, write new and updated documents to both blue and green, and apply deletions to both. Right before the switch, check that the two indexes match on document count and latest modification time.
Making It an Operating Practice
Record metadata: For every index, store the embedding model name and version, the dimensions, the distance function, the chunk size, and the overlap. You should always be able to trace how a given vector was made.Run regression tests: Run the evaluation set on a schedule, and send an alert if Recall@10 falls more than a set margin below the baseline.Keep rollback simple: Keep the blue index for a while after the switch. A rollback only works in practice if you can do it by flipping one setting.Give each component a clear role: Keyword search (BM25) finds product codes and proper nouns, vector search finds paraphrases, and a reranker re-orders the candidates more precisely. In a hybrid setup, keyword search makes up for weaker vector results, which lowers the risk of switching embeddings.How POLYGLOTSOFT Supports RAG Operations
POLYGLOTSOFT designs RAG as a system you keep running, not a feature you build once. We can help with building evaluation query sets from production logs, comparing candidate embedding models, setting up blue-green re-indexing pipelines, defining dual-write and rollback procedures, and running regular quality regression tests. If you're planning an embedding migration, or your retrieval quality has dropped and you don't know why, get in touch with POLYGLOTSOFT. We'll review your current setup and work with you on a safe migration plan.
