Back to Blog
AI

Vector Search Can't Find Part Numbers: Improving Enterprise RAG Accuracy with Hybrid Keyword-Vector Search and Rerankers

Why vector search fails on exact-match queries like part numbers and error codes, and how hybrid BM25-vector retrieval, RRF fusion, and cross-encoder rerankers raise retrieval accuracy in enterprise RAG.

POLYGLOTSOFT Tech Team2026-11-097 min read14
RAGHybrid SearchRerankerBM25Enterprise AI

"It's Right There in the Document, but the AI Can't Find It"

After rolling out an internal RAG system, the most common complaint is less "the answer is wrong" and more "the document exists, but it wasn't found." Ask "What is the tightening torque for part AB-1203-K?" and vector search will happily return documents for AB-1208-K or AB-1203-M — part numbers that look almost the same. Embeddings compress meaning; they were never designed to separate part numbers, standard codes, error codes, or customer names, where a single character changes what the string refers to.

The danger is that these wrong answers sound plausible. The LLM answers confidently from a similar procedure for a similar part, and users have no reason to doubt it. When you trace answer-quality problems back to their source, a large share originate not in the generation model but in retrieval. If the right document never makes it into the candidates, no model can answer correctly.

Keyword Search and Vector Search Excel at Different Things

  • Keyword search (BM25 and similar): checks whether the query terms literally appear in a document. It is strong on proper nouns, codes, and numbers, and rare terms carry more weight — so a single part number is often enough to pinpoint the right page.
  • Vector search: strong when the wording differs but the meaning matches, such as "the machine suddenly stopped" versus "response procedure for emergency stops."
  • For Korean content, check tokenization carefully. Morphological analyzers (such as Nori in Elasticsearch and OpenSearch) strip particles and improve recall, but depending on configuration they may split `AB-1203-K` into `ab`, `1203`, and `k` — and suddenly every document containing 1203 matches. A safer approach is to extract part-number patterns with regular expressions, store them in a separate, non-tokenized field, and normalize case and hyphenation.

    Combining the Two Result Sets

    Normalized score fusion rescales BM25 scores and cosine similarities to 0–1 and takes a weighted sum. It is intuitive, but score distributions shift from query to query, making a single fixed weight hard to get right.

    Reciprocal Rank Fusion (RRF) ignores scores and uses only ranks. Each document accumulates `1 / (k + rank)` from every result list, with k commonly set to 60. Documents ranked highly by both retrievers rise to the top, and there is almost nothing to tune — a good default for a first deployment.

    Add query routing on top: when a query contains a part-number or error-code pattern, lean toward keyword results; for descriptive questions, lean toward vector results.

    Also, apply metadata filters before retrieval. If you don't first narrow by document type, revision date, or department, an obsolete procedure can compete with — and outrank — the current revision. This matters most on manufacturing floors where multiple revisions of the same document coexist.

    Re-Ranking the Final Order

    The standard pattern is two-stage: retrieve broadly with hybrid search, then reorder the candidates with a cross-encoder reranker. A cross-encoder reads the question and the document together and judges relevance directly, which is more precise than comparing separately computed embeddings. The trade-off is cost: the model runs once per candidate, so it is slower and more expensive.

    That makes candidate count the key dial. A common starting point is retrieving 50–100 candidates and passing only the top 5–10 after reranking to the LLM. More candidates raise the chance the right document is included, but latency grows too, so set the number against your internal response-time target.

    Validate the gains with an internal evaluation set. Collect 100–200 real work questions and label the correct document for each, and you can compare before and after on the same footing. Mix in part-number lookups, descriptive questions, and questions that require distinguishing revisions.

    Metrics to Watch in Production

  • Retrieval hit rate: the share of queries where the correct document appears in the top N (Recall@N).
  • Answer accuracy: whether the answer was right, measured separately given the correct document.
  • Keeping them separate reveals the cause. Low hit rate means fix retrieval; high hit rate with wrong answers means fix prompting and generation. Log queries that drew "couldn't find it" feedback or follow-up rephrasings, and refine dictionaries, synonyms, and chunking every two to four weeks — accuracy improves steadily.

    The POLYGLOTSOFT Approach

    Drawing on our MES and WMS delivery experience, POLYGLOTSOFT builds hybrid-search RAG systems designed around the structure of manufacturing and logistics documents — part numbers, drawing numbers, equipment codes, and revision histories. We work with you step by step, from dedicated code-pattern fields and query routing to reranker integration and an internal evaluation set. If internal document search accuracy is holding your team back, get in touch with us today.