Half of Your Retrieval Problem Is Not Retrieval — It's Input
We regularly hear the same complaint: a company deploys an AI assistant trained on internal documents, and the answers are simply wrong. Teams spend weeks swapping embedding models, replacing the vector database, and rewriting prompts. Accuracy still hovers in the low 60s.
When we trace the root cause, more than half of the failures live in the ingestion stage, not the retrieval stage. At one manufacturing client, we reverse-traced 120 failed queries and found that 88 of them involved documents that had never entered the index in usable form. Tables had collapsed into a single line, scanned PDFs had produced empty bodies, and clause numbers had been separated from their text.
Enterprise documents are not tidy Markdown. Twenty-year-old scanned PDFs, spreadsheets full of merged cells, PowerPoint decks with embedded diagrams, and policy manuals split between HWP and Word all coexist. A document broken at the parsing stage cannot be rescued by any retrieval strategy. You cannot search text that does not exist.
Parsing Difficulty and Strategy by Document Type
Splitting Text PDFs from Scanned PDFs
The first step is automatic classification. A practical rule: if extracted characters per page fall below a threshold (roughly 100 characters), treat the file as a scan and route it to OCR. Running OCR on everything multiplies cost and processing time, and it degrades perfectly good text through misrecognition.
Always store OCR confidence alongside the output. Pages whose average confidence falls below a threshold — 0.75 works well as a starting point — should go to a review queue rather than pass through silently.
Preserving Tables as Structure
Tables break more often than anything else. Naive text extraction ignores merged cells and multi-level headers, concatenating cell values into a single line. A fact such as "Q3 2026 defect rate: 0.8%" becomes a meaningless string of numbers like "2026 Q3 0.8 1.2 0.9."
Detect tables with a dedicated parser, convert them to Markdown or HTML, and flatten multi-level headers so each column carries its parent label. Including the caption and the preceding paragraph in the same chunk preserves what the table is actually about.
Diagrams and Clause-Structured Documents
Diagrams are not searchable on their own. Generate captions with a vision model to create a textual surrogate, and attach the original image path as metadata. For contracts and internal regulations, structure is meaning: extract hierarchies such as "Article 3, Paragraph 2" into structured fields at parse time. Trying to recover them later with regular expressions over body text collides with tables and footnotes, and accuracy drops sharply.
Chunking: Moving Beyond Fixed Length
A fixed 512-token split is a starting point, not an answer. We recommend this order of operations.
Chunk metadata determines operational quality. At minimum, carry source document ID, document version, validity period, owning department, security level, and page or clause position. These fields enable filtered retrieval, source citation, and the retirement logic described next.
Where Pipelines Break in Production
Stale Chunks
This is the most common incident. When a policy is revised but old chunks remain in the index, the assistant confidently cites a repealed rule. Re-ingestion must delete all existing chunks by document ID and reinsert atomically. Append-only updates guarantee an eventual incident.
Permission Leakage
If performance review files and company-wide announcements share one index, a single query can break the access boundary. Store access groups in chunk metadata and enforce filters at query time, or separate indexes by classification level. Filtering after retrieval is worse: the top-K results are already contaminated, so answer quality suffers as well.
Quarantining Failures
Silently skipping documents that fail to parse creates the worst user experience — "the document exists, but the AI doesn't know about it." Failures should land in a quarantine queue with a recorded reason and be managed as reprocessing work.
Build Order and Verification
The sequence is clear: golden set → measure parsing quality → tune chunking → tune retrieval. Select 30–50 representative documents across types, have people write the expected outputs, and quantify parsing accuracy first. Skipping this step and jumping to retrieval tuning makes diagnosis impossible.
Run these automated checks at each stage.
Surfacing these metrics on a live dashboard lets you detect degradation on the day a new document type arrives, rather than months later through user complaints.
Building Enterprise AI with POLYGLOTSOFT
POLYGLOTSOFT designs and builds RAG and generative AI systems grounded in enterprise document repositories. We deliver production-oriented architecture: OCR pipelines for scanned archives, structural extraction for tables and diagrams, permission-aware index separation, and automated re-ingestion for revised documents. If you have already deployed AI search and accuracy refuses to improve, we suggest starting with a diagnostic review of your ingestion pipeline. Through our subscription development service, we can move with you from a small validation project to company-wide rollout, one stage at a time.