Vertical AI
RAG is not enough, why retrieved knowledge fails on real work
RAG retrieves what was written. But the knowledge that makes operations work was never written down. Here is why retrieval alone fails on real enterprise work.
Key takeaways
- RAG retrieves written content; roughly 80% of how work actually gets done was never written down.
- Better embeddings, chunking, or rerankers cannot fix a substrate problem.
- Documentation describes the official process; the real process, with workarounds and judgment calls, lives in people.
- RAG and expertise mining are complementary; the sequence is mine first, then retrieve.
I have been building software since I was 8. For the last six years, I have been the technical half of a partnership that shipped over a hundred products for enterprises. For the last two, I have been building agentic AI systems, hands on, inside real companies, under real deadlines.
I say this because what I am about to write might sound like I am against RAG. I am not. RAG is a useful tool. I have built with it. I will build with it again. But it is not enough.
What RAG does well
RAG retrieves relevant written content from a knowledge base and feeds it to a language model at inference time. The model gets grounded in company-specific information instead of relying solely on its training data.
This is genuinely useful. It helps with answering questions about documented policies, surfacing content from large document stores, reducing hallucination for well-documented topics, and making general-purpose models more domain-specific. For tasks where the answer exists in written form, RAG works.
The problem is how much of the organization's critical knowledge is not written at all.
The substrate problem
Every AI approach reads from a substrate. RAG's substrate is written content, documents, wikis, manuals, policy files, knowledge bases. But here is the gap: research suggests that 42% of essential expertise lives only in employees' heads. (Harvard via Synaply) Roughly 80% of how work actually gets done is undocumented. (Tallyfy)
RAG cannot retrieve what was never written. This is not a retrieval problem. You cannot solve it with better embeddings, better chunking, better reranking, or a bigger vector database. The content does not exist.
Where RAG breaks on real work
A RAG-powered system handles straightforward queries well. "What is our return policy?" "What is the SLA for this tier?" These answers exist in writing. RAG finds them. The demo is impressive.
Then the system hits a real operational case. A claim that does not fit neatly into any documented category. A customer situation where the standard policy does not apply. An exception that requires judgment, not just information retrieval. The experienced operator knows exactly what to do. RAG does not.
Documents describe the official process, not the real one. They describe the idealized workflow. The real process has workarounds, shortcuts, unwritten rules about which approvals actually matter, and judgment calls that were never codified. An agent operating on the official version will follow the documented steps precisely, and miss every exception that makes the operation actually work.
Context degrades over time. Documents go stale. The team knows it; the retrieval system does not. An experienced operator has a built-in filter for this. RAG has no such filter.
A different substrate, not a better pipeline
The instinct in the engineering community is to fix this with better infrastructure. Fine-tune the embeddings. Add metadata filtering. Build a more sophisticated reranking model. These are all improvements to the retrieval pipeline. They make RAG better at finding written content. They do not solve the fundamental problem: the most critical operating knowledge was never written down.
RAG and expertise mining are not competing
RAG is strong where the knowledge is written and accessible. Expertise mining is strong where the knowledge was never expressed. In practice, you need both. RAG handles documented retrieval. Expertise mining captures what RAG cannot reach.
But the sequence matters. If you deploy RAG first and build your entire AI pipeline on retrieved documents, you are building on the official process, not the real one. Good demo results, poor production results.
If you start with expertise mining, capture the real operating logic, structure it, validate it, then layer RAG on top for the documented portion, you build on the right foundation. Most of the market has invested heavily in retrieval. Almost nobody has invested in extraction. The companies that do will be the ones whose AI actually works on real cases.
