Why Plain PDF-to-Vector RAG Fails (and What We Built)

Plain PDF-to-vector RAG fails because it cuts documents into chunks that lose their context, breaks tables and cross-references, and retrieves text by similarity rather than meaning. Questions that need facts from several places get partial or wrong answers. The fix is better document parsing, a knowledge graph of how facts relate, and an agent that plans its retrieval.

Key Takeaways

• Basic RAG works for simple lookups but struggles when an answer depends on rules, exceptions and tables spread across many PDFs.
• Most failures start at ingestion: bad PDF parsing and blind chunking, not the language model.
• A knowledge graph stores how facts connect, so the system can follow relationships instead of guessing from similar-sounding text.
• Agentic RAG plans which sources to query and checks its evidence before answering.
• Test with real user questions before launch; a demo on five easy questions proves very little.

What is plain PDF-to-vector RAG?

Retrieval-augmented generation (RAG) lets a language model answer from your own documents. The “plain” version most tutorials teach has four steps: extract text from PDFs, split it into fixed-size chunks, turn each chunk into a vector embedding, and at question time fetch the most similar chunks and pass them to the model.

It is quick to build and looks impressive in a demo. Many teams start here when they want an AI chatbot on company data, and it is a reasonable first step. The trouble starts when real users ask real questions.

Why does PDF RAG give wrong answers?

From our own builds and the published research, the failures fall into five groups.

1. Chunks lose their context

A chunk such as “the company’s revenue grew by 3% over the previous quarter” does not say which company or which quarter. Anthropic used exactly this example in its 2024 research on contextual retrieval. Adding context to each chunk before embedding, plus keyword (BM25) search, cut failed retrievals by 49%, and by 67% when reranking was added. In other words, a large share of RAG errors is a retrieval problem, not a model problem.

2. Tables, headers and layouts get scrambled

Basic text extraction reads a table row by row as loose words. Column headings end up in one chunk and the values in another. Multi-column layouts, footnotes and scanned pages make it worse. The model then answers from fragments that no longer mean what the original page said.

3. Cross-references and exceptions are split apart

Policy documents, contracts and regulations are full of lines like “subject to clause 7.2” or “except where listed in Schedule B”. Vector search finds the main rule but rarely the exception sitting 40 pages away. The answer sounds confident and is wrong.

4. Similar is not the same as relevant

Embeddings match wording. Two visa subclasses, two product versions or two years of the same policy look almost identical as vectors, so the system can return the outdated or wrong one.

5. Multi-step questions need more than one search

Microsoft Research put it plainly in its 2024 GraphRAG paper summary: “Baseline RAG struggles to connect the dots,” especially when an answer requires linking separate pieces of information through shared attributes. A single top-k search cannot do that.

If your chatbot is wrong, look at how documents were parsed, chunked and connected before you blame the model. Better retrieval beats a bigger model.

What did KONZE build instead?

Our clearest example is an agentic-RAG immigration chatbot built on a knowledge graph. Immigration content is a hard test for plain RAG: an answer about an occupation or visa pathway can depend on several lists, conditions and exceptions held in different documents, and the rules change often. We see the same pattern in the occupation-list work behind our post on how Australian migration agents can save time.

Instead of a pile of vectors, the system has three layers:

1. Structured ingestion. Documents are parsed with their headings, sections and tables kept intact, so each piece of text carries its source and context.
2. A knowledge graph. Entities (such as occupations, visa types and requirements) and the links between them are stored explicitly, so “A applies to B unless C” is a relationship, not three unrelated chunks.
3. An agent that plans retrieval. It works out what the question needs, queries the graph and text sources step by step, and answers only from what it found.

Our published agentic RAG knowledge assistant case study shows the same design for internal enterprise knowledge. There, the flow is Understand → Plan → Retrieve → Analyse → Respond. The assistant can pull from internal documents and CRM records together and lives inside Microsoft Teams, where staff already work. The lesson from both projects: the intelligence sits in how knowledge is structured and retrieved, not in the chat window.

Plain RAG vs knowledge graph RAG vs agentic RAG: how do they compare?

You do not always need the most advanced option. Match the approach to the hardest question your users really ask.

How can you fix a RAG chatbot that gives wrong answers?

1. Collect 50–100 real questions from staff or customers, with the correct answers and source pages.
2. Check retrieval separately from answers. For each question, did the right passage come back at all? If not, no prompt will save it.
3. Fix parsing first. Keep headings, tables and page references; use OCR or layout-aware extraction for scans.
4. Chunk by structure, not by character count, and add a short context line to each chunk.
5. Add hybrid search and reranking so exact terms like codes and clause numbers are found.
6. Model relationships in a knowledge graph where rules and exceptions matter.
7. Add an agent and a human check for high-stakes answers, with citations shown to the user.

Measure progress against your question set, not gut feel. This is the same discipline we argue for in whether your AI investment is actually delivering.

When is plain RAG good enough?

Plain RAG is fine when documents are short, well written and mostly prose, and when a wrong answer is low-cost. A help-centre bot over 30 clean FAQ pages may never need a graph.

It is not enough for regulated or rule-heavy content: immigration, education admissions, insurance, legal agreements and finance. The same applies to contracts, where our intelligent document processing case study shows why clause-level context matters. For agencies, our guide on how AI is changing education and immigration consulting covers where these tools fit day to day.

FAQ

Why does my RAG chatbot give wrong answers?

Usually because the right passage was never retrieved. Poor PDF parsing, fixed-size chunks without context and pure similarity search are the common causes. Test retrieval on real questions before changing the model or prompt.

What is the difference between RAG and GraphRAG?

Standard RAG fetches text chunks similar to the question. GraphRAG builds a knowledge graph of entities and relationships from your data and uses it at query time, so it can connect facts spread across documents.

What is agentic RAG?

Agentic RAG adds a planning step: an AI agent decides what information a question needs, runs several searches across documents or systems, and checks its evidence before answering. It suits multi-step business questions.

How do I improve RAG accuracy on PDFs with tables?

Use layout-aware extraction that keeps tables as tables, chunk by section, and store each table with its headings and source page. Hybrid keyword and vector search with reranking then helps find exact values.

Conclusion

Plain PDF RAG is a good prototype and a weak production system for rule-heavy documents. Most wrong answers come from retrieval, not the model. Fix parsing and chunking first, add hybrid search and reranking, then bring in a knowledge graph and an agent where questions join facts across sources. Keep a real question set and measure every change against it.

If your document chatbot answers confidently but wrongly, talk to the KONZE team about our AI and ML solutions and how we structure knowledge for production.