# Next-Generation RAG Search System Development > A 『RAG 2.0』 design that combines hybrid search, rerankers, and stateful orchestration. Built around LlamaIndex + Qdrant + bge-m3, an approach aimed at the precision and operability that single-shot retrieval cannot reach. Date: 2026-04-29 --- ![Architectural concept diagram of a next-generation RAG search system](https://img.wiredxeco.com/wiredxeco/next-gen-rag-search/hero.webp) ## Why 『RAG 2.0』, and why now Of all the projects involving generative AI in business, the most common request right now is "we want to make our internal documents searchable." But over the past year it has become widely understood that **a naive RAG — just dropping things into a vector DB and running similarity search — does not reach production-grade accuracy**. The reason is simple: **semantic closeness and the actual business answer are different things**. Ask "what is the internal policy on invoice payment due dates?" and the top vector matches are likely to be miscellaneous meeting minutes that contain "payment" and "due date." What you really want is a single specific clause from the policy book. Filling that gap is what people call **RAG 2.0** — hybrid search, reranking, query decomposition, and stateful orchestration. We combine these to build a **search experience that holds up in production**. ## Overall development approach The RAG we build is not a one-shot "search → answer." We design it as a **stateful retrieval flow**: decompose the query, gather a primary set via hybrid search, rerank, and re-search if needed. The foundation is LlamaIndex + Qdrant; we use OpenAI Embeddings and bge-m3 for embeddings depending on the use case, and put GPT-class models at the center of generation. The standard scope is to take it **from PoC to a verifiable state in roughly 4 weeks**. ## Retrieval strategy — the heart of 『RAG 2.0』 Our position is that plain vector search is already obsolete. To deliver real precision, you have to **layer multiple techniques**. - **Hybrid search (BM25 + vectors)**: keyword matching plus semantic similarity. BM25 covers areas where vectors are weak — proper nouns, model numbers, clause numbers - **Adding a reranker**: take a wider set from primary retrieval and reorder it with Cohere Rerank or a ColBERT-style model. **The quality of the top 5 improves dramatically** - **Query decomposition**: split compound questions like "tell me the difference between A and B" into multiple sub-queries via an LLM, then search them in parallel - **Multilingual embeddings**: we adopt bge-m3 for Japanese documents. Stable even when English is mixed in A RAG that skips this layer may look fine in a demo but **hits a precision ceiling almost immediately in real use**. Conversely, doing this layer carefully alone makes the perceived quality feel like a different product. ## Orchestration — toward stateful retrieval The recent trend is stateful retrieval. We are moving away from "single search → answer" toward designs that **give the search process itself state**. - **Flow control with LangGraph and similar**: explicitly describe the loop of search → evaluate → re-search if needed as a graph - **Self-reflection**: have the LLM judge whether the retrieved chunks are sufficient to answer the question, and re-fetch with a different query if not - **Conversation state retention**: carry over documents and filters from the previous turn so follow-up questions don't search from scratch - **Tool-call integration**: dynamically invoke internal APIs or computation tools based on retrieval results At this point it is less "search" and more an **agent that has search inside of it**. If traditional RAG is a static dictionary, this is closer to **an assistant that goes out to gather information with intent**. ## How we think about model (LLM) selection The LLM itself depends on the use case, but we tend to propose a two-tier setup: **GPT-class models at the core when accuracy is the priority**, and lighter-weight models in cost-sensitive areas. - Estimate the trade-off between generation quality and inference cost on a per-project basis - Run the same questions across multiple models and compare scores against an evaluation set (**decide by numbers, not by gut**) - When personal information or confidentiality is involved, work backwards from the API path and retention policy The point is not "just use the latest model" but **decide with evaluation data**. ## Should you even be doing RAG? Honestly, in a fair share of consultations we tell the client "this isn't a RAG project." The criteria are clear. - Data is on the order of **a few thousand characters** → ❌ faster and more accurate to put it directly in the prompt - Data is **updated frequently** → ✅ a fit for RAG (no retraining required to keep up) - **Search precision** is business-critical → ✅ a fit for RAG (with the 2.0 stack above) - **Complex flow control** is required → ❌ RAG alone is insufficient. You need agent design If you start without clarity here, you risk **applying RAG to a problem that shouldn't be solved with RAG**, and the operations phase ends up burning cost. The first week is worth spending on requirements alone. ## Evaluation and operations — so it doesn't end at PoC With RAG, **the real game starts after you build it, not before**. We put up the evaluation and operations scaffolding alongside the initial build. - Prepare a **question → expected answer → expected source** evaluation set early - **Continuously score** Recall / Precision / answer correctness - Move the reindexing procedure for document additions and revisions into CI - Accumulate user feedback (👍 / 👎) and feed it back into reranker and search-parameter tuning Without an evaluation set, debating "the precision isn't there" goes in circles. **Building a state in which you can talk in numbers from day one** turns out to be the shortest path. ## For your project too RAG is a domain where **simply lining up libraries does not yield precision**. Hybrid search, rerankers, query decomposition, state management — which to combine and which to skip is what separates the result. We take on **building, evaluating, and operating next-generation RAG** centered on LlamaIndex / Qdrant / bge-m3 end-to-end. Internal knowledge search, semi-automated customer support, Q&A platforms over policies and contracts — please get in touch from the **"will it actually work on our data?"** stage.