Curiosity · Concept

Query Rewriting (for RAG)

User queries are usually too short, ambiguous, or conversational to retrieve well. Query rewriting sits between the user and the retriever and fixes that. Common patterns: expansion (add synonyms and paraphrases), decomposition (break a compound question into sub-queries), conversational rewriting (resolve pronouns against chat history), HyDE-style hypothetical answers, and step-back prompting (ask a more abstract version of the question first). A strong query rewriter is often the highest-leverage RAG improvement after chunking.

Quick reference

Proficiency
Intermediate
Also known as
query reformulation, query expansion
Prerequisites
Retrieval-Augmented Generation

Frequently asked questions

What is query rewriting in RAG?

Query rewriting is the practice of transforming the user's original question before hitting the retriever — using an LLM or a learned model — so that the query better matches how relevant passages are expressed in the corpus.

What are the main query-rewriting patterns?

Expansion (add synonyms / reformulations), decomposition (split compound questions into sub-questions), conversational rewriting (resolve 'it', 'that', 'he' against chat history), HyDE (generate a hypothetical answer and embed it), and step-back prompting (retrieve against a broader abstract version of the question).

When does query rewriting help the most?

Conversational agents (pronouns need resolving), multi-hop questions (decomposition), domain corpora where user vocabulary differs from source vocabulary (expansion), and long-tail questions (HyDE, step-back).

What's the latency cost?

An extra LLM call per query. Most production systems use a small/fast model (Haiku, GPT-4.1-nano class) to keep added latency under 300 ms, or cache rewrites on common queries.

Sources

  1. Ma et al. — Query Rewriting for Retrieval-Augmented Large Language Models — accessed 2026-04-20
  2. Zheng et al. — Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models — accessed 2026-04-20

Related

  • Chunking Strategies (for RAG) — Chunking strategies are the rules by which a RAG pipeline splits documents into retrievable units — choice of size, overlap, and boundary (character, token, sentence, section, semantic) directly controls retrieval quality and answer grounding.
  • HyDE (Hypothetical Document Embeddings) — HyDE is a retrieval technique where the LLM first generates a hypothetical answer to the user query, then embeds that generated answer and uses it — not the query — to search the vector index.
  • Hybrid Search (BM25 + Vector) — Hybrid search is a retrieval strategy that combines sparse keyword scoring (usually BM25) with dense vector similarity, then fuses the two ranked lists — catching both exact-term matches and semantically related passages.
  • All Concepts — Browse the complete concepts library
  • VSET — Programmes offered — B.Tech programs (CSE, AI&ML, AI&DS, Electronics, IIoT)