RAG Chatbot Architecture: Encoder + Decoder

This is a practical build guide for a RAG (Retrieval-Augmented Generation) chatbot. The core idea: use a small encoder to find relevant information, and a large decoder LLM to write the answer. Two models, two jobs — and the split is exactly what makes RAG efficient.

1. Why two models?

A chatbot generates text, so it needs a decoder-only LLM (GPT, LLaMA, Mistral, Qwen). But a plain LLM only knows what it saw during training — it can't answer about your private, recent, or niche documents, and it may hallucinate. RAG fixes that by retrieving the right facts at question time and pasting them into the prompt.

Retrieval is a find the most relevant text problem, not a generate text problem. The best tool for it is an encoder (embedding model), which is tiny and fast compared to the chat LLM. So a RAG chatbot uses both:

RAG chatbot = TWO models, different jobs

1. ENCODER (embedding model, small)   -> RETRIEVAL
     - embed your docs once (indexing)
     - embed the user question
     - find the most similar chunks

2. DECODER (chat LLM, large)          -> GENERATION  (required)
     - takes: retrieved chunks + the question
     - writes the final grounded answer

2. Why the encoder saves resources

The encoder is the efficient choice for retrieval for a concrete reason: it runs one forward pass and outputs one vector. No token-by-token loop. A decoder, by contrast, generates autoregressively (one token at a time) and is far heavier. Embedding models are also simply much smaller:

Model                    Size        Job
-----------------------  ----------  -------------------------
all-MiniLM-L6-v2         ~22M        embeddings (encoder)
BERT-base                ~110M       embeddings (encoder)
A chat LLM (decoder)     7B - 70B+   generation

A retrieval encoder like all-MiniLM-L6-v2 is around 22M parameters — hundreds of times smaller than the chat LLM, and fast even on CPU. Two more efficiencies:

Precompute once — embed all your documents at indexing time and store the vectors. You never re-embed them per question.
Cheap at query time — you only embed the question (one tiny pass) and run a fast vector similarity search.

3. The two phases of RAG

Phase A — Indexing (done once, ahead of time)

documents (PDFs, wiki, FAQ)
   |  1. LOAD
   v
raw text
   |  2. CHUNK        split into passages (a few hundred tokens each)
   v
chunks
   |  3. EMBED        encoder turns each chunk into a vector
   v
vectors
   |  4. STORE        save vectors in a vector database
   v
searchable index

Phase B — Query (every time the user asks)

user question
   |  1. EMBED QUESTION   same encoder -> question vector
   v
question vector
   |  2. SIMILARITY SEARCH   find chunks closest to the question
   v
top-k relevant chunks
   |  3. AUGMENT PROMPT   paste chunks into the prompt as context
   v
augmented prompt
   |  4. GENERATE   decoder LLM writes the grounded answer
   v
final answer (with citations)

Key insight: the LLM is never retrained. RAG changes the input (what context you feed it), not the model's weights. Update your documents and answers update instantly — no training required.

4. How retrieval works: embeddings + cosine similarity

An embedding is a vector that captures the meaning of a piece of text. Texts with similar meaning get vectors pointing in similar directions, even if they share no words. To compare two vectors we measure the angle between them with cosine similarity:

cos(theta) = (A . B) / (||A|| * ||B||)

 1  = identical direction (very similar)
 0  = unrelated (perpendicular)
-1  = opposite

"Find the most relevant chunk" becomes "find the stored vector with the highest cosine similarity to the question vector." That single number drives retrieval.

Question:  "How do I reset my password?"
Chunk A:   "To reset your password, open Settings..."   cos = 0.91  (retrieved)
Chunk B:   "Invoices are billed monthly..."             cos = 0.12
Chunk C:   "Office hours are 9am to 5pm..."             cos = 0.08

5. Full query flow (putting it together)

User question
   |
   +-- encoder embeds question --> vector search --> top relevant chunks
   |                                                        |
   +--------------------------------+-----------------------+
                                    v
          prompt = [retrieved chunks] + [user question]
                                    |
                            DECODER LLM generates
                                    v
                         grounded answer with citations

Encoder does the finding — cheap, small, precomputed.
Decoder does the writing — required; RAG cannot generate text, only the decoder can.

6. A concrete local stack (48GB Apple Silicon + Ollama)

Component            Choice                         Role
-------------------  -----------------------------  ----------------------
Encoder (retrieval)  all-MiniLM-L6-v2               embed docs + questions
                     (via sentence-transformers)    (small, CPU is fine)
Vector store         Chroma / FAISS / pgvector      similarity search
Decoder (chat)       ollama run qwen3:32b           generate the answer
                     (or llama3)

The encoder sips resources; the decoder (e.g. qwen3:32b) is the heavy part that fits comfortably in 48GB. That split is the whole point of RAG — you don't make the big model do the searching.

7. Minimal RAG loop in Python (reference)

Requires pip install sentence-transformers numpy. The retrieval math is plain numpy — no vector DB needed for a small demo.

import numpy as np
from sentence_transformers import SentenceTransformer

# --- INDEXING (done once) ---
docs = [
    "To reset your password, open Settings and click 'Forgot password'.",
    "Invoices are billed monthly on the first day of each month.",
    "Our office hours are 9am to 5pm, Monday through Friday.",
]
model = SentenceTransformer("all-MiniLM-L6-v2")        # small encoder
doc_vecs = model.encode(docs, normalize_embeddings=True)

def cosine(a, b):                                      # normalized -> dot == cosine
    return float(np.dot(a, b))

# --- QUERY (each question) ---
question = "How do I change my login password?"
q_vec = model.encode(question, normalize_embeddings=True)
scores = [cosine(q_vec, d) for d in doc_vecs]
context = docs[int(np.argmax(scores))]                 # best-matching chunk

# --- AUGMENT the prompt ---
prompt = f"""Answer using ONLY the context. If not in context, say you don't know.
Context:
{context}

Question: {question}
Answer:"""

# --- GENERATE with the decoder LLM (Ollama) ---
# import ollama
# reply = ollama.chat(model="qwen3:32b",
#                     messages=[{"role": "user", "content": prompt}])
# print(reply["message"]["content"])

8. RAG vs fine-tuning — which to reach for

                RAG                          Fine-tuning
--------------  ---------------------------  ---------------------------
What changes    the prompt (adds context)    the model's weights
Best for        injecting knowledge/facts    teaching behavior/style/format
Update cost     add a document (instant)     re-train
Hallucination   low (grounded + citations)   unchanged
Hardware        light (small encoder)        heavier (GPU for training)

For a chatbot that must answer from your own documents/FAQ, start with RAG. Reach for fine-tuning only when you need a specific tone, format, or skill that prompting plus retrieval can't deliver — and even then, the two are complementary.

9. Summary

A RAG chatbot is two models: a small encoder for retrieval and a large decoder LLM for generation.
The encoder saves resources — one pass per text, document vectors precomputed once, tiny compared to the LLM.
The decoder is required — only it can generate the reply; RAG just feeds it better context.
RAG changes the input, not the weights, so your knowledge base updates instantly with no retraining.

One line: small encoder finds the context, big decoder writes the reply.

Hello World

Be Happy!