DEVSAIO

RAG in Python: answer questions from your own documents

Build retrieval-augmented generation from scratch: chunk documents, embed them, find relevant passages and answer with citations.

DEVSAIO team · 2 min read
Share on X

RAG (retrieval-augmented generation) lets a model answer questions from your documents without retraining anything: find the relevant passages, then give them to the model with the question.

1. Chunk your documents

python
from pathlib import Path

def chunks(text, size=800, overlap=150):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

docs = []
for p in Path("docs").glob("*.md"):
    for i, c in enumerate(chunks(p.read_text())):
        docs.append({"source": f"{p.name}#{i}", "text": c})

2. Embed every chunk (once)

python
import os, json
from openai import OpenAI

client = OpenAI(base_url="https://devsaio.lol/v1", api_key=os.environ["DEVSAIO_API_KEY"])

def embed(texts):
    r = client.embeddings.create(model="bge-m3", input=texts)
    return [d.embedding for d in r.data]

for i in range(0, len(docs), 64):
    batch = docs[i:i + 64]
    for d, v in zip(batch, embed([d["text"] for d in batch])):
        d["vec"] = v

json.dump(docs, open("index.json", "w"))

3. Retrieve the best chunks

python
import math

def cos(a, b):
    return sum(x * y for x, y in zip(a, b)) / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

def search(q, k=4):
    qv = embed([q])[0]
    return sorted(docs, key=lambda d: cos(qv, d["vec"]), reverse=True)[:k]

4. Answer with the context

python
def answer(q):
    ctx = "\n\n".join(f"[{d['source']}]\n{d['text']}" for d in search(q))
    r = client.chat.completions.create(model="claude-sonnet-4-6", temperature=0.2, messages=[
        {"role": "system", "content": "Answer only from the provided context. Cite sources in [brackets]. If the answer isn't in the context, say you don't know."},
        {"role": "user", "content": f"Context:\n{ctx}\n\nQuestion: {q}"},
    ])
    return r.choices[0].message.content

print(answer("What is the refund policy?"))

Making it production-ready

  • Store vectors in a real vector store (pgvector, Qdrant, Chroma) once you have more than a few thousand chunks.
  • Re-embed only documents that changed.
  • The “say you don't know” instruction plus citations is your best defence against made-up answers.
Try it with one API key

Claude, GPT, Gemini, DeepSeek, image and video models through one OpenAI- and Anthropic-compatible endpoint. Prepaid, pay only for what you use.

Keep reading