DEVSAIO

How chat memory works (and how to keep it cheap)

AI APIs are stateless: you send the whole conversation every time. Here's how to manage history without exploding costs.

DEVSAIO team · 2 min read
Share on X

AI chat APIs don't remember anything between requests. The “memory” in a chatbot is just your code sending the previous messages again each time.

python
messages = [{"role": "system", "content": "You are a helpful tutor."}]

def ask(text):
    messages.append({"role": "user", "content": text})
    r = client.chat.completions.create(model="claude-sonnet-4-6", messages=messages)
    answer = r.choices[0].message.content
    messages.append({"role": "assistant", "content": answer})
    return answer

The cost problem

Turn 1 sends 2 messages. Turn 20 sends 40. Because input is billed every time, the cost of each new message grows as the conversation gets longer.

Strategies

1. Keep a window

Send the system prompt plus only the last N messages:

python
window = [messages[0]] + messages[-12:]

Simple and effective for casual chat.

2. Summarise older turns

When the history gets long, ask a cheap model to summarise the older part, then keep the summary plus recent messages:

python
summary = client.chat.completions.create(model="claude-haiku-5-5", messages=[
    {"role": "user", "content": "Summarise this conversation in 5 bullet points, keeping names, numbers and decisions:\n" + old_text}
]).choices[0].message.content
messages = [messages[0], {"role": "system", "content": "Earlier conversation summary:\n" + summary}] + messages[-8:]

3. Store facts separately

For assistants that need long-term memory (user preferences, account details), store facts in your database and add only the relevant ones to the system prompt.

4. Start fresh for new topics

In UIs, encourage a “New chat” for unrelated questions.

Don't forget the assistant's own messages

Always append the model's answers to history. Without them, the model loses track of what it already said.

Try it with one API key

Claude, GPT, Gemini, DeepSeek, image and video models through one OpenAI- and Anthropic-compatible endpoint. Prepaid, pay only for what you use.

Keep reading