Pick the right model
- Use the smallest model that passes your tests. Small models can be many times cheaper (small vs large).
- Route by difficulty: cheap model by default, expensive only for hard cases (routing).
Send fewer tokens
- Trim conversation history: last N messages or a running summary (memory).
- Retrieve, don't stuff: send relevant passages, not whole documents (RAG).
- Shorten your system prompt. It's paid for on every single request.
- Clean inputs: remove HTML, boilerplate and duplicate text.
- Resize images before sending them to vision models.
Generate fewer tokens
- Ask for concise output and set
max_tokens. Output usually costs more than input. - Ask for diffs, not whole files, when editing code.
Avoid paying twice
- Cache results for repeated identical requests (same prompt, same input).
import hashlib, json
cache = {}
def cached_chat(model, messages):
k = hashlib.sha256(json.dumps([model, messages]).encode()).hexdigest()
if k not in cache:
cache[k] = client.chat.completions.create(model=model, messages=messages, temperature=0).choices[0].message.content
return cache[k]- Don't retry blindly on long requests that may still be running.
Put guard rails in place
- Set monthly limits per key and per-user limits in your app, so a bug or abuse can't run up a bill (API keys).
Measure first
Open Usage Logs, sort by cost, and look at the top few features or models. The biggest savings usually come from one or two places.
Claude, GPT, Gemini, DeepSeek, image and video models through one OpenAI- and Anthropic-compatible endpoint. Prepaid, pay only for what you use.