DEVSAIO

Process thousands of AI requests reliably

Patterns for big jobs: concurrency limits, checkpoints, retries and cost tracking, so a crash halfway doesn't mean starting over.

DEVSAIO team · 2 min read
Share on X

Processing 10,000 items is different from 10. Things will fail halfway. Plan for it.

1. Checkpoint every result

Write each result as soon as you have it, and skip items already done:

python
import json, os

DONE = "results.jsonl"
done_ids = {json.loads(l)["id"] for l in open(DONE)} if os.path.exists(DONE) else set()
todo = [x for x in items if x["id"] not in done_ids]

2. Limit concurrency

python
import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://devsaio.lol/v1", api_key=KEY, max_retries=3, timeout=120)
sem = asyncio.Semaphore(8)
lock = asyncio.Lock()

async def run(item):
    async with sem:
        try:
            r = await client.chat.completions.create(model="claude-haiku-5-5", temperature=0,
                messages=[{"role": "user", "content": PROMPT.format(**item)}], max_tokens=300)
            row = {"id": item["id"], "ok": True, "out": r.choices[0].message.content, "usage": r.usage.model_dump()}
        except Exception as e:
            row = {"id": item["id"], "ok": False, "error": str(e)}
        async with lock:
            with open(DONE, "a") as f:
                f.write(json.dumps(row) + "\n")

async def main():
    await asyncio.gather(*(run(x) for x in todo))

asyncio.run(main())

3. Re-run failures

Failed rows are recorded with ok: false. Delete them from the file (or filter them) and run the script again; only failures are retried.

4. Track cost as you go

Sum usage from the results to estimate cost before the job finishes, and give the job's key a monthly limit as a safety net.

5. Test on 20 items first

Run a small sample, check quality and cost per item, then multiply. It's the cheapest way to avoid an expensive surprise.

Try it with one API key

Claude, GPT, Gemini, DeepSeek, image and video models through one OpenAI- and Anthropic-compatible endpoint. Prepaid, pay only for what you use.

Keep reading