DEVSAIO

Timeouts and long AI responses: avoiding cut-offs

Long generations can exceed default HTTP timeouts. Here's how to set timeouts, stream, and design for slow responses.

DEVSAIO team · 2 min read
Share on X

A long answer from a large model can take a minute or more. Many HTTP clients, proxies and hosting platforms time out much sooner.

Where timeouts hide

  • Your HTTP client: some default to 30–60 seconds.
  • Your web server or proxy: nginx, Apache, Cloudflare and load balancers have their own limits.
  • Serverless platforms: functions often have hard execution limits.
  • The browser: if your page waits for a single long request.

Fixes

1. Raise the client timeout

python
client = OpenAI(base_url="https://devsaio.lol/v1", api_key=KEY, timeout=180)
js
const client = new OpenAI({ baseURL: "https://devsaio.lol/v1", apiKey: KEY, timeout: 180_000 });

2. Stream

Streaming sends data continuously, which keeps connections alive through most proxies and gives users something to read immediately. It's the best fix for interactive apps. See the streaming guide.

3. Move long work to the background

For reports or bulk processing, accept the request, return a job id immediately, process in a worker, and notify or poll for the result.

4. Ask for shorter output

If a feature doesn't need 3,000 words, say so in the prompt and set max_tokens. Shorter output is faster and cheaper.

Don't double-charge yourself

If your client times out but the request kept running upstream, retrying immediately can mean paying twice. Prefer longer timeouts and streaming over aggressive retries for long generations.

Try it with one API key

Claude, GPT, Gemini, DeepSeek, image and video models through one OpenAI- and Anthropic-compatible endpoint. Prepaid, pay only for what you use.

Keep reading