Without streaming, your user stares at a spinner until the whole answer is ready, which can be 20 seconds or more for a long reply. With streaming, the first words appear in under a second. The total time is the same, but the app feels far faster.
What the stream looks like
Add "stream": true to a chat request. Instead of one JSON response you get a series of Server-Sent Events:
data: {"choices":[{"delta":{"content":"Hel"}}]}
data: {"choices":[{"delta":{"content":"lo!"}}]}
data: {"choices":[{"delta":{},"finish_reason":"stop"}]}
data: [DONE]Each data: line is a JSON chunk with a small piece of text in choices[0].delta.content. [DONE] marks the end.
Parsing it correctly
The most common bug: assuming each network chunk contains whole lines. It doesn't. A JSON line can be split across two reads. Always buffer:
const res = await fetch("/api/chat", { method: "POST", body: JSON.stringify({ messages }) });
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split("\n");
buffer = lines.pop(); // keep the incomplete last line for the next read
for (const line of lines) {
if (!line.startsWith("data: ")) continue;
const data = line.slice(6).trim();
if (data === "[DONE]") return;
const json = JSON.parse(data);
output.textContent += json.choices?.[0]?.delta?.content ?? "";
}
}With the SDKs it's simpler
Python:
for chunk in client.chat.completions.create(model="claude-sonnet-4-6", messages=msgs, stream=True):
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")The SDKs handle buffering for you.
Proxying streams through your server
Your API key must stay on the server, so the browser usually streams from your endpoint, which streams from the AI API. Make sure nothing buffers in between:
- Send
Content-Type: text/event-stream(ortext/plain) and flush after each write. - With nginx, add the header
X-Accel-Buffering: no. - Disable compression on the streaming route; gzip often waits for a full buffer.
Usage and cost while streaming
Billing is the same as non-streaming: you pay for the tokens generated. If the user closes the tab mid-answer, your server should stop reading the stream; you only pay for what was produced.
Claude, GPT, Gemini, DeepSeek, image and video models through one OpenAI- and Anthropic-compatible endpoint. Prepaid, pay only for what you use.