The text stops mid-sentence. The client shows a generic network error, the handler logs no exception, and the same endpoint that streamed fine on localhost dies in production after a suspiciously round interval.
Nothing in the code threw. Something else closed the connection, and that something else is a default nobody set.
Streaming is usually described as a user experience feature: show tokens as they arrive so the wait feels shorter. That description is true and it is also the reason the failure is hard to find. The wait did not disappear. It moved into a connection that now has to stay open while the work finishes, and every layer between your model and your browser has its own opinion about how long an open connection may stay quiet.
Key takeaways
- Streaming removes the wait before the first token, not the work. The connection lives for the whole gap between the first token and the last.
- Four layers sit between the model and the browser and their idle timeouts do not agree: nginx reads 60 seconds, Cloudflare 125, Node 300, and Go has none by default.
- nginx times out between two successive reads, not over the whole response, so the longest silence in your stream is the number that kills it, not the total duration.
- SSE already gives reconnection and a Last-Event-ID header. A WebSocket adds a channel back to the server and loses both of them.
- When a run takes minutes, persist the events first and stream them second, because a reconnect has nothing to replay if the events only ever existed inside one response.
Streaming moves the wait, it does not remove it
Time to first token is the time between sending a request and receiving the first token of the response. For reasoning models, the first token counted is the first reasoning token, not the first visible word (Artificial Analysis). That is the number streaming improves.
The total time does not change. The model still needs the same seconds to finish, and the connection exists for all of them. Between the first token and the last, the stream is mostly silence with something written into it every so often, and how often is decided by the model, not by you.
That distinction is the whole article. If your provider sends events every 40 milliseconds, an idle timeout of 60 seconds is generous and irrelevant. If your agent loop spends 90 seconds inside a tool call before the next token, that same 60 seconds is the thing that ends the run.
Four layers, six timeouts, no two agree
Here are the defaults that sit between a model call and a browser, read from the primary documentation on 2026-11-03.
| Layer | Setting | Default | What it does |
|---|---|---|---|
| nginx | proxy_read_timeout |
60s | Closes the upstream connection if nothing arrives between two reads |
| nginx | proxy_buffering |
on | Holds the response in buffers, so early events can arrive at the client late or in a burst |
| Cloudflare | Proxy Read Timeout | 125s | Returns 524 if the origin sends no response in that window |
| Cloudflare | Proxy Write Timeout | 30s | Returns 524 if the write to the origin does not finish in time |
| Node | server.requestTimeout |
300s | Answers 408 if the whole request is not received in time |
| Node | server.timeout |
0 | No socket inactivity timeout at all |
| Go | Server.WriteTimeout |
0 | No write timeout at all |
The numbers come from the nginx proxy module documentation, Cloudflare's connection limits reference, the Node HTTP documentation, and the Go net/http documentation.
Read the table as a list of clocks, not settings. Six of them, in four layers, and the difference between the smallest and the largest is not a percentage. Whichever fires first ends your stream, and that is usually not the one you configured, because the one you configured is the one you thought about.
Two details in that table deserve more than a row.
The first is nginx. proxy_read_timeout is set "only between two successive read operations, not for the transmission of the whole response" (nginx docs). A stream that runs for three minutes and emits a token every five seconds will outlive a 60 second timeout indefinitely. A stream that runs for 90 seconds and thinks for 61 of them does not. Duration is not the risk. The longest silence is.
The second is Cloudflare, because 125 is not the number most people repeat. The widely quoted figure is 100 seconds, from older write-ups; the current reference says 125 for the proxy read timeout and 30 for the write timeout, and only the read timeout is adjustable, and only on Enterprise zones, up to 6,000 seconds (error 524). If the number in your head came from a blog post, it is worth checking against the table before building an argument on it.
This is not a WebSocket problem
The first instinct when a stream dies is to reach for a socket, on the theory that a WebSocket is a more durable kind of connection. It is not. It is a connection with a different lifecycle, and it inherits the same timers, the same load balancer, and the same deploy that drains a node.
What a WebSocket actually buys is a channel back to the server while the response is open. That matters when the reader has to interrupt generation, steer it, or talk at the same time. Token streaming does not need it, because token streaming goes one way.
Server-sent events are the cheaper tool and they come with two things people rebuild badly by hand. An EventSource reconnects on its own after the connection drops (MDN), and the spec defines a Last-Event-ID header that the browser sends on that reconnect, carrying the id of the last event it received (HTML spec). Number your events and resumption is a lookup, not a feature.
The reconnect delay is implementation-defined, "probably in the region of a few seconds" in the spec's own words, and the server can override it with the retry field. That is the knob for a stream that gets killed by a proxy: if your client waits 3 seconds before asking again and your run is four minutes long, you get a reconnect every time the proxy idles, forever.
One more number worth knowing before designing around open streams: over HTTP/2 the number of simultaneous streams is negotiated between client and server and defaults to 100 (MDN). Every open SSE stream is one of those, which is fine until one page opens a stream per widget.
When the run takes four minutes, the stream is not the transport
Here the shape changes, not just the size.
A four minute agent loop is not a request that happens to be slow. It is a job: it can be retried, it can outlive a browser tab, and it can be cancelled halfway. The moment that is true, the response stops being the record of what happened. It becomes a view of a record that has to exist somewhere else.
Cloudflare's own 524 page recommends status polling for long running HTTP processes. That is the vendor telling you the same thing: do not make the connection carry the truth.
The design that survives is boring. Accept the job, return an id, append events to a log with a sequence number as they are produced, and let the stream read from that log. A reconnect with Last-Event-ID becomes a request for "everything after 412" rather than a request to run the work again. The client can close the tab, and the run continues, because the run never depended on the tab.
You cannot resume what you did not write down. That sentence is the whole section.
Cancelling is a decision on the server
Closing the socket is a client action. It says nothing to the model call that is still generating tokens into a buffer nobody will read.
Unless the abort is propagated, the stop button stops the pixels, not the bill. In a streaming handler this is a few lines: listen for the close event, cancel the request to the provider with the same signal, and let the cancellation travel the whole way down. In a job-shaped design it is a message to the worker, plus a rule for what a cancelled job leaves behind.
This is testable without a benchmark. Run the probe below with an abort time, then compare the token usage your provider reports for that request against a run you let finish.
Measure your own number
The table above is a list of other people's defaults. The number that matters is the longest silence your stream can produce, and that is a property of your loop, not of nginx.
Save this as sse-probe.mjs and point it at your endpoint. It reports time to first token, total time, the largest gap between two chunks, and how many event boundaries arrived.
// sse-probe.mjs — node sse-probe.mjs <url> [abortAfterMs]
const url = process.argv[2];
const abortAfter = Number(process.argv[3] || 0);
const start = performance.now();
const ac = new AbortController();
if (abortAfter) setTimeout(() => ac.abort(), abortAfter);
let first = null, last = null, gap = 0, events = 0, bytes = 0;
try {
const res = await fetch(url, { signal: ac.signal });
const reader = res.body.getReader();
const dec = new TextDecoder();
let buf = "";
for (;;) {
const { value, done } = await reader.read();
if (done) break;
const now = performance.now();
bytes += value.byteLength;
if (first === null) first = now;
if (last !== null) gap = Math.max(gap, now - last);
last = now;
buf += dec.decode(value, { stream: true });
events = (buf.match(/\n\n/g) || []).length;
}
} catch (err) {
console.log(`aborted at ${Math.round(performance.now() - start)}ms (${err.name})`);
}
console.log({
ttftMs: first && Math.round(first - start),
totalMs: Math.round((last || start) - start),
largestGapMs: Math.round(gap),
events,
bytes,
});Compare largestGapMs against the smallest timeout in your stack, not against totalMs. If the gap loses, the stream will die in production on exactly the runs where the model thinks hardest, which is the opposite of the runs you want to lose.
What this does not cover
The numbers here are documentation defaults read on 2026-11-03, not measurements taken on your stack. Your reverse proxy may already override them, your platform may add another layer, and an Enterprise zone can move the Cloudflare timeout to a value this table does not describe.
It also does not cover HTTP/3 behaviour, gRPC streaming, or provider-side caps on how long a single response may stay open. Those are real and they behave differently; they are a second article.
If your p95 run duration is longer than the shortest timeout in your stack, the stream is a view over a log, not the transport of truth. Build the log first.