The Reattach Found an Empty Buffer Every Time in Production and Never Once on My Machine

Switch chats and come back, and the stream you left was gone.

Not errored. Gone. The reattach looked up the buffer, found it empty, and returned nothing, and it did that every single time in production and never once on my machine.

Why the generation has to outlive the tab

A turn on this thing can take 20 seconds or more, because it fans out to three hotel suppliers and plans an itinerary and prices the trip against live rates. A chatbot answering from a document can afford to die when you close the tab, because you can just ask again and the second answer costs nothing. An agent that spends 20 seconds and real supplier calls can’t tell you not to touch the tab, and it can’t silently throw that work away when you do.

So the generation and the connection had to come apart. Generation on the server, long running, owns the work. Connection on the client, disposable, owns nothing.

The first half of that is result.consumeStream(), which keeps the model generating after the client disconnects. That shipped and it wasn’t enough, and the way it wasn’t enough is worth being precise about. The generation survived. The tokens didn’t. They were produced into a stream nobody was reading, which means they went straight to /dev/null, and the only thing that ever reached the database was the final assembled message once generation completed.

So the experience was: Click away at second 4, come back at second 12, see a frozen half-message, and then watch a complete reply appear out of nowhere some seconds later when persistence finally caught up. Worse than a spinner, because it looks broken rather than slow.

Buffering

So keep every chunk as it streams. Then a client that reattaches can replay the history it missed and then tail whatever is still coming, and the two of those together look like the stream never dropped.

Two endpoints sit behind that. resume hands back the buffered history and then keeps the connection open for live chunks. stream-status answers the cheaper question of whether there’s anything in flight for this session at all, which the client needs before deciding whether to reattach or just load the persisted messages.

i still think that design is right. The connection became disposable and the stream became durable and the thing behaved exactly as described.

we put the buffer in an in-process Map.

Isolates

Workers spreads requests across many isolates, and an isolate is its own JavaScript context with its own heap. The request that started the generation and filled that Map, and the request that reattaches thirty seconds later, are almost never the same isolate. There’s no affinity to rely on and nothing in the API suggests there is.

So the reattach was looking in a Map that really was empty. A different Map. In a different isolate. Correctly returning nothing, with no error, because an empty buffer and a buffer that was never yours look identical from the inside.

Locally there’s one isolate, so there’s one Map, and the bug can’t reproduce. Not by luck. By construction. Every local run confirms the design works, and every production run fails, and both are telling the truth about the runtime they are on.

The fix was Upstash Redis. The reason it specifically is that it speaks HTTPS rather than a TCP socket, and a Worker can’t open a raw TCP connection, so most of the Redis clients you would reach for first are unusable before you even get to the logic. The in-process Map stayed on top as a same-isolate fast path, so a reattach that does land on the same isolate skips the network entirely and the shared store is only consulted on a miss.

That’s a small thing worth keeping. The wrong fix here is deleting the Map and going to Redis for everything, which trades a correctness bug for a latency tax on every single chunk. The Map was never wrong. It was just never sufficient.

We’d ripped Redis out of this codebase earlier, deliberately, to stay on platform primitives. This one feature is what justified putting a sliver of it back, and only this one. One exception you can state a reason for beats the dogma, and it also beats letting the exception quietly become the new default everywhere.

Anything two requests must share can’t live in a process. That’s obvious written down. It wasn’t obvious with the Map sitting right there, working, in front of us.

And then you have to tell them

Once a generation can finish while the user is somewhere else, you have to say so, or the feature is invisible. The work completing in the background is worth nothing if the only way to discover it is to guess and click back.

So there’s durable stream tracking and a browser notification when a generation completes in the background, plus the client-side bookkeeping to keep the URL and the local state in sync and invalidate the right queries when you return. You leave, the trip plan finishes, the tab tells you it’s done.

The guest session took four goes

The other half of the same seam. A logged out visitor should still get a real conversation that persists, and getting there was four production bugs in a row, each one a different wrong assumption.

First we supported guest sessions at all.

Then the client-only-id bug, which is the one worth reading. The client minted its own session id shaped like local-${Date.now()} instead of creating a database row and using the id that came back. So every saveGuestMessage call pointed at a session that didn’t exist, and failed with NOT_FOUND, and failed silently. People typed, the UI showed their messages because the UI had them locally, and nothing was written anywhere. The fix mints a real CUID-backed chat_session row with user_id = NULL, and our project rules now forbid reintroducing that local- id by name, because it is load bearing in the way that only a bug you already shipped can be.

Then we were littering the database. Creating that real row eagerly on page load meant exactly one empty session row for every visitor who ever opened the page and never typed a word, which on a public marketing surface is most of them. Switched to lazy creation, so the row is minted on the first actual turn.

Then the auth boundary, where a guest who had been chatting and then signed in got orphaned, because the new authenticated session had no relationship to the guest one. Now the guest session is claimed on sign in and attached to the new user id, so the conversation survives the transition instead of being abandoned next to it.

“support guests” is four features wearing one name. Where the id comes from, when the row gets created, what happens when identity changes underneath you, and what the client is allowed to invent.

There was a fifth thing in the same family, much smaller and much more irritating to track down, which is that a protected session query was firing for logged out visitors and causing a visible flicker on the chat page before anything rendered.

The other Workers bug from this product is background I/O that Node allows and the edge doesn’t, which failed the same way this one did. Correct on the runtime the tests run on, illegal on the runtime that serves users, and invisible to both until production disagrees with you.