Cannot Perform I/O on Behalf of a Different Request
A supplier endpoint with a p90 of 15 seconds, four commits to put a timeout on it, one full revert in production, and the same illegal background I/O turning up again a layer down in the cache.
Our availability endpoint had a p50 of 4.8 seconds and a p90 of 15.1. 19% of cold calls came back over 15 seconds and a few of them hung past 40.
The control is what made it undeniable. The details call, on the same hotels, in the same session, over the same auth, through the same integration code, came back in about 0.18 seconds. One endpoint at 0.18 and its neighbour at 15.1 isn’t a network problem and it isn’t your client. we measured it that way specifically because the first thing anyone says when you report latency is that it is probably your side, and a control on the same connection ends that conversation before it starts.
That part was easy. it then took us four commits to put a timeout on it.
The spinner
The call site looked like this.
const rooms = await getOrSet(cacheKey, () => supplier.availability(params))Nothing bounds that await. When the supplier hung, the request hung with it until Cloudflare killed the whole thing, and the user sat reading “Checking live room availability…” until they gave up and closed the tab. An unbounded await on somebody else’s API is a spinner with no exit condition. The supplier decides how long your page hangs.
The easy half
Wrap every supplier request in an AbortController.
const controller = new AbortController()
const timer = setTimeout(() => controller.abort(), timeoutMs)
try {
return await fetch(url, { ...init, signal: controller.signal })
} finally {
clearTimeout(timer)
}The finally matters more than it looks. Without it every fast response leaves a live timer behind, and on a Worker handling real traffic you are then leaking timers at the rate of your request volume. The abort also throws a specific error you have to catch and translate, otherwise a slow supplier surfaces to the user as a stack trace instead of a message.
This went in across all three suppliers. It is correct, it is boring, and i want to point at exactly why it is safe, because that turns out to be the whole article. The abort happens inside the call we are already awaiting. Nothing gets started that outlives the response.
The part we were proud of
A timeout that only fails isn’t much of an upgrade. The supplier is slow, you wait 12 seconds, and then instead of a spinner you get an error. Better, not good.
What we wanted was to race the lookup against a 12 second deadline and return a degraded response when the deadline wins. Mark that supplier degraded in the response envelope, attach a “taking longer than expected” diagnostic the UI can render, and let the page paint with everything else that did come back. The user gets hotel details and photos and the rooms panel says it is still working.
And then the bit we liked. The lookup that lost the race is still running. It is going to finish eventually whether we want it to or not. So let it finish, and let it write its result into the cache on the way out, and the user’s retry thirty seconds later is instant. A free warm cache falling out of a timeout.
Node tests passed. All of them.
Cannot perform I/O on behalf of a different request
A Worker request owns an execution context, and any I/O you start belongs to that context. Once my procedure returned the degraded response, that request was over. The availabilityPromise was still out there doing a fetch and then a cache write on behalf of something that no longer existed, and the runtime refuses to carry it.
The legal way to do this is ctx.waitUntil, which tells the runtime to keep the request alive until the promise settles. we went to add it and couldn’t, because our tRPC context didn’t carry executionCtx at all. The procedure had no reference to the thing that grants permission. There was no legal way to background anything from where we were standing.
So, revert. we kept the AbortController bounds from the previous commit, because those are fine, and went back to a plain await.
What i keep turning over is that Node has no rule like this. A dangling promise in Node is just a promise. My tests ran in Node, which means every test we could have written would have passed, including tests written specifically to catch this. The failure mode isn’t “we forgot to test it”. The failure mode is that the runtime the tests run on doesn’t have the constraint the production runtime has.
Threading executionCtx into trpc
Not clever. Plumbing.
// context.ts
export type Context = {
// ...
waitUntil?: (p: Promise<unknown>) => void
}
// worker.ts
createContext({
// ...
waitUntil: honoCtx.executionCtx.waitUntil.bind(honoCtx.executionCtx),
})The bind isn’t decoration. executionCtx.waitUntil loses its receiver if you pass the bare function reference, which fails at runtime in a way that reads like the context is missing rather than like a this problem.
With that in place the deadline race works the way we wanted it to two commits earlier.
const result = await Promise.race([availabilityPromise, deadline(12_000)])
if (result === TIMED_OUT) {
ctx.waitUntil?.(availabilityPromise)
return degraded('taking longer than expected')
}There’s a regression test on it now. Stalled supplier, degraded response in under a second, never hangs. The optional call is deliberate, so a code path that somehow arrives without a context degrades to dropping the cache warm instead of throwing.
And then the same bug in the cache
Separately, and a while later, we went to raise the availability TTL from 60 seconds to 12 hours. At a p90 of 15.1 seconds a one minute TTL means almost every hotel view pays the full cold cost, which makes the cache decorative.
12 hours on live availability sounds reckless and mostly isn’t, because a fixed check-in and check-out pair rarely changes its inventory inside a day, and the booking flow re-validates price and token before any money moves. A stale read costs a re-validation. It doesn’t cost a wrong charge.
Our cache wrapper supports stale while revalidate, and turning it on was the obvious move. we set staleTtlMs: 0 instead. revalidateInBackground serves the stale value and then fires a background fetch to refresh it, with no waitUntil anywhere near it. That’s the same illegal background I/O, one layer down, in code nobody was thinking about.
Two unrelated pieces of code, same hazard, and both times the safe answer was no unawaited background I/O.
The other half of this problem
While measuring all of the above we found a latency bug that has nothing to do with Workers and is worth the detour.
The hotel detail page called getSupplierDetails, which is fast and mostly static, and getSupplierRooms, which is the slow live availability call, on the same render. tRPC’s httpBatchLink did what it is designed to do and combined them into one request, which then resolved only when both finished. So the header and the photos, which were ready in well under two seconds, sat blank for about four seconds waiting on rooms.
The fix is a splitLink that routes hotel.getSupplierRooms through a plain non-batching httpLink and leaves everything else batched. The header paints at about 1.7 seconds now and rooms stream in behind it.
Batching is normally free. It stops being free the moment the members of a batch have different latencies, because a batch is a shared fate and everything in it waits for the slowest thing in it.
Where it actually landed
The deadline is 20 seconds right now. It was 15 and that was too tight, because our BFF adds a second or two of its own and 19% of cold calls were tripping the bound on hotels that were slow but genuinely fine, which turns a working hotel into a degraded one for no reason.
20 is probably also wrong. we’ll find out when someone complains.
The other Workers bug we hit on this product was chat streams disappearing across isolates, which is the same shape as this one. Fine on Node, illegal at the edge, invisible to the test suite in both directions.