We Turned Simulation Off and Two Bugs Showed Up in the First Five Minutes

The day we flipped simulation off and let real payments through, the full live pricing and payment path ran properly for the first time in a while, and it surfaced two bugs almost immediately.

The first one could hard block a checkout. Our live re-price step asks the supplier to confirm the price before any money moves, which is the correct thing to do and also the step that had never actually run in anger. When a supplier handed back a stale or expired pricing token, that call returned a zero or otherwise invalid total, and the code read that as “we can’t price this” and stopped the checkout dead. A customer with a valid card and a real room, blocked, because a token had aged out.

The second was slower to notice and more embarrassing. Checkout was doing a supplier re-price and then up to three sequential gateway calls, all in series. Nobody had felt it, because the simulated path skipped most of them, so the thing we’d been clicking through for months was a shorter path than the one customers would take.

Neither of those is exotic. Both were invisible for as long as they existed, because the only code path that could produce them was the one we weren’t running.

You can’t just stop mocking

The obvious response is to stop simulating and test for real, which doesn’t survive contact with a booking product. A real checkout charges a real card and consumes real inventory that someone else wanted. Do that on every test run and you’re paying money and burning rooms to find out whether a button works.

The agentic commerce literature is blunt about this. Discovery is easy and checkout is the unsolved problem, and this is exactly why. Mock it and you hide the integration bugs that matter. Run it and it costs.

What worked was moving the decision from a global flag to a per-user one. When the signed-in user is internal staff, checkout runs the entire real flow, the same code in the same order with the same recovery paths, and leaves out the two side effects that can’t be taken back: It doesn’t charge the card and it doesn’t call the supplier to book. Everything else is real. The booking row, the guests, the payment ledger entry, the confirmation email.

So the team drives the live booking path end to end, watching the real ledger and the real emails and the real self-heals fire, with no money moved and no inventory consumed.

The detail that mattered more than we expected is that the decision gets persisted on the payment record at the time it’s made, not re-read later from the live role. An in-flight booking that started as simulated finishes as simulated even if the flag or the role changes underneath it mid-session. Otherwise you get a booking that’s half one thing and half the other, which is a much worse bug than either mode.

The difference from a global mock is the whole point. A global mock means nobody is running the real path. A per-user rule means the real path runs on every single request, and a small set of accounts skip only the irreversible bit at the very end.

The envelope this sits in

All of that lives inside what we call a self-healing tool envelope. Every tool the model can call returns the same shape.

type ToolEnvelope<T> = {
  summary: T | null
  result: T | null
  health: Record<string, 'ok' | 'empty' | 'failed' | 'skipped' | 'degraded'>
  diagnostics: {
    area: string
    severity: 'info' | 'warn' | 'error'
    message: string
    retryHint?: string
  }[]
  error?: string
}

The heals themselves are unremarkable. A same-day check-in the supplier rejects gets repaired forward by a day. A search at an exact landmark with no inventory broadens to the surrounding area. An empty radius auto-expands and says so, “auto expanded radius 0.8km to 2.5km”. A supplier throwing 5xx gets retried, then circuit-broken and substituted. A save or book tool called while signed out returns an auth prompt rather than guessing.

Two rules are what keep that from becoming a mess.

Health can never downgrade

const setHealth = (area, status) => {
  if (health[area] === 'ok' && status !== 'ok') return
  health[area] = status
}

Five lines, one rule. Once an area is ok, nothing later can mark it worse.

That isn’t cosmetic. Search fans out across suppliers and then does post-processing, and one of those post-processing steps fetches review scores after the rooms have already loaded. Without the never-downgrade rule, that late best-effort step throwing would flip the whole hotels area to failed and paint a perfectly good results grid red. The user had their hotels. The late error is a footnote, and monotonic health is how you encode “we delivered something useful” in a way the UI can read.

The failures that genuinely matter all happen before ok is ever set, so nothing real gets hidden by this. That’s the part i’d check first if i were copying the pattern, because a monotonic health field in a pipeline where real failures can land late would be actively dangerous.

The heal is surfaced, not logged

The envelope gets read twice. The model checks health before deciding what to say, so it can tell you it couldn’t find hotels right at the museum and searched the surrounding district instead. The UI renders the same diagnostics in a panel directly above the results.

Same envelope, two consumers, so a recovery is never invisible to the agent or the human.

This is the part the research line tends to skip, and it’s the part that matters most in commerce. A silent fallback that quietly swaps in a worse result is, from where the user is sitting, indistinguishable from a bug. And “we quietly charged you a different price” stops being a reliability question and becomes a trust question, and then a compliance one.

Underneath both rules: Never bubble a raw supplier error to the model or the UI. Catch it at the boundary and convert it into a diagnostic carrying a recovery hint, so the agent has something to offer instead of an apology.

What’s still open

We fixed the hard block with a fallback that trusts the already-validated search price when the live re-price comes back invalid, and it emits a warn diagnostic so we can see how often it fires. i’m not fully comfortable with it. It’s a band-aid over a supplier token lifetime problem, and if a price genuinely moved between search and checkout we’d be honouring the old one.

The serial gateway calls are still on the list.

And one more from a different corner of the same system, because it’s the same disease. Chat title generation once wrapped its model call in a try/catch that returned a static string on any error, and when that model got decommissioned, every title in the product silently became the generic fallback. No signal at all. That’s written up in the model switch that broke nine features, and the short version is that a fallback with no diagnostic is indistinguishable from a bug.

If you’ve got a simulate flag right now, the useful question isn’t whether your tests pass. It’s what you expect to happen on the day you turn it off, and whether you’d rather find out on a Tuesday morning or during a launch.