A circuit breaker for the AI gateway
Any system that depends on an external service will eventually watch that service go down. What matters is how your system behaves while it is gone.
The bad default
The request fails, so you retry. It fails again, so you retry again, and every attempt sits through a full timeout before giving up. Queues pile up, workers are held waiting, and quota is burned on requests that will not succeed.
Then the service comes back, the whole backlog hits it at once, and it gets knocked over again.
The breaker pattern
Three states:
- Closed: everything passes through normally.
- Open: after a set number of consecutive failures, the breaker stops calling the service and fails immediately, with no waiting.
- Half open: after a delay it allows a single attempt. Success closes the breaker; failure opens it again.
Why the third state is the important one
Skip it and you reopen the floodgate onto a service that has not recovered yet. That single probe checks how the service is doing before full traffic resumes.
What must happen while the breaker is open
On its own, the breaker only decides when to stop calling. The part that saves you is what happens to the request in the meantime. In my setup the request survives: it goes into a queue for a later retry, and the staff member gets a note that something is waiting.
When the breaker rejects a request and nothing catches it afterwards, the user still loses; the error just shows up sooner.
Practical numbers
Transient errors happen all the time, so the trip threshold has to be higher than one. Mine is five consecutive failures, and the delay before half-open is measured in minutes rather than seconds; a service that just fell over needs real time to steady itself.
Related reading
Tell me what you want to build. Your first 15 minutes of consulting are free.
Book a consultation