Open the circuit

Week 2 · Production Python · about 3 minutes

What you will be able to do

Fail fast when a dependency is already down, instead of piling retries onto a customer request.

The idea

Count consecutive hard failures (timeouts, connection errors). When they cross a threshold, open the circuit: the next calls do not leave the process. Return a fallback you designed — a cached answer, a queued job, or a clear error.

After a cool-down, let one probe through (half-open). Success closes the circuit. Failure opens it again. Retries still belong inside the closed state, with the cap from the rate-limit lesson. The breaker is what stops those retries from stacking across every user.

import time, httpx

class Circuit:
    def __init__(self, threshold=5, cooldown=30.0):
        self.threshold, self.cooldown = threshold, cooldown
        self.failures, self.opened_at = 0, None

    def call(self, fn, fallback):
        if self.opened_at is not None and time.monotonic() - self.opened_at < self.cooldown:
            return fallback()  # open: don't call the dependency
        try:
            result = fn()  # closed, or the half-open probe after cool-down
        except (httpx.TimeoutException, httpx.ConnectError):
            self.failures += 1
            if self.opened_at is not None or self.failures >= self.threshold:
                self.opened_at = time.monotonic()  # open, or reopen after a failed probe
            raise
        self.failures, self.opened_at = 0, None  # success closes the circuit
        return result

If the gateway timed out for the last five callers, the sixth caller should not wait too.

Key takeaways

• Closed: call the dependency, retry only transient errors.
• Open: do not call. Return the fallback.
• Half-open: one probe decides whether to close.

Part of the Forward Deployed Engineer roadmap. New to the role? Start with what a Forward Deployed Engineer does.