FDE ForgeBeta
Start free
Week 5 · Enterprise AI

A latency budget picks the model

0:00 / 3:00

15 seconds

p95 is 12 seconds.

Clinicians abandon the copilot after four. The big model is accurate — and too slow.

QueryBudget 3sPrimarySlow · richFastWithin budgetAnswerOn time
Lesson notesA latency budget picks the model: the short version+

Week 5 · LLM gateways · about 3 minutes

What you will be able to do

Route Helix questions to a smaller model when the primary blows the latency budget.

The idea

Pick a hard latency budget with the customer (for example p95 under 3 seconds). Measure the primary. If it misses, route to a smaller model or a cached FAQ path.

Accuracy that arrives after the clinician has walked away is worthless. Budget first, then model.

def pick_model(primary_p95_ms: float, budget_ms: float = 3000) -> str:
    if primary_p95_ms > budget_ms:
        return "small"
    return "primary"

The latency budget is a product requirement, not a tuning tip.

Key takeaways

  • Agree a p95 latency with the customer.
  • If primary misses, route to fast.
  • Late answers do not count.

Part of the Forward Deployed Engineer roadmap. New to the role? Start with what a Forward Deployed Engineer does.