Users forgive a slightly worse answer. They do not forgive waiting. So we start every LLM feature with a latency budget.

Spend the budget explicitly

Retrieval, re-ranking, generation and guardrails each get a slice of the 800 ms. If a step goes over, it has to earn it.

Cache what you can

Most questions are not new. A semantic cache in front of the model cut our median latency in half on one project.