Most AI features don’t fail on launch day. They fail slowly: a prompt nobody owns, a model version that changes underneath you, a monthly bill that doubles without a release. These are the habits that keep that from happening.
Give every request a token budget
Every AI call gets a ceiling for input and output tokens, written down next to the feature. When retrieval returns more context than the budget allows, rank and trim rather than sending everything. Budgets make cost predictable, and they force the useful question: which context actually changes the answer?
Write the evals before the prompt
A feature isn’t ready until it has a set of real examples with expected outcomes. Run them on every prompt change and every model upgrade, and block the release when the important cases get worse. Without that, every change is an argument about taste.
Plan the fallback for every call
- A timeout with a clear message, not a spinner that never ends.
- A smaller or cheaper model for retries, when the task allows it.
- A non-AI path: search, a form, or a human queue.
- Cached answers for questions that repeat.
Log enough to debug, not enough to leak
Record the request id, model and version, token counts, latency and the outcome. Redact personal data before anything is stored, and set a retention period on day one. You will need these logs the first time a user says “it gave me the wrong answer yesterday”.
Pin versions and own the upgrade
Pin the model version. Treat an upgrade like any other release: run the eval suite, compare cost and latency, and ship it with the results attached. “The provider changed something” should never be news you learn from a customer.
Give it an owner
Someone owns the prompt, the eval set and the budget, the same way someone owns a service. Ownership is what turns a demo into a feature that is still working in a year.
The short version
- A token budget per request.
- An eval suite that gates releases.
- A fallback for every call.
- Logs that help without leaking.
- Pinned versions, deliberate upgrades.
- A named owner.