The LLM feature launch checklist.

Print this, put it in the launch review, and don’t ship until every box has an answer, even if the answer is “not applicable, because…”.

Purpose and limits

• What decision or task does the feature help with, in one sentence?
• What will it never do? Is that written down and tested?
• Do users know they’re using AI, and how to reach a person?

Data

• Which data reaches the model, and is each field needed?
• Is personal or regulated data redacted or excluded?
• Does the provider’s data use and retention match your contracts?
• Where are prompts and outputs stored, and for how long?

Quality

• Is there a golden dataset of real cases with expected behavior?
• Does the eval suite run on every prompt and model change?
• Which cases are critical, and do they block a release?
• Has a domain expert reviewed a sample of outputs?

Safety

• Tested against instructions hidden in retrieved content (prompt injection)?
• Can one user ever see another user’s data through the model?
• Does it decline harmful or off-topic requests?
• Is there a way for users to report a bad answer?

Cost and performance

• Token budget per request, and expected monthly cost at launch traffic?
• Latency target, and what happens when it’s missed?
• Rate limits per user?
• Caching for repeated questions?

Operations

• Model version pinned? Upgrade process written down?
• Fallback when the provider is slow or down?
• Logs and dashboards for errors, cost and quality?
• A named owner, and an on-call path for incidents?
• A way to switch the feature off quickly?