Shipping AI Features That Actually Work in Production
A field guide to building AI features that hold up outside the demo — evals, cost, latency, and the human in the loop.
Most AI demos are magic. Most AI products are a mess.
The difference isn't the model — it's the surrounding system: evals, retrieval, fallbacks, cost controls, and a very clear picture of who is accountable when the AI gets it wrong.
Start with evals, not prompts
Before you tune a prompt, write ten to fifteen input/output pairs that represent the range of what your system will actually see. These become your eval set. Every prompt change should be judged against them.
Retrieval is the whole ballgame
For most business AI use cases, the model is fine. The retrieval layer — what context you show the model — is where products live or die. Chunk boundaries, embedding choice, re-ranking, and metadata filtering all matter more than which model you use.
Design the human in the loop
Even the best AI systems get it wrong. Your product should make that graceful: confidence scores, edit-before-send flows, and one-click hand-offs to a human.
Cost and latency are product decisions
Every product decision has an AI cost. Batching, caching, streaming, and model tiering (Haiku vs Sonnet vs Opus, say) should be conscious choices — not accidents.
Ship less magic and more reliability. Users will thank you.