How to Ship an AI Feature That Behaves in Production

An AI feature that works in a demo and an AI feature that works in production are two different pieces of software. The demo pipes model output straight to the screen and looks flawless, because the person driving it asks friendly questions. Production users do not. They ask a yes or no question and get an essay. They paste emoji into every field. And eventually they screenshot the one response where the model confidently invented a refund policy you do not have.

The model did not change between the demo and the incident. The engineering around it was never built. Four layers close that gap: validation, fallbacks, cost control and evals. None of them are glamorous. All of them are the difference between a feature you can defend and a feature you apologise for.

Validate every response before a user sees it

Raw model output is untrusted input. Treat it exactly the way you treat a form submission from the open internet: it goes through checks, and if it fails any of them, it does not reach the user.

The checks are not exotic:

• Schema match. If the feature expects structured output, parse it and verify every field. A response that almost matches the schema is a failure, not a warning.
• Length bounds. A one-line answer to a question that needs detail, or ten paragraphs to a yes or no, both fail. Set a floor and a ceiling.
• Prohibited content. Screen for anything that must never appear: personal data, credentials, legal claims, competitor names, whatever your context forbids.

Piping unvalidated output to the front end works right up until it does not, and when it does not, it fails in public.

Retry with feedback, not retry blind

When a response fails validation, do not simply fire the same prompt again and hope. Append the reason for the failure to the retry: state that the previous answer broke the schema, or exceeded the length limit, and restate the constraint. Models correct themselves well when told what went wrong. Cap this at two or three attempts, then move to the fallback path. An uncapped retry loop is a bug and a bill.

Decide how it fails before you decide how it works

The failure path is a product decision, and it should be designed before launch, not improvised during an outage. Build a ladder and walk down it in order: retry with feedback first, then a simpler or cheaper model, then a cached or last-known-good answer, then an honest handoff. The one thing that never appears is a raw error or a blank screen. If the model cannot answer safely, say so in plain language and give the user a route forward.

Wrap the model call the way you would wrap any external API: one wrapper in one place, error handling on every call, backoff between retries. Providers have outages, deprecate versions and change behaviour. If provider downtime takes your whole feature down with it, you have not built a feature, you have built a thin skin over someone else's uptime.

Two items belong on the pre-deploy checklist and are cheap to verify: the fallback chain is actually configured, not just designed, and errors surface useful messages to your monitoring rather than stack traces to your users.

Cost is an architecture decision, not a billing surprise

A public AI endpoint spends your money on every request, which makes it different from every other endpoint you run. One unauthenticated route plus one scripted loop is enough to produce a bill that gets the feature switched off by finance rather than by engineering.

Put a gateway in front of the model, and enforce three things there rather than in application code:

• Authentication before inference. No valid key, no tokens burned. The request should be rejected before it ever reaches the model.
• Payload and context limits. Validate input size and schema at the gate. An oversized prompt from an anonymous user should never touch your most expensive model.
• Per-user spend tracking and caps. Tag every request with an identity, log token consumption, and enforce daily or monthly caps per tier.

With the perimeter handled, work the levers that cut the bill without cutting quality. Cache the repeated parts of your prompts: a static system prompt sent on every call is money spent on the same tokens over and over, and prompt caching serves it far cheaper. Route by complexity: a small classifier that sends simple requests to a lightweight model and reserves the heavyweight model for genuinely hard, multi-step work removes most of the waste in a typical workload. Batch anything that is not urgent, and consider semantic caching, where a query close enough in meaning to a recent one reuses the recent answer.

One warning: routing that quietly degrades answers is not a saving, it is a slower-burning incident. Measure output quality per tier on a schedule, so you know cheap requests are still getting acceptable answers.

Evals are your test suite now

Exact-match assertions do not work on non-deterministic output. The same prompt produces differently worded answers on different runs, so string comparison flakes on every build and teaches the team to ignore red pipelines. The replacement is evaluation: score outputs against criteria, and gate on the score.

• Model-as-judge in CI. A second model scores responses for accuracy, tone, safety and schema compliance against pass or fail thresholds. The pipeline fails when quality drops, exactly as it would for a broken unit test.
• A failure-driven suite. Every bad response a real user reports becomes a permanent test case with an expected score. Over time you accumulate a regression suite of genuine edge cases rather than synthetic ones, and the same mistake cannot ship twice unnoticed.
• Version comparison. Run the same inputs across prompt versions and across models, and compare scores. A prompt change becomes a measured decision instead of a feeling, and a model upgrade becomes an experiment instead of a leap.

Extend the same discipline through deployment. Add an inline cost estimate per deploy, so a prompt change that quietly inflates token usage is flagged before it ships. Then release behind a quality-gated canary: route a small slice of production traffic to the new version, watch quality and latency, and roll back automatically if the score drops. Nobody should be watching a dashboard at midnight to catch a regression a threshold could catch.

An AI reviewer in the pull-request pipeline completes the loop. Prompt it for architecture and business logic rather than style: injection vectors, unhandled edge cases, anything touching payments or data deletion. Gate merges on severity. Together with the canary you get two nets, one before merge and one after deploy.

Keep the agent architecture boring

When a feature grows into multiple agents, resist the elegant design. Start with an orchestrator: one central agent that receives the request, decides which sub-agents to call, collects their outputs and synthesises the answer. Hub and spoke is easy to reason about, easy to debug, and easy to wrap in the validation and fallback layers described above.

The alternative, agents handing work to each other in a chain with no central controller, has real uses in open-ended research workflows. It is also how teams lose weeks to circular agent calls. Migrate a specific sub-workflow to that pattern only when measurement shows the orchestrator is the bottleneck, not because the diagram looks better.

Apply the same restraint to memory. Keep a rolling buffer of recent turns, summarise older exchanges, and persist only the facts that change the agent's future behaviour. Then test it: ask the agent to reference past context, and if it hallucinates memories, fix the retrieval pipeline before adding anything else.

Get the playbook

Everything in this article is condensed into Building With AI, Safely, a free PDF covering validation, fallbacks, cost levers, evals and agent architecture as a single checklist you can run before every AI deploy. Download Building With AI, Safely and keep it next to your release process. It is part of our full engineering series, which lives at /resources.