Which Model Belongs Where in Your Product?

• Which model should we actually use?
• How do we stop it making things up?
• How do we know an upgrade didn’t break it?
• What will this cost us at scale?
• Should we just build it ourselves?

An API key and a prompt box will get you a working feature for about a week. After that the questions start arriving, usually from whoever owns the customer relationship or the cloud bill. The questions below are the ones that come up most often when a team moves from a demo to something they have to stand behind.

§Which model should we actually use?

This is the first question and usually the worst-answered one. Teams pick on reputation, or on whatever a colleague tried last month, then discover they are paying premium rates for a task a smaller model handles fine, or that latency has quietly killed the user experience.

A good answer depends on your task, not on a leaderboard. Accuracy on your own inputs, time to first token, cost per call at your real volume, and where the data is allowed to sit all pull in different directions. The honest answer is usually a mix, and it changes as models are updated underneath you.

§How do we stop it making things up?

Hallucination is the question every stakeholder asks, and the one that sinks the most trust. A model answering from its training data will produce confident fiction the moment someone asks about your pricing, your policy, or your product’s edge cases.

What matters here is whether the model is answering from your content or from memory, and whether a user can see where an answer came from. That changes the architecture, the data work and the testing, which is why it is a decision to make early rather than a patch to apply later.

§How do we know an upgrade didn’t break it?

Most teams have no way to tell. Someone edits a prompt or a provider ships a new model version, and behaviour shifts in ways nobody notices until a customer complains. The feature still returns text, so nothing looks broken.

A good answer depends on having something to compare against: a set of real cases with expected behaviour, run whenever anything changes. Without it, every improvement is a gamble, and you cannot tell drift from progress.

§What will this cost us at scale?

Inference bills do not spike dramatically; they creep. Usage grows, prompts get longer, someone adds a feature that calls the model twice, and the monthly figure drifts upward with no single moment to point at.

The useful answer is not a headline price but a per-feature view of what each call costs and what happens when a provider is slow or down. Caching, routing and fallbacks all sit in this space, and they are far cheaper to design in than to retrofit.

§Should we just build it ourselves?

Plenty of teams can wire an endpoint. Fewer can say, months later, why an answer changed, what it cost, and which parts of the system belong to them. The gap between those two positions is the whole subject, and it is worth being clear-eyed about which side of it you are on.

If you want the working detail behind these answers, the guide to LLM integration services for London teams covers model benchmarking, RAG grounding, evals, guardrails and cost controls in full.