Small Language Models & Right-Sized AI

A logistics client asked us last autumn to move a document classifier from one 7B model to another. Same task, same accuracy bar, a newer base with better throughput on the hardware they'd already bought. We scoped it at two weeks. It took nine.

Nothing exotic went wrong. The model just wasn't a component. It had grown into the codebase over eighteen months, and by the time we came to lift it out, forty or so places in the application knew something specific about it.

I've written before that the smallest model which clears the bar is usually the honest answer. That post covered how you pick it. This one covers the part that comes after, and it's the part that decides whether right-sizing holds up or quietly reverses over three years. If swapping the model is a project, you'll only do it when something forces you to. And then you're back where you started: running the model you happened to choose, because changing it costs more than living with it.

What the nine weeks were actually spent on

Very little of it was inference code. The tally looked like this.

Prompts lived inside application source, in a dozen files, each one tuned to a particular model's habits over months of small fixes nobody wrote down. The output parser had been shaped around one model's formatting quirks: the stray sentence before the JSON, the markdown fence it sometimes added, the way it rendered nulls. Chunking assumed a context window. Retry logic and timeouts assumed a latency profile. A fine-tuned adapter was pinned to a base checkpoint, which is correct, except the training data that produced it was sitting in a notebook on a laptop.

And there was no eval set. That's the one that cost the weeks. Without a held-out set, "is the new model as good as the old one" isn't a question you can answer, so it becomes a question you argue about. Four people spot-checked a few hundred documents and disagreed. The team ended up building the eval during the migration, under time pressure, which is the worst moment to decide what correct means.

"We didn't have a model in production. We had a model grown into production. Those turned out to be different things."

Head of Platform, logistics client

None of those forty touchpoints was a mistake when it was made. Each was a reasonable local fix. The failure is architectural, and it only becomes visible on the day you try to change one thing.

The misconception is that the model is the system

Teams treat the model as the core of the AI system and the surrounding code as plumbing. It's the other way round. The model is the most replaceable part you have, and the one most likely to be replaced. Open weights improve on a cycle measured in months. Hardware you already own gets faster per token. Providers deprecate versions on their schedule. Your own traffic shifts and the task you sized for in January isn't quite the task by December.

Everything around the model has a longer life than the model does. The task definition, the eval set, the labeling guidance, the audit log, the integration into the system of record: those outlive three or four models. They're the durable assets. The weights are the disposable part.

That inversion is easy to say and hard to build, because during development the model feels like the hard bit. It's the thing that's new, the thing being demoed, the thing everybody is looking at. The plumbing feels like a detail. Then the model becomes the only piece anyone wants to change and the plumbing turns out to be the thing holding it in place.

What we build instead

Four things, none of them clever.

A task contract. Input schema, output schema, and a written accuracy bar taken from the process the system replaces. It sits in the repository, versioned, and it's the definition every model gets measured against. If two engineers would label the same record differently, the contract isn't finished.

The eval set as a first-class asset. Pulled from the client's own data estate, labeled well enough to trust, with a documented refresh cadence. It lives in version control next to the code. Any candidate model runs against it before it goes near production, and the result is a number rather than an opinion. This is the slow part. It's also the part that turns a nine-week migration into a two-day one.

One boundary. A single interface the application talks to, expressed in the contract's terms. Prompts are data behind that boundary, versioned and addressable, not string literals in business logic. Output normalization happens there too, so the application never learns that a particular model likes to explain itself before answering. When we did this for the logistics client, the second swap took two days.

A log that answers the audit question. Every call records the model identity and version, the prompt version, the contract version, and the output. When somebody asks in eleven months why a specific record was classified the way it was, that's a query rather than an investigation. For clients in regulated work this isn't a nice-to-have, it's the reason the system is allowed to exist. It also gives you shadow-running: point a candidate model at live traffic, log both answers, compare them against the contract before anyone commits.

"The audit trail was compliance work. It's also the only reason we could prove the new model behaved like the old one."

CTO, insurance client

The trade

You give up the model's best features. Provider-native tool calling, prompt caching, structured output modes, the specific things one vendor does better than the rest: if your boundary is honest, you can't lean on any of them without either leaking them through or writing per-model adapters that reintroduce the coupling you just removed. You end up somewhere near the common ground of your candidate set. On some workloads that costs you real performance, and we'll tell you where.

It's slower at the start. Building the eval set before the first model goes live means weeks of labeling work in front of a system that doesn't exist yet, and it's difficult to defend in a room that watched a prototype work last Friday. The indirection costs you too. There's one more layer between the request and the answer, and one more place to look when something's wrong.

And it's ongoing. The eval set decays as traffic drifts. Somebody has to own it and refresh it, and if nobody does, in two years you have a stale number and false confidence, which is worse than no number.

That's the price. We pay it because the alternative isn't avoiding the cost, it's deferring it to the migration, at a worse moment, with the answer already forced.

The point

Right-sizing isn't a decision you make once. It's a decision you have to be able to make again, cheaply, every time the ground moves. A system where the model is one pinned, measured, replaceable component stays right-sized for a decade. A system where the model has grown into forty files was right-sized exactly once, on the day it shipped, and has been drifting ever since.

The model you pick matters less than whether you can pick again.