When a small model is the honest answer
A government services client sent me their inference bill last spring. One line on it read $51,000 for a single month. The job behind that line was a classifier. It read inbound support messages and sorted each one into one of eleven queues. That was the whole task. Eleven buckets, a few thousand messages a day, no generation, nothing a reader would ever see. The model doing the sorting was the largest one the vendor offered.
Nobody had run an eval. The model got picked because it was the one in the demo, the demo worked, and it shipped.
This is the default now. A team needs a model for something, so they reach for the biggest one available and call that the safe choice. The reasoning rarely gets written down, but it runs about like this: the large model can do everything the small one can and more, so going smaller only adds risk. The extra capability feels free. It is not free. You pay for it on every call, for as long as the system runs, whether or not the task ever uses it.
"We picked the biggest model because nobody gets fired for it. We never asked what the job actually needed."
VP of Engineering, mid-market client
That classifier needed maybe a tenth of the capability it was paying for. A 7B model, and in the end a fine-tuned encoder a fraction of that size, cleared the same accuracy bar on the same held-out set. The gap on the bill was about forty thousand dollars a month. The gap in output, on the eleven-bucket task that was the entire point, sat inside the noise.
The assumption under the big-model default is that capability trades off cleanly against accuracy, so more capability is always a hedge. On a bounded task that assumption is wrong in two places. A larger model is not reliably more accurate on a fixed classification or extraction job. Past a point the accuracy curve flattens, and the extra parameters buy fluency you are not using. And the spend is not a one-time hedge. It is a recurring charge that scales with traffic, so the distance between the model you chose and the model the task needed compounds every month. Forty thousand a month is just under half a million a year, on a job a smaller model does at the same accuracy. That number does not announce itself. It sits on a line in a bill nobody reads closely, because the system works, and a working system rarely gets a second look.
There is a second cost that never reaches the bill. A frontier model behind an API is hard to characterize. You do not control its version. You cannot fully enumerate its failure modes, and when the provider updates the model, its behavior on your task can move without notice. The provider can also deprecate the exact version you built on, on their schedule, not yours. For a tool that drafts marketing copy, fine. For a system of record, or anything that has to stand up an audit trail, that is the part that wakes people up at night. A smaller model you host and version is one you can pin, log, and reason about. You can hold a fixed checkpoint for years. The day the model goes live, you know exactly what it is, and you still know two years later.
There is also the question of where the model runs. A frontier model behind an API means your data leaves your boundary on every call. For a lot of our clients that is not a preference, it is a line they cannot cross. A small model fits on hardware you control. It can sit inside the network that holds the data, behind the same access controls as the system of record it reads from. That is not a cost saving. It is the difference between a system you are allowed to build and one you are not. The largest model in the world is no help if it lives somewhere the data is not permitted to go.
What we do at ByteWorks is unglamorous. Before we recommend a model, we define the task precisely enough to measure it. We build a held-out set out of the data estate you already have. Then we run the candidates against it from smallest to largest and stop at the first one that clears the bar. Most of the time the model that clears it is not the one anyone expected, and it is almost never the largest. We call that result Cost That Makes Sense. It means the smallest model that does the job, hosted where you can audit it, priced against the work instead of against the brochure.
The bar is not a leaderboard score. We set it from the task: the accuracy the current process actually achieves, or the threshold below which the downstream work breaks. A model that beats that bar is done. A model that beats it by twelve points at four times the cost is not better. It is more expensive, and we will say so.
None of this is a clever optimization. It is reading the requirement and then meeting it, which is the boring center of the discipline. The reason it keeps landing like a finding is that the step it depends on, the eval, is the step most teams skip. Building the held-out set is the slow part. It means going into the system of record, pulling a representative sample, and labeling it well enough to trust the number that comes out. That work looks like overhead right up until it saves you half a million dollars, and then it looks like the only sensible thing anyone did.
"The eval took three weeks. It also told us we'd been paying for capability we never called."
Head of Infrastructure, government-services client
The honest answer costs more upfront, and I would rather say that here than have you find it in month three. To choose the small model you have to know the task well enough to measure it. That means defining what correct looks like, assembling a set that represents the real traffic, and committing to a number before you run anything. The frontier API call lets you skip all of it. You wire it in, it returns plausible output, and you never have to write down what the system is actually for. The small-model path does not offer that exit. It needs a more senior team to set up. It costs a few weeks the API approach appears to save, and those weeks are real. It is slower to start and harder to put on a slide. The return shows up later, every month, in a place the person who made the original call is often not looking.
That is the trade. We make it on purpose, and we tell clients which side of it they are standing on before anything gets signed. The version that is slower to start and cheaper to run for a decade is the one we will defend in the room. The version that demos on Friday and bills you forever is the one we will not.
The large model is the right answer when you cannot define the question. That case is real. When it is yours, we will say so plainly and build it. But most of the time you can define the question, which means most of the time the small model is not the cautious choice or the cheap choice. It is the same answer, told honestly. The only reason it still sounds contrarian is that nobody ran the eval.