What is distillation?

Imagine hiring a world-class consultant to answer the same five questions a thousand times a day. Expensive, slow, overkill. What if you could record everything that consultant said, train a local junior on those answers, and have the junior handle 95% of those calls — at a fraction of the cost and with no per-call fee?

That's distillation. A big, expensive, frontier AI model (the learner) generates answers for a narrow task. A much smaller model (the student, or here, the Spirit) learns to mimic those answers. After training, you keep the small one and shut off the expensive one. No more API calls, no more per-token bills, no more rate limits.

Why does this matter for your business?

Three reasons it should be on your radar:

Cost. Frontier AI calls cost real money. A workflow that makes six AI calls per request, a thousand times a day, racks up a four-figure monthly bill quickly. If five of those six calls are narrow and repetitive, distillation can take that workflow from $1,200/month to $30/month — without changing the user-facing behavior.
Latency. A small local model responds in milliseconds. A frontier API call is hundreds of milliseconds at best, sometimes seconds. If your product feels sluggish because you're waiting on AI, distilling the slow steps fixes the symptom.
Ownership. A distilled model lives on your servers. No vendor can change pricing, deprecate the model, throttle you, or read your traffic. For sensitive workloads — health, legal, federal — this matters more than cost.

When is distillation a good idea?

Distillation works well when the task is narrow and repetitive. "Extract the company name from this email" — great. "Write me a thoughtful essay" — bad. The narrower the task, the cheaper and better the resulting Spirit.

A useful rule of thumb: if you can imagine writing the rules for the task in a 5-page document, distillation will probably work. If the task requires open-ended reasoning across many domains, keep the frontier model in the loop.

How do you evaluate a distilled model?

Don't just look at the headline accuracy number. Ask:

Where does it fail? A 90% accurate model that fails quietly on the 10% you care most about is worse than a 70% model that loudly says "I don't know."
What's the held-out evaluation? A model evaluated only on its training data is meaningless. You want a clean test set — what Distillarium calls "Proof" — that the model never saw during training.
Does it generalize? If your real traffic looks different from the training data, the model won't translate. Always evaluate on a sample of your actual production traffic before shipping.
Who's accountable when it's wrong? Distilled models are silent on out-of-distribution inputs. Build a fallback path to the learner (or to a human) for low-confidence outputs.

Questions to ask a vendor pitching distillation

"What learner model did you distill from? When was it last updated?"
"How big is the held-out evaluation set, and what does it look like?"
"Show me the failure cases — not the wins."
"Does this model run on my infrastructure, or yours?"
"What's the plan when the underlying capability shifts?"

A vendor who can't answer these is selling you a black box, not a distilled model.

A short walk through Distillarium

Distillarium is the open-source toolkit that makes the above practical. It uses whiskey-distillery vocabulary on purpose: distillation has needed names for these concepts for decades, and the people who actually distill things have been using them for centuries.

Mash = the seed corpus. The raw inputs your learner will respond to.
The Still = the training run that turns learner behavior into student knowledge.
Cuts = how the data is split into training, validation, and held-out evaluation.
Proof = the held-out accuracy. The number that actually matters.
Tasting Notes = the auto-generated evaluation report. Strengths, weaknesses, failure cases. No hiding.
Spirit = the finished, deployable model. Yours to keep.

Ready to see one in action? Visit the Cellar to inspect the bottled Spirits, or switch to Workbench mode for the code, recipes, and notebooks. If you're evaluating distillation for a specific workflow, get in touch — Automate Capture builds custom Spirits as part of broader AI engagements.