Right now in Podtastic, an LLM is doing the topic classification. I’m sending audio transcript chunks to Gemini, asking it which topic each chunk is about, and showing the user the result. It works. It also costs real money at scale, runs on someone else’s servers, and depends on a third-party pricing decision I don’t control.
It’s also the piece of the stack I’m planning to throw away within twelve months.
The short version of why: frontier LLMs are the cheapest possible way to prove a narrow ML feature should exist. They’re scaffolding. The product is the small model that eventually replaces them, ideally running on the user’s own device.
The scaffolding argument
Five years ago, shipping a machine-learning-backed feature in an app meant six months of dataset collection before you could even prove anyone wanted the feature. You needed labelled examples. You needed enough of them. You needed to handle edge cases. You spent the first half of the project arguing with annotation contractors about taxonomy and never got to actually asking whether users wanted it.
A frontier LLM collapses that pre-work to nothing. You write a prompt, run it against a thousand inputs, and ship a v1 inside a week. If users use it, you keep going. If they don’t, you’ve cost yourself a hackathon and not a product roadmap.
The order of operations is now inverted. Validate first, specialise later. Karpathy calls something close to this Software 3.0, where LLMs become the foundational primitive on top of which everything else is built. I’m with him on the foundation. This post is about what gets built on top.
Why I’m planning to graduate
For a narrow, repeating, well-defined task like topic classification, I don’t think the LLM is the long-term answer. Four reasons, in roughly the order of how much weight I give them.
Self-sovereignty. I don’t want my product’s core intelligence depending on a pricing decision made at a company I don’t work for. The LLM stack is built on infrastructure I don’t control, run by entities whose interests will diverge from mine eventually. For a feature that’s central to what the app does, I want to own the thing that produces it.
Privacy and data sovereignty. Topic classification on a podcast means I’m sending the user’s listening data to a third party. The user almost certainly didn’t sign up for that when they installed the app. If the same classification can happen on the user’s phone with no network call required, that’s the right shape for the product.
Cost at scale. Claude Haiku 4.5 is around $1 per million input tokens and $5 per million output. Cheap. But for genuinely good classification quality I tend to need Sonnet or Opus, which is meaningfully more expensive per call. Multiply by users, then episodes, then chunks per episode, and the marginal cost stops being negligible. A small custom model trained on the same task is, after a one-time training run, basically free per inference.
Latency. A round-trip to a hosted LLM is hundreds of milliseconds even on a good connection. A small on-device model is single-digit milliseconds. For UI that wants to feel reactive, that gap matters.
Stack the four together and the framing is: I want to build the best version of the feature, not the easiest one to ship.
Where this argument might be wrong
Worth being honest about the counterargument, because it has weight.
Frontier model costs keep dropping. Haiku 4.5 already does some tasks that needed Sonnet last year. By 2028, the cost of a Claude API call might be low enough that the savings from running an on-device model don’t pay back the engineering time it takes to ship one. If that happens, the calculus changes.
On-device shipping has its own engineering tax. You’re quantising the model. You’re shipping per-platform builds. You’re managing fallbacks for users on old phones. The cloud version of distillation, where you train a smaller Sonnet-equivalent and keep running it in the cloud, is much less work than the on-device version and captures most of the cost savings.
I might also be wrong about which tasks need the bigger LLMs to classify well in the first place. Anthropic and Google are pushing smaller models toward parity. If they get there before I ship my own model, my argument collapses to “I trained a custom classifier I didn’t need to.”
I’m making a bet, not stating a foregone conclusion. The bet is that the privacy and self-sovereignty reasons will keep outweighing the convenience of the API call for the kind of product I’m building. We’ll see.
What actually makes this work
The piece of this pipeline that gets undersold in most distillation pieces: your LLM-trained dataset is only as good as the labels.
If you point Claude at a million chunks, classify them all, and call that your training set, you’ve shipped Claude’s biases and confidence intervals into a model that pretends to be your own. The hallucinations Claude has on edge cases become baked-in classifier errors. The dataset isn’t yours. You’ve distilled the teacher’s flaws as faithfully as its strengths.
The fix is to build user feedback into the product from day one. Every time the classifier surfaces a topic to the user, the user is implicitly or explicitly validating that classification just by using the feature. A topic they tap into is a label they’ve confirmed. A topic they ignore is a soft negative signal. A topic they actively correct is a strong positive on the corrected label.
That feedback stream is the dataset. Not what Claude said. What Claude said and the user agreed with, separated from what Claude said and the user didn’t.
The product is the labelling tool. The users are the annotators. Your job is to instrument the product so the feedback loop closes on every interaction, then sit on the data for long enough to train a real classifier against it.
What’s missing from most “just distil into a small model” advice is exactly this. The distillation pipeline assumes a clean dataset. The clean dataset doesn’t exist until you’ve shipped the LLM-backed v1 long enough to collect it.
Closing
The interesting skill in 2026 isn’t building either the frontier model or the narrow one. Both of those problems have well-paid teams at well-funded companies solving them. The interesting skill is choosing when to scaffold with the general intelligence and when to commit to specialising. Knowing which features in your product need to graduate and which can stay on the API call forever. Wiring the feedback loops so the data flywheel actually spins.
If you’re building an AI-shaped product feature in 2026 and you’re treating the frontier LLM as the destination, you’re losing ground to whoever’s treating it as scaffolding. The work this year is the workflow between them.