guide
Replace your LLM router with a CPU classifier
If an LLM call in your app picks one answer from a fixed list (an intent, a tool, a queue, allow or block), most of those calls are easy, and a small model running on plain CPU can take them. The trick is knowing which ones.
A small classifier on its own is usually a few points less accurate than the LLM it replaces, and it fails worst on unusual or ambiguous inputs. Swapping it in wholesale trades a cost problem for a quality problem.
Run the small model first. When it is confident, use its answer. When it is not, call your LLM exactly as you do today. Your accuracy stays close to the LLM's, and you only pay for the LLM on the hard share.
- Collect the LLM's own decisions. Your logs already hold inputs and the answer the LLM chose. A few thousand rows is a good start; no hand labeling needed.
- Train a small model on them. Hold rows back that it never trains on.
- Pick a confidence cutoff: the lowest one where the model is still right often enough for you on the held-back rows.
- Judge it by a lower bound, not the raw accuracy. A few hundred rows can look better than the truth, and so can the best of several cutoffs. Use an exact (Clopper–Pearson) bound corrected for the cutoffs you tried, or check the chosen cutoff on fresh rows. How.
- Ship it in front of the LLM. Log what it hands off, plus a small random sample of its confident answers checked by the LLM, and retrain on those alongside your original rows.
On Banking77 (77 intents, 3,080 held-out messages) with Gemini as the LLM, a 33M-parameter model on CPU answered 77.3% of messages by itself at 95.0% accuracy (at least 94.2% with 95% confidence; cutoff chosen on separate rows beforehand) and handed the rest to the LLM. Together they scored 91.2% against 91.7% for the LLM alone, with 77% fewer LLM calls. One decision took 5.4 ms at the median and 15.3 ms at p99 (single stream, 2 threads, GCP e2-standard-4). Full method.
NoGPU runs steps 2–4 for you. Upload examples of the decisions your LLM makes (or run our log exporter), and the free report tells you, on your own data, what share a CPU model can take and how accurate it is. Note: the self-serve model is a lightweight tree classifier, not the benchmark's 33M model, gated the same way; your free report measures exactly the model you would receive. If the numbers work, $99 gets you the model and the script that runs it, offline on any CPU. Your data is deleted within 24 hours.