guide
Put a CPU classifier in front of your LLM calls
The pattern: a small model answers the decisions it is sure of in a few milliseconds, and everything else goes to your LLM exactly as today. Here is the whole integration.
This uses LiteLLM, but any client works the same way: only the hand-off line changes.
import sys
sys.path.insert(0, "/path/to/nogpu") # absolute path to the unpacked bundle
import serve
from litellm import completion
meta, booster, vecs, threshold = serve.load() # once, at startup
def route(text: str) -> str:
d = serve.decide(meta, booster, vecs, threshold, {"text": text})
if d["gate"] == "answer": # sure: answered on CPU, no LLM call
return d["label"]
# not sure: your existing LLM call, unchanged
r = completion(model="gpt-4o-mini", messages=[
{"role": "system", "content": ROUTER_PROMPT},
{"role": "user", "content": text}])
return r.choices[0].message.content
Pass the same fields you uploaded: {"text": ...} if you trained on text,
{"inputs": {...}} if you trained on structured fields. Empty input always goes to your LLM.
Batch jobs can skip Python entirely: cat rows.jsonl | python serve.py prints one
decision per line, and rows marked "cascade" are the ones to send to your LLM.
Keep logging the LLM's answers on the hand-offs, plus a small random sample of the CPU model's confident answers checked by the LLM. Those rows are what the next version trains on, and the sample tells you early if your traffic has drifted away from what the model learned.
Upload examples of the decisions your LLM already makes (or run the
log exporter on your logs). The free report shows, on your own
data, what share the CPU model can take and how often it is right. If the numbers work, $99 gets you
the nogpu folder used above: the model, serve.py, and the measured confidence
cutoff. It runs offline on any CPU. Your upload is deleted within 24 hours.