for the engineers who want the details
How it works
NoGPU trains a small model on the answers your LLM already gave, measures it on rows it never saw, and sets a confidence gate: the model answers only when it is sure, and everything else goes to your LLM as before.
How we measured this
Benchmark numbers load from our measured E2a runs.
- The model your report measured: a fast gradient-boosted tree model (LightGBM) on your fields and text, yours to keep.
- The confidence gate, set so the model answers only above the bar your report certified.
serve.py: reads one question per line and answers, offline, with open-source libraries only.- The report it was built from, so anyone on your team can check the numbers.
$ echo '{"text": "where is my card"}' | python serve.py
{"label": "card_arrival", "confidence": 0.97, "gate": "answer"}
$ echo '{"text": "two cards, one dispute, wrong name"}' | python serve.py
{"label": null, "confidence": 0.54, "gate": "cascade"} # ask your LLM
# or in Python:
from serve import load, decide
model = load()
out = decide(*model, {"text": message})
label = out["label"] if out["gate"] == "answer" else ask_your_llm(message)Logs in OpenAI, LiteLLM, Langfuse, Portkey or Helicone?
Convert them on your own machine first. The script uses only the Python standard library, sends nothing anywhere, and keeps just the last thing the user asked and the answer your model gave. System prompts and earlier messages are dropped.
curl -sO https://nogpu.dev/static/nogpu_export.py
python3 nogpu_export.py your_logs.jsonl > nogpu.jsonl
# custom shape? point at the fields:
python3 nogpu_export.py logs.json --input-field input.query --label-field output.routeWhat does "at least, with 95% confidence" mean?
We test the model on rows it never saw while training. On the benchmark, when it answered, it was right 95.0% of the time. The lower figure (94.2%) comes from a standard statistical test (Clopper–Pearson, one-sided, 95% confidence): the true rate is at least that high, as long as your live traffic looks like the data you uploaded. Your report shows the same numbers for your data.
Which LLMs can it learn from?
Any of them: a frontier API model, your own fine-tuned model, or answers your team reviewed
by hand. If your logs include the model's probabilities as teacher_probs, the
report also shows how the LLM did on the questions the CPU model handed back.
What hardware does it need?
A normal CPU. The encoder benchmark above ran on a 4-core Google Cloud server ($0.13 an hour) using 2 threads, at about 5 ms per answer. The tree model in the self-serve download is lighter still; your report shows its speed on our server.