benchmarks
Where a CPU model can take over, and where it can't
We ran the same pipeline as the free report on public classification datasets and on GitHub issue routing. For each one: the share of held-out rows the CPU model answered alone, how often it matched the label when it did, and a one-sided 95% lower bound on that. A run is certified only if the bound is at least 90%. The runs that failed are listed too.
Protocol: the dataset's train split (train and test merged for prompt-injections), labels with fewer than 5 rows dropped, a uniform random sample of 3,000 rows (seed 7; all rows if fewer; Civil Comments drawn as 30% toxic and 70% ok), then the free-report pipeline unchanged: about 25% held out by a stable hash, exact repeats removed so no held-out row copies a training row, the $99 model tier. The bound is Clopper-Pearson, corrected for the number of cutoffs tried. Measured 2026-11-18. Latency is the model's median per decision on one CPU core, without HTTP.
| Dataset | Decision | Rows | Labels | Alone | Accuracy | 95% bound | Certified | ms |
|---|---|---|---|---|---|---|---|---|
| Bitext customer support | support intent | 2,943 | 27 | 95.3% | 98.8% | 97.6% | yes* | 0.7 |
| jailbreak-classification | jailbreak or benign | 1,031 | 2 | 94.5% | 97.5% | 94.3% | yes* | 1.1 |
| CLINC150 (plus) | assistant intent, incl. out-of-scope | 3,000 | 151 | 57.4% | 96.2% | 93.6% | yes | 2.1 |
| MASSIVE (en-US) | assistant intent | 2,995 | 59 | 45.1% | 93.9% | 90.5% | yes | 1.7 |
| prompt-injections | injection or benign | 662 | 2 | 77.0% | 93.5% | 87.1% | no: too few rows | 0.4 |
| Twitter financial news topic | news topic | 3,000 | 20 | 45.8% | 91.7% | 87.8% | no | 1.5 |
| Civil Comments | toxic or ok | 2,998 | 2 | 53.1% | 89.5% | 85.6% | no | 0.4 |
* Read these two with care. Bitext is synthetic, templated text where the intent word usually appears in the message, so it is mostly keyword matching. In jailbreak-classification the two classes come from different sources, words such as "ChatGPT" and "DAN" appear only in jailbreaks, and many jailbreaks are near-copies of each other; it is not a measure of real-world jailbreak detection.
Other notes: CLINC150's out-of-scope class is treated as an ordinary 151st label, so its numbers are not comparable with the published out-of-scope protocol. MASSIVE (59 of its 60 intents; one had fewer than 5 rows) and CLINC150 are short crowd-written commands where a word from the label name often appears in the text (in 38% and 57% of rows), and near-paraphrases can fall on both sides of the split. prompt-injections mixes English and German, and 68 of its injections are a benign row from the same set with an attack appended, so such pairs can straddle the split; with 124 confident answers among 161 held-out rows, 93.5% accuracy is too small a sample for the 90% bound (the report says about 324 would be needed). In the financial news set, 135 tweets repeat the same text with different links, which exact-repeat removal does not catch; several topics are close to keyword matching ("dividend" appears in 89% of Dividend rows), and all the tweets come from one mid-2022 window. Civil Comments was drawn as 30% toxic (the raw rate is 8%), with "toxic" meaning at least half the raters flagged it; a third of the toxic rows only just cross that line, so the labels themselves are noisy and the miss is label noise as much as model weakness.
A stand-in for ticket routing: each repository's most recent issues (open and closed) that carry exactly one team or area label, text = title + body (templates, code blocks and links removed), labels with at least 20 issues. Here the held-out rows are the newest 25% by creation date, so the model is tested on issues filed after everything it learned from, which is harder than a random split. Measured 2026-11-18.
| Repository | Label family | Issues | Labels | Alone | Accuracy | 95% bound | Certified |
|---|---|---|---|---|---|---|---|
| dotnet/aspnetcore | area- | 2,920 | 11 | 84.6% | 93.8% | 91.3% | yes |
| kubernetes/kubernetes | sig/ | 1,186 | 11 | 69.4% | 95.1% | 90.8% | yes |
| angular/angular | area: | 1,096 | 14 | 50.0% | 93.4% | 87.3% | no |
| flutter/flutter | team- | 2,152 | 15 | 47.0% | 91.7% | 87.1% | no |
| pytorch/pytorch | oncall: | 613 | 3 | 86.4% | 91.7% | 85.1% | no |
| elastic/kibana | Team: | 1,673 | 22 | 41.3% | 78.6% | 71.1% | no |
For uncertified runs (here and above), "Alone" and "Accuracy" are at the strictest confidence cutoff tried; even there the bound missed 90%, so the model would not answer alone. In pytorch/pytorch 88% of the issues carry one label.
- All three intent sets here certified (as did Banking77 in the example): the CPU model took 45% and 57% of decisions alone on the two crowd-written sets, and 95% on synthetic Bitext, with the rest going back to the LLM.
- Toxicity and financial-news topics did not certify at 3,000 rows: accuracy on the rows the model was sure about stayed under about 92%.
- Small sets struggle to certify even when accuracy is high: prompt-injections (662 rows) reached 93.5% but its bound stayed at 87.1%. Your own report says how many rows would clear it.
- Two of six GitHub routing sets certified when tested on newer issues than it learned from.
Your own decisions are the only test that counts: labels your LLM produced on your traffic can be much easier or much harder than any public set.
Download a sample and upload it on the home page; the free report runs the same
pipeline. These samples were changed from the originals: sampled as described above and converted to
{"text", "label"} JSONL with label names as strings.
CLINC150
(source, by CLINC, CC BY 3.0),
MASSIVE en-US
(source, by Amazon, CC BY 4.0),
Civil Comments
(source, by Jigsaw and Google, CC0 1.0; contains offensive text).
The others are on Hugging Face under their own licenses:
Bitext (CDLA-Sharing-1.0),
jailbreak-classification (Apache-2.0),
prompt-injections (Apache-2.0),
Twitter financial news topic (MIT).
GitHub issue text belongs to its authors and is not redistributed here.