benchmarks

Where a CPU model can take over, and where it can't

We ran the same pipeline as the free report on public classification datasets and on GitHub issue routing. For each one: the share of held-out rows the CPU model answered alone, how often it matched the label when it did, and a one-sided 95% lower bound on that. A run is certified only if the bound is at least 90%. The runs that failed are listed too.

Protocol: the dataset's train split (train and test merged for prompt-injections), labels with fewer than 5 rows dropped, a uniform random sample of 3,000 rows (seed 7; all rows if fewer; Civil Comments drawn as 30% toxic and 70% ok), then the free-report pipeline unchanged: about 25% held out by a stable hash, exact repeats removed so no held-out row copies a training row, the $99 model tier. The bound is Clopper-Pearson, corrected for the number of cutoffs tried. Measured 2026-11-18. Latency is the model's median per decision on one CPU core, without HTTP.

DatasetDecisionRowsLabels AloneAccuracy95% boundCertifiedms
Bitext customer supportsupport intent2,9432795.3%98.8%97.6%yes*0.7
jailbreak-classificationjailbreak or benign1,031294.5%97.5%94.3%yes*1.1
CLINC150 (plus)assistant intent, incl. out-of-scope3,00015157.4%96.2%93.6%yes2.1
MASSIVE (en-US)assistant intent2,9955945.1%93.9%90.5%yes1.7
prompt-injectionsinjection or benign662277.0%93.5%87.1%no: too few rows0.4
Twitter financial news topicnews topic3,0002045.8%91.7%87.8%no1.5
Civil Commentstoxic or ok2,998253.1%89.5%85.6%no0.4

* Read these two with care. Bitext is synthetic, templated text where the intent word usually appears in the message, so it is mostly keyword matching. In jailbreak-classification the two classes come from different sources, words such as "ChatGPT" and "DAN" appear only in jailbreaks, and many jailbreaks are near-copies of each other; it is not a measure of real-world jailbreak detection.

Other notes: CLINC150's out-of-scope class is treated as an ordinary 151st label, so its numbers are not comparable with the published out-of-scope protocol. MASSIVE (59 of its 60 intents; one had fewer than 5 rows) and CLINC150 are short crowd-written commands where a word from the label name often appears in the text (in 38% and 57% of rows), and near-paraphrases can fall on both sides of the split. prompt-injections mixes English and German, and 68 of its injections are a benign row from the same set with an attack appended, so such pairs can straddle the split; with 124 confident answers among 161 held-out rows, 93.5% accuracy is too small a sample for the 90% bound (the report says about 324 would be needed). In the financial news set, 135 tweets repeat the same text with different links, which exact-repeat removal does not catch; several topics are close to keyword matching ("dividend" appears in 89% of Dividend rows), and all the tweets come from one mid-2022 window. Civil Comments was drawn as 30% toxic (the raw rate is 8%), with "toxic" meaning at least half the raters flagged it; a third of the toxic rows only just cross that line, so the labels themselves are noisy and the miss is label noise as much as model weakness.

A stand-in for ticket routing: each repository's most recent issues (open and closed) that carry exactly one team or area label, text = title + body (templates, code blocks and links removed), labels with at least 20 issues. Here the held-out rows are the newest 25% by creation date, so the model is tested on issues filed after everything it learned from, which is harder than a random split. Measured 2026-11-18.

RepositoryLabel familyIssuesLabels AloneAccuracy95% boundCertified
dotnet/aspnetcorearea-2,9201184.6%93.8%91.3%yes
kubernetes/kubernetessig/1,1861169.4%95.1%90.8%yes
angular/angulararea: 1,0961450.0%93.4%87.3%no
flutter/flutterteam-2,1521547.0%91.7%87.1%no
pytorch/pytorchoncall: 613386.4%91.7%85.1%no
elastic/kibanaTeam:1,6732241.3%78.6%71.1%no

For uncertified runs (here and above), "Alone" and "Accuracy" are at the strictest confidence cutoff tried; even there the bound missed 90%, so the model would not answer alone. In pytorch/pytorch 88% of the issues carry one label.

Your own decisions are the only test that counts: labels your LLM produced on your traffic can be much easier or much harder than any public set.

Download a sample and upload it on the home page; the free report runs the same pipeline. These samples were changed from the originals: sampled as described above and converted to {"text", "label"} JSONL with label names as strings. CLINC150 (source, by CLINC, CC BY 3.0), MASSIVE en-US (source, by Amazon, CC BY 4.0), Civil Comments (source, by Jigsaw and Google, CC0 1.0; contains offensive text). The others are on Hugging Face under their own licenses: Bitext (CDLA-Sharing-1.0), jailbreak-classification (Apache-2.0), prompt-injections (Apache-2.0), Twitter financial news topic (MIT). GitHub issue text belongs to its authors and is not redistributed here.

Get my free report   See an example report