guide
A confidence threshold you can trust
"Use the model's answer when confidence is above 0.8" is easy to write and easy to get wrong. Here is how to choose the cutoff so the accuracy you measured is the accuracy you get.
- Trying many cutoffs on the same rows and keeping the best. Its accuracy on those rows is flattering by construction. Either choose the cutoff on one held-out set and check it once on another, or try a small fixed set of cutoffs and widen the bound to cover all of them (below).
- Trusting the raw accuracy. 99 right out of 100 sounds like 99%, but with 100 rows the true rate could plausibly be much lower. Judge by a lower bound, not the point estimate.
Out of the n held-out rows above the cutoff, the model got k right. If you compared m candidate cutoffs on these same rows (m = 1 if the cutoff was fixed beforehand on other rows), the exact (Clopper–Pearson) one-sided 95% lower bound, Bonferroni-corrected, is:
from scipy.stats import beta
def lower_bound(k: int, n: int, m: int = 1, conf: float = 0.95) -> float:
return 0.0 if k == 0 else beta.ppf((1 - conf) / m, k, n - k + 1)
lower_bound(99, 100) # 0.953 one cutoff, fixed beforehand
lower_bound(99, 100, m=3) # 0.941 best of 3 cutoffs on the same rows
lower_bound(990, 1000, m=3) # 0.981
Same 99% observed, very different promises. The bound only holds while live traffic looks like the rows you tested on, so keep sampling and re-checking.
- Hold out rows the model never trains on.
- Fix a small set of candidate cutoffs in advance (say 0.8, 0.9, 0.95), and compute each one's lower bound with m = the number of candidates.
- Take the lowest cutoff whose bound clears your target (say 90%). Lower cutoff = more traffic answered by the model. Its share, accuracy and corrected bound are numbers you can stand behind.
NoGPU's free report does exactly this for a small CPU model trained on your LLM's own decisions: three fixed cutoffs, held-out rows the model never trained on, and the corrected 95% lower bound. Full method.