guide
Answer OpenAI Decisions API calls on CPU
If your app asks a decision model to pick one option (the Vercel AI SDK's
experimental_decide, which calls POST /v1/decisions), a small model trained
on your own past decisions can answer the calls it is sure about, locally and offline, and refuse
the rest so your app falls back exactly as it does today.
The app sends some state (text or JSON) and one or more questions. A choice question
lists the allowed options; the answer is one of them, with a confidence and a probability per
option. The decision model may also refuse a question, and the AI SDK then throws a
Experimental_DecisionRefusalError that your code catches to call its LLM instead.
The model you download from NoGPU ships with serve.py. Run it in HTTP mode next to
your app and point the AI SDK's OpenAI provider at it:
python serve.py --http 8787 # listens on 127.0.0.1 only, no auth
// your app
import { experimental_decide } from 'ai';
import { createOpenAI } from '@ai-sdk/openai';
const model = createOpenAI({ apiKey: 'any', baseURL: 'http://127.0.0.1:8787/v1' })
.decisionModel('nogpu');
For each choice question it answers only when both hold: the model's confidence clears the
cutoff in your report, and its label is one of the options offered. Anything else, including
yes/no and score questions, is a refusal. Setting OPENAI_BASE_URL instead works too,
but it also sends your app's other OpenAI calls to this server, which only answers
/v1/decisions.
The model only knows inputs shaped like its training rows. Text parts become the text it reads, and top-level JSON keys named like a training field fill that field. Any other JSON is turned into one string, which a model trained on plain text has never seen, so expect it to hand those calls back to your LLM. So train on the same text your app sends: if you call it with the email body, train on email bodies.
With ai 7.0.165 and @ai-sdk/openai 4.0.111 against the
Banking77 example model (cutoff 0.8): a clear question with its intent among
the options came back as that intent; the same question without its intent among the options,
a nonsense string, and a yes/no question each came back as a refusal the SDK raised for the
caller to catch. The model's own numbers are in its report: on held-out rows it answered 66.7%
alone, 94.1% correct when it did (at least 91.4% with 95% confidence). We have not measured
HTTP latency.
Upload a few thousand of your LLM's past decisions as JSONL ({"text": ..., "label": ...})
and the free report tells you, on your own data, what share a CPU model can answer alone and
how often it is right. If the numbers work, $99 one-time gets you the model and
serve.py. You can also download the free Banking77 sample from the
example page and try --http first. Your data is deleted within 24 hours (24 hours after purchase if you buy).