guide

Answer OpenAI Decisions API calls on CPU

If your app asks a decision model to pick one option (the Vercel AI SDK's experimental_decide, which calls POST /v1/decisions), a small model trained on your own past decisions can answer the calls it is sure about, locally and offline, and refuse the rest so your app falls back exactly as it does today.

The app sends some state (text or JSON) and one or more questions. A choice question lists the allowed options; the answer is one of them, with a confidence and a probability per option. The decision model may also refuse a question, and the AI SDK then throws a Experimental_DecisionRefusalError that your code catches to call its LLM instead.

The model you download from NoGPU ships with serve.py. Run it in HTTP mode next to your app and point the AI SDK's OpenAI provider at it:

python serve.py --http 8787     # listens on 127.0.0.1 only, no auth

// your app
import { experimental_decide } from 'ai';
import { createOpenAI } from '@ai-sdk/openai';
const model = createOpenAI({ apiKey: 'any', baseURL: 'http://127.0.0.1:8787/v1' })
  .decisionModel('nogpu');

For each choice question it answers only when both hold: the model's confidence clears the cutoff in your report, and its label is one of the options offered. Anything else, including yes/no and score questions, is a refusal. Setting OPENAI_BASE_URL instead works too, but it also sends your app's other OpenAI calls to this server, which only answers /v1/decisions.

The model only knows inputs shaped like its training rows. Text parts become the text it reads, and top-level JSON keys named like a training field fill that field. Any other JSON is turned into one string, which a model trained on plain text has never seen, so expect it to hand those calls back to your LLM. So train on the same text your app sends: if you call it with the email body, train on email bodies.

With ai 7.0.165 and @ai-sdk/openai 4.0.111 against the Banking77 example model (cutoff 0.8): a clear question with its intent among the options came back as that intent; the same question without its intent among the options, a nonsense string, and a yes/no question each came back as a refusal the SDK raised for the caller to catch. The model's own numbers are in its report: on held-out rows it answered 66.7% alone, 94.1% correct when it did (at least 91.4% with 95% confidence). We have not measured HTTP latency.

Upload a few thousand of your LLM's past decisions as JSONL ({"text": ..., "label": ...}) and the free report tells you, on your own data, what share a CPU model can answer alone and how often it is right. If the numbers work, $99 one-time gets you the model and serve.py. You can also download the free Banking77 sample from the example page and try --http first. Your data is deleted within 24 hours (24 hours after purchase if you buy).

Get my free report   See an example report