Decision models vs large language models
Most AI products today send every task to a large language model, even when the answer is just "yes" or "no". This research looks at a different kind of AI, the decision model, which picks an answer instead of writing one. First I explain how the two work and where each one wins. Then I test both on a real problem with a prototype.
Think of an exam. A decision model answers a multiple-choice question: it looks at the options and marks one. A large language model (LLM) answers an essay question: it writes the answer word by word. Both can answer "Is this email spam?", but in very different ways.
It reads the question, the allowed answers and the input once, then gives every answer a score.
Used for routing, triage, tagging, moderation, guardrails. In this research: Laya.
It predicts the next small piece of text (a token), adds it, and repeats until the answer is finished.
ChatGPT, Claude, Gemini, Llama. In this research: Qwen2.5-3B.
The decision model gets the question, the allowed answers and the command as one sequence. It runs the network once and reads one score per answer from special marker positions. A softmax turns the scores into probabilities. Output length is irrelevant, so the cost is O(1) in the output.
The LLM first processes the prompt (prefill). Then it runs the network once per output token, each time reading a key-value cache that grows with the text. A 14-token JSON answer means 14 more sequential passes, so the cost is O(N) in output tokens. The JSON still has to be parsed and checked afterwards.
rm -rf /var/log/*. It got this one wrong, but it was only 72.5% sure.| Decision model | LLM | |
|---|---|---|
| Passes through the network | 1 | 1 prefill + 1 per output token |
| Output | One label from a fixed list | Free text that has to be parsed |
| Invalid output possible? | No, by construction | Yes. The 0.5B model I tried first wrote a fake terminal listing instead of JSON |
| Confidence | Softmax probability per label | A number the model writes, if asked |
| Speed (Part 2) | ≈ 10 ms | ≈ 775 ms |
| Cost per answer | Fixed: no text is generated | Grows with every input and output token |
| Can explain its answer? | No | Yes |
| Good at | Volume, latency, predictable cost | Nuance, rules that need reading, open-ended answers |
Routing a ticket, tagging content, intent detection, first-pass linting.
Explaining why a command is risky, rewriting it safely, summarising.
Agent guardrails like this one, moderation, triage with escalation.
AI agents, like coding assistants, can now run commands on real computers. One wrong command, such as rm -rf ~ or DROP TABLE users, can destroy data in a second. So before every command runs, something has to decide: is it destructive (block it) or safe (let it run)?
This is a good test for the research question. The answer is one of two labels, the check runs on every command so it has to be fast, and some commands are tricky: the risk hides in a flag like --force or a missing WHERE. I built a prototype that answers this question in three ways and compared them.
Every command is judged by the decision model. Fast, but it has to be right on its own.
Every command is judged by the LLM. This is how most products do it today.
The decision model answers first. If it is not sure enough, the LLM takes over.
The decision model is about faster than the LLM on the same Mac, and generates zero tokens.
The LLM is more accurate, especially on commands where the risk hides in a flag.
Most commands get a 10 ms answer. Only the uncertain reach the LLM, cutting LLM tokens by .
78 labeled commands: 58 I used during development, plus 20 held-out commands I wrote after finishing the prompts, to check they generalize. 32 are "hard", where the risk sits in a flag, a target or a missing clause. Each command ran 10 times through System 1 and 3 times through System 2 on an .
| Metric | Decision model | LLM only | Hybrid (90%) |
|---|
Commands per second = 1000 ms ÷ average time per command, checking one command after another on one Mac. The hybrid's average is pulled up by the 40% of commands that wait for the LLM, so it handles about 3 per second: 2.4× the LLM alone, far below the decision model alone.
The hybrid takes System 1's answer when its probability is at least the threshold, and asks the LLM otherwise. Move the slider to replay all 78 recorded commands at any threshold.
A router only works if confidence separates right answers from wrong ones. I measured this two ways. Expected Calibration Error (ECE) asks whether "90% sure" really means right 90% of the time. AUROC asks whether right answers get higher confidence than wrong ones: 0.5 means no signal, 1.0 means perfect.
| ECE (lower is better) | AUROC (higher is better) | Distinct confidence values |
|---|
On paper the LLM looks better calibrated (lower ECE). That is only because it is right 94% of the time and always says 1.0. Its confidence never changes, so it cannot tell you which answers to doubt. Its AUROC is exactly 0.5.
The decision model is overconfident: when it says 97%, it is right 81% of the time. But its probability does rank its errors (AUROC 0.64), and that ranking is what makes the hybrid work. Its weak spot is a handful of commands it is confidently wrong about, such as crontab -r and > /etc/hosts at 99% "safe". No threshold below 99% catches those.
Pick any of the 78 benchmarked commands to see what each engine answered on the M4. The hybrid uses the threshold from the slider above. To test your own commands live, run the prototype on your Mac.
Dashed examples are hard cases. Threshold in use: 90%.
The prototype runs both models locally on Apple silicon with MLX. Nothing leaves your machine. One script sets everything up and opens a dashboard where you type a command and see the decision model, the LLM and the hybrid answer it live, with latency and token counts.
git push --force origin main. The decision model is wrong but only 80% sure, so the hybrid escalates and the LLM blocks it.git clone https://github.com/USER/REPO.gitcd ~/Downloads/decide-or-generate-mainbash run.sh
If you cloned with git, cd decide-or-generate instead. The first run creates a virtual environment, installs mlx, mlx-lm and laya-mlx, and downloads the two models (~2 GB). After that it starts in seconds.http://127.0.0.1:8000. Type any shell or SQL command, move the threshold, and compare. Press Ctrl+C in the terminal to stop.PYTHONPATH=src .venv/bin/python benchmarks/run_benchmark.py
One pass vs one pass per token gave a ~75x gap on the same hardware. The decision model also never produced invalid output, while the small LLM I tried first sometimes ignored the JSON format completely.
73% overall and 53% on hard commands. It recognises obvious keywords (DROP, DELETE) but misses risk hidden in flags and targets (git push --force, mkfs, crontab -r).
A clear definition of "destructive" lifted the LLM from 78% to 95%. The same definition dropped the decision model from 72% to 59%: it wants a short question. Asking the LLM to explain its reasoning first made it slower and less accurate (88%).
Qwen2.5-0.5B labeled every command destructive (45%). Moving to the 3B model was the single largest improvement in the project.
The LLM said 1.0 every time, so it cannot drive routing. The decision model's probability is overconfident, but it ranks errors, and ranking is what a router needs.
My first run in a CPU-only Linux VM showed 1.1 s vs 19 s. Native on the M4 GPU it is 10 ms vs 775 ms. I also saw the first request after idle take 40–50 ms, because the GPU needs to warm up.
A decision model reads the input once and scores a fixed list of answers. Its cost does not depend on output length, its answer is always valid, and it returns a probability. An LLM writes its answer one token at a time, so cost grows with every token and the output is free text you have to parse.
Anywhere the answer is one of a few labels and volume is high: guardrails like this one, routing, triage, tagging, linting. Here the LLM spent about 198 tokens and 775 ms to say one word.
Use the decision model's probability as a router. If it is confident, take its answer (System 1, fast). If not, escalate to the LLM (System 2, slower, better at nuance). The confidence threshold is the dial between speed and accuracy.
rm -rf node_modules rather than force a label.