Illustration by tuput
English
Google DeepMind says Gemini Decision 1 takes one input and up to 1,000 questions and returns a probability for every option in 150 to 250 milliseconds, at $0.10 per million input tokens. Artificial Analysis, an independent tester, has ranked this class of model since 13 October 2026, and its second version scores whether the stated confidence is honest.
Google DeepMind introduced Gemini Decision 1 and the Gemini Decisions API on 11 November 2026. The model does not write an answer word by word. In Google’s description it “scores your candidate options in a single forward pass” and returns a probability for each one. A request can carry up to 1,000 independent questions about a shared input of text, images, audio, video or PDFs of up to 256K tokens. Google lists the end-to-end time at 150 to 250 milliseconds and the price at $0.10 per million input tokens, with no output tokens billed.
Gemini Decision 1 belongs to a class that the testing firm Artificial Analysis calls decision models. Since 13 October 2026 the firm has run a public index for them, and its second version, published on 8 November, scores whether a model’s answer is right and also whether its stated confidence can be trusted. What follows sets out what these models do, what the index measures, and which numbers come from the makers.
A multiple-choice test that returns odds
Artificial Analysis explains the idea with a customer message: “Help! My payouts have been failing for 3 days and my rent is due tomorrow.” A decision model is handed three questions about it, which team should get the message (billing, technical or account), whether it is urgent (yes or no) and how upset the writer is (calm, frustrated or angry). It answers all three in one call, with a probability for every option. A language model given the same task writes a structured reply that names one answer to each question, with no probabilities attached. For how those language models are built and why they write one word at a time, see how a large language model is trained.
Google’s API supports three kinds of question. A yes/no question returns the probability that a condition holds. A choice question takes 2 to 255 options and returns the winner plus a normalised probability for every candidate. A score question grades the input against an ordered rubric and returns a weighted score and a probability for each level. Every answer carries a confidence value from 0 to 1, “so your code can route ambiguous cases to a fallback or human reviewer,” in the post’s words.
Google names its own uses. Gemini Notebook uses the API as a router that sends each query to a model tier and a thinking level, which Google says serves everyday queries with up to 40% lower latency. The post says the 150 to 250 millisecond figure holds for contexts up to 16K tokens. It also says the shared input is billed once per request however many questions are asked, with each question adding only its own instruction and option text.
The same idea from other makers
Gemini Decision 1 arrived into a field with several entrants. Liquid AI introduced its d1 on 5 October 2026 as “our first decision model”, now supporting text and images. Liquid says it reads the questions in one forward pass, returns probabilities without generating any tokens, and takes 200 to 300 milliseconds for a text decision. On 16 November it released d1.2 and d1.2-ultra, which add audio input and five new tasks, such as reading a value off an invoice, finding every piece of personal data in a note, or drawing a box around a signature on a scanned form.
On 11 November, the same day as Google’s launch, Subquadratic released SubQ Snap 1.2. The company prices it at $0.04 per million input tokens, says it handles inputs up to 256K tokens, and published LongDecisionBench, nine long-document tests that it describes as open and says it will put on GitHub and Hugging Face shortly.
Academic work had already begun by then. A paper by Delip Rao and Chris Callison-Burch, first submitted on 24 September 2026, compares two decision models, Jev and Perplexity’s Decider, with language models used as judges of written answers.
Why a probability is the point
A probability is useful only if it is honest. When a model says 80%, it should be right about 80 times in 100. Researchers call this calibration. Guo and colleagues reported at the ICML conference in 2017 that modern neural networks, unlike those of a decade earlier, are poorly calibrated, and that temperature scaling, a one-number adjustment, was surprisingly effective at fixing it on most of the datasets they tried. Kadavath and colleagues reported in 2022 that larger language models are well calibrated on multiple-choice and true/false questions “when they are provided in the right format”.
Artificial Analysis states the practical use: a decision model’s answer is typically acted on when its confidence clears a threshold, and otherwise passed to a person or a more capable model. Google describes the same pattern. The setup works only if a “90% sure” label means something close to 90%.
What the independent index measures
Artificial Analysis launched its Decision Index v1.0 on 13 October 2026 with four evaluations, each weighted 25%: AA-Omniscience-Decision (500 private questions, each with four answers and an “I don’t know” option), HLE-MultiChoice (the 513 text-only multiple-choice questions of Humanity’s Last Exam), When2Call (NVIDIA’s test of whether a model should answer, call a tool, ask for more information or decline) and WildGuardTest (a safety classification set from Ai2). Jev 1.13 led with 58, ahead of GLiDE and Microsoft-Decision-1 on 54. Costs ran from $17 to $199 per million tasks and median times from 0.06 to 0.5 seconds per task.
The v1.0 article also reported that every decision model was overconfident. Across almost all probability levels, accuracy fell below the probability the model gave, a gap Artificial Analysis traced mainly to the knowledge and reasoning tests. On tool calling and classification, most were relatively well calibrated. The “I don’t know” option connects to the problem described in our explainer on AI hallucination: the index counts how often a model that is wrong says it does not know instead of guessing.
Version 2, published on 8 November, moved 45% of the index onto private datasets, up from 25%, to guard against models being tuned to public tests. It added a Decision Score. A right answer scores between 50 and 100 and a wrong one between 50 and 0, and confidence moves the score within each half. Across an evaluation the score equals 50 times (1 plus accuracy minus the Brier score), where the Brier score is the average squared gap between the probability given and whether the answer was right, and 0 means certain and always right.
The firm gives one example of the effect. On its new AA-Paperclip test, a private set of 1,500 resumes, receipts, invoices and research papers, Clef passes 36% of documents and Celeris-1.1 Decision passes 24%, but the two match on Decision Score because Celeris-1.1 Decision has the better Brier score, 0.15 against 0.27.
Version 2 also showed that list prices do not settle the cost. Cost per million tasks across decision models varied by more than 60 times, from $14 for Celeris-1.1 Decision to $859 for Clef. Artificial Analysis says most decision models, including Drex Max v1.1 and Decider v1.2, bill a document’s input once for all its questions, while GLiDE v1.2 bills it for every question. That made GLiDE v1.2 cost $780 against $102 for Drex Max v1.1, although its list price was only 33% higher.
Does the answer change when the options are shuffled?
A known weakness of language models is sensitivity to order. Pezeshkpour and Hruschka reported in August 2023 a performance gap of approximately 13% to 75% across benchmarks when answer options were reordered. Wang and colleagues reported in May 2023 that when a language model judges two answers, the ranking can be skewed by changing the order in which they appear. An example in their abstract has Vicuna-13B beating ChatGPT on 66 of 80 test queries with ChatGPT as the judge.
Version 2 of the index tests this with AA-Symmetry: 120 base questions and 440 variations, with options reordered, labels swapped and yes/no questions negated. Mammoth 27B picked the same answer across variations on 99% of questions, against 91% for Drex Max v1.1, 88% for GLiDE v1.2 and 67% for Clef. Artificial Analysis says the lead on this test is driven by a high rate of admitting they do not know. Its methodology page records that version 2.1, in use since November 2026, changed two of the variation types in the test.
Subquadratic reports a related measurement of its own. On a public set of 600 enterprise workflow decisions, it says, reordering the options changed the top answer on 12% to 21% of questions for Snap 1.1, Jev 1.14 and d1-ultra. It says Snap 1.2 uses an architectural change at inference time that “nearly completely eliminates permutation sensitivity” on several benchmarks. That is the company’s account of its own model, measured on its own tests.
Where the models stand on 19 November 2026
Subquadratic said on 11 November that Snap 1.2 scored 73 on index v2.1, “tied for first place”. Liquid AI said on 16 November that d1.2-ultra scored 75, two points ahead of the previous best decision model. The leaderboard data on Artificial Analysis’s live v2.1 page, read on 19 November, is consistent with both reports. The table gives rounded index scores, with cost per million tasks and median time per task as the page lists them.
| Model | Maker | Decision Index v2.1 | Cost per 1M tasks | Time per task |
|---|---|---|---|---|
| Claude Haiku 5.5 (Max), language model | Anthropic | 76 | about $6,218 | about 32.3 s |
| GPT-6 Luna (Max), language model | OpenAI | 76 | about $1,548 | about 11.4 s |
| d1.2-ultra | Liquid AI | 75 | about $133 | 218 ms |
| Drex Max v1.1 | Nace.AI | 73 | about $103 | 283 ms |
| SubQ Snap 1.2 | Subquadratic | 73 | about $137 | 162 ms |
| GLiDE v1.2 | Fastino | 73 | about $780 | 284 ms |
| Mammoth 27B | Base Labs | 71 | about $264 | 106 ms |
| Gemini Decision 1 | 66 | about $345 | 211 ms |
Gemini Decision 1 sits at 66 on this index and has a median time per task of about 211 milliseconds, in the same range as the other decision models listed. Google’s post says the model “leads across the suite, with the lowest calibration error” on benchmarks that cover agent routing, reasoning, ranking and multimodal understanding. The post gives no scores, and the suite is Google’s own, so the claim cannot be set against Artificial Analysis’s figures. The index scores also depend on version: the v2 article of 8 November gave 77 and 76 for the two language models and $1,530 for GPT-6 Luna (Max), slightly different from the live v2.1 page.
Liquid AI adds a calibration claim of its own. Across 13 benchmarks and 22,505 questions answered by all five models it compared, it reports a calibration error of 0.8 points for d1.2-ultra, against 1.2 for d1.2, 2.7 for Jev 1.14, 3.7 for Perplexity’s Decider v1.3 and 7.2 for GPT-6 Luna on its Decisions API. Those are Liquid’s tests, run by the maker, and the page does not describe an independent replication.
What the evidence does not settle
The index has limits that its own pages state. One of its six evaluations, EvalSafe-O*NET-AA, has no ground truth for its questions: the methodology page says the reference answer is the average of two language models, GPT-6 Astra and Claude Fable 5.1, and it keeps only the questions on which both agree. The private datasets protect against tuning, and also mean readers cannot inspect most of the questions. For wider context on why leaderboards need this kind of checking, see our review of benchmark errors and contamination.
Language models still lead on the headline number. On AA-Paperclip, Claude Haiku 5.5 (Max) passed 92% of documents in the v2 article, against 63% for Drex Max v1.1, at the cost and time gaps in the table. The When2Call component tests the tool-choice step that AI agents depend on, and in v1.0 the leading decision models scored 84 on it against 77 and 73 for the two language models tested in the Low setting.
The Rao and Callison-Burch paper tests how far a decision model can stand in for a language model as a judge. Its abstract reports that the language-model judges cost 16 to 357 times as much and took 5 to 350 times as long, yet each decision model trailed them significantly in at most 6 of 27 comparisons. The two kinds of judge also erred alike: over 85% of the verdicts from the three flash-tier language-model judges on a decision model’s most confident errors repeated them. Sending uncertain verdicts to those judges as a fallback gained 2.7 points at best over the strongest single judge, even with oracle thresholds.
Sources & further reading
- Google: Gemini Decisions API, fast, calibrated multimodal decisions (11 November 2026)
- Artificial Analysis: Introducing the Decision Index v2 (8 November 2026)
- Artificial Analysis: Decision Index v1.0 (13 October 2026)
- Artificial Analysis: Decision Index v2.1 leaderboard and evaluation page
- Artificial Analysis: Decision Index benchmarking methodology and version history
- Subquadratic: SubQ Snap 1.2, order-robust decisions and higher long-context accuracy (11 November 2026)
- Liquid AI: d1.2, state-of-the-art omnimodal decision models (16 November 2026)
- Liquid AI: Introducing d1, the most capable decision model, now with vision (5 October 2026)
- arXiv 1706.04599: Guo, Pleiss, Sun and Weinberger, On Calibration of Modern Neural Networks (ICML 2017)
- arXiv 2207.05221: Kadavath et al., Language Models (Mostly) Know What They Know (2022)
- arXiv 2308.11483: Pezeshkpour and Hruschka, Large Language Models Sensitivity to the Order of Options in Multiple-Choice Questions (22 August 2023)
- arXiv 2305.17926: Wang et al., Large Language Models are not Fair Evaluators (29 May 2023)
- arXiv 2609.29769: Rao and Callison-Burch, Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places (24 September 2026)
Researched and written with the help of AI tools and edited for accuracy. Provided for general information and discussion only, not professional advice. See our editorial standards and disclaimer. Spotted an error? Tell us.
Enjoyed this? Get the next one.
One good read at a time, straight to your inbox. No spam, unsubscribe anytime.