CoCoCo Bench: how consistently do LLMs quantify consequences in 2026?

Introduction

Language models are doing more and more of our decisions on our behalf. There are at least three ways in which they decide for us, all becoming more prominent by the day. First, we were just consulting them with our small and big decisions. Second, they started acting autonomously on our behalf, making multiple little decisions for us as they work, without us ever even noticing. Thirdly and most recently, with the advent of decision models, we have started incorporating decisions in all parts of our business processes and software stacks: when to raise an alarm, who gets to speak to a human, which model should serve whom?

These decisions are judgements about consequences, about which outcome matters more and by how much. Those judgements are less stable than they look. Ask a leading model how much losing a customer would hurt your company, and the answer can change depending on whether it compares it with a week of downtime or a delayed product launch. When you think about it, a decision maker whose sense of scale shifts with the question is hard to trust. So far, the solution has mostly been to not think about it. As systems act more autonomously, and we delegate more and more of our lives to these models, consistent sense of the consequences is becoming imperative to trusting them.

A year ago we published an evaluation of how consistently large language models quantify consequences. Already back then frontier models knew a surprising amount about the impacts of products and services, but looking at their decisions more closely revealed that their quantitative judgements were locally plausible and globally inconsistent. Sizes did not survive transitivity checks and underweighted well-researched harms.

Today, as the stakes are getting higher, we've brushed up our methodology and publish it as a continuously updated CoCoCo Bench, the Consequence Comparison Consistency Bench. This post explains what the benchmark asks, what changed since 2025, and what we can learn from it.

What CoCoCo Bench measures

Imagine being asked whether the greenhouse gas emissions from chocolate production are a bigger deal than the joy it brings. Or if the joy is worth the impact on one’s health. Now extend that to thousands of such comparisons across hundreds of impacts, answered one at a time, in a way that stays logically consistent: if impact A is ten times B, and C is five times A, then C must be fifty times B — all without remembering or keeping track of your previous answer. Doing this well takes consistent counterfactual reasoning, causal attribution, and a balanced value system.

CoCoCo Bench asks each model 3,000 such questions across five products, pairing each of the 25 relevant impacts with the others. The answers are scored on how well they agree with each other. Each model answers with its estimated ratio of the impact sizes, and from all of a model’s answers for a product we fit the one set of impact sizes that best explains them. Each comparison then earns credit for how closely its stated ratio matches the ratio of those fitted sizes: full credit for an exact match, none at a gap of one standard deviation of the model’s own fitted ratios. The CoCoCo Score is the average credit over all 3,000 comparisons. A model scores 100 only when its answers agree with one underlying set of sizes; random answers score about 8.

(The full question wording, the scoring rule and a downloadable results file are on the CoCoCo Bench page.)

Results

The new Fable 5.2 is in a league of its own with a CoCoCo Score of 81.0, 5.6 points ahead of the next five models, which score within 1.4 points of each other.

No model comes near 100. Fable 5.2’s 81 corresponds to answering, on average, within a factor of about 1.3 of the value implied by its other answers. Below the leader, consistency comes cheaper: Gemini 3.7 Flash stands out on the Pareto frontier, landing within ten points of the leader at a sixth of its price. GPT-5.6 Luna and, surprisingly, Llama 4 Maverick score in the 60s for under a dollar per 1,000 questions. We suspect Maverick does well because consistency benefits more from the broad general knowledge of a MoE behemoth than from sharp reasoning.

The frontier has moved since 2025. GPT-4o (2024) scores 51 against 81 for the current leader. GPT-3.5 Turbo (2023) scored 22 on the published products before OpenAI retired it. The reverse check agrees: on a 600-question subset of the 2025 questions, scored with the 2025 CoCoCo % metric, the best model of that edition reaches 76 % and GPT-6 Astra 93 %.

More reasoning does not seem to help with consistency. In our pilots, raising the reasoning effort moved scores by less than the resampling range, which is why the leaderboard runs each model at the lowest enabled setting. The one model we also ran with thinking off, Alibaba’s Qwen3.5 Plus, scored higher that way (57 against 54).

Generalist vs specialist decision models

Generalist language models have grown far more consistent, but they still contradict themselves. In a separate study, we asked GPT-6 Astra, Fable 5.1 and Opus 5.5 to quantify all impacts of products and companies. When asking the same question three times each, the size a model gave an impact category changed by a factor of two or more in about a quarter of cases, and by three or more in about one in twelve. We suspect consistency comes from a model’s architecture and what it has learned, rather than from how long it thinks.

Specialist models, like Upright’s Eevo-1, can of course do better. Eevo-1 scores 83.7 at $0.28 per 1,000 questions at its API price, 2.7 points above the best general-purpose model (a gap inside the tie band) at 1/40 of its price (show on the chart). It is trained for consistency on Upright’s impact-sizing data, and holds one size per impact so the size it gives an impact does not move when the comparison partner is swapped.

Want to know more?

If your work touches on AI decision making, alignment, or evaluation and you would like to discuss CoCoCo Bench, or have a model evaluated, write to hello@uprightproject.com. We are also hiring engineers who want to work on better impact data; see our jobs page.