CoCoCo Bench
Current AI systems are guided by prompts and skills with predefined rules. As they take on greater autonomy for real-world operations, they will need to weigh competing outcomes consistently, reaching the same judgements regardless of how alternatives are presented. Without this, decisions risk becoming contradictory, arbitrary, and difficult to trust.
CoCoCo Bench tests this through 3,000 pairwise comparisons of products and their various impacts on the world. The results show that models are growing a consistent value system but still struggle to make internally consistent judgements.
Leaderboard
- Score
- 81.0
- Rank
- 1st
- Public set
- 82.4
- Answered
- 100%
- USD per 1,000
- 11.1
- Score
- 75.4
- Rank
- 2nd, shared by 5
- Public set
- 76.4
- Answered
- 100%
- USD per 1,000
- 9.5
- Score
- 75.3
- Rank
- 2nd, shared by 5
- Public set
- 74.6
- Answered
- 100%
- USD per 1,000
- 9.6
- Score
- 75.2
- Rank
- 2nd, shared by 5
- Public set
- 79.7
- Answered
- 100%
- USD per 1,000
- 18.3
- Score
- 74.1
- Rank
- 2nd, shared by 5
- Public set
- 78.3
- Answered
- 100%
- USD per 1,000
- 3.6
- Score
- 74.0
- Rank
- 2nd, shared by 5
- Public set
- 76.5
- Answered
- 100%
- USD per 1,000
- 15.6
- Score
- 71.6
- Rank
- 3rd, shared by 3
- Public set
- 77.3
- Answered
- 100%
- USD per 1,000
- 1.9
- Score
- 69.7
- Rank
- 3rd, shared by 3
- Public set
- 73.8
- Answered
- 100%
- USD per 1,000
- 2.9
- Score
- 69.6
- Rank
- 3rd, shared by 3
- Public set
- 72.3
- Answered
- 100%
- USD per 1,000
- 3.2
- Score
- 65.5
- Rank
- 4th, shared by 3
- Public set
- 64.3
- Answered
- 100%
- USD per 1,000
- 0.35
- Score
- 65.4
- Rank
- 4th, shared by 3
- Public set
- 65.1
- Answered
- 100%
- USD per 1,000
- 11.5
- Score
- 63.0
- Rank
- 4th, shared by 3
- Public set
- 63.7
- Answered
- 100%
- USD per 1,000
- 0.39
- Score
- 62.4
- Rank
- 5th
- Public set
- 62.7
- Answered
- 100%
- USD per 1,000
- 6.6
- Score
- 58.4
- Rank
- 6th
- Public set
- 57.0
- Answered
- 100%
- USD per 1,000
- 0.80
- Score
- 54.0
- Rank
- 7th, shared by 4
- Public set
- 57.4
- Answered
- 99.7%
- USD per 1,000
- 17.9
- Score
- 51.8
- Rank
- 7th, shared by 4
- Public set
- 54.9
- Answered
- 100%
- USD per 1,000
- 3.9
- Score
- 51.6
- Rank
- 7th, shared by 4
- Public set
- 55.9
- Answered
- 99.9%
- USD per 1,000
- 0.47
- Score
- 51.2
- Rank
- 7th, shared by 4
- Public set
- 57.5
- Answered
- 100%
- USD per 1,000
- 2.3
- Score
- 46.0
- Rank
- 8th
- Public set
- 44.3
- Answered
- 97%
- USD per 1,000
- 0.29
- Score
- 39.8
- Rank
- 9th
- Public set
- 40.9
- Answered
- 100%
- USD per 1,000
- 0.41
Full table
Public and private set scores, answered shares, and prices.
| Model | Score | Public set | Answered | USD per 1,000 | Released |
|---|---|---|---|---|---|
| Fable 5.2lowAnthropic | 81.0 | 82.4 | 100% | 11.1 | 2026-11-09 |
| Opus 5lowAnthropic | 75.4 | 76.4 | 100% | 9.5 | 2026-07-24 |
| Opus 5.5lowAnthropic | 75.3 | 74.6 | 100% | 9.6 | 2026-09-22 |
| GPT-6 AstralowOpenAI | 75.2 | 79.7 | 100% | 18.3 | 2026-09-03 |
| GPT-6.1 SollowOpenAI | 74.1 | 78.3 | 100% | 3.6 | 2026-09-29 |
| Fable 5.1lowAnthropic | 74.0 | 76.5 | 100% | 15.6 | 2026-09-01 |
| Gemini 3.7 FlashlowGoogle | 71.6 | 77.3 | 100% | 1.9 | 2026-08-13 |
| Sonnet 5lowAnthropic | 69.7 | 73.8 | 100% | 2.9 | 2026-06-30 |
| GPT-6 SollowOpenAI | 69.6 | 72.3 | 100% | 3.2 | 2026-09-22 |
| GPT-5.6 LunalowOpenAI | 65.5 | 64.3 | 100% | 0.35 | 2026-07-09 |
| GPT-5.6 SollowOpenAI | 65.4 | 65.1 | 100% | 11.5 | 2026-07-09 |
| Llama 4 MaverickfixedMeta | 63.0 | 63.7 | 100% | 0.39 | 2025-04-05 |
| Grok 4.6lowxAI | 62.4 | 62.7 | 100% | 6.6 | 2026-08-12 |
| GLM 5.3lowZ.ai | 58.4 | 57.0 | 100% | 0.80 | 2026-08-14 |
| Qwen3.5 PluslowAlibaba | 54.0 | 57.4 | 99.7% | 17.9 | 2026-02-16 |
| Haiku 4.51kAnthropic | 51.8 | 54.9 | 100% | 3.9 | 2025-10-15 |
| Mistral Large 3fixedMistral AI | 51.6 | 55.9 | 99.9% | 0.47 | 2025-12-02 |
| GPT-4oOpenAI | 51.2 | 57.5 | 100% | 2.3 | 2024-08-06 |
| Ministral 3 14BMistral AI | 46.0 | 44.3 | 97% | 0.29 | 2025-12-02 |
| gpt-oss-120blowOpenAI | 39.8 | 40.9 | 100% | 0.41 | 2025-08-05 |
Cost efficiency
The Pareto frontier line connects the models that no other model beats on both score and price.
Findings
- The new Fable 5.2 is in a league of its own with a CoCoCo Score of 81.0, 5.6 points ahead of the next five models, which score within 1.4 points of each other.
- Extra reasoning does not help with consistency: in our pilots, raising the reasoning effort moved scores by less than the resampling range, which is why the leaderboard runs each model at the lowest enabled setting.
- Consistency does not have to be expensive anymore: Gemini 3.7 Flash lands a decent 72 at a sixth of Fable 5.2's price. Specialist decision models can beat the frontier generalists at an even cheaper price.
Generalist vs specialist decision models
Why do even the frontier models contradict themselves, and how to solve the problem.
Generalist language models have grown far more consistent, but they still contradict themselves. In a separate study, we asked GPT-6 Astra, Fable 5.1 and Opus 5.5 to quantify all impacts of products and companies. When asking the same question three times each, the size a model gave an impact category changed by a factor of two or more in about a quarter of cases, and by three or more in about one in twelve.
We suspect consistency comes from what a model has learned rather than from how long it thinks. More reasoning has driven most of the recent gains in model capability, but in our pilots raising the reasoning effort did not yield more consistent answers.
Specialist models, like Upright’s Eevo-1, can of course do better. Eevo-1 scores 83.7 at $0.28 per 1,000 questions, 2.7 points above the best general-purpose model (a gap inside the tie band) at 1/40 of its price (show on the chart). Three things set it apart:
- It is trained for consistency. Eevo-1’s training data includes Upright’s impact-sizing data in the same three-line question format, for many products and impacts (though not for the private benchmark products, and with none of the benchmark’s questions or answers).
- It answers in a single pass. Eevo reads the same questions and returns the same three lines, but reads each number off a probability distribution over size ranges instead of writing it as text. Asked the same eight questions again, it changed no size by more than 10%.
- It holds one size per impact. The size Eevo gives an impact barely moves with the comparison partner: a typical spread of ×1.16 across the impact’s 48 questions, against ×1.5 to ×1.7 for the leading general-purpose models.
We keep Eevo-1 and Leena-2, Upright’s impact-sizing model, off the leaderboard: they are trained for this task, they answer differently, and we built them.
Methodology
Pairwise comparisons
A model makes 3,000 pairwise comparisons. For five diverse products, it compares a relevant set of 25 of the product’s impacts, in both orders. Each question names the product, describes the two impacts, and asks for three lines: the size of each impact in cents per dollar of the product’s revenue, and their ratio. Answers must be positive numbers; there is no option to decline. Models run at their lowest enabled reasoning setting, one question at a time, with no tools or web search.
Target products
The official leaderboard uses five undisclosed products in food and beverage, medical, technology and finance sectors. In addition, we publish a second, public product set, along with the models' answers so that anyone can confirm the scores. The public products are: Electric passenger cars, Social media platforms, Large language models, Chocolate, Student loans.
Target impacts
Frontier models first sized the candidate impacts from Upright’s impact taxonomy of hundreds of positive and negative impacts. Then the 25 most significant impacts were selected per product, with all four impact categories represented (environment, health, society and knowledge).
Limitations
- Five products and 125 impacts is a relatively small sample. Redrawing them moves most scores by about five points either way, so gaps of a few points between neighbours are not meaningful.
- The benchmark measures consistency, not accuracy. A model can be consistent and still wrong about the world.
- Each comparison is a single question at the lowest enabled reasoning setting, without tools or web search. It tests the coherence of the model's judgement and knowledge, which influences but does not fully determine how reliable the model is when it can research.
Example question
“How much does chocolate 'treat cardiovascular diseases' and 'create CO2 emissions' in cents per dollar of chocolate revenue, and what's the ratio?”
GPT-6 Astra answered:
Z: 0.1176 Y: 7.84 R: 0.015
The full question
Exactly as models received it, with tips and answer format.
Please estimate the scale of Impact Z compared to Impact Y. Details below.
Provide answers to the following questions:
1. What is the scale of the cost or benefit of Impact Z, expressed in cents per dollar of revenue generated by the product or service?
2. What is the scale of the cost or benefit of Impact Y, expressed in cents per dollar of revenue generated by the product or service?
3. What is the scale of Impact Z compared to Impact Y?
The answers should be consistent, i.e. the answer to 3 should be computable as the ratio of answers 1 and 2.
Use precise positive decimal values, with enough digits to reflect the magnitudes. The unit of answers 1 and 2 is the cent — express each as a count of cents, NOT as the dollar-equivalent fraction of revenue. Costs and benefits are both expressed as non-negative numbers (the type is implicit from the impact description). Avoid rounding to round numbers when a more accurate estimate is available.
The costs and benefits should include all types of costs and benefits (including internal, external, social, environmental, private, public, everything that is relevant to the core question).
<tips>
- Only consider how big the given impacts are in context of the core question, not how important the impact topics are in general.
- Consider all parts of a product's life cycle. Apply appropriate weight on end-use, which often (but not always) yields the largest impacts of products/services/companies.
- Indirect impacts for which the product is only one of many contributors should be taken account only on a basis that takes into account the product's (share of) contribution.
- Uncertainty or ambiguity about definitions lowers the estimated impact size when it matters. For example, if the impact topic is "provides access to essential services", it is unclear what "essential services" includes, and the product contributes services that some people would consider essential while others would not, the estimate should be lower than if there were no such uncertainty. The same applies to ambiguity about the relevant counterfactual(s), causal attribution, and similar.
</tips>
<response-format>
Output exactly three lines, each in the literal format `<label>: <value>` (use period as decimal separator). Do NOT use markdown bold, list markers, or any other formatting.
Line 1: `Z: <number>` — cents per dollar of revenue for Impact Z
Line 2: `Y: <number>` — cents per dollar of revenue for Impact Y
Line 3: `R: <number>` — ratio of Impact Z to Impact Y (i.e. line 1 divided by line 2)
All numbers must be positive; if an impact is very small, give a small positive estimate rather than 0. No other text, no explanation.
</response-format>
You are doing analysis relating to the following core question:
<core-question>
What is the impact of the product or service known as "Chocolate"? (a preparation of roasted and ground cacao seeds that is made in the form of a liquid, paste, or in a block, which may also be used as a flavoring ingredient in other foods.)
</core-question>
Consider the following impacts (i.e. consequences) in context of the core question:
<sub-impact-Z>
<description>
treats cardiovascular diseases
Additional explanation that clarifies what is meant by the impact:
Managing heart and blood vessel disorders by modulating hemodynamics, cardiac rhythm, or vascular tone in humans.
</description>
</sub-impact-Z>
<sub-impact-Y>
<description>
creates carbon dioxide emissions
Additional explanation that clarifies what is meant by the impact:
The release of CO2 into the atmosphere.
</description>
</sub-impact-Y>
Estimate the scale of Impact Z ("treats cardiovascular diseases") compared to Impact Y ("creates carbon dioxide emissions").
Answer in the exact response format above, with no other text.
Scoring
For each product we fit one size per impact that best explains the model’s stated ratios (least squares in log space). A comparison earns full credit when its stated ratio equals the ratio of the fitted sizes, and loses credit in proportion to the gap, down to 0 at one standard deviation of the fitted ratios. Unanswered or unparseable comparisons earn 0. The CoCoCo Score is the mean credit over the 3,000 comparisons, with the products weighted equally. Because the tolerance follows each model’s own spread of ratios, raising the ratios to a common power leaves the score unchanged, and random answers score about 8.
In the case of the example above, fitted together, GPT-6 Astra’s 600 chocolate answers imply a ratio of 0.0053 for this pair, so its stated 0.015 is 2.8 times too large. Its fitted chocolate ratios spread by a factor of 8.3 (one standard deviation), so the gap uses about half of the tolerance and the question earns half of the maximum credit.
Data and access
Download the leaderboard table. The results are released under the Creative Commons Attribution 4.0 licence.
To protect the benchmark, the questions themselves are not published openly, but researchers and AI labs can request the public set’s questions. The private set is reserved for reputable independent evaluators. To make a request, or to have a model evaluated, write to hello@uprightproject.com.
Patent & citing
Upright has a U.S. patent pending on key techniques underlying this benchmark.
To reference the 2026 report that accompanies this page, or the 2025 report that introduced the method, you may use the citation information below.
2026 report
Nurmi, W., & Ojala, J. (2026). CoCoCo Bench: how consistently language models quantify consequences (Technical Report No. UPTR-202602). Upright Oy. https://www.uprightproject.com/blog/cococo
@techreport{cococo2026,
author = {William Nurmi and Juho Ojala},
title = {CoCoCo Bench: how consistently language models quantify consequences},
institution = {Upright Oy},
year = {2026},
number = {UPTR-202602}
}2025 report
Ojala, J. (2025). Evaluating the ability of LLMs to consistently quantify a wide range of consequences (Technical Report No. UPTR-202501). Upright Oy. https://www.uprightproject.com/blog/evaluating-llms
@techreport{cococo2025,
author = {Juho Ojala},
title = {Evaluating the ability of LLMs to consistently quantify a wide range of consequences},
institution = {Upright Oy},
year = {2025},
number = {UPTR-202501}
}