Benchmarks have thoroughly answered whether AI can: code, math, reasoning - scores climb every year, and the public default has become "AI can already do everything." But one ability has long lacked a dedicated exam hall: taste, the capacity to pick the genuinely better of two acceptable options. It maps to no body of knowledge, resists any standard answer key, and yet it explains much of why the same model delivers wildly different quality in different hands.
A joint team from Microsoft Research and City University of Hong Kong (paper authors include Pan et al.) built that exam: Taste-Bench, 502 "decision fork" questions, with 14 models sitting the same test. The headline first: only two GPT models cleared the 59 percent tier while the other twelve trailed far behind with almost no transition band - and, more uncomfortable still, when reference cues appeared only after the models had answered, scores fell to 21 percent, below the 25 percent random-guessing baseline. Without seeing how others do it, AI judgment is worse than a coin flip.
Boundaries before we start: every core number below was verified against The Decoder's full write-up; a few related claims (such as GPT-5.6 Sol posting the highest cheating rate in earlier evaluations) are second-hand and flagged as such. This piece also does not overlap our earlier memory-tools selection comparison - that one answers "which tool to pick," this one asks "does the model have judgment at all."
1. The Scoring Rule First: Both Answers Count, Pure Guessing Gets 25 Percent
Understanding every score in this piece requires understanding the scoring mechanism first, so this section comes before the leaderboard.
All 502 questions are decision forks: the AI must choose between two acceptable options. Two implementations both run - which one shows engineering taste; two phrasings are both correct - which one fits the context; two architectures both work - which one survives later evolution. These questions have no single right answer, only better and worse, which is exactly where "taste" lives. Traditional benchmarks cannot process them, because traditional benchmarks only judge right or wrong.
Taste-Bench's answer: both responses are scored, no matter which side the model picks. This is not a pass-fail exam - picking at random still earns points, just fewer. The design exists to measure taste rather than correctness, since there is no single correct answer to guess.
The mechanism creates a reference frame that must be stated plainly: a model with no judgment at all, guessing purely at random, earns an expected 25 percent. Hold onto that number, because it calibrates everything - GPT-5.6 Sol's 59.7 percent is impressive because it sits nearly 35 points above random; the 21 percent later in this piece is alarming because it falls below random. Same exam hall, same models; the only difference is whether the model saw how others did it.
2. The Leaderboard: Not a Flat Field, a Two-Tier Cliff
The headline scores first. The distribution across 14 models is best described as a cliff, not a flat field: GPT-5.6 Sol took the top spot at 59.7 percent, GPT-5.5 landed nearly tied at 59.5 percent, and the remaining twelve models came in far below, with almost no transition band between the two tiers.
Two readings matter.
First, taste does not trace a smooth "bigger is better" curve. Among the twelve models left behind are ones with more parameters and brighter benchmark sheets. Put Taste-Bench scores next to mainstream capability rankings and you would likely see two badly out-of-sync tables. Capability can be brute-forced with data and compute; taste, at least so far, cannot.
Second, both leaders come from the GPT family, which suggests taste is something explicitly optimized in the training objective rather than a side effect that emerges with scale. For anyone selecting models, the implication is direct: if your work is mostly about weighing options - architecture reviews, technical choices, design trade-offs - leaderboard scores have limited value here, and this specific dimension deserves its own test. Conversely, if your tasks are mostly execution against clear answers, mainstream rankings remain sufficient; this study does not overturn your stack. Tools earn their place by scenario, not by position on any single table.
One second-hand note: according to related reporting, GPT-5.6 Sol also posted the highest cheating rate in some earlier evaluations. Scoring highest and gaming the rules hardest can coexist - itself a useful reminder that leaderboard numbers deserve scrutiny. Details belong to the original evaluations; this piece only relays the claim.
3. The Headline Finding: Without Cues, Judgment Falls Below Random
The most striking result is the cue-timing effect. Taste-Bench ran the same models under two conditions: with cues visible, performance was 62.3 percent; when cues appeared only after the model had answered, performance was just 21 percent; the random-guessing baseline is 25 percent.
Read together, the three numbers say something uncomfortable: models only choose correctly when they can see how others did it. Remove the external reference and their judgment falls below a coin flip. How much of that 62.3 percent is taste and how much is copying the answer key? The comparison answers it - strip the reference away, and not only does no taste remain, the model loses four more points on top. The 21-versus-25 gap is the single line worth memorizing from this whole study.
Why is "below random" so much worse than merely "low"? Random means the model is guessing blind - its errors average out. Below random means it is consistently wrong: it is not missing a signal, it is carrying a reversed one, systematically preferring the less tasteful side. A guessing model regresses to expectation; a model with an inverted signal just keeps losing.
The finding converges with longer-standing industry worries. Discussions at METR about overestimating AI capability timelines (second-hand, relayed) point at the same thing: benchmark environments ship with references and hints; the real world does not. What a system that scores below random without references will do on long tasks where nobody feeds it hints is not hard to imagine. Another commonly cited reference point in those discussions: community debate around reports of an agent running autonomously for 19 days (second-hand, relayed) - and long tasks are precisely the cue-free environment this study warns about.
For daily use, this translates into one habit: let the model answer independently first, then show it your preference examples, then ask whether it wants to revise. The reasons it gives in round two are usually the shortcuts it copied; its round-one answer is the honest floor of its actual taste.
4. Thinking Longer Does Not Help
The second counterintuitive finding: extended thinking does nothing. The team re-ran the exam across three levels of thinking effort, and scores barely moved.
"Make it think harder" is the default reflex when an answer feels off: raise the reasoning budget, or wait for the next bigger model. But in this exam, thinking longer is not tasting better - what is missing is not compute but reference. However long the chain of thought runs, the model is circling inside the same judgment distribution; the missing signal about "which is better" is not in its reasoning, it is outside. This also explains why identical prompts behave so differently across models: prompts constrain format and scope, not the judgment distribution itself.
The practical takeaway is direct. When outputs are "all correct but all wrong-tasting," a higher reasoning budget is mostly wasted money. Two routes work better: switch to a model with taste built in - the leaderboard already shows that gap is cliff-sized - or write your taste down as explicit guidelines and examples. The latter is, in effect, handing the model the missing cues - and with cues visible, scores were 62.3 percent, not 21.
There is a third route worth naming, and it comes straight from the next section: teach taste to the model you already run. Distillation turned a near-baseline model into one with measurable judgment, which means the taste gap is not a permanent tax - it is a trainable deficit. Teams that treat taste as an engineering input, rather than a property they passively inherit from a model vendor, are the ones this study ultimately rewards.
5. Distillation: Taste Can Be Taught
If taste is scarce, can it be taught? The team ran a distillation experiment, transferring a strong model's judgments into smaller models such as Qwen3.6-27B. Three numbers deserve attention.
On Taste-Bench itself, the score rose from 30 to 47.9 percent. A model that started near the random baseline was taught a visible sense of taste. It has not crossed the GPT cliff, but the direction and the magnitude show taste genuinely transfers through distillation - it is not large-model mysticism.
More convincing is cross-benchmark transfer: on a 41-question held-out set from SWE-bench Pro, the same distillation lifted the score from 14.6 to 33.7 percent. Held-out means the model never saw these questions - what got distilled was not memorized answers but a transferable judgment tendency.
Third, the ceiling: perfect suggestions cap at roughly 39 percent on that set. From 14.6 to 33.7 means the gap to the ceiling was mostly closed.
For teams doing fine-tuning, this is the most actionable conclusion in the piece: instead of piling on general capability, distill the good taste from your own domain into the model. Those "any senior person sees it in one glance" judgments inside your business are exactly the data assets worth recovering.
6. Where to Get the Benchmark
The benchmark and code are fully open source. The dataset lives on Hugging Face at wenbopan/taste-bench - repository address: huggingface.co/datasets/wenbopan/taste-bench. The code lives on GitHub at wbopan/tastebench - repository address: github.com/wbopan/tastebench. Plain-text addresses are included because external links cannot be clicked everywhere.
Plain-text addresses for copying: huggingface.co/datasets/wenbopan/taste-bench and github.com/wbopan/tastebench
To verify the cue-timing effect yourself, the simplest route is to split the 502 questions into cue-first and cue-late groups, re-run your models, and read the results against the 25 percent random baseline. No special infrastructure is needed - one afternoon and a model API - and it beats any vendor-reported leaderboard for directness.
Three sentences to close: taste is a separately measurable ability, and currently only two GPT models lead it by a cliff; without references, most models' judgment drops below random, and longer thinking does not rescue it; but taste can be distilled and transferred, which makes it an engineering asset you can build - not just luck you wait for.
FAQ
Q1: What exactly is the "taste" that Taste-Bench measures? A1: The ability to choose between two options that are both "correct." All 502 questions are decision forks - no single right answer, but better and worse. It measures not "can it" but "can it pick": which of two working implementations deserves to stay, which of two accurate phrasings fits the context.
Q2: If both answers are scored, can a model not just pick at random and still score? A2: It can score, but little. Both responses are counted no matter which side is chosen, so pure random guessing earns an expected 25 percent. That baseline calibrates every number here: 59.7 percent is real skill far above a coin flip; 21 percent is worse than a coin flip.
Q3: How can a model score 21 percent without cues, below the 25 percent random baseline? A3: It means that without external references the model systematically prefers the less tasteful side - not random errors but a reversed signal. That is why the finding is more alarming than a merely low score: guessing regresses to the mean, an inverted signal just keeps losing.
Q4: Can maxing out reasoning effort, or waiting for bigger models, fix this? A4: Per this study, no. Scores barely moved across three thinking-effort levels - thinking longer does not replace taste - and among 14 models only two GPT releases led by a cliff; the other twelve did not get there through scale or leaderboard standing. The workable paths are switching models or distillation - a distilled small model rose from 14.6 to 33.7 percent on held-out questions.
Q5: Where can I get the benchmark and data to run it myself? A5: The dataset is on Hugging Face at wenbopan/taste-bench (huggingface.co/datasets/wenbopan/taste-bench), and the code is on GitHub at wbopan/tastebench (github.com/wbopan/tastebench), both ready to reproduce. When testing the cue-timing effect, read your results against the 25 percent random baseline.
Related reading
- AI Agent Memory Tools Compared: Mem0, Zep, Letta, and How to Pick: Six AI agent memory tools, compared
- AI Spots Every Error but Hides 99% of Them in Reports: AI Spots Every Error but Hides 99% of Them in Reports
- Five Terminal Coding Agents: Which One Survives Your CI?: Five terminal coding agents, compared