Taste-Bench finds top AI agents pick the right long-term path under 60% of the time
A new benchmark measuring strategic judgment in long-running agent tasks shows even GPT-5.6 Sol tops out just below 60% on simple binary choices, and extra reasoning effort doesn't close the gap.
AI agents that plan well over many steps are harder to build than the demo reels suggest. A new evaluation called Taste-Bench, reported by The Decoder, measures what its authors call strategic taste: whether an agent picks the better of two paths while working through a long task. The benchmark's headline result is that even the strongest model tested, GPT-5.6 Sol, selects the correct option on just under 60 percent of straightforward binary questions.
The format is deliberately plain. Agents face pairs of choices during extended tasks, and the benchmark scores how often they take the branch that leads somewhere useful versus the one that wastes steps or paints them into a corner. Under 60 percent accuracy means the best system is wrong on roughly two out of every five decisions. For a task that requires dozens of correct turns in a row, that error rate compounds fast.
The other finding undercuts a common assumption about reasoning models. Giving the agent more time to think did not improve its choices. Longer deliberation produced no meaningful gain on strategic selection, which suggests the failure is not about insufficient compute spent per decision. Something else is missing: a sense of which option is worth pursuing before the consequences are visible.
That distinction matters for anyone shipping agent products. Benchmarks like SWE-bench and WebArena largely reward task completion once the goal is fixed. Taste-Bench probes the step before that, where the agent has to judge which goal-adjacent move is the better investment. Most current agent stacks lean on the model's raw judgment at exactly this point, with little scaffolding to catch a bad fork early.
The practical read is that reliability in long-horizon agents may be bottlenecked by decision quality rather than tool use or context length. A model that is right 58 percent of the time on a two-way choice will drift off course across a long run even if every individual action executes perfectly. Teams building multi-step workflows should expect compounding errors and design checkpoints accordingly. The open question is whether better prompting, search over candidate plans, or a new training objective can move the number, and by how much.
The Decoder's coverage did not detail which other models Taste-Bench evaluated or how the benchmark constructs its choice pairs, both of which will shape how the results are read. Watch for the benchmark's public release and for follow-up work that separates judgment failures from execution failures. If more thinking does not help, the fix likely has to come from training and evaluation design, not from turning up the reasoning budget.