Google's Android Bench 2.0 Scores AI Agents on Multi-Step Coding, Not Just One-Shot Answers
Android Bench 2.0 introduces long-horizon tasks, agent-based evaluation and continuous scoring to test whether AI can carry an Android development job through many steps instead of answering a single prompt.
Google has shipped Android Bench 2.0, a rewrite of its benchmark for measuring how well AI models and agents handle Android development work. The update, detailed by InfoQ contributor Sergio De Simone, adds long-horizon tasks, agent-based evaluation and continuous scoring. The thrust is simple: stop grading models on isolated prompts and start measuring whether they can finish a job that spans many steps.
The headline addition is long-horizon tasks, or LHTs. In the original benchmark, a model typically faced a self-contained question with a checkable answer. LHTs instead chain dependencies together, so a decision early in the run constrains what is possible later. That mirrors real Android work, where a change to a Gradle configuration, a manifest entry or a dependency version ripples through builds and tests.
Agent-based evaluation changes who or what is being scored. Rather than a single model call, the harness can assess an agent that plans, edits files, runs commands and reacts to compiler or test output. This matters because agentic coding tools are now the primary way developers interact with models on large codebases, and a model that answers well in isolation can still flounder when it has to recover from a failed build.
Continuous scoring replaces the pass/fail grading common in older benchmarks. Partial credit lets a run earn points for progress toward a working solution even when the final build or test suite does not pass cleanly. That produces a finer-grained signal across models, which is useful when several systems cluster near the top of a binary leaderboard and the ranking stops discriminating between them.
Android is an unusually demanding target for this kind of testing. Projects mix Kotlin and Java, XML resources, Gradle build logic and platform APIs that shift between releases, and tooling errors are often oblique. A benchmark that rewards multi-step recovery and partial progress is a closer proxy for what an engineering team actually needs from a coding agent than a set of trivia-style questions.
The open question is whether benchmark gains translate to real repositories. Android Bench 2.0 measures performance on tasks Google designed, and agents can be tuned to the harness rather than to production codebases with legacy modules, flaky tests and undocumented conventions. Watch for third-party runs on private codebases and for whether other platform vendors copy the long-horizon, continuous-score format.