A new benchmarking tool, SO-Arena, has been introduced to evaluate the capability spectrum and decision reliability of System One models. The research, published on November 13, 2026, on arXiv (2611.15777) by lead authors Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, and Xinchao Wang, aims to address the limitations of aggregate performance metrics in understanding these models' strengths and weaknesses. The open-source implementation is available on GitHub.

Comprehensive Benchmarking Across Domains

SO-Arena assesses System One models across 11 domains and five capability families, encompassing 2,919 tasks. These include 2,300 static decisions, 282 tool-use episodes in simulators, and 337 text-based Web tasks. The benchmark introduces fine-grained difficulty tiers (Easy, Medium, Hard, and Very Hard for math) to expose nuanced performance differences that aggregate scores often obscure.

Key features of SO-Arena include evidence-grounded evaluation, where labels are manually verified against model-visible inputs, and realistic interactive testing. Models are evaluated on their ability to observe, choose, and execute actions in tool and Web sandboxes, with scoring based on outcomes and progress.

Decision Reliability as a Key Metric

Beyond accuracy, SO-Arena evaluates decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option exists. These metrics highlight that strong overall performance does not necessarily translate to reliable decision-making. For instance, the top-performing System One model in terms of accuracy ranked only seventh in reliability.

Implications for System One Models

The findings reveal that scaling and reasoning gains vary significantly across domains, underscoring the need to view capability spectrum and decision reliability as complementary dimensions. This systematic approach provides deeper insights into the strengths and limitations of System One models, which are designed for fast, general-purpose decision-making without lengthy token generation.

The SO-Arena project is open-source and can be accessed on GitHub at https://github.com/holdrain/SO-Arena.