CATArena: New AI Tournament Platform Evaluates Learning Ability of LLM Agents

A team of researchers has unveiled CATArena, a new tournament-style evaluation platform for assessing the learning abilities of Large Language Model (LLM) agents. The platform addresses shortcomings in existing benchmarks, which often focus on end-to-end performance in fixed scenarios and struggle to evaluate the learning and adaptation skills crucial for advanced AI. CATArena employs an iterative, competitive peer-learning framework where agents refine their strategies through repeated interactions and feedback. The platform features four diverse board and card games with open-ended scoring, allowing for continuous and dynamic evaluation of rapidly improving agent capabilities. The core of CATArena is its ability to systematically measure and analyze fundamental sub-abilities of agents, including strategy coding, self-improvement, and peer learning. Agents participate in iterative rounds of competition, revising their strategies based on outcomes and policies observed in previous rounds. This process generates performance rankings and provides insights into an agent's learning capabilities. Experiments conducted on both minimal and commercial code agents demonstrate that CATArena offers reliable, stable, and scalable benchmarking for core agent abilities. The platform's extensible architecture allows for easy adaptation to other types of open-ended tasks, ensuring its continued relevance as agent capabilities advance.