Current AI benchmarks struggle to keep up with modern models. While useful for measuring a model’s performance on a particular task, it can be difficult to know whether a model trained on internet data is actually solving the problem or just remembering an answer it’s already seen. As a model approaches 100% on a given benchmark, it also becomes less effective at revealing meaningful performance differences. As we continue to invest in new and more challenging benchmarks, the path to general intelligence requires us to continue to look for new evaluation methods. More recently, the move to dynamic, human-judged tests has solved these memorization and saturation problems, but in their place has introduced new difficulties due to the inherent subjectivity of human preferences.
While we continue to evolve and pursue current AI benchmarks, we also always aim to test new approaches to evaluating models. That’s why today we’re introducing Kaggle Game Arena. It is a new public AI benchmarking platform where AI models compete directly in strategic games and provide verifiable, dynamic measures of their capabilities.
