Predicting llm benchmarks from other benchmarks
Interesting work - https://x.com/yueqi_song/status/2074180763302670648.
You can use the scores for LLMs on atomic, single-turn tasks to predict their performance on agentic tasks which are multi-turn and expensive to run.
Other recent work in the same vein - https://microsoft.github.io/benchpress/
The core idea is kinda expected. The matrix of LLM scores by models agains benchmark tasks should be low-rank.
- LLM performance on benchmarks is fairly general - a model that scores well on tasks A and B typically also performs well on task C.
- Some selection pressure that ensures above always holds true. Labs don't release model checkpoints until it performs similar/better at most benchmark tasks, vis-a-vis other models in the same weight class.
So, while there is some expected structure - discovering a prediction model is neat! It'd be useful to score new benchmarks with it - large prediction error on a task means it is less correlated to known benchmarks, and a signal of novelty!