Predicting llm benchmarks from other benchmarks

Interesting work - https://x.com/yueqi_song/status/2074180763302670648.

You can use the scores for LLMs on atomic, single-turn tasks to predict their performance on agentic tasks which are multi-turn and expensive to run.

Other recent work in the same vein - https://microsoft.github.io/benchpress/

The core idea is kinda expected. The matrix of LLM scores by models agains benchmark tasks should be low-rank.

• LLM performance on benchmarks is fairly general - a model that scores well on tasks A and B typically also performs well on task C.
• Some selection pressure that ensures above always holds true. Labs don't release model checkpoints until it performs similar/better at most benchmark tasks, vis-a-vis other models in the same weight class.

So, while there is some expected structure - discovering a prediction model is neat! It'd be useful to score new benchmarks with it - large prediction error on a task means it is less correlated to known benchmarks, and a signal of novelty!