#llms

• Poking at LLM benchmarks, 1 - Automation Bench

  TLDR: Good task shape, but not a well-vetted benchmark. Hard to trust the leaderboard.

  First of a few posts looking at the benchmarks used to evaluate LLMs.

  Each new model release decimates the older benchmarks but outside models from the Openai/Anthropic duopoly and Deepseek, I still don't find the others reliable enough to be a daily driver. Would be interesting to see which benchmarks are actually a good proxy for things I use the models for -interactive coding (no software factories here).

  continue reading →

• Training for Specialization

  What do LLMs that are designed to be fine-tuned later look like? For ex, if you are Thinking Machines, where the main product is a fine-tuning platform, do you train the base model any differently besides trying to cover a lot of different domains?

  • How do you make sure the model weight are amenable to new behavior via further training rather than fighting it?

  • Can you preserve general solution strategies, and meta heuristics in the model weights; while making it easy to swap out more factual knowledge? Does this split even make sense?

  • What should the model be fine-tuned with, and what should you leave for in-context learning at query time?

  • Do you have to sacrifice performance on typical frontier model evals, for faster/better specialization?

  • What are good proxies for how fine-tunable a model artifact is? In-context learnability proportional to gains feasible with finetuning?

• PACE: A Proxy for Agentic Capability Evaluation

  Interesting work on predicting LLM benchmark scores, from other benchmarks - you fit a regression model over the scores on atomic, single-turn tasks to predict eval scores for agentic tasks, which are generally multi-turn and expensive to run.

  Other recent work in the same vein - https://microsoft.github.io/benchpress/

  The core idea is kinda expected. The matrix of LLM scores by models agains benchmark tasks should be low-rank.

  • LLM performance on benchmarks is fairly general - a model that scores well on tasks A and B typically also performs well on task C.

  • Some selection pressure that ensures above always holds true. Labs don't release model checkpoints until it performs similar/better at most benchmark tasks, vis-a-vis other models in the same weight class.

  So, while there is some expected structure - discovering a prediction model is neat.

  Feel like it'd be useful to score new benchmarks with it - large prediction error on a task means it is less correlated to known benchmarks, and a signal of novelty!