Computers are now non-deterministic too, great progress.
At least it apologizes if you ask?
PACE: A Proxy for Agentic Capability Evaluation
Interesting work on predicting LLM benchmark scores, from other benchmarks - you fit a regression model over the scores on atomic, single-turn tasks to predict eval scores for agentic tasks, which are generally multi-turn and expensive to run.
Other recent work in the same vein - https://microsoft.github.io/benchpress/
The core idea is kinda expected. The matrix of LLM scores by models agains benchmark tasks should be low-rank.
-
LLM performance on benchmarks is fairly general - a model that scores well on tasks A and B typically also performs well on task C.
-
Some selection pressure that ensures above always holds true. Labs don't release model checkpoints until it performs similar/better at most benchmark tasks, vis-a-vis other models in the same weight class.
So, while there is some expected structure - discovering a prediction model is neat.
Feel like it'd be useful to score new benchmarks with it - large prediction error on a task means it is less correlated to known benchmarks, and a signal of novelty!
-
State machines -> async/await -> durable workflows -> agents
Log of personal notes about async/await and durable workflows and state machines, and how they are all ways to model asynchronous execution. Not very readable, I need to make it more coherent.
React server components precompute the UI tree
I was trying to make sense of why newer React code shows up with
use clientanduse serverannotations everywhere. Apparently, React frameworks have this new construct calledServer componentsnow.Composable SQL abstractions with FunSQL
Metrics layer tools
Reading about multiple projects on Twitter that provide a
metrics layeragainst the database. For example - dbt metrics, Transform, Supergrain. Thoughts from what I make of it below.Node and Wire editors
This was prompted by looking at multiple tools for data exploration and analytics, for ex - Datablocks, Basis Data and more. A common substrate across many of these is an infinite canvas, where you write small bits of functionalities in nodes and then wire them together to model data flow. A visual workspace allows richer interactions and visual tooling that are hard to replicate with a text based IDE.
Anatomy of a recommendation system
At a previous workplace, we created a recommendation engine to feed into an adaptive learning system for school students. This descriptive post was never polished enough to go on the technical blog, so here is the draft.