07 · Measuring Models
Benchmark
A standardized model test
A standardized test to compare models. Labs heavily market scores — but be skeptical: benchmarks can leak into training data (the model has effectively seen the answers), and high scores don't always mean real-world usefulness.
Concrete example
A press release touts a record benchmark score, yet the model still fumbles your actual task — a common gap.
Why it matters
The headline numbers in model launches — useful, but easy to game and over-read.