Today I Learned

A collection of small things I've learned along the way.

Aug 7, 2026

Mean Reciprocal Rank (MRR)

Top-1 accuracy is all-or-nothing: a ranker that puts the right answer second scores exactly zero. MRR grades the whole ranking instead; take the reciprocal of the position where the correct answer landed (1st → 1.0, 2nd → 0.5, 4th → 0.25, missing → 0) and average it across all queries. Handy intuition: 1 ÷ MRR ≈ where the right answer typically sits in the list. I met it while comparing a lexical name-matcher against a sentence-transformer for ranking schema-field suggestions: their top-1 counts were close, but MRR (0.48 vs 0.60) revealed the model was missing higher up the list, which is the metric that actually matters when a human confirms answers from a ranked list

Aug 6, 2026

Thresholded abstention

A classifier should decline to predict when its confidence falls below a threshold, trading coverage for accuracy. The modern literature calls it selective prediction / selective classification (Geifman & El-Yaniv, 2017), and related framings include open-set recognition and out-of-distribution detection, all variants of "know when none of your known answers apply.