Does confidence mean anything?
Forty real benchmark decisions, scored live against the annotators. See whether high confidence earns its name, and what a threshold buys you.
Try this: Drag the threshold slider. Watch how much you could automate, and how much more accurate that automated slice becomes.
Scoring the benchmark cases, live
0 of ... decisionsEight cases from the public benchmark, five questions each. The answers are compared against the majority vote of the 60 annotators who labelled them. Nothing here is cached: the model is reading them in this tab, right now.
Does high confidence earn its name?
Each bar is a confidence band. Its height is how often the model was actually right in that band. Perfect calibration puts every bar on the dotted line.
Bars above the line mean it is underconfident in that band; below means overconfident.
What a threshold buys you
Automate everything the model is at least this sure about, and send the rest to a person.
-
automated
-
right, of those
-
sent to a human
Accuracy over all 0 decisions is -.
This is 40 decisions - enough to feel how confidence behaves, and a small sample by design so you can watch it happen live. The figures measured across all 2,000 benchmark decisions are on the performance page.