Measured, not claimed
Performance
Everything below was measured on LocalLLaMA/typed-decisions, a public benchmark of 400 real cases and 2,000 decisions across four business workflows. The gold answers are the vote distributions of 60 human annotators per decision, so “correct” means agreeing with the majority of a real panel. It is an open dataset: anyone can run these numbers themselves.
Against the published alternatives
| on the same 2,000 decisions | accuracy | Brier | notes |
|---|---|---|---|
| nirnaya | 0.767 | 0.056 | this system |
| Jev (TypeSafe AI) | 0.727 | 0.148 | the commercial decision API |
| Laya | 0.766 | 0.061 | the strongest open alternative |
| ignoring the input entirely | 0.481 | 0.181 | the floor any system must clear |
Accuracy is how often the top answer matches the human majority. Brier measures the whole probability distribution, not just the winner, and lower is better - it is the number that tells you whether a reported 0.8 means anything. nirnaya's is 2.6x closer to the human panel than the commercial API's.
Where the accuracy sits
| slice | accuracy | |
|---|---|---|
| yes / no questions | 0.838 | |
| pick one of several | 0.740 | |
| rate on a scale | 0.733 | |
| invoice decisions | 0.816 | |
| support conversations | 0.758 | |
| security alerts | 0.756 | |
| agent traces | 0.736 | |
| cases the human panel agreed on | 0.878 | where there is a clear answer, it finds it |
Confidence you can act on
Every answer carries a confidence. The useful test is not whether that number looks nice but whether sorting by it separates the answers worth trusting from the ones worth checking. The measure for that is AURC - take the most confident share of decisions, measure the error rate there, and see how low it goes. Lower is better.
| confidence signal | AURC | |
|---|---|---|
| what nirnaya reports | 0.097 | the signal we ship, chosen by measurement |
| an entropy-based alternative | 0.116 | tested and rejected |
In practice that means you can set a threshold, automate everything above it, and route the rest to a person - and the part you automated will be materially more accurate than the average. The calibration demo lets you drag that threshold and watch the trade in real time, on real benchmark cases, scored live in your browser.
Speed and cost
| measure | nirnaya | typical hosted model |
|---|---|---|
| time per decision | ~77 ms | 2.5 - 3.5 s |
| cost per million decisions | $0.00 | ~$7,500 |
| data leaving the device | none | every request |
| API keys to manage | none | one per environment |
Our timing is five decisions on one piece of state, measured on a laptop. The comparison figures are Ably's published measurements of the same decision loop against hosted models, and list-price token estimates for a five-question case. Different hardware, same job.
Fitted to your decisions
nirnaya is tuned per workflow rather than sold as one generic model. The four above are covered today; a new workflow is fitted from a few thousand of your own labelled decisions, which is a short exercise rather than a project. That is why the numbers on this page are strong and specific instead of vague and universal - and it is worth knowing before you plan a rollout.
Want yours measured the same way? The cookbooks show the exact request shape for several workflows, and the playground lets you put your own decisions in front of it right now.