Decision coverage for recorded choices
How often does a recorded choice component answer when it abstains below a confidence threshold, and how often is it wrong? EvalArc 0.16.0 chooses the threshold on calibration decisions, then reports coverage and selective error on held-out decisions. Invalid responses stay visible instead of being counted as wrong answers.
Use it
python -m pip install evalarc==0.16.0 evalarc decision-coverage RECORDS.json --max-error 0.2 --output runs/decision-review # Open runs/decision-review/index.html; the release guide has a synthetic input example.
Scope
This reviews saved choice decisions only. The bundled example is synthetic and says nothing about a provider’s accuracy. Reported confidence is not calibrated by this command. The new HTML report passed structure and escaping tests; its layout and keyboard behavior were not checked in a real browser.
Upgrade and rollback
Install EvalArc 0.16.0 to add this command. Reinstall 0.15.0 to roll back; existing inputs and generated reports are left alone.