fix: fail the diff gate when a check has fewer assessed attempts than the baseline
evalarc diff now has a blocking change kind, less_covered. A check gets it when the current run assessed fewer attempts than the baseline and the pass rate did not fall. A smaller sample can no longer be reported as improved or unchanged. For example, 2/2 to 1/1, 1/2 to 1/1, and 2/2 to 1/1 with one unassessed attempt are all less_covered, and the command exits 1. The existing kinds still take precedence. A lower pass rate is still reported as regressed or less_reliable. A check that passed before and now has zero assessed attempts is still unassessed. The CLI now reads its list of blocking kinds from BLOCKING, so it prints lines such as ' less_covered: cancel-pending / match'. With --output, diff.json keeps schema evalarc.results-diff.v1 and lists less_covered in counts and changes. summary.md shows a 'less covered' row with the baseline and current fractions, and the index.html row uses the existing failed styling. Comparisons where no check lost assessed attempts give the same results as before. docs/ci-gate.md and docs/ci-gate.zh-CN.md list less_covered in the change table. Controller-executed checks on image sha256:3effc5c7...: On base commit e99bc9e, the new tests/test_feature_3412504c3ce6.py failed 6 of 15 tests. The trimmed comparison exited 0 and reported 'Unchanged: 16', which confirms the gap. On candidate commit d4a4afd, all of the following exited 0: that test file, the full pytest suite (including the unchanged tests/test_results_diff.py), ruff check, ruff format --check, scripts/build_site.py and node scripts/check_site.cjs. The earlier review finding that ruff format --check would fail is resolved by that passing run. I did not run these checks myself.
Use it
Prerequisites: an EvalArc checkout at commit d4a4afd or a release that contains it, Python 3.11+, and the package installed from the checkout. Run from the repository root:
python -m pip install -e .
Create a sample input. This is a synthetic, manually trimmed copy of the recorded Inspect example with the epoch-2 sample of cancel-pending removed. The status stays success.
python - <<'EOF'
import json, pathlib
src = pathlib.Path('examples/results-diff/inspect/baseline.json')
doc = json.loads(src.read_text(encoding='utf-8'))
before = len(doc['samples'])
doc['samples'] = [s for s in doc['samples'] if not (s.get('id') == 'cancel-pending' and s.get('epoch') == 2)]
assert len(doc['samples']) == before - 1, 'expected to drop exactly one sample'
pathlib.Path('trimmed.json').write_text(json.dumps(doc), encoding='utf-8')
EOF
Compare it with the baseline. The output folder must not exist yet.
evalarc diff examples/results-diff/inspect/baseline.json trimmed.json --output trimmed-diff
echo "exit code: $?"
Expected result: the output includes the line ' less_covered: cancel-pending / match' and the exit code is 1. trimmed-diff/summary.md has a 'less covered' row for cancel-pending / match showing 2/2 against 1/1. trimmed-diff/diff.json has schema evalarc.results-diff.v1 and a less_covered count. trimmed-diff/index.html marks that row as failed. In CI, use the same command on your committed baseline and current log, for example: evalarc diff evals/baseline.json results/current.json --output evalarc-diffScope
Tests passing on fixtures do not show that the rule is right for every workflow. The rule compares raw assessed-attempt counts per check. It has no statistical tolerance, flag or opt-out, so any drop in assessed attempts blocks, even one lost attempt out of many. The rule lives in the shared _classify, so promptfoo and JUnit comparisons with fewer assessed attempts also block. However, the acceptance tests cover only a trimmed copy of the recorded Inspect example and small synthetic Inspect logs, not real promptfoo or JUnit exports. Out of scope: comparing run-level configuration such as identity.epochs, handling duplicate IDs, detecting non-comparable runs, changing malformed-input messages (these inputs still exit 2), changing the GitHub Action (its outputs come from the same result), and any claim about upstream Inspect issue #5434. The controller's existing site checks (build_site.py and check_site.cjs) passed. Their log covers showcase pages at widths of 1440, 390 and 320 px and offline modes. They are regression checks of the existing site, not a visual or browser review of this feature's index.html diff report. No screenshots of that report were reviewed. It reuses the existing failed-row styling.
Upgrade and rollback
Upgrade: install a release that contains commit d4a4afd. No data or configuration migration is needed. diff.json stays at schema evalarc.results-diff.v1 and gains one new kind value, less_covered. Any consumer that lists kinds explicitly should treat it as blocking. This change tightens the gate. A comparison where a check's assessed attempts fell used to exit 0 and now exits 1. If the gate starts failing, rerun the current evaluation with the same epochs and samples as the baseline, or review and commit a new baseline on purpose. Exit codes 0, 1 and 2 keep their meanings. Rollback: pin the previous version, which restores rate-only classification. Saved result files and existing report folders are unaffected in either direction.