fix: reject duplicate Inspect samples instead of counting them as extra attempts
evalarc diff now rejects an Inspect AI log that contains the same (sample id, epoch) pair more than once. This applies to JSON logs and .eval archives. Before this change the copy was silently counted as an extra attempt, which inflated the assessed count and could hide a less_covered change. Ids are compared after str(), so an integer 7 and a string '7' in the same epoch count as duplicates. A duplicate in either the baseline or the current file makes load_results raise ValueError. evalarc diff then prints the message to stderr and exits 2 before any comparison, so no diff.json is written. The message names the sample id and epoch and points to re-exporting with inspect log dump. The id is shown with repr() and truncated to 80 characters, so control characters stay escaped. Samples without an integer epoch are not checked. The same id in different epochs still counts as repeated attempts. The diff.json schema, change kinds and gate logic are unchanged. docs/ci-gate.md and docs/ci-gate.zh-CN.md add this case to the exit-2 list. Both explain the 7 vs "7" collision and why counting a copy would misstate coverage and mask less_covered. Both give the recovery step inspect log dump LOG.eval > LOG.json (or rerun) and note that samples without an integer epoch are not checked. The commit sets the version to 0.17.5. Controller results: on base 1d62bef, tests/test_feature_e0a029ce3b1b.py had 5 failures and 1 pass. Duplicates on either side exited 1 instead of 2, and the 7/'7' collision, duplicate .eval members and control-character ids raised no error. On candidate 37e8719 these all passed: that test file, the full pytest suite, ruff check, ruff format --check, scripts/build_site.py and node scripts/check_site.cjs.
Use it
Prerequisites: a source checkout of EvalArc at commit 37e8719 or later, with Python and the checkout installed (python -m pip install .). The last published release without this fix is 0.17.4 (Action noteflowai/evalarc@v0.17.4). The fix is meant to ship as 0.17.5 / v0.17.5. Use that version only after it is actually published. Inputs: the repository's recorded examples, plus a SYNTHETIC broken copy that you create in step 1.
1. Create the synthetic log with a duplicated sample:
python - <<'PY'
import json, pathlib
src = pathlib.Path("examples/results-diff/inspect/current.json")
doc = json.loads(src.read_text())
doc["samples"].append(doc["samples"][0]) # same id and epoch
pathlib.Path("current-dup.json").write_text(json.dumps(doc))
PY
2. Compare it with the recorded baseline:
evalarc diff examples/results-diff/inspect/baseline.json current-dup.json --output dup-diff; echo "exit $?"
Expected: exit 2. stderr shows: Inspect log has duplicate sample <id> epoch <n>: each sample and epoch must appear once, or the copy would count as an extra attempt. Re-export the log from Inspect (inspect log dump) instead of merging or editing it. No dup-diff/diff.json is written.
3. Confirm the valid recorded pair still gives a verdict:
evalarc diff examples/results-diff/inspect/baseline.json examples/results-diff/inspect/current.json --output inspect-diff; echo "exit $?"
Expected: exit 1 with 3 blocking changes: refund-duplicate includes and match regressed, and cancel-pending match less_reliable.
To repair a real log, run inspect log dump LOG.eval > LOG.json or rerun the evaluation, then compare again.
Tests: python -m pytest -q tests/test_feature_e0a029ce3b1b.py tests/test_results_diff.pyScope
Only Inspect JSON and .eval logs are checked. Repeats in promptfoo and repeated JUnit cases are intended attempts and are not checked. Samples whose epoch is missing or not an integer are not checked, so duplicates among them still count as attempts. Logs are never repaired or deduplicated automatically. The tests use synthetic copies of the recorded examples. The .eval test uses a ZIP_DEFLATED archive, not a real zstd Inspect export. Passing tests show that the specified cases behave as designed. They do not prove that every hand-merged or edited log is caught. The controller ran scripts/build_site.py and scripts/check_site.cjs, and both passed. These are the existing showcase and browser regression checks (1440, 390 and 320 px widths, keyboard and offline states). They are not a visual review of this feature, which has no UI. No screenshots were reviewed. In this exchange I wrote only this delivery text and executed nothing. The controller ran every check. The fix is not in 0.17.4 or the v0.17.4 Action. It is available only from a source install until a 0.17.5 release is published.
Upgrade and rollback
This is a patch-level fix (0.17.4 to 0.17.5) with no new flags, Action inputs or schema changes. Logs that were accepted before despite a duplicated (sample id, epoch) now exit 2. Re-export them with inspect log dump or rerun the evaluation, and replace any committed baseline that contains such duplicates. CI steps that treat exit 2 as a pipeline error will report these logs as broken input, not as a regression. To roll back, pin evalarc==0.17.4 and noteflowai/evalarc@v0.17.4, or revert commit 37e8719 in a source install. Upgrading or rolling back does not modify any stored data, baselines or reports.