fix: reject Inspect samples without a usable id instead of inventing case None
evalarc diff and load_results now reject Inspect samples that have no usable identity. This covers JSON logs and .eval archives. Each sample is checked in recorded order, before any case bookkeeping and before the duplicate-sample guard. A sample that is not a JSON object fails with ValueError: 'Inspect log sample at position N is not an object (got TYPE)'. A sample whose id is missing, null, boolean, float, a list or an object fails with: 'Inspect log sample at position N has an invalid id SHOWN (TYPE): expected an integer or a string'. Both messages end with re-export guidance. The CLI prints evalarc: MESSAGE, exits 2, writes no diff.json, and leaves the --markdown file unchanged. Two null-id samples now get the invalid-id message instead of 'duplicate sample None epoch 1'. Integer, string and empty-string ids behave as before. docs/ci-gate.md and docs/ci-gate.zh-CN.md describe the accepted id types, both messages and the re-export command. Controller-executed evidence: at base c722b75, tests/test_feature_b64744a8af0c.py had 13 failures and 1 pass. The failures were KeyError 'id', TypeError for non-object samples, no error for null/true/1.5/list/object ids, the misleading duplicate message, and CLI exit 1 (a gate verdict computed on an invented case) instead of 2. At candidate e835b05, the controller ran the feature tests, the full pytest suite, ruff check, ruff format --check, scripts/build_site.py and scripts/check_site.cjs. All exited 0. I did not run anything myself.
Use it
Prerequisites: a checkout of EvalArc at e835b05 or later, installed in the active Python environment (for example pip install -e .), run from the repository root. The malformed log below is a synthetic example made from the shipped sample log.
# 1. Create a synthetic malformed current log: the first sample's id is set to null.
python - <<'EOF'
import json, pathlib
src = pathlib.Path("examples/results-diff/inspect/current.json")
doc = json.loads(src.read_text())
doc["samples"][0]["id"] = None
pathlib.Path("bad-current.json").write_text(json.dumps(doc))
EOF
# 2. Prepare an existing step summary and run the gate.
echo existing > summary.md
evalarc diff examples/results-diff/inspect/baseline.json bad-current.json --output review --markdown summary.md
echo "exit=$?"
# stderr: evalarc: Inspect log sample at position 0 has an invalid id None (null): expected an integer or a string. ... Re-export the log from Inspect (inspect log dump) instead of editing it.
# exit=2; review/ has no diff.json; summary.md still reads 'existing'.
# 3. Python callers get the same message:
python -c "from evalarc.results_diff import load_results; load_results('bad-current.json')" # raises ValueError
# Fix a real log: inspect log dump LOG.eval > LOG.json, or make the exporter write an integer or string id for every sample.Scope
Only Inspect sample identity is validated, in JSON logs and .eval archives, through evalarc diff and load_results. This change does not validate promptfoo or JUnit identity, duplicate JUnit testcases, or other sample fields (scores, epoch type, output, target). No report keys are added. Behavior change, per the base-commit test log: a missing id key raised KeyError 'id', and non-object samples raised TypeError (for example 'list indices must be integers or slices, not str'), not AttributeError. Null, boolean, float, list and object ids raised no error and were compared under invented case names such as None or True. All of these now raise ValueError and make the CLI exit 2. Integer and string ids keep str() case ids, so 7 and '7' still collide in the existing duplicate guard. The test fixtures are synthetic and derived from the shipped example logs. They are not evidence from real exports or users, and passing tests do not prove the change handles every malformed exporter output. The feature has no UI. The controller's check_site.cjs run covered existing showcase pages at 1440, 390 and 320 px, keyboard use and offline modes, and all checks passed. That is regression evidence for existing pages, not visual review of this feature, and no screenshots were reviewed.
Upgrade and rollback
No migration is needed. Existing diff folders and evalarc.results-diff.v1 reports stay valid, and valid Inspect logs (integer, string or empty-string ids) give the same results. The full pytest suite passed at e835b05. If a pipeline that used to pass or fail now exits 2 with the new message, the log has a sample without a usable id. Re-export it with inspect log dump LOG.eval > LOG.json, or fix the exporter. Do not edit ids by hand. Python code that caught KeyError or TypeError from load_results for these inputs should catch ValueError instead. To roll back, reinstall the previous EvalArc version. No data or report format changes in either direction, but rolling back brings back the invented None/True cases and internal error text.