fix: reject duplicated promptfoo result rows instead of counting them as attempts
evalarc diff now rejects a promptfoo --output JSON file that contains the same result row twice, instead of counting the copy as an extra attempt. The promptfoo loader keys each row of results.results[] by (testIdx, promptIdx, provider). The key is checked only when testIdx is an integer and not a bool. If a key repeats, loading stops with a ValueError, the same way duplicate Inspect samples are already handled. The message names the duplicated testIdx, promptIdx and provider and the test label. The label and provider are escaped and truncated with the existing _shown_id helper. The message also tells the user to re-export with promptfoo eval --output. evalarc diff exits 2 for a duplicate in either the baseline or the current file, prints the message to stderr and gives no gate verdict. A list or object promptIdx used to crash the loader with an unhandled TypeError. It is now reported as malformed input with exit 2. Output from --repeat, which gives each repeat a new testIdx, still loads as attempts of one test. The unmodified promptfoo example pair still exits 1. docs/ci-gate.md and docs/ci-gate.zh-CN.md describe the rule, exit 2 and how to recover. Controller results: before the fix, the new test file failed on the base code (9 failed, 5 passed). The duplicate cases did not raise, and the non-scalar promptIdx cases crashed with TypeError at results_diff.py line 543. At commit f18d1890 the controller ran the new tests, the full pytest suite, ruff check, ruff format --check, scripts/build_site.py and scripts/check_site.cjs, and all exited 0. The author had no tools and ran none of these checks.
Use it
Prerequisites: a checkout of this repository at the release commit, with EvalArc from that checkout installed in the active Python environment. Run the commands from the repository root. No model key or network is needed.
Step 1. Build a synthetic duplicated file by hand. It copies the committed promptfoo example and appends its first result row again, unchanged, as a merge script might:
python - <<'EOF'
import json, pathlib
doc = json.loads(pathlib.Path("examples/results-diff/promptfoo/current.json").read_text())
rows = doc["results"]["results"]
rows.append(json.loads(json.dumps(rows[0])))
pathlib.Path("merged-current.json").write_text(json.dumps(doc))
EOF
Step 2. Compare it with the committed baseline:
evalarc diff examples/results-diff/promptfoo/baseline.json merged-current.json --output promptfoo-diff; echo "exit: $?"
Expected: exit 2 and no gate verdict. stderr shows a message that starts with "promptfoo output has duplicate result testIdx". It names the promptIdx, the provider and the test label, says each test, prompt and provider must appear once, and tells you to re-export with promptfoo eval --output.
Step 3. Check that valid input is unaffected:
evalarc diff examples/results-diff/promptfoo/baseline.json examples/results-diff/promptfoo/current.json --output promptfoo-diff-ok; echo "exit: $?"
This exits 1 with the same regressed and improved rows as before.
To recover in a real pipeline, run promptfoo eval --output again, or remove the copied row in a reviewed commit, then compare again. Afterwards, delete merged-current.json and any output folders.Scope
Scope: this release adds one duplicate-row check in the promptfoo loader used by evalarc diff, rejects a list or object promptIdx as malformed input, and adds the matching error message, exit 2 and docs. It does not find a copied row whose testIdx was edited. Rows with a missing, string or bool testIdx are not checked. It does not prove that a file is an unmodified promptfoo export. JUnit repeated test cases still count as repeated attempts. Inspect behavior is unchanged. Files are not repaired or deduplicated, and there are no new flags. Compatibility: a promptfoo file that used to load with duplicated rows now exits 2 on purpose, and so does one with a list or object promptIdx (which used to crash). The tests use synthetic changes to the committed recorded examples (promptfoo 0.123.1 output), built at runtime in a temporary folder. Other promptfoo versions and real merged files from the field were not tested. Passing tests show the specified cases behave as intended. They do not prove that every kind of merged or edited file is detected. UI and browser evidence: this change has no UI. The controller's check_site.cjs run did exercise the existing site showcases at widths 1440, 390 and 320 and offline, and those checks passed. They are regression checks of existing pages, not a visual review of this feature. No screenshots were reviewed. No live provider or model is involved.
Upgrade and rollback
Upgrade: install an EvalArc release, or a pinned action tag or commit SHA, that contains commit f18d1890. Input formats are unchanged. For valid inputs, diff.json, summary.md, index.html and exit codes 0 and 1 are unchanged. If a CI step starts exiting 2 with a promptfoo duplicate result error, the input file has a copied row. Regenerate it with promptfoo eval --output, or remove the copied row in a reviewed commit, then compare again. Do not deduplicate it automatically without review. A non-scalar promptIdx error means the file is malformed and should be re-exported. Rollback: pin the EvalArc version, or the noteflowai/evalarc action tag or commit SHA, that you used before. Rolling back brings back the old behavior, where duplicated rows are silently counted as extra attempts and a list or object promptIdx crashes the loader. Committed baselines and existing output folders stay readable in either direction.