fix: fail the diff gate when the baseline run did not finish
evalarc diff and the GitHub Action now fail the gate (exit 1) when the baseline run did not finish. Before this change, an Inspect baseline log whose status was not success could pass. Checks missing from it were classified as 'added', and 'added' never blocks. Change classification, counts and blocking rules are unchanged. When the baseline is incomplete, diff.json gains "baseline_incomplete": true and gate_passed is false. When the baseline is complete, the key is left out, so those reports keep their previous content apart from created_at and version. After the existing summary line, default stdout prints: 'The baseline run is incomplete (status error); N check(s) appear only in the current run and were not compared. Rerun the baseline to completion; the gate fails.' summary.md and --markdown add a line below the table: 'The baseline run did not finish (status `error`); the gate fails because checks missing from it cannot be compared. N check(s) appear only in the current run. Rerun the baseline to completion.' The heading reads 'EvalArc: the baseline run is incomplete' only when there are no blocking changes and the current run finished. Otherwise it shows the existing blocking heading or 'the baseline and current runs are incomplete'. index.html shows 'Gate failed' with the reason 'the baseline run is incomplete'. When there are no blocking rows, it says 'Rerun the baseline to completion and compare again.' evalarc verify recomputes the same result. docs/ci-gate.md and the README CI-gate paragraph document the rule. The controller ran the checks; the author did not run anything. On base commit dc848cc, tests/test_feature_f107c0a41d3a.py failed (5 failed, 1 passed), which confirms the gap. On candidate bc6089b, these all exited 0: the feature test, the full pytest suite, ruff check, ruff format --check, scripts/build_site.py and node scripts/check_site.cjs.
Use it
Prerequisites: evalarc at commit bc6089b or later is installed (for example, pip install -e . in a repository checkout), and the commands run from the repository root, which contains examples/results-diff/inspect/baseline.json.
Step 1. Create a SYNTHETIC unfinished baseline. This copies the bundled example log, sets status to error and removes the refund-duplicate samples, imitating an interrupted Inspect run:
python - <<'EOF'
import json, pathlib
src = pathlib.Path('examples/results-diff/inspect/baseline.json')
log = json.loads(src.read_text(encoding='utf-8'))
log['status'] = 'error'
log['samples'] = [s for s in log['samples'] if s.get('id') != 'refund-duplicate']
pathlib.Path('/tmp/unfinished-baseline.json').write_text(json.dumps(log), encoding='utf-8')
EOF
Step 2. Compare it with the finished example log. The --output folder must not already exist:
evalarc diff /tmp/unfinished-baseline.json examples/results-diff/inspect/baseline.json --output /tmp/diff-review --markdown /tmp/summary.md
echo "exit: $?"
Expected results:
- Exit code 1.
- Stdout contains 'The baseline run is incomplete (status error); 2 check(s) appear only in the current run and were not compared. Rerun the baseline to completion; the gate fails.'
- /tmp/diff-review/diff.json contains "baseline_incomplete": true.
- /tmp/diff-review/index.html shows 'Gate failed', 'the baseline run is incomplete' and 'Rerun the baseline to completion and compare again.'
- /tmp/summary.md contains 'The baseline run did not finish (status `error`)'.
Step 3. Confirm the saved folder recomputes:
evalarc verify /tmp/diff-review --json
In CI: evalarc diff evals/baseline.json results/current.json --output runs/diff-review --markdown "$GITHUB_STEP_SUMMARY". If this fails because the baseline is incomplete, regenerate a finished baseline (for example, rerun inspect eval to completion) and compare again.Scope
Scope: one gate rule for an unfinished baseline. It appears in the existing terminal, JSON, Markdown and HTML outputs. docs/ci-gate.md and the README CI-gate paragraph state the rule. The translated docs, docs/ci-gate.zh-CN.md and README.zh-CN.md, are not updated with it. Not included: new flags or override modes, relabelling 'added' checks, reconstructing missing samples, duplicate-ID or malformed-record handling, and other EA-02 corpus items. The rule depends on the run status that the export records. Exports with no run status, such as JUnit, are never marked incomplete, so the rule cannot fire for them. The acceptance tests use synthetic fixtures derived from the bundled example Inspect log. No live or real interrupted Inspect run was tested. Passing tests show the specified behavior on those fixtures, not correctness across every Inspect version or log variant. The new HTML text is covered by string assertions only. The controller's node scripts/check_site.cjs run passed the existing showcase browser regression checks (keyboard, links and downloads at 1440, 390 and 320 px widths). Those checks do not exercise this feature's diff-report banner. No screenshot or visual review of the new state was performed. Behavior change: pipelines that passed with an interrupted baseline now fail. Report folders saved by 0.17.2 with an incomplete baseline will show a recomputation mismatch in evalarc verify, because they recorded a pass. Folders with finished baselines are unaffected.
Upgrade and rollback
To upgrade, install the patch release that contains commit bc6089b (the CHANGELOG 0.17.3 entry) or pin the matching Action tag. No migration is needed, and existing result files and report folders are not modified. The new diff.json key appears only when the baseline is incomplete, so schema-v1 consumers keep working. Expect pipelines that compared against an interrupted baseline to start exiting 1. The intended fix is to regenerate a finished baseline. To roll back, pin noteflowai/evalarc@v0.17.2 or evalarc==0.17.2. Nothing is rewritten, so rollback loses no data. However, rollback restores the old behavior, in which an unfinished baseline can pass the gate. If you rerun evalarc verify on a folder saved by 0.17.2 with an incomplete baseline, expect a mismatch. Regenerate that report with the new version instead of editing it.