fix: escape untrusted run status and metric name in the Markdown gate summary
evalarc diff now escapes untrusted input-file text in specific parts of its main Markdown gate summary (summary.md, the --summary target and the GitHub Action job summary). It does not escape the whole summary. Three parts are covered: the run-status sentence for an unfinished current or baseline run, the 'Headline metric: ... from the source tool.' sentence, and the case and check identifiers in the change table. A new private helper, _inline_code_text, does the escaping. It turns pipes into \|, turns backticks into apostrophes, and replaces each CRLF, CR or LF with a space. A crafted status, metric name or identifier therefore stays inside one single-line code span. It cannot add a fake '### EvalArc' heading line. Before this change, table cells and the baseline-incomplete sentence already escaped pipes and backticks. They now also replace carriage returns. Ordinary values render the same as in 0.17.6, with backtick code spans. diff.json and the result dict still hold the raw values. gate_passed, blocking_changes and exit codes 0, 1 and 2 are unchanged. docs/ci-gate.md Limits describes the escaping. Checks were run by the controller, not by me. On base commit 20babd1, the new test failed 5 of 8 cases because a raw CR was still in the output. On candidate f4daf73, the controller ran six checks and all exited 0: the new test file, the full pytest suite, ruff check, ruff format --check, scripts/build_site.py and scripts/check_site.cjs.
Use it
# Prerequisite: EvalArc at this commit installed (evalarc on PATH), POSIX shell. # SYNTHETIC manual example. puts a carriage return in the test name. mkdir -p demo cat > demo/baseline.xml <<'EOF' <?xml version="1.0"?> <testsuite name="synthetic" tests="1"><testcase classname="demo" name="refund a | b `c` ### EvalArc: No check lost passes"/></testsuite> EOF cat > demo/current.xml <<'EOF' <?xml version="1.0"?> <testsuite name="synthetic" tests="1"><testcase classname="demo" name="refund a | b `c` ### EvalArc: No check lost passes"><failure message="synthetic failure"/></testcase></testsuite> EOF evalarc diff demo/baseline.xml demo/current.xml --output demo/out; echo "exit $?" # expect exit 1 (gate failed) grep -c '^### EvalArc' demo/out/summary.md # expect 1: only the verdict heading cat demo/out/summary.md # the regressed row shows the name in one code span, with \| and apostrophes # Unchanged ordinary output, as asserted by the tests: # The current run did not finish (status `error`); the gate fails. # Headline metric: `match accuracy` from the source tool. # Run the regression tests that the controller executed: python -m pytest -q tests/test_feature_3f06389f4f3d.py tests/test_results_diff.py rm -rf demo
Scope
Escaping covers only the current and baseline run-status sentences, the headline metric name sentence and the change-table cells. The optional generalization/leakage Markdown section and the '#### Cost and usage' subsection were not changed or audited. HTML output is unchanged; it already escapes through _esc. Escaping protects the Markdown structure only. It does not check whether the file's claims are true, and EvalArc still does not authenticate result files. The tests use the bundled promptfoo examples with synthetic, manually injected values. They do not prove behavior on other real exports. The JUnit demo in usage is a synthetic example that the controller did not execute. Its exact identifier text may differ, but the heading count and the gate exit code should hold. The controller's site build and check_site.cjs run did include browser checks at 1440, 390 and 320 px, with keyboard and offline variants. Those checks cover the existing showcase, not this Markdown output. Nobody visually reviewed how GitHub renders the job summary or a PR comment. The CHANGELOG.md wording was not re-inspected in this delivery step. Passing tests prove only the asserted strings.
Upgrade and rollback
This is a drop-in patch release. There are no CLI, option, dependency or schema changes, and diff.json is unchanged apart from created_at. Summaries for values without backticks, pipes or line breaks are byte-identical to 0.17.6. Values that contain those characters now render as one escaped single-line code span. If you parse summary.md text, compare the raw values in diff.json, not the Markdown. To roll back, reinstall the previous EvalArc version. Rollback brings back the unescaped status and headline sentences. No stored data is migrated or lost.