EvalArcPERMISSIVE / NO REQUIRED DIMENSION
OPEN SOURCE / RESEARCH PREVIEW 0.10.0
Look past
the score.
A tool returned an error. The write already happened.
A retry added the same note twice.
EvalArc tests whether an agent's grader catches the difference between plausible progress and a completed task.
Inspect the acceptance gate ↘ Every repeated attempt ↓ Compare revision regressions ↓RECORDED CONTROL / 01NOT RESOLVED
93.75%
Partial score. Duplicated write.
add_notecommittedresponsetransient errorretry with new keyduplicate note
Scripted negative control · public seed 17
No model call or live customer data.
Generate Python or JavaScript candidates and audit independent controls against the same task contracts. Try both runtimes ↗. The recorded showcases below retain their original versions and fingerprints.
NEW / RECORDED MODEL EVIDENCE
A skill loaded. Did the task pass?
27 real Qwen3-8B trials compare no skill, direct loading and MCP delivery on attributed robot recordings. Inspect every tool receipt, candidate, independent score and ATIF trajectory. Three engineering profiles, including negative results.
L40S recordings; one public development task. No skill accuracy gain or general model ranking is claimed.
2task packs
15declared faults caught
JSONdownloadable evidence
CPUreproduce without a model API
NEW IN 0.5 / DECLARE YOUR ACCEPTANCE RULES
Same score. Different gate.
One frozen policy.
Two explicit acceptance rules.
Both support jobs score 93.75% and resolve 0/2 attempts. A deliberately permissive gate accepts the result; requiring every notes check to pass rejects it. The task outcome stays the same.
Loading the recorded suite…
PROTECTED / EVERY NOTES CHECK REQUIRED
Notes must be correct
Inspect both attempts ↗Gate acceptance is a configured decision, separate from full task resolution. The permissive gate intentionally accepts an unresolved policy. No score is averaged across coding and support jobs.
Shared candidate, gate decisions & provenance
Recorded Docker suite: three jobs, five attempts, 31 case executions. JUnit distinguishes a rejected gate from an environment error; a hosted CI importer was not exercised. This is scripted development evidence.
Prefer a filterable table or Python? Open the Hugging Face Casebook ↗ and compare gate_accepted with fully_resolved. The original records accompany every row.
NEW IN 0.4 / REPEAT ONE FROZEN CANDIDATE
One run is not the whole record.
Same cases. Fresh state.
Every attempt stays visible.
Three Docker attempts per scripted control. The reference passes every time; the duplicate-write policy keeps its 93.75% score and fails acceptance every time. Switch controls and open any attempt's full evidence.
Loading the recorded attempts…
RESOLVED ATTEMPTS
MEAN ASSESSED SCORE
INVALID ATTEMPTS
OBSERVED VARIABLE CHECKS
Frozen candidate & execution conditions
These v0.4 recordings show fixed public cases, with no observed check variation in either control. They are not stochastic model trials or an estimate of reliability on unseen tasks. The v0.3 comparison below retains its original records and grading fingerprint.
NEW IN 0.3 / COMPARE TWO REVISIONS
Better score. New failure.
Two closure checks improve.
A previously passing note check fails.
These recorded policies use the same task, seeds, grader and runtime. Select a changed case to compare the observed outcomes.
Loading the recorded comparison…
BASELINE / CLOSE-UNRESOLVED
CURRENT / NEW-KEY-ON-RETRY
Before
Final ticket state
After
Final ticket state
Reproduce this comparison without running candidates
evalarc compare examples/comparison/baseline.json \
examples/comparison/current.json \
--output runs/comparison-001
# Exit 1: a check regressed, despite the higher score.Use a fresh output directory. The CLI validates report consistency and matching execution conditions before comparison. It does not authenticate the producer of the reports.
FOLLOW THE EVIDENCE
One score never tells the whole story.
Switch the control. Inspect the checks.
Follow each recorded state change.
Loading recorded audit evidence…
WEIGHTED SCORE
ACCEPTANCEcases passed
Action & result
State changes in this step
Initial and final state
The coding grader checks observable responses and restart behavior. This record contains the failure message and a transcript fingerprint; it does not include the agent's coding process or a full response transcript.
Reproduction record & fingerprints
HAND OFF EVIDENCE
Download the evidence.
Check every gate.
Recompute the original suite configuration, plan, five attempts, acceptance decisions and JUnit without executing a candidate.
Download suite evidence (ZIP) · Offline verification guide ↗
EVALARC 0.8+ / READ-ONLY
# Unzip the downloaded archive first.
evalarc verify suite-evidence --json
# Require all configured gates to accept:
evalarc verify suite-evidence \
--json --require-acceptedThis example is consistent (exit 0), but one gate rejects it (exit 1 with --require-accepted). Two jobs are accepted; one is fully resolved. No grader rerun or producer authentication is implied.
FROM THE BROWSER TO YOUR TERMINAL
Make the grader
earn your trust.
Run the known-good reference and the declared faulty controls. Keep the outcomes, seeds, runtime limits and fingerprints together.
Installation & execution guide ↗TRUSTED BUNDLED CONTROLS / LINUX / PYTHON 3.11+
git clone https://github.com/noteflowai/evalarc.git
cd evalarc
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .
evalarc audit --task support-routing \
--backend local --trust-local \
--output runs/supportLocal mode runs with your user privileges. Use the documented Docker backend for candidate isolation.
What this evidence establishes
The bundled references pass and the 15 declared faulty implementations fail their intended checks. These are public development tasks and scripted policies. They do not establish frontier-model performance, coverage of arbitrary reward hacks, a human time horizon, or RL training gains. The browser replays saved reports; it does not execute submissions.