EvalArcOPEN SOURCE / RESEARCH PREVIEW 0.12.1
Look past
the score.
A tool returned an error. The write already happened.
A retry added the same note twice.
Review an agent change before you accept it. Compare failed checks, follow the recorded actions, and keep the evidence behind your decision.
Find the regression ↘Interactive recorded example · no account, install or model key
Run your first local review ↗RECORDED CONTROL / 01NOT RESOLVED
93.75%
Partial score. Duplicated write.
add_notecommittedresponsetransient errorretry with new keyduplicate note
Scripted negative control · public seed 17
No model call or live customer data.
Watch the 30-second walkthrough
Four annotated views of saved scripted Docker controls. No audio or new agent run. Transcript and recording method ↗
3task packs
21declared faults caught
JSONdownloadable evidence
CPUreproduce without a model API
COMPARE TWO RECORDED REVISIONS
Better score. New failure.
Two closure checks improve.
A previously passing note check fails.
These recorded policies use the same task, seeds, grader and runtime. Select a changed case to compare the observed outcomes.
Loading the recorded comparison…
BASELINE / CLOSE-UNRESOLVED
CURRENT / NEW-KEY-ON-RETRY
After
Final ticket state
Reproduce this comparison without running candidates
evalarc compare evalarc-evidence-explorer/comparison/baseline.json \
evalarc-evidence-explorer/comparison/current.json \
--output runs/comparison-001
# Exit 1: a check regressed, despite the higher score.Download the records using the first-review guide and use a fresh output directory. The CLI validates report consistency and matching execution conditions before comparison. It does not authenticate the producer of the reports.
NATIVE STRANDS EVALS REPORTS
Inspect the state rules.
Recheck the same saved states with two native Strands evaluators. The eight-row mean improves from 75% to 87.5%; the notes rule still regresses.
Two equally weighted state rules, separate from EvalArc’s five-dimension score. Saved scripted controls; no new agent or AWS run.
Inspect the Strands state review →
All eight checks · expected and observed state · offline download
FOLLOW THE EVIDENCE
One score never tells the whole story.
Switch the control. Inspect the checks.
Follow each recorded state change.
Loading recorded audit evidence…
WEIGHTED SCORE
ACCEPTANCEcases passed
Action & result
State changes in this step
Initial and final state
The coding grader checks observable responses and restart behavior. This record contains the failure message and a transcript fingerprint; it does not include the agent's coding process or a full response transcript.
Reproduction record & fingerprints
DECLARE YOUR ACCEPTANCE RULES
Same score. Different gate.
One frozen policy.
Two explicit acceptance rules.
Both support jobs score 93.75% and resolve 0/2 attempts. A deliberately permissive gate accepts the result; requiring every notes check to pass rejects it. The task outcome stays the same.
Loading the recorded suite…
PERMISSIVE / NO REQUIRED DIMENSION
Partial progress allowed
Inspect both attempts ↗PROTECTED / EVERY NOTES CHECK REQUIRED
Notes must be correct
Inspect both attempts ↗Gate acceptance is a configured decision, separate from full task resolution. The permissive gate intentionally accepts an unresolved policy. No score is averaged across coding and support jobs.
Shared candidate, gate decisions & provenance
Recorded Docker suite: three jobs, five attempts, 31 case executions. JUnit distinguishes a rejected gate from an environment error; a hosted CI importer was not exercised. This is scripted development evidence.
Prefer a filterable table or Python? Open the Hugging Face Casebook ↗ and compare gate_accepted with fully_resolved. The original records accompany every row.
REPEAT ONE FROZEN CANDIDATE
One run is not the whole record.
Same cases. Fresh state.
Every attempt stays visible.
Three Docker attempts per scripted control. The reference passes every time; the duplicate-write policy keeps its 93.75% score and fails acceptance every time. Switch controls and open any attempt's full evidence.
Loading the recorded attempts…
RESOLVED ATTEMPTS
MEAN ASSESSED SCORE
INVALID ATTEMPTS
OBSERVED VARIABLE CHECKS
Frozen candidate & execution conditions
These v0.4 recordings show fixed public cases, with no observed check variation in either control. They are not stochastic model trials or an estimate of reliability on unseen tasks. The v0.3 comparison below retains its original records and grading fingerprint.
THREE TASK PACKS / DECLARED FAULTS
A perfect score.
How much coverage remains?
A fault detected by one case loses its coverage if that case is removed or weakened. Open each dependency to inspect the recorded checks, status and seed. Margins count distinct case IDs, not repeated runs.
durable-kv
8/8 declared faults detected
3 single-case dependencies
- partial-batch Sole case: rollback-batch
- boolean-equals-one Sole case: cas-type-sensitivity
- accept-nonstring-keys Sole case: reject-and-continue
support-routing
7/7 declared faults detected
2 single-case dependencies
- skip-retry Sole case: retry-before-commit
- new-key-on-retry Sole case: retry-after-commit
robot-evidence-review
6/6 declared faults detected
1 single-case dependencies
- assume-complete Sole case: incomplete-recording
These summaries derive from the original scripted audit records. They are not new agent runs or a guarantee against unseen faults.
YOUR NEXT REVIEW
Bring a change
you need to trust.
Start with two EvalArc evaluations, or prepare a saved AgentCore export using the documented input contract. Review locally, then share a minimal finding.
Export-to-review walkthrough ↗
Bounded import format; no automatic cloud collection. The scored trace controls are synthetic. Live AgentCore evaluation has not been validated by this demo.
FIRST-USE FEEDBACK
Where did the review help?
Tell us what you were checking, where you got stuck, and whether the result changed a decision. A failed setup is useful feedback too.
Share a first-use report ↗Use a minimal redacted example. No production traces, private prompts or credentials are needed.
SAVED TRACE REVIEW
Zero, skipped, or missing?
A zero can be a valid judgment. A skipped evaluator needs context. A required skill may never have loaded. Inspect each against a versioned golden case, with recording identity and explicit acceptance rules.
The five controls are synthetic. The separate MCP record contains a real local load and no evaluator scores. Review your own saved AgentCore Evaluate export offline with evalarc trace-import.
FIXED TRACE, REPEATED JUDGMENTS
Same trace. Same verdict?
Keep the execution fixed. Inspect three saved judgments per target: a changing score, a flipped acceptance decision, and unavailable assessments are different findings.
| Control | Judge run 1 | Judge run 2 | Judge run 3 |
|---|---|---|---|
| same-pass | 1 | 1 | 1 |
| same-reject | 0 | 0 | 0 |
| gate-flip | 1 | 0 | 1 |
| score-only-change | 0.8 | 0.9 | 1 |
| partial-judgments | 1 | skipped | missing |
Agreement is not accuracy: three rejections still reject the case. Agent reruns remain separate in the execution repeatability report.
RECORDED MODEL EVIDENCE
A skill loaded. Did the task pass?
27 real Qwen3-8B trials compare no skill, direct loading and MCP delivery on attributed robot recordings. Inspect every tool receipt, candidate, independent score and ATIF trajectory. Three engineering profiles, including negative results.
L40S recordings; one public development task. No skill accuracy gain or general model ranking is claimed.
HAND OFF EVIDENCE
Download the evidence.
Check every gate.
Recompute the original suite configuration, plan, five attempts, acceptance decisions and JUnit without executing a candidate.
Download suite evidence (ZIP) · Install and verify locally ↗
EVALARC 0.8+ / READ-ONLY
# Unzip the downloaded archive first.
evalarc verify suite-evidence --json
# Require all configured gates to accept:
evalarc verify suite-evidence \
--json --require-acceptedThis example is consistent (exit 0), but one gate rejects it (exit 1 with --require-accepted). Two jobs are accepted; one is fully resolved. No grader rerun or producer authentication is implied.
FROM THE BROWSER TO YOUR TERMINAL
Make the grader
earn your trust.
Run the known-good reference and the declared faulty controls. Keep the outcomes, seeds, runtime limits and fingerprints together.
Installation & execution guide ↗TRUSTED BUNDLED CONTROLS / LINUX / PYTHON 3.11+
git clone https://github.com/noteflowai/evalarc.git
cd evalarc
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .
evalarc audit --task support-routing \
--backend local --trust-local \
--output runs/supportLocal mode runs with your user privileges. Use the documented Docker backend for candidate isolation.
What this evidence establishes
The saved audits detect 21 declared faults across 3 task packs. These are public development tasks and scripted policies. They do not establish frontier-model performance, coverage of arbitrary reward hacks, a human time horizon, or RL training gains. The browser replays saved reports; it does not execute submissions.