One fixed recording · repeated judgments
Same trace.Same verdict? SYNTHETIC Five authored controls with three judgment sets on identical simulated spans. Scores and placeholder identities are synthetic; no model or AWS evaluation.
Separate a changing score, a flipped acceptance decision and an unavailable judgment. All-reject agreement is still failure; repeated agreement is not accuracy.
Descriptive agreement of imported judgments on the same declared recording and rubric. No agent or judge is executed. Repetition IDs and evaluator revisions are caller-declared; agreement does not establish independence, accuracy, calibration or task success.
3 Saved judgments per target
1 Targets with observed gate disagreement
1 Targets with missing assessments
4/5 required targets have all judgments; 2 show observed score variation. 1 were rejected in every judgment. 0 are not applicable and excluded from coverage.
Import your own saved judgments Save each repeated evaluation of the same recording as a Trace Workbench input with a distinct run_id. Freeze evaluator revisions and configuration. Agent reruns with new spans belong in a separate execution comparison.
evalarc trace-stability judge-1.json judge-2.json --output runs/judge-review
evalarc trace-stability-verify runs/judge-review All Gate disagreement Incomplete Same gate Search case, evaluator or explanation
same-pass Same observed gate Inspect the same-pass condition.
authored-goal 3/3 expected judgments assessed · 3 pass · 0 reject · 0 unassessed
Observed values: 1 . Score variation: not observed.
Frozen target, rating rules and overall case gates {
"session_id": "session-0",
"trace_id": null,
"span_id": null,
"revision": "control-1; no model or AWS call",
"rating": {
"kind": "numeric",
"min": 0,
"max": 1,
"pass_at_least": 0.8
},
"case_gates": [
{
"run_id": "authored-judgment-1",
"gate": "accepted"
},
{
"run_id": "authored-judgment-2",
"gate": "accepted"
},
{
"run_id": "authored-judgment-3",
"gate": "accepted"
}
]
}
same-reject Same observed gate Inspect the same-reject condition.
authored-goal 3/3 expected judgments assessed · 0 pass · 3 reject · 0 unassessed
Observed values: 0 . Score variation: not observed.
Frozen target, rating rules and overall case gates {
"session_id": "session-1",
"trace_id": null,
"span_id": null,
"revision": "control-1; no model or AWS call",
"rating": {
"kind": "numeric",
"min": 0,
"max": 1,
"pass_at_least": 0.8
},
"case_gates": [
{
"run_id": "authored-judgment-1",
"gate": "rejected"
},
{
"run_id": "authored-judgment-2",
"gate": "rejected"
},
{
"run_id": "authored-judgment-3",
"gate": "rejected"
}
]
}
gate-flip Gate disagreement Inspect the gate-flip condition.
authored-goal 3/3 expected judgments assessed · 2 pass · 1 reject · 0 unassessed
Both pass and reject observed.
Observed values: 0, 1 . Score variation: yes.
Frozen target, rating rules and overall case gates {
"session_id": "session-2",
"trace_id": null,
"span_id": null,
"revision": "control-1; no model or AWS call",
"rating": {
"kind": "numeric",
"min": 0,
"max": 1,
"pass_at_least": 0.8
},
"case_gates": [
{
"run_id": "authored-judgment-1",
"gate": "accepted"
},
{
"run_id": "authored-judgment-2",
"gate": "rejected"
},
{
"run_id": "authored-judgment-3",
"gate": "accepted"
}
]
}
score-only-change Same observed gate Inspect the score-only-change condition.
authored-goal 3/3 expected judgments assessed · 3 pass · 0 reject · 0 unassessed
Observed values: 0.8, 0.9, 1 . Score variation: yes.
Frozen target, rating rules and overall case gates {
"session_id": "session-3",
"trace_id": null,
"span_id": null,
"revision": "control-1; no model or AWS call",
"rating": {
"kind": "numeric",
"min": 0,
"max": 1,
"pass_at_least": 0.8
},
"case_gates": [
{
"run_id": "authored-judgment-1",
"gate": "accepted"
},
{
"run_id": "authored-judgment-2",
"gate": "accepted"
},
{
"run_id": "authored-judgment-3",
"gate": "accepted"
}
]
}
partial-judgments Incomplete judgments Inspect the partial-judgments condition.
authored-goal 1/3 expected judgments assessed · 1 pass · 0 reject · 2 unassessed
Observed values: 1 . Score variation: not observed.
Frozen target, rating rules and overall case gates {
"session_id": "session-4",
"trace_id": null,
"span_id": null,
"revision": "control-1; no model or AWS call",
"rating": {
"kind": "numeric",
"min": 0,
"max": 1,
"pass_at_least": 0.8
},
"case_gates": [
{
"run_id": "authored-judgment-1",
"gate": "accepted"
},
{
"run_id": "authored-judgment-2",
"gate": "incomplete"
},
{
"run_id": "authored-judgment-3",
"gate": "incomplete"
}
]
} Preserved input files and recording identity Fixed recording SHA-256:
8bd84193b28c09bffe8d140d9dacaf46d729c5300f9babc4b709a06d7dcb021c {
"configuration": {
"model": "synthetic-control",
"model_parameters": {},
"prompt_sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
"tools_sha256": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
"skills": {}
},
"evaluators": [
{
"id": "authored-goal",
"revision": "control-1; no model or AWS call",
"level": "session",
"rating": {
"kind": "numeric",
"min": 0,
"max": 1,
"pass_at_least": 0.8
}
}
]
}