EvalArc / Strands state review

STRANDS EVALS 1.3.0 / RECORDED STATE REVIEW

The mean improved.
One rule regressed.

The retry wrote the same note twice. Inspect the expected state beside both observed outcomes, using the native report’s case and evaluator identities.

Four saved scripted Docker cases · two deterministic state rules · no account, installation or model key needed to browse.

Baseline mean75%Not every rule passed
Current mean87.5%Not every rule passed
Regression gateREJECTED1 regression · 2 improvements

This is the equally weighted mean of eight native state-rule rows. EvalArc’s original 90% → 93.75% uses five weighted dimensions. These are different scoring scopes; the notes regression appears in both.

CASE + EVALUATOR / EVERY CHECK RETAINED

Find the change in the state.

ticket.notes compares the complete ordered list, including duplicates. ticket.status expects “closed” for a resolved ticket and “open” otherwise.

All 8 checks are available below. Expand a check to inspect its state.

route-open-ticket@17ticket.notesUNCHANGED

Expected in both revisions 

[
  "Customer contacted support.",
  "Reviewed request T-85a0bcc1."
]

Baseline PASS

[
  "Customer contacted support.",
  "Reviewed request T-85a0bcc1."
]

Current PASS

[
  "Customer contacted support.",
  "Reviewed request T-85a0bcc1."
]

Link to this check

close-resolved-ticket@17ticket.notesUNCHANGED

Expected in both revisions 

[
  "Customer contacted support.",
  "Reviewed request T-c41f9dfd."
]

Baseline PASS

[
  "Customer contacted support.",
  "Reviewed request T-c41f9dfd."
]

Current PASS

[
  "Customer contacted support.",
  "Reviewed request T-c41f9dfd."
]

Link to this check

retry-before-commit@17ticket.notesUNCHANGED

Expected in both revisions 

[
  "Customer contacted support.",
  "Reviewed request T-3fb81d27."
]

Baseline PASS

[
  "Customer contacted support.",
  "Reviewed request T-3fb81d27."
]

Current PASS

[
  "Customer contacted support.",
  "Reviewed request T-3fb81d27."
]

Link to this check

retry-after-commit@17ticket.notesREGRESSION

Expected in both revisions 

[
  "Customer contacted support.",
  "Reviewed request T-cd68615c."
]

Baseline PASS

[
  "Customer contacted support.",
  "Reviewed request T-cd68615c."
]

Current FAIL

[
  "Customer contacted support.",
  "Reviewed request T-cd68615c.",
  "Reviewed request T-cd68615c."
]

Link to this check

route-open-ticket@17ticket.statusIMPROVEMENT

Expected in both revisions 

"open"

Baseline FAIL

"closed"

Current PASS

"open"

Link to this check

close-resolved-ticket@17ticket.statusUNCHANGED

Expected in both revisions 

"closed"

Baseline PASS

"closed"

Current PASS

"closed"

Link to this check

retry-before-commit@17ticket.statusIMPROVEMENT

Expected in both revisions 

"open"

Baseline FAIL

"closed"

Current PASS

"open"

Link to this check

retry-after-commit@17ticket.statusUNCHANGED

Expected in both revisions 

"closed"

Baseline PASS

"closed"

Current PASS

"closed"

Link to this check

KEEP THE EVIDENCE

A review you can hand off.

Download the HTML, both native reports and original EvalArc inputs together. Extract the ZIP and open strands-review/index.html; all checks and filters work offline.

Download Strands review (ZIP)

Native baseline JSONNative current JSONComparison JSONOriginal baseline JSONOriginal current JSONFile checksums
Native report fingerprints
{
  "baseline": "b5c6d53203cf7561c26c7597e7afecc797ed705bde506da16ae6691595fd1ce9",
  "current": "49994b4bc5ed730428e362df4f00697c652ed0b840cc270a7b765c5d283f805b"
}

Check links identify the case, evaluator and both native report files. File hashes establish content identity, not the identity of the producer.

The site build rechecks the saved rule outcomes against the original state. CI separately runs the real SDK with network attempts rejected. Neither process reruns an agent or calls AWS. Strands already includes per-row verdicts; this independent companion displays them.

Reproduce with the pinned SDK ↗ · Share a first-use finding ↗