EVALARC / DEVELOPMENT AUDIT
Test the grader.
Then trust the signal.
✓ 7 of 7 declared faults detected
2 fault(s) depend on a single detecting case.
Next: Add cases that detect the single-case faults independently.
Behavioral controls for task-specific agent evaluations. Correct and faulty submissions are evaluated against the same externally enforced contract.
7/7
declared faults detected
1.000
reference correctness
docker
execution backend
Which faults depend on one case?
2 of 7 declared faults have exactly one distinct detecting case in the assessed records. Repeating a case under more seeds does not increase this margin. A surviving fault has margin zero; an environment failure remains unassessed.
Margins below are derived from the saved failing case IDs. Rendering an older audit does not rerun its controls.
Does the grader detect plausible defects?
| Negative control | Target | Candidate score | Result | Margin | Evidence |
|---|---|---|---|---|---|
claim-without-actions | routing | 0.350 | Detected | 4 | Inspect recorded cases |
wrong-ticket | scope | 0.200 | Detected | 4 | Inspect recorded cases |
wrong-queue | routing | 0.700 | Detected | 4 | Inspect recorded cases |
duplicate-note | notes | 0.750 | Detected | 4 | Inspect recorded cases |
close-unresolved | closure | 0.900 | Detected | 2 | Inspect recorded cases |
skip-retry | notes | 0.812 | Detected | 1 | Inspect recorded cases |
new-key-on-retry | notes | 0.938 | Detected | 1 | Inspect recorded cases |
Inspect the detecting cases
claim-without-actions · 4 detecting cases
Target: routing. Detecting case IDs: close-resolved-ticket, retry-after-commit, retry-before-commit, route-open-ticket.
route-open-ticket · seed 17 · failed
{
"seed": 17,
"case_id": "route-open-ticket",
"checks": {
"routing": false,
"notes": false,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.713709,
"transcript_sha256": "dc89878d8618444cc445bec619924daccdde74b03fca5826ccd7ef1d02d4245e",
"initial_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "finish",
"action": {
"finish": true,
"message": "Completed all requested work."
}
}
],
"tool_calls": 0
}close-resolved-ticket · seed 17 · failed
{
"seed": 17,
"case_id": "close-resolved-ticket",
"checks": {
"routing": false,
"notes": false,
"closure": false,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.617806,
"transcript_sha256": "dc89878d8618444cc445bec619924daccdde74b03fca5826ccd7ef1d02d4245e",
"initial_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "finish",
"action": {
"finish": true,
"message": "Completed all requested work."
}
}
],
"tool_calls": 0
}retry-before-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-before-commit",
"checks": {
"routing": false,
"notes": false,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.635735,
"transcript_sha256": "dc89878d8618444cc445bec619924daccdde74b03fca5826ccd7ef1d02d4245e",
"initial_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "finish",
"action": {
"finish": true,
"message": "Completed all requested work."
}
}
],
"tool_calls": 0
}retry-after-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-after-commit",
"checks": {
"routing": false,
"notes": false,
"closure": false,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.642983,
"transcript_sha256": "dc89878d8618444cc445bec619924daccdde74b03fca5826ccd7ef1d02d4245e",
"initial_state": {
"T-cd68615c": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-cd68615c": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "finish",
"action": {
"finish": true,
"message": "Completed all requested work."
}
}
],
"tool_calls": 0
}wrong-ticket · 4 detecting cases
Target: scope. Detecting case IDs: close-resolved-ticket, retry-after-commit, retry-before-commit, route-open-ticket.
route-open-ticket · seed 17 · failed
{
"seed": 17,
"case_id": "route-open-ticket",
"checks": {
"routing": false,
"notes": false,
"closure": true,
"scope": false,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.66702,
"transcript_sha256": "13bb6318b8c0c1e61289c5ad6878ae37d2cb7a2591279528eea61699f94cd047",
"initial_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "queue-0-0",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket.",
"Reviewed request T-85a0bcc1."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "U-6a06e9ab"
}
},
"result": {
"ok": true,
"ticket": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "U-6a06e9ab",
"queue": "queue-0-0"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"U-6a06e9ab": {
"before": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
},
"after": {
"category": "billing",
"queue": "queue-0-0",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "U-6a06e9ab",
"body": "Reviewed request T-85a0bcc1.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"U-6a06e9ab": {
"before": {
"category": "billing",
"queue": "queue-0-0",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
},
"after": {
"category": "billing",
"queue": "queue-0-0",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket.",
"Reviewed request T-85a0bcc1."
]
}
}
}
},
{
"step": 3,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 3
}close-resolved-ticket · seed 17 · failed
{
"seed": 17,
"case_id": "close-resolved-ticket",
"checks": {
"routing": false,
"notes": false,
"closure": false,
"scope": false,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.688977,
"transcript_sha256": "af34c4df1f605b18475bb88594ec02c95aa692cdedc06945af120e864d7b20d1",
"initial_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "queue-1-0",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket.",
"Reviewed request T-c41f9dfd."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "U-b48438b5"
}
},
"result": {
"ok": true,
"ticket": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "U-b48438b5",
"queue": "queue-1-0"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"U-b48438b5": {
"before": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
},
"after": {
"category": "billing",
"queue": "queue-1-0",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "U-b48438b5",
"body": "Reviewed request T-c41f9dfd.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"U-b48438b5": {
"before": {
"category": "billing",
"queue": "queue-1-0",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
},
"after": {
"category": "billing",
"queue": "queue-1-0",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket.",
"Reviewed request T-c41f9dfd."
]
}
}
}
},
{
"step": 3,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 3
}retry-before-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-before-commit",
"checks": {
"routing": false,
"notes": false,
"closure": true,
"scope": false,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.661649,
"transcript_sha256": "dce95ae090de1ec11795608277ea5d9d4ea46545741bdd9f89e8b351d2221909",
"initial_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "queue-2-2",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket.",
"Reviewed request T-3fb81d27."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "U-62590992"
}
},
"result": {
"ok": true,
"ticket": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "U-62590992",
"queue": "queue-2-2"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "U-62590992",
"queue": "queue-2-2"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"U-62590992": {
"before": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
},
"after": {
"category": "account",
"queue": "queue-2-2",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "U-62590992",
"body": "Reviewed request T-3fb81d27.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"U-62590992": {
"before": {
"category": "account",
"queue": "queue-2-2",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
},
"after": {
"category": "account",
"queue": "queue-2-2",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket.",
"Reviewed request T-3fb81d27."
]
}
}
}
},
{
"step": 4,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 4
}retry-after-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-after-commit",
"checks": {
"routing": false,
"notes": false,
"closure": false,
"scope": false,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.558768,
"transcript_sha256": "cfc5d50e5cf0c9e319b39869d751e334459cfa948ea4693b7a24d9652ee8d30e",
"initial_state": {
"T-cd68615c": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-cd68615c": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-51436d1f": {
"category": "account",
"queue": "queue-3-2",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket.",
"Reviewed request T-cd68615c."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "U-51436d1f"
}
},
"result": {
"ok": true,
"ticket": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "U-51436d1f",
"queue": "queue-3-2"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"U-51436d1f": {
"before": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
},
"after": {
"category": "account",
"queue": "queue-3-2",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "U-51436d1f",
"body": "Reviewed request T-cd68615c.",
"request_id": "review"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {
"U-51436d1f": {
"before": {
"category": "account",
"queue": "queue-3-2",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
},
"after": {
"category": "account",
"queue": "queue-3-2",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "U-51436d1f",
"body": "Reviewed request T-cd68615c.",
"request_id": "review"
}
},
"result": {
"ok": true,
"replayed": true
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 4,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 4
}wrong-queue · 4 detecting cases
Target: routing. Detecting case IDs: close-resolved-ticket, retry-after-commit, retry-before-commit, route-open-ticket.
route-open-ticket · seed 17 · failed
{
"seed": 17,
"case_id": "route-open-ticket",
"checks": {
"routing": false,
"notes": true,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.576444,
"transcript_sha256": "06bfdd936f8a75a1f79fac629e71818d6f8750e756c87b7318481642641860f9",
"initial_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "queue-0-0",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-85a0bcc1"
}
},
"result": {
"ok": true,
"ticket": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-85a0bcc1",
"queue": "queue-0-0"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-0-0",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-85a0bcc1",
"body": "Reviewed request T-85a0bcc1.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "queue-0-0",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-0-0",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
}
}
}
},
{
"step": 3,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 3
}close-resolved-ticket · seed 17 · failed
{
"seed": 17,
"case_id": "close-resolved-ticket",
"checks": {
"routing": false,
"notes": true,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.552313,
"transcript_sha256": "18b3105a6a7f077107630156bf9124018c8271feaf28f6fd778cad881d67e0b2",
"initial_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "queue-1-1",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-c41f9dfd"
}
},
"result": {
"ok": true,
"ticket": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-c41f9dfd",
"queue": "queue-1-1"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-1-1",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-c41f9dfd",
"body": "Reviewed request T-c41f9dfd.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "queue-1-1",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-1-1",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "close_ticket",
"arguments": {
"ticket_id": "T-c41f9dfd"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "queue-1-1",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
},
"after": {
"category": "billing",
"queue": "queue-1-1",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
}
}
}
},
{
"step": 4,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 4
}retry-before-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-before-commit",
"checks": {
"routing": false,
"notes": true,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.55564,
"transcript_sha256": "8bb84e30d03030ac20e834eae63758c4041341b0325cebc35e1aeca01d59ccfe",
"initial_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "queue-2-0",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-3fb81d27"
}
},
"result": {
"ok": true,
"ticket": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-3fb81d27",
"queue": "queue-2-0"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-3fb81d27",
"queue": "queue-2-0"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-2-0",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-3fb81d27",
"body": "Reviewed request T-3fb81d27.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "queue-2-0",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-2-0",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
}
}
}
},
{
"step": 4,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 4
}retry-after-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-after-commit",
"checks": {
"routing": false,
"notes": true,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.538827,
"transcript_sha256": "3076729f9cfaeec8c787fb9216588135070ea5aee843c853840be4436629ef92",
"initial_state": {
"T-cd68615c": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-cd68615c": {
"category": "billing",
"queue": "queue-3-1",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-cd68615c"
}
},
"result": {
"ok": true,
"ticket": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-cd68615c",
"queue": "queue-3-1"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-3-1",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-cd68615c",
"body": "Reviewed request T-cd68615c.",
"request_id": "review"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-1",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-3-1",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-cd68615c",
"body": "Reviewed request T-cd68615c.",
"request_id": "review"
}
},
"result": {
"ok": true,
"replayed": true
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 4,
"type": "tool_call",
"action": {
"tool": "close_ticket",
"arguments": {
"ticket_id": "T-cd68615c"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-1",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
},
"after": {
"category": "billing",
"queue": "queue-3-1",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 5,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 5
}duplicate-note · 4 detecting cases
Target: notes. Detecting case IDs: close-resolved-ticket, retry-after-commit, retry-before-commit, route-open-ticket.
route-open-ticket · seed 17 · failed
{
"seed": 17,
"case_id": "route-open-ticket",
"checks": {
"routing": true,
"notes": false,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.585063,
"transcript_sha256": "eba30506565d7d2797e622ec54845d26c0b3a9fc9e323c298cbc76dc390e082c",
"initial_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1.",
"Reviewed request T-85a0bcc1."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-85a0bcc1"
}
},
"result": {
"ok": true,
"ticket": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-85a0bcc1",
"queue": "queue-0-2"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-85a0bcc1",
"body": "Reviewed request T-85a0bcc1.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-85a0bcc1",
"body": "Reviewed request T-85a0bcc1.",
"request_id": "duplicate"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
},
"after": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1.",
"Reviewed request T-85a0bcc1."
]
}
}
}
},
{
"step": 4,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 4
}close-resolved-ticket · seed 17 · failed
{
"seed": 17,
"case_id": "close-resolved-ticket",
"checks": {
"routing": true,
"notes": false,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.568052,
"transcript_sha256": "7b2dd2bb2f48dc19e31cd072abca3fd5d43d9d7369a1a19077e3bcf0c45ae229",
"initial_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd.",
"Reviewed request T-c41f9dfd."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-c41f9dfd"
}
},
"result": {
"ok": true,
"ticket": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-c41f9dfd",
"queue": "queue-1-0"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-c41f9dfd",
"body": "Reviewed request T-c41f9dfd.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-c41f9dfd",
"body": "Reviewed request T-c41f9dfd.",
"request_id": "duplicate"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
},
"after": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd.",
"Reviewed request T-c41f9dfd."
]
}
}
}
},
{
"step": 4,
"type": "tool_call",
"action": {
"tool": "close_ticket",
"arguments": {
"ticket_id": "T-c41f9dfd"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd.",
"Reviewed request T-c41f9dfd."
]
},
"after": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd.",
"Reviewed request T-c41f9dfd."
]
}
}
}
},
{
"step": 5,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 5
}retry-before-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-before-commit",
"checks": {
"routing": true,
"notes": false,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.538005,
"transcript_sha256": "34dc374c3e62cf1911796b91f4c3d7b6d1f7c14a4ac997c8bcd8d7ed2c7f6fdd",
"initial_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27.",
"Reviewed request T-3fb81d27."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-3fb81d27"
}
},
"result": {
"ok": true,
"ticket": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-3fb81d27",
"queue": "queue-2-1"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-3fb81d27",
"queue": "queue-2-1"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-3fb81d27",
"body": "Reviewed request T-3fb81d27.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
}
}
}
},
{
"step": 4,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-3fb81d27",
"body": "Reviewed request T-3fb81d27.",
"request_id": "duplicate"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
},
"after": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27.",
"Reviewed request T-3fb81d27."
]
}
}
}
},
{
"step": 5,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 5
}retry-after-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-after-commit",
"checks": {
"routing": true,
"notes": false,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.595631,
"transcript_sha256": "877a29b2016a4c4dd8565f1fb9987faf298ad23e39683d63b32762d7abfcadf2",
"initial_state": {
"T-cd68615c": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-cd68615c": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c.",
"Reviewed request T-cd68615c."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-cd68615c"
}
},
"result": {
"ok": true,
"ticket": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-cd68615c",
"queue": "queue-3-0"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-cd68615c",
"body": "Reviewed request T-cd68615c.",
"request_id": "review"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-cd68615c",
"body": "Reviewed request T-cd68615c.",
"request_id": "review"
}
},
"result": {
"ok": true,
"replayed": true
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 4,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-cd68615c",
"body": "Reviewed request T-cd68615c.",
"request_id": "duplicate"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 5,
"type": "tool_call",
"action": {
"tool": "close_ticket",
"arguments": {
"ticket_id": "T-cd68615c"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c.",
"Reviewed request T-cd68615c."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 6,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 6
}close-unresolved · 2 detecting cases
Target: closure. Detecting case IDs: retry-before-commit, route-open-ticket.
route-open-ticket · seed 17 · failed
{
"seed": 17,
"case_id": "route-open-ticket",
"checks": {
"routing": true,
"notes": true,
"closure": false,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.554959,
"transcript_sha256": "3b8ea18d1468afe56a69b06b884469141aa5aed9ab211a2c60378f64aaf456e5",
"initial_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-85a0bcc1"
}
},
"result": {
"ok": true,
"ticket": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-85a0bcc1",
"queue": "queue-0-2"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-85a0bcc1",
"body": "Reviewed request T-85a0bcc1.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "close_ticket",
"arguments": {
"ticket_id": "T-85a0bcc1"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
},
"after": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
}
}
}
},
{
"step": 4,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 4
}retry-before-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-before-commit",
"checks": {
"routing": true,
"notes": true,
"closure": false,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.540602,
"transcript_sha256": "4d7abd25df6f1a223a236570e6b1d0f5d31f8f0343a6497a27b12f396f65e2ca",
"initial_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-3fb81d27"
}
},
"result": {
"ok": true,
"ticket": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-3fb81d27",
"queue": "queue-2-1"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-3fb81d27",
"queue": "queue-2-1"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-3fb81d27",
"body": "Reviewed request T-3fb81d27.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
}
}
}
},
{
"step": 4,
"type": "tool_call",
"action": {
"tool": "close_ticket",
"arguments": {
"ticket_id": "T-3fb81d27"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
},
"after": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
}
}
}
},
{
"step": 5,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 5
}skip-retry · single-case dependency
Target: notes. Detecting case IDs: retry-before-commit.
retry-before-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-before-commit",
"checks": {
"routing": false,
"notes": false,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.566178,
"transcript_sha256": "7700d40c0510e616f031164599fb05256c512aae5f280105701d533ec030f826",
"initial_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-3fb81d27"
}
},
"result": {
"ok": true,
"ticket": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-3fb81d27",
"queue": "queue-2-1"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {}
},
{
"step": 2,
"type": "finish",
"action": {
"finish": true,
"message": "Could not complete the operation."
}
}
],
"tool_calls": 2
}new-key-on-retry · single-case dependency
Target: notes. Detecting case IDs: retry-after-commit.
retry-after-commit · seed 17 · failed
{
"seed": 17,
"case_id": "retry-after-commit",
"checks": {
"routing": true,
"notes": false,
"closure": true,
"scope": true,
"protocol": true
},
"passed": false,
"status": "failed",
"error": null,
"duration_seconds": 0.58689,
"transcript_sha256": "4d1123f9ef2ebc43d80c84e4f2054b3ed62a22db7d6c565f0f7f3a9f28c662b4",
"initial_state": {
"T-cd68615c": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-cd68615c": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c.",
"Reviewed request T-cd68615c."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-cd68615c"
}
},
"result": {
"ok": true,
"ticket": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-cd68615c",
"queue": "queue-3-0"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-cd68615c",
"body": "Reviewed request T-cd68615c.",
"request_id": "review"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-cd68615c",
"body": "Reviewed request T-cd68615c.",
"request_id": "review-retry"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 4,
"type": "tool_call",
"action": {
"tool": "close_ticket",
"arguments": {
"ticket_id": "T-cd68615c"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c.",
"Reviewed request T-cd68615c."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 5,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 5
}Positive control
routing1.00 4/4
notes1.00 4/4
closure1.00 4/4
scope1.00 4/4
protocol1.00 4/4
Reference execution evidence
route-open-ticket · seed 17 · passed
{
"seed": 17,
"case_id": "route-open-ticket",
"checks": {
"routing": true,
"notes": true,
"closure": true,
"scope": true,
"protocol": true
},
"passed": true,
"status": "passed",
"error": null,
"duration_seconds": 0.802482,
"transcript_sha256": "7b5b4973e934a60c55b95bda6eedf79fcff62764fbc1b81723c5126608caf4bd",
"initial_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-85a0bcc1": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
},
"U-6a06e9ab": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-85a0bcc1"
}
},
"result": {
"ok": true,
"ticket": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-85a0bcc1",
"queue": "queue-0-2"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-85a0bcc1",
"body": "Reviewed request T-85a0bcc1.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-85a0bcc1": {
"before": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-0-2",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-85a0bcc1."
]
}
}
}
},
{
"step": 3,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 3
}close-resolved-ticket · seed 17 · passed
{
"seed": 17,
"case_id": "close-resolved-ticket",
"checks": {
"routing": true,
"notes": true,
"closure": true,
"scope": true,
"protocol": true
},
"passed": true,
"status": "passed",
"error": null,
"duration_seconds": 0.704571,
"transcript_sha256": "a260768c11b589ce3f58658cac6aae8a01d36e388a47fc47c7995f8d5c36708e",
"initial_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-c41f9dfd": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
},
"U-b48438b5": {
"category": "billing",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-c41f9dfd"
}
},
"result": {
"ok": true,
"ticket": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-c41f9dfd",
"queue": "queue-1-0"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-c41f9dfd",
"body": "Reviewed request T-c41f9dfd.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "close_ticket",
"arguments": {
"ticket_id": "T-c41f9dfd"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-c41f9dfd": {
"before": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
},
"after": {
"category": "billing",
"queue": "queue-1-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-c41f9dfd."
]
}
}
}
},
{
"step": 4,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 4
}retry-before-commit · seed 17 · passed
{
"seed": 17,
"case_id": "retry-before-commit",
"checks": {
"routing": true,
"notes": true,
"closure": true,
"scope": true,
"protocol": true
},
"passed": true,
"status": "passed",
"error": null,
"duration_seconds": 0.710904,
"transcript_sha256": "b1d1d91dc01fc4bfb3243aa3739e953a40aa3920caf63a4b2ab73c87ecabda66",
"initial_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-3fb81d27": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
},
"U-62590992": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-3fb81d27"
}
},
"result": {
"ok": true,
"ticket": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-3fb81d27",
"queue": "queue-2-1"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-3fb81d27",
"queue": "queue-2-1"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-3fb81d27",
"body": "Reviewed request T-3fb81d27.",
"request_id": "review"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-3fb81d27": {
"before": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "technical",
"queue": "queue-2-1",
"resolved": false,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-3fb81d27."
]
}
}
}
},
{
"step": 4,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 4
}retry-after-commit · seed 17 · passed
{
"seed": 17,
"case_id": "retry-after-commit",
"checks": {
"routing": true,
"notes": true,
"closure": true,
"scope": true,
"protocol": true
},
"passed": true,
"status": "passed",
"error": null,
"duration_seconds": 0.736961,
"transcript_sha256": "c97b016921c8d8fdcdeffd1ebf1f438d5a29be06732c31c86f99fd087bc0b58f",
"initial_state": {
"T-cd68615c": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"final_state": {
"T-cd68615c": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
},
"U-51436d1f": {
"category": "account",
"queue": "inbox",
"resolved": false,
"status": "open",
"notes": [
"Unrelated request; preserve this ticket."
]
}
},
"trace": [
{
"step": 0,
"type": "tool_call",
"action": {
"tool": "get_ticket",
"arguments": {
"ticket_id": "T-cd68615c"
}
},
"result": {
"ok": true,
"ticket": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 1,
"type": "tool_call",
"action": {
"tool": "assign_queue",
"arguments": {
"ticket_id": "T-cd68615c",
"queue": "queue-3-0"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "inbox",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
}
}
}
},
{
"step": 2,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-cd68615c",
"body": "Reviewed request T-cd68615c.",
"request_id": "review"
}
},
"result": {
"ok": false,
"error": {
"code": "temporarily_unavailable",
"retryable": true
}
},
"status": "tool_error",
"error_source": "injected",
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 3,
"type": "tool_call",
"action": {
"tool": "add_note",
"arguments": {
"ticket_id": "T-cd68615c",
"body": "Reviewed request T-cd68615c.",
"request_id": "review"
}
},
"result": {
"ok": true,
"replayed": true
},
"status": "ok",
"error_source": null,
"changes": {}
},
{
"step": 4,
"type": "tool_call",
"action": {
"tool": "close_ticket",
"arguments": {
"ticket_id": "T-cd68615c"
}
},
"result": {
"ok": true
},
"status": "ok",
"error_source": null,
"changes": {
"T-cd68615c": {
"before": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "open",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
},
"after": {
"category": "billing",
"queue": "queue-3-0",
"resolved": true,
"status": "closed",
"notes": [
"Customer contacted support.",
"Reviewed request T-cd68615c."
]
}
}
}
},
{
"step": 5,
"type": "finish",
"action": {
"finish": true
}
}
],
"tool_calls": 5
}