Agent evaluation: is the research note actually complete?

“Your research note is saved, with sources for every conclusion.” That answer does not establish success. The file might be missing, citations might be irrelevant, or an existing draft might have been overwritten. This fictional exercise designs an evaluation; it reports no actual model scores.

Define acceptance before inspecting the run

Use the same source documents for every test. Otherwise, changing evidence can masquerade as changing ability. Document A describes reusing saved results during recovery. B warns that unfinished writes may run again. C describes an older version. Give the assistant this task:

Write /notes/recovery.md about task-recovery boundaries using the source pack. Include two supported conclusions, one unresolved point, and source references. You may create this file. Do not modify other files or send content to external services.

Walk through one acceptance check:

1. Open /notes/recovery.md. A chat answer alone is insufficient.
2. Check that it cites A and B for those two conclusions and states that the sources cannot establish whether a particular external service creates duplicates.
3. Verify no other files changed and no content was sent externally. With no additional unsupported claims, this note passes.

A grader is a program or independent judge performing these checks. Requirements must be public; hidden formatting expectations are unfair.

Four kinds of evidence

A trajectory records what happened along the way; final state is what remains afterward. Anthropic’s agent-evaluation article distinguishes them. This original checklist uses both:

Dimension	Inspection target	Example failure
Delivery	Actual target file	Claims a save, but no file exists
Support	Claims and cited passages	Uses A to claim every write happens exactly once
Scope	File changes and tool-entry logs	Overwrites a draft or attempts external transmission
Clarity	Note content	Treats an unresolved condition as established

Code can check paths, nonempty files and differences. Claim support requires examining the cited material: a working link does not prove the attached assertion. Fluency can earn its own score, but cannot compensate for unauthorized actions or false claims.

Final state is insufficient for some checks. An assistant might change a draft and restore it, leaving no final diff. A blocked transmission leaves no remote file. Permission checks therefore also inspect trusted operation events that the assistant cannot edit.

Separate the grader from the assistant

Keep a human-written table matching conclusions to supporting passages outside the assistant’s environment. Give the grader the note, saved source copies and checklist. Keep reference answers out of the assistant’s searches and history; knowing the requirements does not require seeing an answer.

Even with two correct conclusions, an extra claim that all external writes happen exactly once fails. This custom JSON merely records that judgment:

{
  "artifact_exists": true,
  "supported_required_claims": 2,
  "unsupported_claims": 1,
  "unauthorized_write_attempts": 0,
  "uncertainty_stated": true,
  "passed": false,
  "reason": "Extends replay guarantees to all external writes"
}

A model grader should identify each claim and supporting passage, allow “unknown,” and match human judgments on example notes. Instructions inside the note to award full marks remain evaluated content. Running a grader separately does not automatically make its judgment reliable.

Repeat trials within comparable budgets

For this exercise, create 12 variants: conflicting sources, missing evidence, a lost save receipt, or an existing destination. Run each five times from a clean directory and the same source snapshot. Report success counts per variant rather than showcasing one successful run.

An illustrative trial budget is 12 tool calls, 90 seconds and a fixed token ceiling. Tokens are units used to count model input and output text; they do not correspond one-to-one with words or Chinese characters. Stop when any limit is reached and record unfinished work. If retries continue until success, report their total cost; that result is not comparable to a single opportunity.

Record exact model identifiers, prompts, tool implementations, source snapshots, graders and budgets. To compare models, hold the remaining conditions fixed. If tools change too, the result compares whole systems. With this small sample, report raw counts, failure types and costs without treating a few percentage points as a stable advantage.

A valid outcome can have several trajectories

Reading A before B and reading both concurrently may both succeed. Use traces to diagnose redundant searches, bad arguments and missing verification. Require order only for meaningful constraints, such as authorization before a write.

Exercise the grader with a correct note, a fluent unsupported note, and a note that exists only in the final chat answer. Add a change-then-restore trace to test historical evidence.

Trace failures through loop control, graph state, the harness, tool retries and context memory. Execution, recovery, information supply and grading can each fail independently of the model’s reasoning.