RepoDailyIndependent open-source software evaluation.
AI agent evaluation guide · Updated 2026-06-30
A practical checklist for choosing or building agent evaluation datasets before comparing web agents, coding agents, MCP tools, memory layers, and self-hosted agent runtimes.
Agent evaluation fails when a team tests only one impressive demo task. A good dataset should separate task domains, tool permissions, hidden state, expected evidence, pass/fail oracles, safety probes, latency and cost logging, and regression cadence.
This checklist connects RepoDaily-covered tools — Agent-Reach, Firecrawl, Playwright, Jina Reader, codebase memory, Cognee, skills, SkillSpector, AI Website Cloner Template, and DeerFlow — to public evaluation patterns from SWE-bench, WebArena, VisualWebArena, OSWorld, GAIA, τ-bench, MCP-Bench, and MCP-specific eval notebooks.
The goal is not to crown one universal benchmark. The goal is to pick the smallest dataset that matches the behavior under review: code patches, web tasks, GUI control, tool use, memory recall, permission boundaries, collaborative user interaction, or production regression.
RepoDaily verdict
Use benchmark leaderboards for orientation, not as your rollout gate. A production-ready agent evaluation set should include domain-matched tasks, deterministic oracles, source evidence requirements, unsafe-action probes, cost and latency logs, and repeated regression runs whenever tools, models, prompts, memory, or permissions change.
This is not a benchmark of SWE-bench, WebArena, OSWorld, GAIA, τ-bench, MCP-Bench, or any model. RepoDaily built a deterministic local fixture with machine-readable task cards and simulated run records to test whether the checklist can distinguish valid runs from missing evidence, stale sources, unsafe tool use, wrong-tenant memory, and regression drift.
| Evidence item | Fixture result | Why it matters | Limitation |
|---|---|---|---|
| Task-card and run cases | 10/10 simulated evaluation cases returned the expected classification. | Shows that the checklist can be represented as machine-checkable task cards and run artifacts. | Synthetic deterministic fixture only; no live model or benchmark environment is connected. |
| Evidence and artifact enforcement | Missing evidence is classified separately from a successful run. | Prevents narrative-only success claims when the expected artifact is absent. | The fixture checks presence and identity, not semantic quality of a real artifact. |
| Freshness and scope probes | Stale source and wrong-tenant cases are detected as separate failure classes. | Agent evaluation needs to distinguish correctness from unsafe scope or freshness violations. | Tenant and freshness state are simulated metadata, not a live retrieval system. |
| Tool safety and regression drift | Forbidden tool use and mismatched regression keys are detected. | A run can complete the visible task while still violating policy or losing baseline comparability. | The fixture does not measure model capability, latency, cost, or benchmark difficulty. |
| Evaluation surface | Best public pattern | What it reveals | What it misses |
|---|---|---|---|
| Coding-agent patch quality | SWE-bench / SWE-bench Verified style | Whether an agent can modify a real repository and pass tests for issue-like tasks | Team policy, secret handling, PR review burden, and non-Python or internal-code quirks |
| Browser and website automation | WebArena / VisualWebArena style | Whether an agent can navigate realistic self-hosted websites or visual web tasks | Your private auth flows, vendor rate limits, cookies, and prompt-injection exposure |
| Full computer-use agents | OSWorld style | Whether an agent can operate real desktop applications with execution-based checks | Enterprise desktop policy, human approval rules, and device-specific constraints |
| General assistant tool use | GAIA style | Whether the assistant can combine reasoning, web browsing, multimodality, and tool use | Repeatability, cost controls, and task-specific operational safety |
| Collaborative user-facing agents | τ-bench style | Whether the agent can coordinate with a simulated user toward shared objectives | Real customer ambiguity, policy escalation, and abuse handling |
| MCP and tool-selection behavior | MCP-Bench / MCP eval style | Whether the agent can discover tools, choose parameters, use intermediate evidence, and complete multi-step tasks | Your exact MCP server semantics, permissions, logging, and incident paths |
| Agent memory and skills | RepoDaily fixture style plus seeded datasets | Whether memory, skills, or context compression improve task correctness without leakage | Long-term retention, deletion, cross-tenant isolation, and stale fact behavior unless explicitly tested |
Score the dataset before using it to compare models or tools. A flashy benchmark is not useful if it cannot reproduce failures or map to the rollout decision.
| Dimension | 0 points | 1 point | 2 points | Reviewer question |
|---|---|---|---|---|
| Task-domain fit | Generic demos only | Some matching tasks | Tasks mirror the agent job-to-be-done | Does this dataset test the behavior we will ship? |
| Oracle quality | Subjective review only | Partial rubric | Deterministic pass/fail or well-scoped human rubric | How do we know a task passed? |
| Evidence requirement | No sources or logs | Optional citations | Required source, tool, diff, screenshot, or state evidence | What artifact proves the answer? |
| Permission and safety probes | No unsafe-action cases | Some manual review | Explicit deny/prompt/allow and abuse probes | Can the dataset catch harmful success? |
| Reproducibility | One-off run | Pinned prompt/model only | Pinned data, environment, tools, seeds, and evaluation scripts | Can another reviewer rerun it? |
| Regression cadence | No repeat schedule | Manual after big changes | Runs on model, tool, prompt, memory, and policy changes | When is the eval mandatory? |
| Cost and latency visibility | Not measured | Run-level totals | Per-task tokens, wall time, retries, tool calls, and failures | Can we compare operational cost, not just accuracy? |
Run this before trusting a public benchmark, adapting one internally, or comparing agent tools for production rollout.
Pick one behavior to evaluate: repo patching, website navigation, computer use, MCP tool use, memory recall, skill selection, or permission gating.
Success checkThe eval has one primary behavior, not a vague “agent quality” goal.
Choose a closest public benchmark pattern such as SWE-bench, WebArena, OSWorld, GAIA, τ-bench, or MCP-Bench.
Success checkThe chosen pattern matches the interaction mode and oracle style.
Require at least one pass artifact per task: test result, final state check, source citation, browser state, screenshot, tool trace, or reviewer rubric.
Success checkA reviewer can explain why a run passed without trusting the model narrative.
Add stale data, forbidden tool, wrong tenant, unsafe command, prompt injection, partial extraction, or irrelevant tool distractor cases.
Success checkThe dataset can catch dangerous or misleading success.
Compare a no-tool/no-memory baseline and the candidate agent stack on the same task subset.
Success checkThe evaluation shows whether the tool layer improves outcome quality.
Define when to rerun: model upgrade, prompt change, MCP server change, memory schema change, permission policy change, or new data source.
Success checkThe dataset becomes a release gate, not a one-time demo.
| Scenario | Dataset design | Pass signal |
|---|---|---|
| Coding-agent rollout | Use issue-like tasks with repository snapshots, tests, patch diff review, and forbidden command probes | Patch passes tests and does not violate command or secret policy |
| Research agent with web access | Use web tasks with source freshness, conflicting pages, prompt injection, and citation requirements | Answer cites correct current sources and ignores injected page instructions |
| Browser automation agent | Use self-hosted website tasks with account state, forms, file uploads, and final-state checks | Final state matches expected state and all browser actions are logged |
| Computer-use agent | Use desktop tasks with screenshots, app state, setup scripts, and cleanup checks | Execution artifacts prove the GUI task completed without hidden manual help |
| MCP tool server pilot | Use fuzzy user requests, distractor tools, parameter validation, and intermediate evidence capture | Agent chooses correct tool chain and grounds answer in tool outputs |
| Agent memory layer | Seed true, stale, forbidden, deleted, and cross-tenant facts | Agent recalls scoped current facts and refuses or ignores forbidden/stale data |
| Skill library rollout | Pair task variants with and without skill injection and version-pinned repo context | Skill improves task outcome without overreaching or conflicting with repo policy |
A high score on a public benchmark can still miss your private tools, permissions, customer workflows, or compliance boundary.
If reviewers decide pass/fail ad hoc, improvements may reflect reviewer mood rather than agent behavior.
Public tasks may be memorized or tuned against. Keep a private holdout set for rollout decisions.
An agent can complete the user-visible task while using forbidden tools, leaking data, or bypassing approvals.
Browser, OS, MCP, and package environments can break for reasons unrelated to model quality. Log environment failures separately.
Accuracy without tool-call count, latency, retry, and token cost can hide operationally unusable agents.
Store instruction, starting state, allowed tools, forbidden tools, expected artifact, oracle, owner, and rerun trigger for every task.
Require source links, patches, screenshots, final-state JSON, tool traces, or test output before marking a task correct.
Add tasks that should prompt, refuse, or fail safely — not only tasks where success is desired.
Use public benchmarks for orientation and a private fixture set for rollout and regression decisions.
Compare no-tool, source-only, tool-enabled, memory-enabled, and full-agent runs to isolate which layer helps.
Record model, prompt, tool version, MCP server version, memory schema, policy version, and run artifacts for every eval.
Short answers for teams choosing agent evaluation datasets.
No. Coding agents, browser agents, MCP tool agents, memory agents, and user-collaboration agents need different interaction modes and oracles.
Use them for orientation, not rollout. Add private tasks, safety probes, and cost/latency logging before making a team policy decision.
Ten to twenty tasks can be useful if they include clear pass artifacts, negative probes, and regression rules tied to a real rollout decision.
Rerun whenever the model, prompt, tool permissions, MCP server, memory schema, data source, or allowed action policy changes.
Feedback
Anonymous feedback helps RepoDaily improve what is actually useful.