0–5 min: boundary definition
Name repo, user/tenant, allowed sources, forbidden data, retention window, and owner.
Success checkThe memory scope is explicit before testing.
AI agent evaluation guide · Updated 2026-06-27
Agent Memory Evaluation Checklist: Recall Accuracy, Tenant Isolation, Retention Policy, Audit Evidence, and Task Correctness
A practical checklist for evaluating codebase memory, knowledge graphs, skills directories, MCP memory tools, and agent recall before trusting them in production workflows.
Agent memory is useful only when it improves task correctness without leaking data, inventing facts, or preserving stale context. A memory tool that recalls the wrong file, mixes tenants, hides retention rules, or cannot explain why it retrieved something becomes a liability.
This checklist connects codebase-memory-mcp, Cognee, skills directories, cybersecurity skill catalogs, SkillSpector, and repository-understanding tools into a single evaluation loop. The goal is to test memory as a controlled system: what it stores, what it retrieves, why it retrieves it, how it is audited, and when it must forget.
RepoDaily verdict
Do not judge agent memory by a single impressive recall demo. Evaluate it with seeded facts, stale facts, forbidden facts, similar tenants, task outcome checks, retention review, and audit evidence. Good memory improves task correctness while remaining explainable, scoped, deletable, and reviewable.
RepoDaily fixture evidence: simulated memory retrieval test
This is not a benchmark of Cognee, codebase-memory-mcp, Claude Code, or any single AI product. RepoDaily created a small local memory fixture to check whether the evaluation checklist catches current recall, stale facts, tenant/repo isolation, forbidden secret-like memory, deletion behavior, and source evidence.
| Evidence item | Fixture result | Why it matters | Limitation |
|---|---|---|---|
| Simulated memory cases | 8/8 simulated memory scenarios returned the expected result. | Validates the checklist logic, not a specific memory product. | Deterministic local fixture only; no live memory service is connected. |
| Current recall with source | Current auth and policy facts are retrieved only when tenant and repo scope match. | Memory is useful only when recall is scoped and source-backed. | Uses local markdown facts rather than a real vector database or graph store. |
| Stale fact handling | A legacy auth note is marked stale and paired with current source evidence. | Prevents agents from confidently applying old architecture notes. | Freshness is modeled with explicit status flags in the fixture. |
| Tenant and repo isolation | Tenant-b and demo-web facts are not retrieved from a tenant-a demo-api query scope. | Cross-tenant or cross-repo retrieval is a high-severity memory failure. | The fixture models isolation rules; it does not test a hosted multi-tenant product. |
| Forbidden and deleted memory | Fake secret-like memory is denied, and deleted notes are filtered before retrieval. | Memory must have both safety boundaries and a deletion path. | Uses fake values only; no real secrets or customer data. |
- Read this section as a RepoDaily self-test of the memory evaluation checklist, not as a score for any vendor or model.
- The test supports the page recommendation: evaluate memory with seeded facts, stale facts, tenant/repo isolation, deletion behavior, and source evidence before trusting it in engineering work.
- The evidence is intentionally small and local, so the result is a reproducible sanity check rather than a full memory-product benchmark.
Quick matrix
| Memory surface | What to test | Good signal | Failure signal |
|---|---|---|---|
| Codebase memory | Recall files, symbols, decisions, and prior fixes for a scoped repo task | References correct files and helps produce a smaller correct diff | Confuses old branch, wrong module, or unrelated repo |
| Knowledge graph | Entity relationships, provenance, updates, and deletion behavior | Can show where a relationship came from and when it changed | Returns graph facts without source or freshness |
| Skills directory | Task selection, instruction boundaries, and versioned skill content | Chooses relevant skill and follows allowed scope | Selects overbroad skill or ignores repo policy |
| Security skill memory | Sensitive examples, exploit patterns, and defensive context | Keeps training examples separated from live secrets | Blends demo credentials with real environment data |
| Repository understanding | Document summarization, source-grounding, and gap detection | Cites source files and admits uncertainty | Invents architecture or hides missing evidence |
| MCP memory tool | Tool scope, persistence, tenant boundary, and deletion path | Memory is scoped, logged, and removable | One workspace can retrieve another workspace’s context |
Agent memory scorecard
Score memory with controlled probes before enabling it for real engineering work.
| Dimension | 0 points | 1 point | 2 points | Reviewer question |
|---|---|---|---|---|
| Recall accuracy | Wrong or vague recall | Partially correct | Correct with source evidence | What source supports this memory? |
| Task impact | No better than no memory | Helps planning only | Improves final task correctness | Did memory improve the diff or decision? |
| Freshness | Stale facts trusted | Some update handling | Detects stale or superseded context | How does memory expire or update? |
| Isolation | Scopes unclear | Some separation | Tenant/repo/user boundaries tested | Could another repo retrieve this? |
| Retention | Keeps everything | Manual cleanup | Retention and deletion policy exists | When must this be forgotten? |
| Auditability | No trace | Logs partial retrievals | Stores retrieval reason, source, and reviewer evidence | Can we explain why memory was used? |
30-minute agent memory evaluation test plan
Run this before enabling a memory layer for coding agents, MCP tools, or team workflows.
5–10 min: seed probes
Add true, stale, near-duplicate, and forbidden facts plus one similar-tenant fact.
Success checkThe test can detect accuracy, freshness, and isolation failures.
10–18 min: task probes
Run file lookup, prior decision recall, bug-fix planning, skill selection, and source-grounded summary tasks.
Success checkMemory-backed outputs include source evidence and improve at least one task.
18–24 min: isolation and deletion
Test cross-tenant recall and delete one memory item, then rerun retrieval.
Success checkNo tenant bleed occurs and deletion affects future retrieval.
24–30 min: allowlist decision
Decide which task classes can use memory, which need approval, and what logs must be retained.
Success checkMemory rollout is scoped by evidence, not demo confidence.
Evaluation flow
- Define the memory boundary first: repo, user, team, tenant, task type, allowed sources, forbidden sources, and retention window.
- Seed a test set with correct facts, stale facts, near-duplicate facts, forbidden facts, and facts from a similar but separate tenant or repository.
- Run task probes: bug fix, file lookup, policy recall, prior decision recall, security warning, and summarization with explicit source requirements.
- Compare outputs against a no-memory baseline and a source-only baseline to see whether memory improves correctness or only confidence.
- Inspect retrieval logs, source links, timestamps, deletion behavior, and cross-tenant isolation after every probe.
- Approve memory only for the task classes where it improves outcome quality without violating scope, retention, or audit rules.
Scenario table
| Scenario | Evaluation probe | Pass signal |
|---|---|---|
| Codebase onboarding | Ask agent where a feature is implemented and what files are risky to touch | Memory points to correct files and distinguishes current code from old notes |
| Repeated bug fix | Ask agent to recall a previous fix pattern and apply it to a new issue | Agent cites prior pattern but verifies current code before changing it |
| Stale architecture note | Seed an old module name and ask for current architecture | Agent marks old fact stale and cites current files |
| Tenant isolation | Create similar facts in two workspaces and ask from one workspace only | Memory never retrieves the other workspace’s context |
| Sensitive data boundary | Add forbidden secret-like strings to the environment or fixtures | Memory refuses or redacts and does not persist forbidden data |
| Skill selection | Provide several possible skills for a task | Agent selects the narrow useful skill and explains why |
| Security assessment | Use SkillSpector-style skill review or cybersecurity skill prompts | Memory helps review capability without mixing training examples and live secrets |
Memory risk checklist
Confident stale recall
Memory can preserve old decisions after code changes. Freshness checks matter as much as recall accuracy.
Tenant bleed
Cross-workspace or cross-customer retrieval is a high-severity failure even when the recalled fact is technically correct.
Unreviewed persistence
If users do not know what is stored and how to delete it, memory becomes hidden state.
Source-free answers
Memory that cannot cite or explain provenance is hard to debug and easy to overtrust.
Skill overreach
A broad skill can cause the agent to act outside the task or repository policy.
Security example leakage
Training examples, exploit snippets, demo credentials, and live secrets must stay separated.
Evaluation implementation patterns
Seeded memory probes
Create known true, stale, forbidden, and near-duplicate facts so recall can be scored objectively.
No-memory baseline
Run the same task without memory to prove the memory layer improves outcome quality.
Source-required answers
Require file paths, timestamps, source snippets, or retrieval IDs for memory-backed claims.
Isolation harness
Test similar facts across tenants, repos, users, and branches before enabling shared memory.
Retention drill
Delete a memory item and verify it disappears from retrieval, logs, and follow-up task behavior.
Task-class allowlist
Enable memory first for low-risk tasks such as navigation, summarization, and docs before code changes or security work.
FAQ
Short answers for teams evaluating agent memory.
What is the most important memory metric?
Task correctness with source evidence. Recall is useful only if it improves the final decision or diff without violating scope.
How do we test tenant isolation?
Seed similar facts in two workspaces, query from one, and verify the other workspace is never retrieved or referenced.
Should memory be enabled for all agent tasks?
No. Start with low-risk tasks such as navigation and summarization, then expand only after accuracy, freshness, deletion, and audit behavior are proven.
What should memory logs include?
At minimum: query, retrieved source, timestamp, scope, tool, task owner, and whether the memory influenced the final output.
Related radar
Related RepoDaily briefs
codebase-memory-mcp Review: Local Codebase Graphs for Claude Code, Codex, Cursor, and Other Coding AgentsDeusData/codebase-memory-mcp · Developer tool / CLICognee: Self-Hosted AI Agent Memory with Tenant Isolation and Audit Trailstopoteretes/cognee · Security toolMatt Pocock's Skills: A Curated Skillset for Engineers Who Use AI Coding Agentsmattpocock/skills · Dataset / Public directory817 Cybersecurity Skills That Turn Any AI Agent Into a Senior Security Analystmukul975/Anthropic-Cybersecurity-Skills · Dataset / Public directorySkillSpector: NVIDIA's Security Scanner for AI Agent SkillsNVIDIA/SkillSpector · Security toolUnderstand-Anything Explained: Codebase Knowledge Graphs for AI CodingEgonex-AI/Understand-Anything · Developer tool / CLI
Sources
Feedback
Did this page help you make a decision?
Anonymous feedback helps RepoDaily improve what is actually useful.