Repository-aware code intelligence · Updated 2026-07-20

Code Intelligence Evaluation Checklist: Accuracy, Freshness, Provenance, Privacy, and Agent Outcomes

A repeatable checklist for evaluating repository graphs, code memory, GraphRAG, and MCP context tools before coding agents rely on them.

Code intelligence tools should be judged by whether they help a developer or coding agent make fewer wrong edits—not by how impressive the graph looks.

Run this checklist on the same representative repository, branch, and seeded change tasks. Record both retrieval quality and downstream editing outcomes.

RepoDaily verdict

Do not adopt a repository intelligence layer until it proves symbol coverage, freshness, provenance, privacy boundaries, and a measurable reduction in wrong-file edits or unnecessary context.

Continue the decision path

Use the comparison to choose a candidate, then check the Radar for the current evidence-backed status.

Compare the four repository-intelligence approaches See current Radar verdicts

Reuse this checklist

Copy a Markdown version into an issue, architecture decision record, or team evaluation document.

# Code Intelligence Evaluation Checklist: Accuracy, Freshness, Provenance, Privacy, and Agent Outcomes

A repeatable checklist for evaluating repository graphs, code memory, GraphRAG, and MCP context tools before coding agents rely on them.

## RepoDaily verdict

Do not adopt a repository intelligence layer until it proves symbol coverage, freshness, provenance, privacy boundaries, and a measurable reduction in wrong-file edits or unnecessary context.

## Quick matrix

| Area | Minimum evidence | Failure signal | Decision |
| --- | --- | --- | --- |
| Coverage | Key symbols, imports, calls, and ownership paths appear | Important files or relationships are absent | Hold |
| Freshness | A changed symbol is reflected after the documented refresh path | Stale edges survive a branch or file change | Hold |
| Provenance | Every answer can point back to files, symbols, or commits | Generated explanation has no traceable source | Watch |
| Privacy | Artifacts, caches, and network calls are documented | Private code leaves the approved boundary | Hold |
| Agent outcome | Fewer wrong-file edits or less irrelevant context | Graph adds latency without improving decisions | Do not adopt |
| Exit cost | Indexes and local artifacts can be removed cleanly | Opaque state remains after uninstall | Assess carefully |

## Code intelligence evidence scorecard

Score evidence, not marketing claims. A tool should reach at least 8/12 before a controlled trial.

| Dimension | 0 points | 1 point | 2 points | Reviewer question |
| --- | --- | --- | --- | --- |
| Symbol coverage | Major symbols missing | Partial coverage | Representative symbols and relations verified | What important code is invisible? |
| Freshness | Manual rebuild unclear | Refresh works sometimes | Branch and incremental changes are reliable | How quickly does context become stale? |
| Provenance | No source links | File-level only | File, symbol, and change traceability | Can every claim be checked? |
| Privacy | Unknown data path | Local by default but unclear artifacts | Documented local/network/storage boundary | Where can private code travel? |
| Agent outcome | No measured effect | Subjective improvement | Repeatable reduction in wrong edits or context | Did the agent make a better decision? |
| Operational fit | Fragile setup | Usable with manual care | Repeatable install, refresh, and cleanup | Can the team maintain it? |

## Evaluation flow

1. Choose one representative repository with multiple languages, generated files, tests, and at least one non-trivial dependency path.
2. Seed five questions and three change tasks before installing any tool so every candidate receives the same test.
3. Record a baseline using grep, LSP, repository map, or the coding agent without the candidate tool.
4. Install and index under a documented environment; record time, disk use, network calls, and artifacts.
5. Test symbol lookup, dependency path, ownership, branch change, deleted file, and stale-index recovery.
6. Run the same coding tasks and compare wrong-file edits, irrelevant context, tool calls, latency, and reviewer corrections.
7. Document what was not assessed and schedule a recheck after a release or repository-shape change.

## Scenario tests

| Scenario | Required test | Pass condition |
| --- | --- | --- |
| New developer onboarding | Find entry point and business flow | Sources are correct and understandable without reading the whole tree |
| Agent bug fix | Locate owner, callers, and tests | Agent edits the right files and cites the dependency path |
| Refactor blast radius | Change a shared symbol | Affected callers and tests are found with low false confidence |
| Branch isolation | Index two branches with conflicting symbols | Answers reflect the active branch |
| Freshness recovery | Rename and delete files | Old relationships disappear after the documented refresh |
| Private repository | Inspect storage and network behavior | No unapproved code or artifact leaves the boundary |

## Risk checklist

- **Convincing but incomplete graph:** A polished graph can hide parser gaps. Verify known symbols and missing-language behavior.
- **Stale context:** Old edges can make an agent confidently edit the wrong file. Test branches, renames, and deletions.
- **Untraceable summaries:** Treat generated explanations as hints unless they link to concrete files and symbols.
- **Private-code artifacts:** Inspect indexes, embeddings, logs, exports, and remote API calls before using sensitive repositories.
- **Benchmark theater:** Token reduction alone is not enough; measure wrong edits, reviewer corrections, and task success.

## 60-minute smoke test

This is an L2 smoke test, not a production-readiness claim.

- **0–10 min: baseline:** Run five repository questions without the tool.  
  **Success:** Baseline answers, latency, and context are recorded.
- **10–20 min: install and index:** Install, index, and record artifacts and network behavior.  
  **Success:** Setup is reproducible and data boundaries are known.
- **20–35 min: accuracy:** Test symbols, callers, dependencies, ownership, and tests.  
  **Success:** Answers cite correct files and known gaps are explicit.
- **35–45 min: freshness:** Rename, delete, and switch branches.  
  **Success:** Stale relationships disappear through a documented refresh.
- **45–60 min: agent task:** Run one bug-fix or refactor task with the same prompt.  
  **Success:** The agent chooses better files or uses less irrelevant context without new critical errors.

## Evidence boundary

A deterministic local fixture passed 10/10 checks. This validates the evaluation method only; it is not a benchmark of any candidate tool and does not raise a candidate Evidence Level.

- The method distinguishes coverage from provenance and makes unsupported files explicit.
- Rename, deletion, and branch-isolation cases prevent a visually convincing but stale index from passing.
- The fixture preserves a no-tool baseline and inventories local artifacts.
- Live products still require separate installation, version recording, and task-outcome tests before reaching L2.

RepoDaily checklist-methodology evidence

A deterministic local fixture passed 10/10 checks. This validates the evaluation method only; it is not a benchmark of any candidate tool and does not raise a candidate Evidence Level.

ID	Area	Expected	Actual	Result
CI01	coverage	found_with_provenance	found_with_provenance	PASS
CI02	change impact	caller_and_test_visible	caller_and_test_visible	PASS
CI03	multi language	python_and_typescript_visible	python_and_typescript_visible	PASS
CI04	known gaps	unsupported_explicit	unsupported_explicit	PASS
CI05	branch isolation	branch_scoped	branch_scoped	PASS
CI06	freshness rename	old_removed_new_added	old_removed_new_added	PASS
CI07	freshness delete	deleted_removed	deleted_removed	PASS
CI08	provenance	complete	complete	PASS
CI09	artifact inventory	documented_local_only	documented_local_only	PASS
CI10	baseline	baseline_and_added_value_recorded	baseline_and_added_value_recorded	PASS

The method distinguishes coverage from provenance and makes unsupported files explicit.
Rename, deletion, and branch-isolation cases prevent a visually convincing but stale index from passing.
The fixture preserves a no-tool baseline and inventories local artifacts.
Live products still require separate installation, version recording, and task-outcome tests before reaching L2.

Quick matrix

Area	Minimum evidence	Failure signal	Decision
Coverage	Key symbols, imports, calls, and ownership paths appear	Important files or relationships are absent	Hold
Freshness	A changed symbol is reflected after the documented refresh path	Stale edges survive a branch or file change	Hold
Provenance	Every answer can point back to files, symbols, or commits	Generated explanation has no traceable source	Watch
Privacy	Artifacts, caches, and network calls are documented	Private code leaves the approved boundary	Hold
Agent outcome	Fewer wrong-file edits or less irrelevant context	Graph adds latency without improving decisions	Do not adopt
Exit cost	Indexes and local artifacts can be removed cleanly	Opaque state remains after uninstall	Assess carefully

Code intelligence evidence scorecard

Score evidence, not marketing claims. A tool should reach at least 8/12 before a controlled trial.

Dimension	0 points	1 point	2 points	Reviewer question
Symbol coverage	Major symbols missing	Partial coverage	Representative symbols and relations verified	What important code is invisible?
Freshness	Manual rebuild unclear	Refresh works sometimes	Branch and incremental changes are reliable	How quickly does context become stale?
Provenance	No source links	File-level only	File, symbol, and change traceability	Can every claim be checked?
Privacy	Unknown data path	Local by default but unclear artifacts	Documented local/network/storage boundary	Where can private code travel?
Agent outcome	No measured effect	Subjective improvement	Repeatable reduction in wrong edits or context	Did the agent make a better decision?
Operational fit	Fragile setup	Usable with manual care	Repeatable install, refresh, and cleanup	Can the team maintain it?

60-minute smoke test

This is an L2 smoke test, not a production-readiness claim.

0–10 min: baseline

Run five repository questions without the tool.

Success checkBaseline answers, latency, and context are recorded.

10–20 min: install and index

Install, index, and record artifacts and network behavior.

Success checkSetup is reproducible and data boundaries are known.

20–35 min: accuracy

Test symbols, callers, dependencies, ownership, and tests.

Success checkAnswers cite correct files and known gaps are explicit.

35–45 min: freshness

Rename, delete, and switch branches.

Success checkStale relationships disappear through a documented refresh.

45–60 min: agent task

Run one bug-fix or refactor task with the same prompt.

Success checkThe agent chooses better files or uses less irrelevant context without new critical errors.

Evaluation flow

Choose one representative repository with multiple languages, generated files, tests, and at least one non-trivial dependency path.
Seed five questions and three change tasks before installing any tool so every candidate receives the same test.
Record a baseline using grep, LSP, repository map, or the coding agent without the candidate tool.
Install and index under a documented environment; record time, disk use, network calls, and artifacts.
Test symbol lookup, dependency path, ownership, branch change, deleted file, and stale-index recovery.
Run the same coding tasks and compare wrong-file edits, irrelevant context, tool calls, latency, and reviewer corrections.
Document what was not assessed and schedule a recheck after a release or repository-shape change.

Scenario table

Scenario	Required test	Pass condition
New developer onboarding	Find entry point and business flow	Sources are correct and understandable without reading the whole tree
Agent bug fix	Locate owner, callers, and tests	Agent edits the right files and cites the dependency path
Refactor blast radius	Change a shared symbol	Affected callers and tests are found with low false confidence
Branch isolation	Index two branches with conflicting symbols	Answers reflect the active branch
Freshness recovery	Rename and delete files	Old relationships disappear after the documented refresh
Private repository	Inspect storage and network behavior	No unapproved code or artifact leaves the boundary

Risk checklist

Convincing but incomplete graph

A polished graph can hide parser gaps. Verify known symbols and missing-language behavior.

Stale context

Old edges can make an agent confidently edit the wrong file. Test branches, renames, and deletions.

Untraceable summaries

Treat generated explanations as hints unless they link to concrete files and symbols.

Private-code artifacts

Inspect indexes, embeddings, logs, exports, and remote API calls before using sensitive repositories.

Benchmark theater

Token reduction alone is not enough; measure wrong edits, reviewer corrections, and task success.

Evaluation patterns

Seeded fact set

Write expected answers before testing so reviewers do not move the goalposts.

Baseline first

Compare against grep, LSP, repo-map, or the unassisted agent.

Artifact inventory

Record every database, cache, embedding store, export, log, and network destination.

Failure injection

Rename files, delete symbols, switch branches, and introduce unsupported syntax.

Outcome review

Have a human reviewer score file choice, dependency reasoning, and correction effort.

FAQ

Use these answers when setting evaluation expectations.

Is a lower token count enough to pass?

No. Lower context is useful only when task success, file choice, and reviewer effort stay equal or improve.

Should every repository use a graph?

No. Small repositories may be better served by grep, LSP, or a lightweight repository map.

What evidence level does this checklist produce?

A completed smoke test can support L2. Repeated scenario tests support L3; long-running production-like evidence is L4.

What has RepoDaily tested directly?

RepoDaily directly tested the checklist methodology with a synthetic local repository. It has not yet completed a normalized hands-on test of the four candidate products.

Related radar

AI Agent Tools Radar

Related RepoDaily briefs

codebase-memory-mcp Review: Local Codebase Graphs for Claude Code, Codex, Cursor, and Other Coding AgentsDeusData/codebase-memory-mcp · Developer tool / CLI Understand-Anything Explained: Codebase Knowledge Graphs for AI CodingEgonex-AI/Understand-Anything · Developer tool / CLI graphify review: a local GraphRAG skill that turns any folder into a queryable knowledge graphsafishamsi/graphify · Infrastructure / Runtime code-review-graph: a local-first SQLite knowledge graph that feeds MCP coding tools the exact blast radius of every changetirth8205/code-review-graph · Infrastructure / Runtime

Sources

Feedback

Did this page help you make a decision?

Anonymous feedback helps RepoDaily improve what is actually useful.

Report outdated or missing evidence