Agent memory benchmarks are mostly incomparable — here is our complete measurement
99.9% vs 96.6% — neither number means what you think. Why recall@5, answer accuracy, and eval sets keep getting conflated, and the full, reproducible measurement behind CoreMem's claims.
Every agent-memory project publishes a LongMemEval number. Ours is 99.9%. MemPalace reports 96.6%. mem0 reports 93.4%. None of these are the same measurement — and picking between them on the number alone is how you pick the wrong memory layer.
This post publishes everything behind our claims: the datasets, the metrics, the models, the harness, the failures. It’s the measurement we wish competitors published.
The metric problem
LongMemEval is a family: Oracle (500 questions over short sessions) and S (500 harder questions over ~48 dense sessions each), each scored on several metrics. The three headline numbers above are not the same metric:
- Our 99.9% is session recall@5 on Oracle — the fraction of sessions containing the answer that appear in the top-5 retrieval results. This is a retrieval metric.
- A claim of ~96.6% on LongMemEval in the current open-source landscape is an answer-accuracy style number from a different evaluation procedure (LLM-generated answer judged by an LLM over a different context selection).
- mem0’s 93.4% is yet another pipeline and procedure.
Recall@5 is not answer accuracy. Answer accuracy depends on the answer model, the judge, the prompting, and the context budget. None of these numbers rank against each other.
Our complete measurement
Retrieval — LongMemEval, k=5, zero LLM calls
Default strategy (episodic), deterministic retrieval, no LLM on the retrieval path:
| Set | Metric | Score |
|---|---|---|
| Oracle (500 q) | session_recall@5 | 0.999 |
| Oracle | message_recall@5 | 0.867 |
| S (500 q) | session_recall@5 — shipped default | 0.984 |
| S | 95.0% baseline → +0.034 validated improvements (temporal decomposition + preference routing) | — |
| S (all modes) | abstention (false positives) | 0.0% |
The S number deserves the footnote. 95.0% is the pre-improvement episodic baseline. The shipped default adds temporal query decomposition and preference routing — measured at +0.034 session recall overall on the full 500-question set, with zero regressions. We quote 98.4% as the shipped number, and the baseline is in every table so nothing fades.
Methodology: eval_output/lme-oracle/results.json, eval_output/lme-s/results.json — full question-level outputs, resumable harness in scripts/, k=5, evaluation_scope=per_question_haystack, journal model ollama-cloud:deepseek-v4-flash.
End-to-end answer accuracy — the metric most people actually mean
LLM answer (deepseek-v4-flash) → anonymous shuffled judge, 500 S questions, our default evidence-first 4k bundles:
| Context selection | Accuracy | Abstention | Context |
|---|---|---|---|
| 4k bundles, evidence-first (default) | 0.678 | 0.867 | 6,016 chars |
| session_cap=2 | 0.656 | — | 11,866 |
LLM query expansion (expanded) |
0.642 | — | 4,587 |
| 16k bundles (pre-0.13 default) | 0.608 | — | 14,744 |
| message top-5 only | 0.528 | — | 7,302 |
Mean retrieval: 0.234s per question. Judge and answer model are the same for every row — the context selection is the only variable. That’s the comparison we’d like to see from everyone.
If you want to compare us against any other system, use this row: answer accuracy 0.678, abstention 0.867, 6k context. Same metric, same judge, same questions — and the full question-level JSON is in the repo.
The part nobody publishes: what we tried and rejected
The number above is the surviving combination of a program that measured and discarded. All falsified at S-scale, all reproducible in scripts/:
- L-12 cross-encoder: +0.018 oracle-style, cancels the temporal win on S (−0.004) → rejected
- Fact-augmented key expansion: neutral on S → rejected
- BGE-reranker-v2-m3: −0.040 message recall at 7× slower → rejected
- Graph-based retrieval (8 research-grounded edge types): neutral-to-negative across 500 q → parked
- LoCoMo CoN transfer: zero effect → rejected
The composition lesson: individually-positive improvements do not always sum. Shipping the validated combination only — L-6 + temporal decomposition + preference routing — is why the default is +0.034 with zero regressions, not +0.10 with three silent ones.
Reproduce it
git clone https://github.com/open-assistants-lab/CoreMem && cd CoreMem
uv sync
# retrieval: LongMemEval Oracle + S, resumable harness
uv run python scripts/eval_longmemeval.py --dataset oracle
uv run python scripts/eval_longmemeval.py --dataset s
# answer accuracy (500 S questions, needs an LLM key)
uv run python scripts/eval_answer_longmemeval.pyBenchmark claims in the CoreMem docs link to the outputs. If you run the harness and get different numbers, file an issue — that’s the whole point.