Benchmarks

LongMemEval Oracle + S tables, end-to-end answer accuracy, the validated-improvements program, the versioned-memory perf gate — and the composition lesson.

LongMemEval results

Oracle (500 questions, ~2 sessions each, k=5)

Metric	`direct`	`expanded`	`episodic` (default)
session_recall@5	0.938	0.951	0.999
message_recall@5	0.754	0.854	0.867
session_hit@5	0.972	0.972	1.000
message_hit@5	0.904	0.951	0.947
context_chars_mean	4,937	3,928	4,540

S (500 questions, ~48 sessions each, k=5)

Numbers are the pre-improvement episodic baseline; the validated improvements add +0.034 session recall overall (see below).

Metric	`direct`	`episodic` (default)
session_recall@5	0.865	0.950
message_recall@5	0.670	0.617
session_hit@5	0.968	0.981
message_hit@5	0.768	0.768
context_chars_mean	—	3,991

Recommendation

Use recall(strategy=“episodic”) (the default) — strongest zero-LLM mode on both evaluations. direct for single-session factual questions (best message precision), expanded when highest precision is needed, fusion for session diversity (2× compute). All modes abstain correctly on unanswerable questions (0% false-positive rate).

End-to-end answer accuracy

LLM answer → LLM judge over 500 S questions (scripts/eval_answer_longmemeval.py, deepseek-v4-flash as answer model and judge, anonymous shuffled judging, evidence-first bundle formatting):

Context	Accuracy	Context chars
4k bundles (CE-ranked, evidence-first) — the default	0.678	6,016
cap=2 session selection (`session_cap=2`)	0.656	11,866
LLM query expansion (`expanded`)	0.642	4,587
16k bundles (pre-0.13 default)	0.608	14,744
message top-5 only	0.528	7,302

Abstention accuracy 0.867 for the top modes.

Validated improvements (2026-08, all zero-LLM, folded into the default)

Measured on LongMemEval-S (500 questions) against the episodic baseline:

Improvement	Validated delta	Status
Temporal query decomposition (from/to, since/when, ago-event cues)	+0.037 session / +0.029 message recall on 133 temporal questions	✅ folded into the default
Preference union routing (per-variant top-40 union)	+0.033 session recall on 30 preference questions	✅ folded into the default
4k bundles + evidence-first ordering	+0.070 answer accuracy vs 16k (0.678 vs 0.608), ~60% less context	✅ folded into the default (v0.13)
Session-cap selection (`session_cap=2`)	+0.124 message recall / +0.048 answer accuracy, at −0.058 session recall	⚠️ opt-in
Batch ingest (`ingest_many`)	550 messages 49.9s → 11.5s (4.3×), identical retrieval	✅ shipped
L-12 cross-encoder (`COREMEM_CROSS_ENCODER_MODEL`)	+0.018 oracle-style, cancels the temporal win on S (−0.004)	⚠️ opt-in; L-6 stays default
Graph-based retrieval (8 edge types)	neutral-to-negative across 500 S questions	❌ parked

falsified, honestly Also tested and rejected: fact-augmented key expansion (neutral on S), BGE-reranker-v2-m3 (−0.040 message recall, 7× slower), LoCoMo CoN transfer (zero effect). The eval harness in scripts/ is resumable, so negative results are reproducible too.

The composition lesson

Individually-positive improvements do not always sum — a combined 500/500 S-scale validation showed the L-12 reranker cancels the temporal decomposition’s session gains. The default strategy ships only the validated combination (L-6 + temporal decomposition + preference routing): +0.034 session recall overall with zero regressions.

Versioned memory perf gate (0.14.0)

10k messages, median of 3 (gate in docs/versioned-memory-design.md §7):

Metric	Result	Gate
Ingest overhead	+8.2%	≤15% ✅
Recall latency	63.5 → 66.7 ms warm	unchanged ✅
Recall metrics	identical (0.965 / 0.537)	✅
Storage	1.13×	✅
Rollback	3.85s / 1k rows (~4ms/row)	documented (HybridDB batched-rollback follow-up shipped in 0.7.0)
verify_chain	0.05s @ 12k events	✅

Results: eval_output/lme-oracle/results.json, eval_output/lme-s/results.json, results/eval_answer_s500.json.

Source of truth for this page: CoreMem · open-assistants-lab/CoreMem