Benchmarks
LongMemEval Oracle + S tables, end-to-end answer accuracy, the validated-improvements program, the versioned-memory perf gate — and the composition lesson.
LongMemEval results
Oracle (500 questions, ~2 sessions each, k=5)
Measured before cross-encoder reranking became the default (July 2026). A current-pipeline Oracle run is not published; the S results below are the current pipeline.
| Metric | direct (zero-LLM) |
expanded (1 LLM call/q) |
|---|---|---|
| session_recall@5 | 0.938 | 0.951 |
| message_recall@5 | 0.754 | 0.854 |
| session_hit@5 | 0.972 | 0.972 |
| message_hit@5 | 0.904 | 0.951 |
| context_chars_mean | 4,937 | 3,928 |
S (500 questions, ~48 sessions each, k=5)
Temporal decomposition and preference routing are folded into the default and validated on their question subsets (+0.037 session recall on 133 temporal questions, +0.033 on 30 preference questions) — not re-measured as a full-set aggregate. The default rows below are measured memorycore_episodic_reranked runs: bge-small (default embedder since v0.13.4) and the previous all-MiniLM default.
| Metric | direct |
episodic (bge-small) |
episodic (MiniLM) |
|---|---|---|---|
| session_recall@5 | 0.865 | 0.957 | 0.952 |
| message_recall@5 | 0.670 | 0.628 | 0.618 |
| session_hit@5 | 0.968 | 0.981 | 0.981 |
| message_hit@5 | 0.768 | 0.774 | 0.770 |
| context_chars_mean | — | 3,733 | 3,880 |
Use recall(strategy=“episodic”) (the default) — strongest zero-LLM mode on both evaluations. direct for single-session factual questions (best message precision), expanded when highest precision is needed, fusion for session diversity (2× compute). All modes abstain correctly on unanswerable questions (0% false-positive rate).
End-to-end answer accuracy
LLM answer → LLM judge over 500 S questions (scripts/eval_answer_longmemeval.py, deepseek-v4-flash as answer model and judge, anonymous shuffled judging, evidence-first bundle formatting):
| Context | Accuracy | Context chars |
|---|---|---|
| 4k bundles (CE-ranked, evidence-first) — the default | 0.678 | 6,016 |
cap=2 session selection (session_cap=2) |
0.656 | 11,866 |
LLM query expansion (expanded) |
0.642 | 4,587 |
| 16k bundles (pre-0.13 default) | 0.608 | 14,744 |
| message top-5 only | 0.528 | 7,302 |
Abstention accuracy 0.867 for the top modes.
Validated improvements (2026-08, all zero-LLM, folded into the default)
Measured on LongMemEval-S (500 questions) against the episodic baseline:
| Improvement | Validated delta | Status |
|---|---|---|
| Temporal query decomposition (from/to, since/when, ago-event cues) | +0.037 session / +0.029 message recall on 133 temporal questions | ✅ folded into the default |
| Preference union routing (per-variant top-40 union) | +0.033 session recall on 30 preference questions | ✅ folded into the default |
| 4k bundles + evidence-first ordering | +0.070 answer accuracy vs 16k (0.678 vs 0.608), ~60% less context | ✅ folded into the default (v0.13) |
Session-cap selection (session_cap=2) |
+0.124 message recall / +0.048 answer accuracy, at −0.058 session recall | ⚠️ opt-in |
Batch ingest (ingest_many) |
550 messages 49.9s → 11.5s (4.3×), identical retrieval | ✅ shipped |
L-12 cross-encoder (COREMEM_CROSS_ENCODER_MODEL) |
+0.018 oracle-style, cancels the temporal win on S (−0.004) | ⚠️ opt-in; L-6 stays default |
| Graph-based retrieval (8 edge types) | neutral-to-negative across 500 S questions | ❌ parked |
falsified, honestly Also tested and rejected: fact-augmented key expansion (neutral on S), BGE-reranker-v2-m3 (−0.040 message recall, 7× slower), LoCoMo CoN transfer (zero effect). The eval harness in scripts/ is resumable, so negative results are reproducible too.
Individually-positive improvements do not always sum — a combined 500/500 S-scale validation showed the L-12 reranker cancels the temporal decomposition’s session gains. The default strategy ships only the validated combination (L-6 + temporal decomposition + preference routing), validated on their question subsets with zero regressions.
Versioned memory perf gate (0.14.0)
10k messages, median of 3 (gate in docs/versioned-memory-design.md §7):
| Metric | Result | Gate |
|---|---|---|
| Ingest overhead | +8.2% | ≤15% ✅ |
| Recall latency | 63.5 → 66.7 ms warm | unchanged ✅ |
| Recall metrics | identical (0.965 / 0.537) | ✅ |
| Storage | 1.13× | ✅ |
| Rollback | 3.85s / 1k rows (~4ms/row) | documented (HybridDB batched-rollback follow-up shipped in 0.7.0) |
| verify_chain | 0.05s @ 12k events | ✅ |
Results: eval_output/lme-oracle/results.json (Oracle direct + expanded, committed), eval_output/lme-s/results.json (S direct, committed). The S episodic runs are reproducible with scripts/eval_agent_journal_longmemeval.py --mode memorycore_episodic_reranked (per-question outputs are gitignored, not published).
Source of truth for this page: CoreMem · open-assistants-lab/CoreMem