Half our experiments failed. That's why the ones that pass mean something.
The reranker that looked better, the graph edges we believed in, the embedder swap that was a wash — and the one rule that kept the default pipeline honest: the composition lesson.
Most memory products publish their wins. We publish the losses too — because a benchmark program that only reports what worked is a marketing department, and a memory layer you can’t audit is a liability. CoreMem’s default pipeline is what survived our own falsification loop. Here’s what didn’t, and the rule that saved us from the best-looking mistake.
The rule: the composition lesson
Individually-positive improvements do not always sum. We learned this the expensive way: a combined 500/500 S-scale validation showed the L-12 cross-encoder cancels the temporal decomposition’s session gains — +0.018 on the oracle-style subset, but −0.004 on S where it mattered. Two “wins” that cancel into a regression.
Since then the rule is: the default ships only the validated combination. L-6 reranker + temporal decomposition + preference routing — measured at +0.034 session recall overall with zero regressions. Everything else is opt-in or not shipped.
The falsified lever list
| Lever | Measured result | Verdict |
|---|---|---|
| L-12 cross-encoder | +0.018 oracle-style, −0.004 on S (cancels temporal win) | ⚠️ opt-in; L-6 stays default |
| Fact-augmented key expansion | neutral on LongMemEval S | ❌ rejected |
| BGE-reranker-v2-m3 | −0.040 message recall at 7× slower | ❌ rejected |
| Graph-based retrieval (8 edge types) | neutral-to-negative across 500 S questions | ❌ parked |
| LoCoMo CoN transfer | zero effect | ❌ rejected |
| Hash embedding fallback | 5.3× accuracy cliff (nDCG 0.059 vs 0.315, near-random) | ⚠️ smoke-tests only |
Every one of these had a plausible story before we measured it. Graph edges came from published research on temporal memory; fact augmentation is a standard trick in this space; the bigger reranker is supposed to be better. The harness is resumable, so the negative results are reproducible — scripts/ in the CoreMem repo, 500-question S-scale runs.
Why “0 LLM calls” is a design constraint, not a slogan
The default retrieval path makes no LLM calls — which is also why the falsification program works at all. LLM pipelines hide their failures inside a token bill: a query-expansion model that misfires costs you $0.40 and degrades silently. Deterministic retrieval fails visibly and measurably — every lever’s effect shows up in the recall table, and the abstention rate stays at a verifiable 0% false positives across all modes.
That’s the audit trail that matters: not just “we’re local and private,” but “retrieval is deterministic, journals are source-quote-verified, and every claim in the product page traces to a JSON file in the repo.”
The takeaway for choosing memory
When you evaluate an agent-memory layer, ask for its falsified list as well as its headline. Our headline numbers — 99.9% session recall@5 on Oracle, 0.678 answer accuracy with 0.867 abstention on S under a fixed judge — are only meaningful because the full measurement and the full failure log are published next to them. Any vendor can promise recall; the ones who can show you what they rejected are the ones you can bet a production agent on.