Core concepts

Recall strategies, deterministic heuristics, session bundles, the AgentJournal, versioned memory governance, and metadata pre-filtering — the mental model behind CoreMem.

Ingestion model

Everything is grouped by turn. ingest() stores a single message and returns its turn_id; ingest_turn() stores a whole conversation turn under one turn_id; ingest_many() bulk-ingests and returns message ids. Return conventions matter downstream: compile_turn(turn_id=...) needs the turn id from ingest.

Recall strategies

recall() is the single retrieval entry point, with four strategies:

Strategy	LLM calls	Pipeline
`episodic` (default)	0	Temporal query decomposition → hybrid search per variant → RRF fusion (preference questions: per-variant top-40 union) → cross-encoder rerank → MMR diversity
`direct`	0	Single hybrid search + deterministic heuristics
`expanded`	1	LLM query rephrasing, then the direct pipeline per variant
`fusion`	0	RRF fusion of `direct` + `episodic`

Recommendation: use recall(strategy="episodic") (the default) — the strongest zero-LLM mode on both evaluations. Use direct for single-session factual questions (best message precision), expanded when highest precision is needed, and fusion when session diversity is critical (2× compute).

All modes abstain correctly on unanswerable questions (0% false-positive rate).

Deterministic heuristics

Zero-LLM scoring boosts applied to every result:

Heuristic	What it catches
`keyword_overlap`	Exact + fuzzy (difflib) + bigram matches between query and content
`temporal_boost`	Queries with “latest”, “current”, “recently”
`recency_decay`	Unconditional exponential decay (30-day half-life)
`person_name_boost`	Proper name mentions in content
`quoted_phrase_boost`	Exact phrase matches in quotes

from coremem import SearchHeuristics

score = SearchHeuristics.apply_all(
    query="latest project",
    content="Just finished the Q3 project report",
    score=0.75,
    ts="2026-05-28T10:00:00Z",
)

Versioned memory

default since 0.15.0

With versioned=True (the default for new stores since 0.15.0) the messages and journal_records tables carry a tamper-evident hash chain (HybridDB 0.6.0+): every insert/delete is recorded, and the governance API is available:

core.checkpoint_memory("before-cleanup")     # named restore point
core.rollback_memory("before-cleanup")       # chain never rewinds — recorded as new versions
core.memory_log(limit=100)                   # change log, newest first
core.memory_history("m1")                    # provenance timeline per message
core.memory_diff(1, 3)                       # added / removed / changed
core.as_of_memory(3)                         # what the agent knew at seq 3
core.verify_memory_chain()                   # tamper-evidence check

Versioning is create-time only

Versioning applies to NEW stores — an existing store opened without it stays un-versioned (no in-place migration in v1). Pass versioned=False to opt out. Measured gate at 10k messages: ingest overhead +8.2%, recall latency unchanged, storage 1.13×.

Metadata pre-filtering

v0.16.0

Recall equality filters (role, session_id, user_id, agent_id) are pushed into the Chroma scan before the vector search (via HybridDB 0.7.0’s search(where=)) instead of relying on over-fetch + Python post-filtering. Filtered recall can no longer starve.

0.16.1 — chronological timestamps

ts_after/ts_before filters now compare chronologically across mixed naive/aware formats — a naive 23:00 message is no longer excluded by an earlier UTC filter (the documented limitation since 0.13.2).

The AgentJournal

The AgentJournal subsystem compiles conversation turns into dense, retrieval-optimized daily journal pages (markdown + frontmatter), with deterministic validation of every claim against its source:

The LLM compiler (openai:gpt-4o-mini by default) produces a structured plan
The deterministic compiler validates every claim against source messages (exact quote substrings, role/evidence-type compatibility) before it is written
dream() consolidates daily pages — analysis and promoted facts append to DREAMS.md (MEMORY.md is compiler-owned)
rebuild_index() rebuilds weekly/monthly/index navigation

Set COREMEM_LLM_MODEL (e.g. ollama:llama3.2) to change the compilation model. If a turn changes after compilation, compile_turn raises unless you pass force=True.

Embeddings and reranking

Setting	Default	Notes
Embedder	ChromaDB-bundled MiniLM	`COREMEM_EMBEDDING_MODEL` is not the switch — HybridDB’s `embedding_fn` is
Cross-encoder	`ms-marco-MiniLM-L-6-v2`	`COREMEM_CROSS_ENCODER_MODEL` to override (L-12 is opt-in: +0.018 oracle-style, −0.004 on S — cancels the temporal win)
`DISABLE_CROSS_ENCODER=1`	off	Skips reranking (eval scripts)

The composition lesson

Individually-positive improvements do not always sum: a 500/500 S-scale validation showed the L-12 reranker cancels the temporal decomposition’s session gains. The default ships only the validated combination (L-6 + temporal decomposition + preference routing) — +0.034 session recall overall, zero regressions.

Source of truth for this page: CoreMem · open-assistants-lab/CoreMem