Benchmarks

Search accuracy on real BEIR benchmarks, DuckDB mirror speedups at scale, write-path tuning, and how to run the benchmark suite yourself.

Measured 2026-08-20, commit 00a361c, Apple Silicon MacBook Pro, CPython 3.13. Search accuracy uses real IR benchmarks (BEIR); speed numbers are smoke-scale (50 docs / 50 graph nodes / 1,000 analytics rows) unless noted. Full-scale runs (--benchmark-full) exist but were not part of this study.

Search accuracy (BEIR)

Evaluated on two BEIR datasets with real queries and relevance judgments: NFCorpus (3,633 medical docs, 324 queries, graded 0–3) and SciFact (5,183 scientific abstracts, 301 queries, binary). Embeddings: ChromaDB’s bundled MiniLM.

Fusion-weight sensitivity (hybrid nDCG@10)

Findings

1. Hybrid fusion is the right default. It beats both single modes on every metric in both domains (+11% nDCG over keyword on NFCorpus, +5% on SciFact). RRF captures the lexical and semantic regimes.
2. fts_weight=0.5 is the sweet spot and the curve is flat (spread < 0.03) — no tuning needed.
3. The hash embedding fallback is a 5.3× accuracy cliff (nDCG 0.059 vs 0.315, near-random). The fallback exists for offline smoke tests, not production.

Analytics: DuckDB mirror vs raw SQLite

Same queries on the SQLite table (raw_query) and the DuckDB mirror (olap.query), median of 5 runs.

Findings

1. The DuckDB mirror pays off immediately and compounds with scale. Even at 10k rows it is 4–7× faster; at 1M rows, 58–352×. There is no “SQLite wins at small scale” regime for analytical queries.
2. Filtered aggregation is the columnar sweet spot (352× at 1M rows) — vectorized scans over WHERE clauses.
3. Point lookups are the mirror’s weak spot (2.4× at 1M) — the mirror has no index. Use get()/query() for point lookups.
4. The mirror is cheap to maintain: full sync copies 1M rows in ~0.6s; incremental journal sync adds no measurable write overhead.

Write path

Profiling insert_batch revealed the bottleneck was not journal writes but a fresh SQLite connection per row — schema lookups opened ~200k connections per 100k rows. Fixes: the batch cursor is threaded through the hot paths, per-row schema reads are hoisted, and _process_journal uses one connection end-to-end.

The journal remains the write bottleneck (~2,000 rows/s) — dominated by per-row Python + journal overhead. Optimization candidates: batched journal writes, dropping the per-row re-select.

Versioned-table rollback (0.7.0)

• Removal-heavy rollbacks ~20× faster: 1,000 removed rows 1,980ms → 92ms (batched path; ~122ms with Chroma + DuckDB live). Root cause: rollback() looped per removed row through public delete(), paying one connection + transaction and ~10 statements per row. Removals are now set-based — one transaction, chunked DELETE … WHERE pk IN (…), hash-chain tombstones via executemany.
• Batched restores: update-heavy rollbacks re-apply state as new versions — 1,000 restored rows were 3,931ms (41× the removal path); now 107ms at 100k-row tables via split batched INSERT/UPDATE. Chroma re-embedding of restored LONGTEXT remains by design.

Running the benchmarks yourself

Benchmarks are separate from the default test run:

uv run python -m pytest -q
# 262 passed, 48 skipped (benchmarks under tests/benchmarks/)

uv run python -m pytest tests/benchmarks -q --run-benchmarks --benchmark-disable
# 48 passed — smoke scale: 50 docs, 50 graph nodes, 1,000 analytics rows

uv run python -m pytest tests/benchmarks -q --run-benchmarks
# timed loops; full-scale via --benchmark-full

Source studies live in the repo: docs/PERFORMANCE.md and docs/BENCHMARKS.md.

Source of truth for this page: HybridDB · open-assistants-lab/HybridDB