Core concepts

Column types, search modes, metadata pre-filtering, versioned tables, the self-healing journal, and long-document chunking — the mental model behind HybridDB.

Column types

HybridDB maps Python-friendly types to SQLite storage and automatically sets up the right search indexes:

TEXT columns get automated FTS5 keyword search. LONGTEXT columns get both FTS5 and ChromaDB semantic search.

Search modes

Three modes, one API — hybrid is the default:

db.search("docs", "body", "hello", mode="keyword")    # BM25 lexical
db.search("docs", "body", "hello", mode="semantic")   # vector ANN
db.search("docs", "body", "hello", mode="hybrid")     # RRF fusion of both

• TEXT columns support keyword search; LONGTEXT columns support all three modes.
• Hybrid search fuses keyword and vector results with reciprocal-rank fusion.
• search_all() and search_columns() query every searchable text column at once.
• Empty queries return [].

Recency scoring

results = db.search(
    "messages", "content", "project update",
    recency_weight=0.3,        # 30% weight to recency
    recency_column="timestamp",
)

Metadata pre-filtering

where= filters are pushed into the Chroma ANN scan before the vector query when the keys are scalar columns mirrored into Chroma metadata (TEXT / INTEGER / REAL / BOOLEAN — not LONGTEXT / JSON). Equality and Chroma operators are supported; the Python post-filter still runs on top, so results are correct in every mode.

# multi-tenant scoping at the vector-scan level
db.search("memories", "content", "standup notes",
          mode="hybrid", where={"user_id": "u2"})

# operator form
db.search("notes", "body", "roadmap", where={"score": {"$gte": 50}})

Versioned tables

Opt in per table with versioned=True. Versioned tables keep an append-only, hash-chained history ({table}__history) of every insert/update/delete, while the main table stays the current state — FTS5 and Chroma keep indexing current data only.

db.create_table("docs", {"id": TEXT, "body": LONGTEXT}, versioned=True, hash_chain=True)
db.author = "agent-1"                       # optional, recorded per event

db.upsert("docs", {"id": 1, "body": "v1"})
db.upsert("docs", {"id": 1, "body": "v2"})

db.log("docs")                              # change log, newest first
db.history("docs", key=1)                   # every version of a row
db.diff("docs", from_seq=1, to_seq=2)       # added/removed/changed
db.as_of("docs", seq=1)                     # point-in-time read

cp = db.checkpoint("docs", "before-edit")
db.rollback("docs", checkpoint="before-edit")   # state re-applied as new versions
db.verify_chain("docs")                     # -> {"valid": True, "checked": N, ...}
db.archive("docs", "exports/docs", format="parquet")
db.prune("docs", before_seq=5)              # retention; keeps the chain verifiable

Key semantics:

• Append-only: rollback records the restored state as new versions — nothing is erased, so the audit trail stays complete.
• verify_chain() detects any direct modification of the history store.
• Pruning records a chain anchor; the retained tail stays verifiable. Rewind depth is bounded by retention.
• Write overhead is ~13% (measured at 100k rows). Schema changes are rejected on versioned tables.
• Rollback cost: removal-heavy rollbacks are set-based and ~20× faster since 0.7.0; update-heavy restores re-embed changed LONGTEXT rows by design.

upsert() also works on non-versioned tables (plain insert-or-update).

Self-healing journal

All ChromaDB and DuckDB mutations are journaled in SQLite. Inserts process the journal immediately by default; defer it for batches:

db.insert_batch("contacts", big_list_of_rows, sync=False)
db.process_journal()      # sync everything at once

If the process crashes mid-write, the journal replays pending entries on next startup — no ghosts, no drift. Health and repair:

health = db.health("contacts")
# {"sqlite_rows": 5000, "chroma_docs": {"contacts_bio": 5000}, "status": "ok"}

result = db.reconcile("contacts")
# {"ghosts_deleted": 0, "missing_added": 3, "metadata_updated": 0}

Long-document chunking

One embedding per LONGTEXT cell is right for messages and memory entries. For multi-page knowledge documents, index chunks as rows so retrieval works at paragraph granularity:

from hybriddb.chunking import chunk_text

db.create_table("doc_chunks", {"doc_id": TEXT, "chunk_seq": "INTEGER", "content": LONGTEXT})
for i, chunk in enumerate(chunk_text(full_text)):   # ~300-token chunks, never mid-sentence
    db.insert("doc_chunks", {"doc_id": doc_id, "chunk_seq": i, "content": chunk})

# search chunks, then join back to the parent document
hits = db.search("doc_chunks", "content", "quarterly roadmap", mode="hybrid")

Splitting rules: paragraph boundaries first, then sentences — never mid-sentence; adjacent pieces merge until ~1200 chars (~300 tokens for MiniLM-class models); max_chars is tunable (default 1200); overlap=True (default) prepends the previous chunk’s final sentence. Chunks are ordinary rows, so versioning, checkpoints, and rollback work on the chunk table unchanged.

Custom embeddings

Default is ChromaDB’s bundled local MiniLM — no API key. Plug in any embedding function:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
db = HybridDB("./data", embedding_fn=lambda text: model.encode(text).tolist())

db = HybridDB(
    "./data",
    embedding_fn=lambda text: openai_client.embeddings.create(
        input=text, model="text-embedding-3-small"
    ).data[0].embedding,
    embedding_model_name="text-embedding-3-small",
)

Continue to the API reference for every public method.

Source of truth for this page: HybridDB · open-assistants-lab/HybridDB