Benchmarks
The measured fusion story — panel synthesize 95.0% vs 90.0% best single vs 82.5% self-consistency on matched GSM1K, with evidence feeding the router.
Fusion, measured — not marketed
matched GSM1K, n=40| Strategy | Score |
|---|---|
| ModelPanel fusion panel (synthesize) | 95.0% |
| Best single model | 90.0% |
| Self-consistency baseline | 82.5% |
GSM1K is a level playing field: ScaleAI’s post-cutoff math set with deterministic last-number grading — none of the panel members appear in its reference leaderboard, so fusion “gains” can’t be contamination noise. Members and judge run at temperature 0.0 (greedy, benchmark-fair; SelfConsistency keeps 0.7 sampling).
GSM8K is substantially contaminated in training corpora. The GSM8K-vs-GSM1K delta is the overfitting probe: a model that “wins” GSM8K and collapses on GSM1K memorized, not reasoned. Only clean-data matched runs count as evidence.
How the panel is selected
The benchmark harness (modelpanel/) runs adapters per benchmark (GSM8K, GSM1K, MATH, GPQA, LiveCodeBench) against every candidate model, then evidence feeds the router: benchmark JSONs → panel_selector --apply → registry chains.
- Reasoning panel members selected from matched gsm8k+gsm1k evidence (nemotron / minimax / gpt-oss, n=80 pooled); other rosters gated on their own evidence —
modelpanel/fusion-codeand-generalship only when their benchmarks support them. - Members and judge default to temperature 0.0; SelfConsistency keeps 0.7 sampling (it’s the control).
reasoning_effort=highis accepted by Groq/NIM on gpt-oss-120b (verified: CoT grows 1137→1382 tokens), enabling calibration toward official model-card conditions.
The cost model
Fusion costs ~4 calls per prompt (3 members + 1 judge) — on low-cost tiers that’s latency, not money. If the members+judge phase exceeds the latency budget, the gateway degrades to the best single member automatically (x-modelpanel-mode: fusion-degraded) — the answer never fails because the panel is slow.
The health loop
The gateway is only half the product. The probe pipeline re-verifies the free-tier landscape every 8 hours — 30+ providers, /models + 1-token inference, dead-marking on failures — and the free-tier index publishes the result. Registry, router, benchmarks, and the public index all read from the same truth:
probes (every 8h) ──▶ registry.json ──▶ gateway hot-reload ──▶ routing + fusion rosters
benchmarks ──▶ panel_selector --apply ──┘Source evidence: research/benchmarks/ in the repo (timestamped run JSONs), the design spec §13.4 (fusion promoted to MVP on this evidence), and the GSM1K reference leaderboard (10 verified paper models — none of the panel appear, keeping the field clean).
Source of truth for this page: ModelPanel · open-assistants-lab/ModelPanel