Core concepts

Canonical normalization across three dialects, tier-aware routing with fallback-before-first-token, notices, the fusion sidecar, and the registry the gateway hot-reloads.

Three dialects, one canonical request

Every request — OpenAI Responses, Chat Completions, or Anthropic Messages, streaming or not — parses into one CanonicalRequest and remembers the caller’s dialect for the return leg. That’s what makes cross-format fallback possible: an Anthropic request can fail over to OpenAI-compatible egress.

Tier-aware keys

The vault knows what kind of key it holds, and the router treats them accordingly:

This is the cost-arbitrage mechanism: fusion runs a panel (~4 calls per prompt), which is affordable on low-cost tiers — and safe on paid keys because they never fire unless you say so.

Routing and fallback

• Task aliases — free/auto, free/reasoning, free/code, free/fast — resolve to ordered candidate chains; direct ids (groq/openai/gpt-oss-120b) route to that model.
• Budget skip: local token-bucket budgets per provider model; a candidate with no remaining budget is skipped.
• Fallback before the first token: on connection errors, 5xx, 429s, or model-gone 4xx the router moves to the next candidate before anything streams. Mid-stream failures surface as an error chunk (retrying is the client’s call — documented behavior).
• Sticky routing: notices never auto-switch your model.

Notices

One surface, two kinds — a header plus a body field (and terminal stream events), deduped per (model, provider-pair) over 24h, never blocking:

• Failure notices: your first choice died; here’s who actually served (x-modelpanel-fallback-from).
• Opportunity notices: the model you pinned is also served by another healthy provider you have keys for (same or better tier, with headroom) — x-modelpanel-notice, plus a notice body field.

The fusion sidecar

modelpanel/fusion is served by a separate Python (stdlib-only) sidecar the gateway spawns on 127.0.0.1:8791:

• A task-routed panel — 3 models run in parallel from a recipe
• A separate judge synthesizes the answers (streamed)
• Degradation: if the panel exceeds the latency budget, the gateway serves the best single member automatically — x-modelpanel-mode: fusion-degraded
• Your recipes: modelpanel/fusion/<your-combo> reads ~/.modelpanel/combos.json — {members, method, judge}

Rosters ship for reasoning, fast, code, and general — gated on benchmark evidence per task (see Benchmarks).

Registry-driven, hot-reloaded

The gateway boots from registry/registry.json (fail-fast without it) and fs.watch hot-reloads it — a fresh probe run changes routing with no restart. Every model in the catalog carries a task class and a tier; /v1/models reflects probe-verified models per provider.

Adding a provider is a registry entry, not code: adapters are config-driven (three shapes cover most — OpenAI-compatible, Gemini-native, Cloudflare/ Cohere-style oddballs).

The probe pipeline (Python, this repo) re-verifies provider health every 8 hours on the Open Assistants infra — 30+ providers, /models + 1-token inference — feeding the registry, the router, and the free-tier index.

Security posture

• Keys in ~/.modelpanel/keys.json (chmod 600) or env overrides — env wins over the vault file.
• Keys are sent only to the provider that owns them; never logged (redacted), never resold.
• Prompts are never persisted (in-memory only); logs are metadata (provider, latency, status).
• Binds 127.0.0.1 by default — MODELPANEL_HOST/MODELPANEL_PORT to change.

Source of truth for this page: ModelPanel · open-assistants-lab/ModelPanel