[bellman-sionic]

[Submission] OpenGateway og/auto (jev-omni judge + task table) [sionic-gajae]

What

• Router: OpenGateway og/auto. A judge model (sionic-ai/jev-omni, OpenGateway decisions API) labels each query's task. A fixed task table then picks the serving model:
  • multiple_choice, factual_recall, other → google/gemini-3.7-flash (reasoning effort none)
  • math, passage_qa, and the fallback → google/gemini-3.8-flash (reasoning effort none)
  • translation → z-ai/glm-5.3-flash (served on OpenGateway UltraFast)
  • coding → deepseek/deepseek-v4.1-flash (reasoning effort low, served on OpenGateway UltraFast)
• Every generation was served through live OpenGateway. Each prediction row carries router_usage with the judge's actual token counts. The judge is priced at the OpenGateway current price, input $0.0063/1M tokens, output $0 (model_cost/model_profiles.yaml, model_cost/model_cost.json).
• Files: router_inference/config/opengateway-og-auto.json, router_inference/predictions/opengateway-og-auto.json (8,400 rows with generations), router_inference/predictions/opengateway-og-auto-robustness.json (420 routing-only rows), plus the jev-omni price entries (model_cost/model_cost.json, model_cost/model_profiles.yaml, universal_model_names.py).

Local evaluation (llm_evaluation/run.py, re-run 2026-11-12 on 60d621f at main's standard prices, incl. Gemini 3.7/3.8 Flash $1.50 / $7.50 from #263)

Metric	Value
Arena	75.55
Accuracy	81.62
Cost / 1K queries	$1.92 (judge cost included; total $16.096291, of which router calls $0.064119; scripts/pricing/compute_cost.py agrees)
Robustness	96.67

Notes

• Earlier figures (Arena 77.14, $0.96 / 1K) were computed at the Gemini introductory price. Predictions and token counts are unchanged; only the price table moved.
• The routes come from the judge's reading of the query, not from dataset or template matching. The judge sees only the query text, never a dataset name, and the same table applies to every query the judge labels as a given task.
• LiveCodeBench_419 first came back empty: deepseek-v4.1-flash@low spent all 32,000 output tokens on reasoning and hit the cap. The same prompt was resent to the same model and effort and finished normally (17,677 output tokens, finish_reason stop). With that row, the local re-score reports 0 abnormal entries.
• Earlier commits on this branch (391076e, ce08909, a0db2d4) are previous versions of this submission and stay in its history.

[sionic-gajae]

[bellman-sionic]

/evaluate

[bellman-sionic]

/evaluate

[github-actions]

Router Evaluation Results

Router: opengateway-og-auto
Dataset Split: full
Evaluator: main @ 44cc1ff

RouterArena Metrics

Metric	Value
RouterArena Score	0.7712
Accuracy	81.60%
Total Cost	$8.061135
Avg Cost per Query	$0.000960
Avg Cost per 1K Queries	$0.9597
Number of Queries	8400
Abnormal Entries	1
Robustness Score	0.9667

⚠️ 1 of 8400 queries (0.0%) had no valid generation (inference failed / empty answer) and were scored as incorrect (0). These queries still count toward the denominator, so accuracy and cost reflect the full query set. Please regenerate predictions for these queries and resubmit for a complete evaluation.

---

Evaluation completed by RouterArena automated workflow

[bellman-sionic]

/evaluate

[github-actions]

Router Evaluation Results

Router: opengateway-og-auto
Dataset Split: full
Evaluator: main @ 44cc1ff

RouterArena Metrics

Metric	Value
RouterArena Score	0.7713
Accuracy	81.61%
Total Cost	$8.070033
Avg Cost per Query	$0.000961
Avg Cost per 1K Queries	$0.9607
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.9667

---

Evaluation completed by RouterArena automated workflow

[bellman-sionic]

@yl231 Re-evaluation finished clean (abnormal 0, CI green). Could you take a look when you get a chance? [sionic-gajae]

[yl231]

Thanks, @bellman-sionic. The integrity checks are clean:

• the answers are your own: 4 of 6,983 long answers match another file;
• they score in line with other calls to the same models, with a same-model gap of −0.1 points on the public queries and −0.2 elsewhere.

One thing before merging: the judge's price needs a public source. sionic-ai/jev-omni isn't listed on OpenRouter, and the profile has source: null with "OpenGateway current price". Could you link a public page where OpenGateway publishes that price for anyone to buy?

If the judge instead runs on your own infrastructure with no public price, it falls under the self-hosted rule (#266). Its price then follows from its size: give its active parameter count and weight precision as local: {active_params_b: ..., bits: ...}, and python scripts/pricing/local_price.py --write sets the price.

Gemini prices are going up. Gemini 3.7 and 3.8 Flash are now priced at Google's standard $1.50 / $7.50 rather than the $0.75 / $3.75 introductory rate (#263). Your leaderboard cost will be computed at that price, like every other router's.

[bellman-sionic]

@yl231 Thanks for the review.

Judge price source. sionic-ai/jev-omni is sold publicly on OpenGateway to anyone with an account:

• Model page: https://opengateway.ai/models/sionic-ai/jev-omni
• Machine-readable: https://apis.opengateway.ai/v1/models (entry sionic-ai/jev-omni: list $0.126/MTok input, discount_percent: 95, no until, so $0.0063/MTok input, output $0)

The 95% discount is now standing (no end date), so $0.0063 is the current public price, not a time-limited promo. I pushed 199f150, which sets source to the model page and updates as_of/note. The price itself is unchanged.

Gemini. Understood. Please compute our cost at the standard $1.50 / $7.50 like every other router.

[sionic-gajae]

[bellman-sionic]

@yl231 Re-ran the local evaluation on the current head (60d621f, which already includes #263) at the standard Gemini price: Arena 75.55, accuracy 81.62, $1.92 per 1K queries (total $16.096291, judge $0.064119). Predictions are unchanged. PR description updated. [sionic-gajae]

[bellman-sionic]

Closing this submission. Gemini's standard price changed the trade-off, so we are re-optimizing the router for the current price table and will open a fresh PR. Thanks for the review, @yl231. [sionic-gajae]