[tjonesit]

Tirtha is an OpenAI-compatible gateway by Spanda Works (https://tirtha.ai). This entry is Tirtha 1.0 as served in production on 2026-11-10.

In auto mode a small model answers first; a second model gives a second opinion and a yes/no checker compares the two. The request climbs to a larger model only when the check fails. The served model is the routing choice.

Predictions. Every prompt of the full split (8,400) and the robustness split (420) was sent to the production setup and the served model recorded. Model pool: google/gemma-4-31b-it-turbo (Base), deepseek-ai/DeepSeek-V4.1-Flash-1017 (Plus, and the second opinion), Qwen/Qwen3.8-Max (Max; not reached in this setup).

Prices. Our public per-level prices (https://api.tirtha.ai/v1/pricing, read 2026-11-11), not provider list prices, because they are what a customer pays. The checker (baseten/elephant-1) is priced at its maker's list price.

Router-side calls are in router_usage. The first answer on climbed rows is measured. The second-opinion call and the checker call are estimated from measured averages (second opinion = the first answer's token counts; checker = 515 input tokens), as stated in each row's purpose field. Together they are about 63% of the cost.

Data use. This is Tirtha 1.0 as served in production on 2026-11-10. Its answer checker was chosen on 2026-11-08 by a pre-registered test on GSM8K and TriviaQA questions taken from their original public sources, not from RouterArena's files; some of those public questions may also be in RouterArena's set. Later that day we ran RouterArena's sub_10 split to compare setups, as the README suggests. A climb-to-larger-model variant that scored best there is not this submission. No part of the router was trained, fitted or tuned on RouterArena data or labels. Every model call the router makes is listed in router_usage.

Local check on main @ 18383b4: check_config_prediction_files.py tirtha full and ... robustness both pass; compute_cost.py tirtha --split full = $0.3320 per 1K queries.

[tjonesit]

/evaluate

[github-actions]

Router Evaluation Results

Router: tirtha
Dataset Split: full
Evaluator: main @ 18383b4

RouterArena Metrics

Metric	Value
RouterArena Score	0.7338
Accuracy	75.11%
Total Cost	$2.788474
Avg Cost per Query	$0.000332
Avg Cost per 1K Queries	$0.3320
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.8262

---

Evaluation completed by RouterArena automated workflow