Repository navigation
Conversation
…lease dates Cards for OpenAI's GPT-6.2 Astra and GPT-6.2 Sol, released 2026-11-17, written before their onboarding. The change touches no published artifact. With no --model, eval-no-tools now sends requests to both models along with the rest of config.MODELS. - Route: both ids join GPT_6_MODELS, so they take the Responses-API transport, the local litellm registration and the Responses pricing branch. OpenAI's API changelog says tool calling on these models requires the Responses API. - Price: Astra $8 / $40 per 1M input/output tokens and $0.40 cached input; Sol $2 / $10 and $0.10 (OpenAI's model and pricing pages, read 2026-11-17). Cache reads are a twentieth of input, as on the GPT-6.1 rows, so each override carries its cache_read rate. Cache writes at the 30-minute rate are 1.25x input, the harness default. The listed long-context rates (double input and cached input, 1.5x output above 272k input tokens) are the structure the harness already applies to every Responses row. - Cards: written from OpenAI's documentation before any request to either model, and their notes say so. They take the GPT-6.1 Sol and Terra treatment: the forced answer tool, thinking_budget=True (the 16,384-token starting budget and 300s timeout) and no reasoning parameter. OpenAI's pages state no default reasoning effort and do not say whether a forced tool_choice works on these models; the onboarding gauntlet settles both and supplies the expected cost per scenario. The module docstring now says a card written before its model's onboarding says so in its notes. - Release dates: 2026-11-17 for both, from OpenAI's API changelog and its Developer Community announcement, in paper_results and the app's mirror. - Tests: the cards' treatments, cost reconstruction at the listed standard and long-context rates, the request shape (forced tool; no reasoning, temperature, top_p or logprobs), the listed cache rates and litellm entries, and the release dates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check found nothing above a nit. This takes four of its notes: Sol's long-context comment no longer says more than the pricing page does, the forum post is called a forum announcement, a test docstring stops claiming that the models reason by default, and one docstring is reflowed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Both cards now begin "Onboarded 2026-11-19" and give the onboarding gauntlet's results, run under the cards' own request fields: the forced answer tool passed 3/3 and 16/16 whole-scenario for each model. Each card gains the expected cost per scenario the gauntlet suggested, $0.057 for Astra and $0.019 for Sol. A two-household smoke on each card parsed all 36 outputs with no budget escalation and reasoning tokens on every request, and every response named the requested model with no dated snapshot. No request field changes: the contract, the 16,384-token starting budget and the 300s timeout stay as they were. The module docstring drops its clause about cards written before onboarding, since no card is one now. The tests pin the two expected costs and price the smoke's four requests at the overrides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cards gave each expected cost per scenario as "at $8/$40 per 1M" and "at $2/$10 per 1M", but the suggested $0.057 and $0.019 include cache-write pricing. The notes now say "at OpenAI's list rates", with the cache-write rate beside the input and output rates. Each card's comment gives the inferred reconstruction: the whole-scenario probe sends the manifest's first scenario, the smoke's first household, so its request had 2,236 prompt tokens, and it wrote the 2,104 tokens to the cache that the smoke's first household read. With the probe's completion tokens that gives $0.056896 and $0.018914; list input and output rates alone would round to 0.053 and 0.018. The cards' cost test now rebuilds both values from those counts at the price overrides, as the Gemini 4 cards' test does. Sol's comment now says the aborted first gauntlet stopped after 150s on the gauntlet's clock while litellm's error cites the 300s timeout, so the sentence does not read as the model's latency. Both notes now say that both smoke responses name the requested model. No request field changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This adds the serving card, list price and release date for Google's Gemini 4 Neon so the board can run it. No request has reached the model, and this changes no published artifact. - Release: the Gemini API changelog lists gemini-4-neon as generally available on 2026-11-18, on the paid tier, and Google's launch post of that day says it is rolling out that day in the Gemini API (both read 2026-11-19). Route: the first-party Gemini API, gemini/gemini-4-neon, the model page's one stable version with no dated snapshot. - Price: $3 / $15 per 1M input/output tokens (output includes thinking tokens) and $0.15 cached input, Google's paid-tier standard rates, one rate at every prompt length, with no introductory rate, no dated change and no free tier. litellm 1.95.0's bundled map has no gemini-4 id, so the override prices every request, as it does for Gemini 4 Carbon and Hydrogen. - Card: written from Google's documentation and from Gemini 4 Hydrogen's card, whose default thinking level, high, Neon shares. It takes the board's forced answer tool, sends no reasoning parameter, takes the thinking class's 300s timeout and the Gemini family's 16,384-token starting budget, and caps escalation at 65,536 tokens, the Models API's outputTokenLimit and the model page's default synchronous output. It sets no per-scenario cost. The onboarding gauntlet settles the contract, the reasoning, the timeout and the cost. - Module docstring: a card written before its model's onboarding says so in its notes, as Neon's does. - Tests: the card's treatment and escalation ceiling, the override's reconstruction, Neon in the Gemini 4 listed-rate property (renamed to cover three models), the request shape, and the release date. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ogen's contract. The price comment said the page lists no cache storage charge; the page states no storage or write price at all, so the comment now says only what it lists. The card's notes said Neon takes Hydrogen's treatment, which would include Hydrogen's 600s timeout; they now say Hydrogen's contract, as the card's comment and the request test do. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…the measured output rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd correct the card's raw-probe wording. The card now says its 475 to 516s is a constant-rate prediction from calls of at most 16,384 tokens, that the record's slowest rate (the short raw call at level low) predicts about 624s and why the card sets that call aside, and that no page read for Neon gives a model-specific timeout. It names the gauntlet's not-scorable verdict at its fixed budget, describes the raw question as tax before refundable credits, gives the raw calls' thought and answer tokens separately, and says litellm sends one non-streaming request with no long-decode continuation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ache writes as assumed usage. OpenAI's pricing page lists a long-context 30-minute cache write of $20 per 1M for GPT-6.2 Astra and $5 for GPT-6.2 Sol, the rates the harness's long-context doubling already gives. The two price comments now state every long-context rate with its source and read date, and say that batch rows are priced on the same synchronous basis. The GPT-6.2 pricing test prices a long-context cache write for each model, and the roster-wide cost property names these two rates as listed. The gauntlet reports give no prompt or cache counts, so the cards' comments and the card cost test now give the probe's 2,104 cache writes as assumed usage and say what the record does and does not show. The notes attribute the rounded cost estimates to onboarding. The LiteLLM cache-rate test is renamed for the effective map entry it reads, and its docstring now lists only what it asserts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"with only the gauntlet run on the model before it" could be read as a noun phrase; the comments and the card test's docstring now say that only the gauntlet had run on the model before the smoke's first household. Comments and a docstring only. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Brings in the GPT-6.2 cards' answers to their review's first round (comments, docstrings, test names and two pricing test cases). No conflict; nothing of Gemini 4 Neon's changes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…mples. The six-decimal costs in the cards' comments and the card test rest on 2,236 prompt tokens, 2,104 cache writes and zero cache reads; the comments and the docstring now say all three, and the test passes the zero reads explicitly. They no longer say that only the gauntlet had run before the smoke's first household: the smoke's cache read does not show which request wrote the cache. The card values are unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Brings in the GPT-6.2 cards' answer to their review's second round (comment and docstring wording, one explicit test argument). No conflict; nothing of Gemini 4 Neon's changes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The gemini/ branch of _uncapped_completion_controls returned the Gemini family's 16,384-token budget before it read the card, so completion_token_cap had no effect on a gemini/ id. A gemini/ card that sets the field now starts its requests there (base and cap both, as a thinking-budget card does on other providers, still clamped to the card's escalation ceiling), and every other gemini/ id keeps 16,384. No card sets the field in this commit, so no request changes. The onboarding gauntlet's probes, the Gemini batch body and the run fingerprint take their budget from the same controls, so they follow such a card too. The ModelCard field comment now says the field also sets a gemini/ card's starting budget. Tests pin the 16,384-token budget of every gemini/ id that sets no cap, check that each gemini/ card reads the field on first and answers-only repair requests, and check that a card starting at its ceiling sends one request and records a length finish as budget_exhausted_at_ceiling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two-household smoke of 2026-11-19 ran on the board's request at the Gemini family's 16,384-token starting budget. One household's first two requests used the whole 16,384- and 32,768-token budgets with no answer before its third, at 65,536, answered; each of those two was billed in full and discarded. Neon's card now sets completion_token_cap to 65,536, its escalation ceiling, so no request goes out below the ceiling. The forced answer tool, the provider-default thinking level and the 600 s timeout stay. The card's comment block and notes record the smoke per request and no longer say that the smoke has not run or that it will settle the contract. The expected cost per scenario stays unset until a smoke at the new starting budget. The treatment table and the Neon request test pin the 65,536-token start on first and answers-only repair requests and in the run fingerprint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in its card notes. The notes said the record's requests ran at 127 to 138 completion tokens a second, but the raw call at thinking level low, which the next sentence sets aside, ran at about 105. They now name the gauntlet's probes, the raw call at the default level and the smoke's requests, as the comment block does. The comment block's thinking_budget bullet is reflowed, and the notes' sentence on budget_exhausted_at_ceiling now says the harness records it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de false or too broad. The Gemini 4 Carbon and Hydrogen card comments said the harness gives every gemini/ id a 16,384-token starting budget. Since a gemini/ card can now set its own, they say that these cards set no completion_token_cap and so get the Gemini family's 16,384. Neon's timeout comment said the record shows no slowdown with length; the raw call at 7,930 tokens ran faster than the 32,768-token smoke request, so it now says the smoke shows none. Comments only: the module's syntax tree is unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ond smoke. - The card no longer says the harness discards a truncated response. It says the first smoke's two truncated requests returned no usable output, and that on a length finish the harness keeps what came back and, at the ceiling, marks each output still missing budget_exhausted_at_ceiling. - The second smoke, at the 65,536-token start, replaces the pending-smoke sentences: both households answered every output on their first response. - The output-rate passage gives the measured range instead of a claim about length, and keeps the 65,536-token times conditional on it. - Per-call figures are rounded; "for every model" is gone. - Neon's price comment says the rates reconstruct synchronous token cost from reported usage. - Three test docstrings name _initial_completion_budget_tokens and say "one request" for what the tests assert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and coverage. The card comment and notes now say the harness keeps a length-finished response's usable values and explanations, with valid cells from earlier requests, and name the two records it writes at the ceiling: failure source budget_exhausted_at_ceiling for an output still missing its value, and "budget_exhausted_at_ceiling (explanation)" in the error for one that has its value but lacks a usable explanation. The contract bullet says the forced tool worked for both households in each smoke, with no comparison to the JSON contract. Two test docstrings now say the request controls are checked with and without explanations and the initial-budget helper with explanations, and that the 600s timeout comes from extrapolating measured rates to 65,536 tokens. Comments and docstrings only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gth. The card comment and notes said the harness asks no more for missing outputs at the ceiling. It stops only when a request at the ceiling ends at length; after another finish, repair requests still ask again. The notes and comment now say so, matching the PR description. Comment and the Neon notes string only; no executable code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds Gemini 4 Neon (
gemini-4-neon), which Google released on 2026-11-18, to PolicyBench. The harness starts Neon at its 65,536-token synchronous output limit. In a first two-household smoke at the Gemini family's 16,384-token start, one household answered only on its third request, at 65,536, after two truncated responses. In a second smoke at the new start, both households returned all requested values and explanations on their first response, through the forced answer tool.gemini/card'scompletion_token_capnow sets that model's starting budget. Before this PR the field had no effect on agemini/id. No othergemini/card sets it, so every othergemini/id keeps 16,384.completion_token_capandprovider_max_completion_tokensto 65,536, the Models API'soutputTokenLimitand the model page's default synchronous output. It sets a 600 s request timeout from the measured output rate and no expected cost per scenario.gemini-4id. PolicyBench uses these rates to reconstruct synchronous token cost from reported usage, as it does for Gemini 4 Carbon and Hydrogen.A bare
eval-no-toolswith no--modelnow sends requests to Neon too, as it does for every roster model. This PR targetsmainafter #268 merged, and its commit list still includes #268's original commits.Harness change
_uncapped_completion_controlsinpolicybench/eval_no_tools.pyreturned the Gemini family's 16,384-token budget from itsgemini/branch before it read the model card, socompletion_token_capdid nothing for agemini/id. Now agemini/card that sets the field starts its requests there. As in the thinking-budget branch that other providers' cards take, the value serves as both the base and the cap, and_completion_controlsstill clamps it to the model's escalation ceiling. The new tests pin 16,384 for every othergemini/roster id and card and for a card-lessgemini/id, at 1, 3, 16 and 20 outputs, with and without explanations. They also check that eachgemini/card reads the field. TheModelCardfield comment now says the field also sets agemini/card's starting budget.For a
gemini/card that sets the field (today only Neon's), the new start also reaches:_probe_requestinpolicybench/onboard.pytakes its budget from the same controls and probes a carded model under its card, so a rerun of the gauntlet on Neon would send 65,536-token probes.GeminiBatchAdapter.build_request_bodyinpolicybench/batch_eval.pytakesmax_output_tokensfrom the same request, so a Neon batch body would ask for 65,536.initial_completion_budget_tokens65,536 for Neon, so a run folder started at the 16,384-token start does not resume under this head.scripts/freeze_snapshot.pychecks a run's fingerprint against the harness. Itsshared_completion_budget_tokensalready read the card's field for thinking-budget cards, so it now matches the request Neon sends.Onboarding record
Gauntlet and raw calls
On 2026-11-19 the onboarding gauntlet ran under this card, which then had the thinking class's 300 s timeout and the Gemini family's 16,384-token start. The Gemini API accepted the board's forced function call, but on the 3-variable tool probe the model reasoned through the whole 16,384-token budget and ended at length with 0 of 3 outputs parsed. The 3-variable JSON probe ended the same way. The gauntlet sends each probe once at that budget, with no escalation, so its report reads "not scorable" at that budget. It did not reach its whole-scenario probe and derived no card.
Two direct generateContent calls with no tools asked the model one plain tax question. Thought tokens made up most of each response: about 7,600 of about 7,900 tokens with no
thinkingConfig(the default, high), and about 1,400 of about 1,700 at thinking levellow, which the board does not send. Both responses name the model asgemini-4-neon, with no dated snapshot. The first asked formaxOutputTokens65,536 on one non-streaming call, and the API accepted it.First smoke, at the 16,384-token start
Later on 2026-11-19 a two-household smoke ran on the board's request as it then was: the forced answer tool, no reasoning parameter, a 16,384-token first request doubled on a length finish to 32,768 and then 65,536, and the 600 s timeout, in one supervisor round. Costs in both smokes are estimates at Google's list rates.
On a length finish the harness keeps the usable values and explanations the response returned, with valid cells from earlier requests, and asks again at the next budget for the outputs still missing. In this smoke the two truncated responses returned nothing usable. The card now starts at the ceiling, so no request runs at a lower budget. When a request at the ceiling ends at length, the harness keeps the usable cells and sends no further request for an output still missing. An output still missing its value gets the failure source
budget_exhausted_at_ceiling. One that has its value but lacks a usable explanation keeps the value, and the harness addsbudget_exhausted_at_ceiling (explanation)to the error.Second smoke, at the 65,536-token start
On 2026-11-21 the same two households ran on the same request, with the first request at 65,536, in one round.
Output rate and timeout
The gauntlet's probes, the raw call at the default level and both smokes' responses ran at about 127 to 138 completion tokens a second, on responses of about 7,900 to 35,000 tokens. No request in the record used the whole 65,536-token ceiling. At those rates, one that did would take about 475 to 516 s, inside the 600 s timeout; at any rate below about 109 tokens a second it would time out. The raw call at thinking level
lowaveraged about 105 tokens a second, which would predict about 624 s, but the board does not send that level, so the card sets that call aside.Google's generateContent long-decode guide (read 2026-11-14 for the Carbon and Hydrogen cards, not re-read for Neon) says the server closes a non-streaming connection after 10 minutes. It also recommends a 20-minute client read timeout, and its option table gives 20 minutes per request. If the server closes at 10 minutes, a longer client timeout could not help such a request. If the 20 minutes applies, 600 s still exceeds the 475 to 516 s these rates predict.
Test plan
tests/test_model_cards.py, the Gemini request, price and budget tests intests/test_eval_no_tools.pyand the Gemini probe tests intests/test_onboard.pypass (152 tests).ruff check .,ruff format --check .andpytest -m "not slow", which also selects the release-date and app-mirror tests intests/test_paper_results.py.🤖 Generated with Claude Code