Skip to content

Add the Gemini 4 Neon model card and metadata - #272

Draft
MaxGhenis wants to merge 22 commits into
mainfrom
gemini-4-neon-cards
Draft

MaxGhenis wants to merge 22 commits into
mainfrom
gemini-4-neon-cards

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This adds Gemini 4 Neon (gemini-4-neon), which Google released on 2026-11-18, to PolicyBench. The harness starts Neon at its 65,536-token synchronous output limit. In a first two-household smoke at the Gemini family's 16,384-token start, one household answered only on its third request, at 65,536, after two truncated responses. In a second smoke at the new start, both households returned all requested values and explanations on their first response, through the forced answer tool.

  • Harness: a gemini/ card's completion_token_cap now sets that model's starting budget. Before this PR the field had no effect on a gemini/ id. No other gemini/ card sets it, so every other gemini/ id keeps 16,384.
  • Model card: Neon takes Gemini 4 Hydrogen's contract, the board's forced answer tool on the whole-scenario request. It sends no reasoning parameter, so the model thinks at its default level, which Google's thinking guide gives as high, as for Hydrogen.
  • Budget and timeout: the card sets completion_token_cap and provider_max_completion_tokens to 65,536, the Models API's outputTokenLimit and the model page's default synchronous output. It sets a 600 s request timeout from the measured output rate and no expected cost per scenario.
  • List price: Google lists Standard paid-tier rates of $3 input, $0.15 cached input and $15 output per 1M tokens, output including thinking tokens (read 2026-11-19). It lists one rate at every prompt length, with no introductory rate, dated change or free tier. litellm 1.95.0's bundled map has no gemini-4 id. PolicyBench uses these rates to reconstruct synchronous token cost from reported usage, as it does for Gemini 4 Carbon and Hydrogen.
  • Release date: 2026-11-18, from the Gemini API changelog's general-availability entry and Google's launch post, in the paper registry and the app's model metadata.
  • Module docstring: a card written before its model's onboarding says so in its notes, as Neon's does.
  • Tests cover the card's treatment and escalation ceiling, the price override (Neon joins the Gemini 4 listed-rate property), the request shape, the release date and the harness change.

A bare eval-no-tools with no --model now sends requests to Neon too, as it does for every roster model. This PR targets main after #268 merged, and its commit list still includes #268's original commits.

Harness change

_uncapped_completion_controls in policybench/eval_no_tools.py returned the Gemini family's 16,384-token budget from its gemini/ branch before it read the model card, so completion_token_cap did nothing for a gemini/ id. Now a gemini/ card that sets the field starts its requests there. As in the thinking-budget branch that other providers' cards take, the value serves as both the base and the cap, and _completion_controls still clamps it to the model's escalation ceiling. The new tests pin 16,384 for every other gemini/ roster id and card and for a card-less gemini/ id, at 1, 3, 16 and 20 outputs, with and without explanations. They also check that each gemini/ card reads the field. The ModelCard field comment now says the field also sets a gemini/ card's starting budget.

For a gemini/ card that sets the field (today only Neon's), the new start also reaches:

  • The onboarding gauntlet: _probe_request in policybench/onboard.py takes its budget from the same controls and probes a carded model under its card, so a rerun of the gauntlet on Neon would send 65,536-token probes.
  • The Gemini batch path: GeminiBatchAdapter.build_request_body in policybench/batch_eval.py takes max_output_tokens from the same request, so a Neon batch body would ask for 65,536.
  • Run fingerprints: the supervisor's treatment fingerprint and the resume sidecar's treatment record initial_completion_budget_tokens 65,536 for Neon, so a run folder started at the 16,384-token start does not resume under this head. scripts/freeze_snapshot.py checks a run's fingerprint against the harness. Its shared_completion_budget_tokens already read the card's field for thinking-budget cards, so it now matches the request Neon sends.
  • Later requests: repair and explanations-only requests start at the same budget.

Onboarding record

Gauntlet and raw calls

On 2026-11-19 the onboarding gauntlet ran under this card, which then had the thinking class's 300 s timeout and the Gemini family's 16,384-token start. The Gemini API accepted the board's forced function call, but on the 3-variable tool probe the model reasoned through the whole 16,384-token budget and ended at length with 0 of 3 outputs parsed. The 3-variable JSON probe ended the same way. The gauntlet sends each probe once at that budget, with no escalation, so its report reads "not scorable" at that budget. It did not reach its whole-scenario probe and derived no card.

Two direct generateContent calls with no tools asked the model one plain tax question. Thought tokens made up most of each response: about 7,600 of about 7,900 tokens with no thinkingConfig (the default, high), and about 1,400 of about 1,700 at thinking level low, which the board does not send. Both responses name the model as gemini-4-neon, with no dated snapshot. The first asked for maxOutputTokens 65,536 on one non-streaming call, and the API accepted it.

First smoke, at the 16,384-token start

Later on 2026-11-19 a two-household smoke ran on the board's request as it then was: the forced answer tool, no reasoning parameter, a 16,384-token first request doubled on a length finish to 32,768 and then 65,536, and the 600 s timeout, in one supervisor round. Costs in both smokes are estimates at Google's list rates.

  • Both households answered every output, values and explanations, through the forced tool: 16 of 16 and 20 of 20.
  • The 20-output household answered on its first request, with about 15,000 completion tokens (about $0.23).
  • The 16-output household's first two requests used their whole 16,384- and 32,768-token budgets and returned no usable output. Its third request, at 65,536, answered with about 24,000 completion tokens. The household cost about $1.12, $0.75 of it on the two truncated requests, which used about 55% of the smoke's completion tokens.

On a length finish the harness keeps the usable values and explanations the response returned, with valid cells from earlier requests, and asks again at the next budget for the outputs still missing. In this smoke the two truncated responses returned nothing usable. The card now starts at the ceiling, so no request runs at a lower budget. When a request at the ceiling ends at length, the harness keeps the usable cells and sends no further request for an output still missing. An output still missing its value gets the failure source budget_exhausted_at_ceiling. One that has its value but lacks a usable explanation keeps the value, and the harness adds budget_exhausted_at_ceiling (explanation) to the error.

Second smoke, at the 65,536-token start

On 2026-11-21 the same two households ran on the same request, with the first request at 65,536, in one round.

  • Both households answered every output, values and explanations, on their first response, through the forced tool. No response ended at length, the harness ran no repair, and no request erred or timed out.
  • The 20-output household's response used about 19,000 completion tokens and the 16-output household's about 35,000, nearly all thinking, at an estimated $0.29 and $0.53. Together they cost $0.83, down from $1.36 in the first smoke.
  • Both responses used more tokens than the same households' answering responses in the first smoke. Two households cannot show whether the larger first budget or run-to-run variation accounts for that.

Output rate and timeout

The gauntlet's probes, the raw call at the default level and both smokes' responses ran at about 127 to 138 completion tokens a second, on responses of about 7,900 to 35,000 tokens. No request in the record used the whole 65,536-token ceiling. At those rates, one that did would take about 475 to 516 s, inside the 600 s timeout; at any rate below about 109 tokens a second it would time out. The raw call at thinking level low averaged about 105 tokens a second, which would predict about 624 s, but the board does not send that level, so the card sets that call aside.

Google's generateContent long-decode guide (read 2026-11-14 for the Carbon and Hydrogen cards, not re-read for Neon) says the server closes a non-streaming connection after 10 minutes. It also recommends a 20-minute client read timeout, and its option table gives 20 minutes per request. If the server closes at 10 minutes, a longer client timeout could not help such a request. If the 20 minutes applies, 600 s still exceeds the 475 to 516 s these rates predict.

Test plan

  • Locally, tests/test_model_cards.py, the Gemini request, price and budget tests in tests/test_eval_no_tools.py and the Gemini probe tests in tests/test_onboard.py pass (152 tests).
  • CI runs ruff check ., ruff format --check . and pytest -m "not slow", which also selects the release-date and app-mirror tests in tests/test_paper_results.py.
  • Onboarding gauntlet (no card derived) and both two-household smokes, above.
  • Independent review.

🤖 Generated with Claude Code

MaxGhenis and others added 6 commits November 17, 2026 14:37
…lease dates

Cards for OpenAI's GPT-6.2 Astra and GPT-6.2 Sol, released 2026-11-17, written
before their onboarding. The change touches no published artifact. With no
--model, eval-no-tools now sends requests to both models along with the rest
of config.MODELS.

- Route: both ids join GPT_6_MODELS, so they take the Responses-API
  transport, the local litellm registration and the Responses pricing
  branch. OpenAI's API changelog says tool calling on these models requires
  the Responses API.
- Price: Astra $8 / $40 per 1M input/output tokens and $0.40 cached input;
  Sol $2 / $10 and $0.10 (OpenAI's model and pricing pages, read
  2026-11-17). Cache reads are a twentieth of input, as on the GPT-6.1 rows,
  so each override carries its cache_read rate. Cache writes at the
  30-minute rate are 1.25x input, the harness default. The listed
  long-context rates (double input and cached input, 1.5x output above 272k
  input tokens) are the structure the harness already applies to every
  Responses row.
- Cards: written from OpenAI's documentation before any request to either
  model, and their notes say so. They take the GPT-6.1 Sol and Terra
  treatment: the forced answer tool, thinking_budget=True (the 16,384-token
  starting budget and 300s timeout) and no reasoning parameter. OpenAI's
  pages state no default reasoning effort and do not say whether a forced
  tool_choice works on these models; the onboarding gauntlet settles both
  and supplies the expected cost per scenario. The module docstring now says
  a card written before its model's onboarding says so in its notes.
- Release dates: 2026-11-17 for both, from OpenAI's API changelog and its
  Developer Community announcement, in paper_results and the app's mirror.
- Tests: the cards' treatments, cost reconstruction at the listed standard
  and long-context rates, the request shape (forced tool; no reasoning,
  temperature, top_p or logprobs), the listed cache rates and litellm
  entries, and the release dates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check found nothing above a nit. This takes four of its notes: Sol's
long-context comment no longer says more than the pricing page does, the
forum post is called a forum announcement, a test docstring stops claiming
that the models reason by default, and one docstring is reflowed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Both cards now begin "Onboarded 2026-11-19" and give the onboarding gauntlet's results, run under the cards' own request fields: the forced answer tool passed 3/3 and 16/16 whole-scenario for each model. Each card gains the expected cost per scenario the gauntlet suggested, $0.057 for Astra and $0.019 for Sol. A two-household smoke on each card parsed all 36 outputs with no budget escalation and reasoning tokens on every request, and every response named the requested model with no dated snapshot.

No request field changes: the contract, the 16,384-token starting budget and the 300s timeout stay as they were. The module docstring drops its clause about cards written before onboarding, since no card is one now. The tests pin the two expected costs and price the smoke's four requests at the overrides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cards gave each expected cost per scenario as "at $8/$40 per 1M" and "at $2/$10 per 1M", but the suggested $0.057 and $0.019 include cache-write pricing. The notes now say "at OpenAI's list rates", with the cache-write rate beside the input and output rates. Each card's comment gives the inferred reconstruction: the whole-scenario probe sends the manifest's first scenario, the smoke's first household, so its request had 2,236 prompt tokens, and it wrote the 2,104 tokens to the cache that the smoke's first household read. With the probe's completion tokens that gives $0.056896 and $0.018914; list input and output rates alone would round to 0.053 and 0.018. The cards' cost test now rebuilds both values from those counts at the price overrides, as the Gemini 4 cards' test does.

Sol's comment now says the aborted first gauntlet stopped after 150s on the gauntlet's clock while litellm's error cites the 300s timeout, so the sentence does not read as the model's latency. Both notes now say that both smoke responses name the requested model. No request field changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This adds the serving card, list price and release date for Google's Gemini 4 Neon so the board can run it. No request has reached the model, and this changes no published artifact.

- Release: the Gemini API changelog lists gemini-4-neon as generally available on 2026-11-18, on the paid tier, and Google's launch post of that day says it is rolling out that day in the Gemini API (both read 2026-11-19). Route: the first-party Gemini API, gemini/gemini-4-neon, the model page's one stable version with no dated snapshot.
- Price: $3 / $15 per 1M input/output tokens (output includes thinking tokens) and $0.15 cached input, Google's paid-tier standard rates, one rate at every prompt length, with no introductory rate, no dated change and no free tier. litellm 1.95.0's bundled map has no gemini-4 id, so the override prices every request, as it does for Gemini 4 Carbon and Hydrogen.
- Card: written from Google's documentation and from Gemini 4 Hydrogen's card, whose default thinking level, high, Neon shares. It takes the board's forced answer tool, sends no reasoning parameter, takes the thinking class's 300s timeout and the Gemini family's 16,384-token starting budget, and caps escalation at 65,536 tokens, the Models API's outputTokenLimit and the model page's default synchronous output. It sets no per-scenario cost. The onboarding gauntlet settles the contract, the reasoning, the timeout and the cost.
- Module docstring: a card written before its model's onboarding says so in its notes, as Neon's does.
- Tests: the card's treatment and escalation ceiling, the override's reconstruction, Neon in the Gemini 4 listed-rate property (renamed to cover three models), the request shape, and the release date.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ogen's contract.

The price comment said the page lists no cache storage charge; the page states no storage or write price at all, so the comment now says only what it lists. The card's notes said Neon takes Hydrogen's treatment, which would include Hydrogen's 600s timeout; they now say Hydrogen's contract, as the card's comment and the request test do.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Nov 21, 2026 1:36pm UTC

Request Review

MaxGhenis and others added 3 commits November 19, 2026 23:02
…the measured output rate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd correct the card's raw-probe wording.

The card now says its 475 to 516s is a constant-rate prediction from calls of at most 16,384 tokens, that the record's slowest rate (the short raw call at level low) predicts about 624s and why the card sets that call aside, and that no page read for Neon gives a model-specific timeout. It names the gauntlet's not-scorable verdict at its fixed budget, describes the raw question as tax before refundable credits, gives the raw calls' thought and answer tokens separately, and says litellm sends one non-streaming request with no long-decode continuation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 3 commits November 20, 2026 08:06
…ache writes as assumed usage.

OpenAI's pricing page lists a long-context 30-minute cache write of $20 per 1M for GPT-6.2 Astra and $5 for GPT-6.2 Sol, the rates the harness's long-context doubling already gives. The two price comments now state every long-context rate with its source and read date, and say that batch rows are priced on the same synchronous basis. The GPT-6.2 pricing test prices a long-context cache write for each model, and the roster-wide cost property names these two rates as listed.

The gauntlet reports give no prompt or cache counts, so the cards' comments and the card cost test now give the probe's 2,104 cache writes as assumed usage and say what the record does and does not show. The notes attribute the rounded cost estimates to onboarding.

The LiteLLM cache-rate test is renamed for the effective map entry it reads, and its docstring now lists only what it asserts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"with only the gauntlet run on the model before it" could be read as a noun phrase; the comments and the card test's docstring now say that only the gauntlet had run on the model before the smoke's first household. Comments and a docstring only.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Brings in the GPT-6.2 cards' answers to their review's first round (comments, docstrings, test names and two pricing test cases). No conflict; nothing of Gemini 4 Neon's changes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
MaxGhenis and others added 2 commits November 20, 2026 17:58
…mples.

The six-decimal costs in the cards' comments and the card test rest on 2,236 prompt tokens, 2,104 cache writes and zero cache reads; the comments and the docstring now say all three, and the test passes the zero reads explicitly. They no longer say that only the gauntlet had run before the smoke's first household: the smoke's cache read does not show which request wrote the cache. The card values are unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Brings in the GPT-6.2 cards' answer to their review's second round (comment and docstring wording, one explicit test argument). No conflict; nothing of Gemini 4 Neon's changes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis changed the base branch from gpt-6.2-cards to main November 20, 2026 18:18
Main now holds the GPT-6.2 cards (#268, squash-merged as 610f495); this branch already carried the same content from the gpt-6.2-cards branch. No conflict.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
MaxGhenis and others added 4 commits November 20, 2026 19:36
The gemini/ branch of _uncapped_completion_controls returned the Gemini family's 16,384-token budget before it read the card, so completion_token_cap had no effect on a gemini/ id. A gemini/ card that sets the field now starts its requests there (base and cap both, as a thinking-budget card does on other providers, still clamped to the card's escalation ceiling), and every other gemini/ id keeps 16,384. No card sets the field in this commit, so no request changes.

The onboarding gauntlet's probes, the Gemini batch body and the run fingerprint take their budget from the same controls, so they follow such a card too. The ModelCard field comment now says the field also sets a gemini/ card's starting budget. Tests pin the 16,384-token budget of every gemini/ id that sets no cap, check that each gemini/ card reads the field on first and answers-only repair requests, and check that a card starting at its ceiling sends one request and records a length finish as budget_exhausted_at_ceiling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two-household smoke of 2026-11-19 ran on the board's request at the Gemini family's 16,384-token starting budget. One household's first two requests used the whole 16,384- and 32,768-token budgets with no answer before its third, at 65,536, answered; each of those two was billed in full and discarded. Neon's card now sets completion_token_cap to 65,536, its escalation ceiling, so no request goes out below the ceiling. The forced answer tool, the provider-default thinking level and the 600 s timeout stay.

The card's comment block and notes record the smoke per request and no longer say that the smoke has not run or that it will settle the contract. The expected cost per scenario stays unset until a smoke at the new starting budget. The treatment table and the Neon request test pin the 65,536-token start on first and answers-only repair requests and in the run fingerprint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in its card notes.

The notes said the record's requests ran at 127 to 138 completion tokens a second, but the raw call at thinking level low, which the next sentence sets aside, ran at about 105. They now name the gauntlet's probes, the raw call at the default level and the smoke's requests, as the comment block does. The comment block's thinking_budget bullet is reflowed, and the notes' sentence on budget_exhausted_at_ceiling now says the harness records it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…de false or too broad.

The Gemini 4 Carbon and Hydrogen card comments said the harness gives every gemini/ id a 16,384-token starting budget. Since a gemini/ card can now set its own, they say that these cards set no completion_token_cap and so get the Gemini family's 16,384. Neon's timeout comment said the record shows no slowdown with length; the raw call at 7,930 tokens ran faster than the 32,768-token smoke request, so it now says the smoke shows none. Comments only: the module's syntax tree is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ond smoke.

- The card no longer says the harness discards a truncated response. It says
  the first smoke's two truncated requests returned no usable output, and that
  on a length finish the harness keeps what came back and, at the ceiling,
  marks each output still missing budget_exhausted_at_ceiling.
- The second smoke, at the 65,536-token start, replaces the pending-smoke
  sentences: both households answered every output on their first response.
- The output-rate passage gives the measured range instead of a claim about
  length, and keeps the 65,536-token times conditional on it.
- Per-call figures are rounded; "for every model" is gone.
- Neon's price comment says the rates reconstruct synchronous token cost from
  reported usage.
- Three test docstrings name _initial_completion_budget_tokens and say "one
  request" for what the tests assert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and coverage.

The card comment and notes now say the harness keeps a length-finished
response's usable values and explanations, with valid cells from earlier
requests, and name the two records it writes at the ceiling: failure source
budget_exhausted_at_ceiling for an output still missing its value, and
"budget_exhausted_at_ceiling (explanation)" in the error for one that has
its value but lacks a usable explanation. The contract bullet says the
forced tool worked for both households in each smoke, with no comparison
to the JSON contract. Two test docstrings now say the request controls are
checked with and without explanations and the initial-budget helper with
explanations, and that the 600s timeout comes from extrapolating measured
rates to 65,536 tokens. Comments and docstrings only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gth.

The card comment and notes said the harness asks no more for missing
outputs at the ceiling. It stops only when a request at the ceiling ends
at length; after another finish, repair requests still ask again. The
notes and comment now say so, matching the PR description.

Comment and the Neon notes string only; no executable code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 6aa35257 Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant