Skip to content

Cost fallback: apply cache rates or give no cost, and none for GPT models - #273

Draft
MaxGhenis wants to merge 18 commits into
mainfrom
fix/analysis-cost-fallback
Draft

MaxGhenis wants to merge 18 commits into
mainfrom
fix/analysis-cost-fallback

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Draft: two published board costs are filled by this code path, so this PR stays a draft until Max decides. On the frozen board the change leaves both costs unchanged, to the bit.

Two terms below: the fill is invoked for a model when model_cost_latency calls it, and a model gets a filled cost when the fill returns one.

What changes

When a model's summed cost is empty (no row records one, or, in malformed data, recorded costs of +inf and -inf sum to NaN; see "When the fill is invoked") and the model has a PRICE_OVERRIDES_PER_1M entry, analysis.model_cost_latency invokes a fill from that entry. On main the fill applies the entry's input and output rates to the summed prompt and completion tokens of the rows that record both counts, whatever else the entry lists. That differs in three ways from what eval_no_tools._reconstruct_token_cost returns for a response's counts under the same entry:

  • It ignores the cache-read and cache-write rates an entry lists. Thirteen entries outside the Responses-API GPT models list one: Claude Fable 5, 5.1 and 5.2, Opus 5.5, Sonnet 5.5 and Haiku 5.5 list both; DeepSeek V4.1 Flash, the three Gemini 4 entries, Grok 4.7, Muse Wonder and Muse Spark 1.3 list a cache-read rate.
  • For the ten Responses-API GPT entries it ignores their cache rates: the entry's cache-read rate where it lists one, otherwise a tenth of the input rate, and cache writes at 1.25 times the input rate.
  • For the same ten it ignores the long-context rate: above 272,000 prompt tokens _reconstruct_token_cost doubles the input and cache rates and multiplies the output rate by 1.5.

For a model with at least one row that holds both token counts, the new _override_fallback_cost returns the following. A model with no such row gets no cost (Claude Fable 5 on the frozen board):

Override Fill
Input and output rates only, and not a Responses-API GPT model (18 entries) As on main: the same rates on the same rows' sums
A cache-read or cache-write rate, and not a Responses-API GPT model (13 entries) Cached and cache-write tokens at those rates, when every row that has both token counts records one response (live_calls and cache_hit_calls of 1 and 0, or 0 and 1), a cached count and, if the entry lists a cache-write rate, a cache-write count. Otherwise no cost
Responses-API GPT model (10 entries) No cost

Why one response per row: the reconstruction bounds a response's cached and cache-write counts by that response's prompt count. A row can sum several responses (test_rows_summing_two_responses_fill_no_cost builds one), and bounding a sum is not summing the bounds. With one response per row, a row holds that response's counts or an equal share of them (how run_no_tools_eval, rows_from_unit and a row repair write a row, as the claims checks read and ran them), and dividing commutes with the bound, so the rows price to their share of what the reconstruction returns for the response. A random check below compares the two on rows built as equal shares.

Why an empty count gives no cost: an empty cache count can be a count the provider did not report, or a column another model's rows brought into the frame. full_run_export.load_predictions concatenates per-model frames, fold_board concatenates the base board with each added model's frame, and pandas pads a column one frame lacks with NaN.

This PR changes two scripts that read these costs. It does not change the other readers I know of (paper/index.qmd, scripts/tool_pilot_note.py, the release drivers, the app).

  • scripts/freeze_snapshot.py indexed costUsd on each no-tools board row in regenerate_analysis, so a row without it raised KeyError('costUsd') there. freeze_run calls regenerate_analysis after it has deleted the run directory and rewritten its payload. require_published_costs now names the board models whose row has no costUsd, and separately those whose costUsd is not a finite number of zero or more, and exits. main() calls it on the published dashboard before remove_stale_artifacts, and regenerate_analysis calls it in place of the indexing. This is a behaviour change for releases. Take a model with an override whose rows record no cost and hold both token counts. On main the fill gave it the flat input and output cost, so its row had a costUsd. Where the fill now returns no cost (a cache-rate entry whose rows do not each record one response and the counts above; a Responses-API GPT entry), the exported row has no costUsd, and the freeze stops and names the model unless a release driver carries a cost onto the row, as the drivers' CARRIED_USAGE does for Claude Fable 5.
  • scripts/effort_sensitivity.py: run_stats reports null for a cost or latency model_cost_latency leaves out, for a call count some row does not record, and for output tokens and the request window when no row records them. On main a cost or latency left out raised KeyError, as did an absent live_calls, completion_tokens or request-time column, and an absent cache_hit_calls column when every row recorded live_calls (when some row did not, main reported the call counts as null without reading that column); a window no row timed raised ValueError; and empty completion_tokens reported 0 output tokens. lockfile_litellm(None) returns a null pin. --check-board reads each frozen figure with get: a figure one side has and the other lacks fails the check, and a cost or latency neither side has passes.

When the fill is invoked

The rule is this PR's code: model_cost_latency invokes the fill for a model that has an override when the model's summed total_cost_usd is empty. The sum is empty when no row of that model in the frame records a cost (an empty cell and an absent column both count as none), and also when the recorded costs include both +inf and -inf, whose sum is NaN. Main has the same trigger; round 4 found the second case, and #277 lists it. This PR does not change which rows record a cost.

What follows is what I observed by running the named functions on stub responses for Gemini 3.6 Flash, which has an input-and-output override. There was no provider call. The script is runs/r3-fix-273/when_a_cost_is_recorded.py in the review record. It is not an account of when the harness records a cost: I ran only these cases, and I have not enumerated the repair, retry and replay paths.

  • The synchronous _extract_usage_metadata, given prompt_tokens of 100,000 and completion_tokens of 1,000, recorded the token reconstruction ($0.1575), and still did when the stub also carried a provider charge (usage.cost). Given the prompt count and no completion count, it recorded the provider charge when the stub carried one and no cost when it did not.
  • Given completion_tokens of 0, or prompt_tokens of 0, on a stub with no provider charge, it extracted that count as empty and recorded no cost. (The claims checks before round 4 found two more things: with a usage.cost on the same stubs it recorded the charge, and it keeps a 0 reported under the other names it reads, input_tokens and output_tokens.) I tagged the two zero-count results as one live call each with _request_call_provenance; _aggregate_request_results on them gave both counts, no cost and live_calls of 2. I built two rows from that aggregate by hand, and the fill priced them at $0.1575.
  • The batch rows_from_unit kept a completion_tokens of 0 and priced that result ($0.15 for 100,000 prompt tokens). A result with a prompt count and no completion count got no cost.
  • test_a_batch_cell_merged_from_two_partial_count_responses_invokes_the_fill takes a result with a prompt count only and a repair result with a completion count only through rows_from_unit, merge_attempt_rows and _apply_spend_ledger_costs (it does not run run_batch_eval). The merged row has both counts, no cost and live_calls of 2, and the fill prices it at $0.1575.

So the fill can be invoked for a model whose responses the recording functions can price: in both examples the model is Gemini 3.6 Flash, the same functions priced its both-count stub at $0.1575, and no row records a cost. In both examples each row records two responses. On this branch an input-and-output override prices such rows, as it does at any response count. With cached and cache-write counts of 0 added, a cache-rate override gives the merged batch row no cost, because the row records two responses (the same test checks this). I have not established why the rows of the two published models below record no cost.

Published models the fill is invoked for (verified)

I checked the tracked predictions files: the frozen board run (paper/snapshot/20260501/runs/us_full_run_20260612_policyengine_4_16_1_populace) and the seven sensitivity/data/*-predictions.csv.gz files. On the frozen board three models record no cost on any row and have an override, so the fill is invoked for them:

Model Override Rows with both token counts Published costUsd Filled cost on this branch
gemini-3.6-flash input and output only 1,984 of 1,984 5.4831164999999995 5.4831164999999995
grok-build-0.1 input and output only 1,984 of 1,984 4.487201 4.487201
claude-fable-5 cache rates 0 of 1,984 54.109099999999955, carried by the release drivers none

In each of the seven sensitivity files, the one model records a cost, so the fill is not invoked there. test_frozen_filled_costs_are_the_published_ones pins the two filled figures and checks that those two are the frozen board's models with no recorded cost, an override, and both token counts on some row. test_frozen_usage_reproduces_every_published_model_but_fable_5 compares each published cost, latency and token field of every other model with model_cost_latency on the frozen predictions. The round-2 and round-3 reviewers each report comparing both figures with main and the frozen payload and finding them equal bit for bit.

Each no-tools board row of the committed snapshot carries a cost, so the new freeze check passes there with the same costs in the same order (test_the_committed_snapshot_has_a_cost_for_every_board_model). For the two board models in the committed effort-sensitivity summary (Claude Fable 5.2 and Claude Opus 5.5), run_stats on their frozen rows equals the summary's board figures in each field run_stats returns (test_board_rows_reproduce_the_committed_run_figures).

Review rounds and what changed

The rounds were independent reviews (subfleet run --task review --tier hard). After rounds 1 and 2, a read-only verification pass checked the findings before the fixes (runs/r1-verify-273/ and runs/r2-verify-273/ in the review record). Round 2's pass has one check per finding. Round 1's has six checks; it examined the premise of the GPT finding and did not reproduce it as a defect.

Round 1 (job 20261120-184326-pb273-r1) found three P1s and two P2s. 200981b6 and 9985a803 answer four of them: a categorical model column priced an empty group under pandas 2; rows summing several responses were mispriced; NaN-padded cache columns read as zero; effort_sensitivity.py raised on a model with no cost. I declined the fifth, a P2 asking the fill to price a GPT household whose complete total is at most 272,000 prompt tokens. Round 2 found the first and third closed, the second closed for cache-rate overrides (an input-and-output override still prices a row's sums, as on main), and the fourth and fifth open in part.

Round 2 (job 20261120-225741-pb273-r2, head 0de3248e) asked for changes:

  • P1, --check-board raised on a frozen row without costUsd or latencySeconds. The dashboard writes such rows: the committed gpt-5.5 row has no latencySeconds. --check-board reads only the Claude Fable 5.2 and Claude Opus 5.5 rows, which carry both keys, so the committed check passed before the fix and passes now. Fixed in 3a9abe3d; b3f95422 and 3797be1d read the other frozen figures the same way. Four test functions (eight cases with their parameters) call check_board on the committed snapshot's payload and board rows: as committed, with one frozen key removed, and with Fable 5.2's recorded costs cleared.
  • P1, a board that froze on main could fail at costUsd. Answered by the named refusal above (f68eff27, b7cb1c68). The freeze still stops for such a board. It now stops before the write steps of freeze_snapshot.main, and says which models.
  • P1, this description said current runs reach the fallback only when they had no price. Wrong. Round 3 found the correction wrong in a different way; see below.
  • P2, run_stats raised on untimed rows and on rows written before the call-count columns. Fixed in 375b86fe and aacced1a. The placeholder rows run_no_tools_eval writes for a household when run_single_no_tools raises hold no usage and no request times, so run_stats takes the run's output tokens and window from the rows that record them.
  • P2, the cache-rate fill accepted count pairs such as (-1, 2) and (0.5, 0.5), and with a caller-supplied cache-read rate of zero it could price a fully cached prompt at -8.9e-22 (no shipped entry has a zero cache rate). Fixed in c9451f2c: (1, 0) and (0, 1) pass, and each row's uncached count is taken after the bounds and then summed. The input-and-output return, which prices the two published costs, comes before both lines.
  • P2, GPT models get no cost even where a row proves the long-context tier. The reviewer is right that a one-response row above 272,000 prompt tokens proves it. I corrected the docstring and the test header (314c2738) and filed the fill as Cost fallback follow-ups from the review of #273 #277, which sketches it. It would price nothing on the frozen board: the largest prompt share on a gpt- row there is 1,072 tokens, and the rows of the Responses-API GPT models there all record a cost.

Round 3 (job 20261121-134356-pb273-r3, scoped, head b7cb1c68) found round 2's three P1s closed and the published figures unchanged, and raised three findings about wording:

  • P1, the corrected explanation made false claims about when the harness records a cost. It said a response that reports one of the two token counts gets no cost. The reviewer ran the synchronous extractor on a response with a missing count and a usage.cost of $0.15, and it recorded the $0.15. The explanation also did not separate the counts a provider reports from the counts the extractor keeps: in the reviewer's two stubs the extractor dropped a 0. The section "When the fill is invoked" now states this PR's rule and the observations above, and makes no general claim about the harness. The same claim is gone from the model_cost_latency docstring and the batch test's docstring (9135560f), and from Cost fallback follow-ups from the review of #273 #277.
  • P2, the freeze's refusal said no row records a cost for a model whose recorded cost is invalid. The refusal now reports a missing cost and an invalid one separately (9135560f).
  • P3, the description said a model "reaches the fallback" both for invoking the fill and for getting a cost. It now uses the two terms at the top, and the table lists Claude Fable 5.

Before round 4, two rounds of read-only checkers went through this description and the comments the branch adds, claim by claim, against the code, the tests and the data (runs/r3-claims-check/ and runs/r3-claims-check-2/ in the review record). 3797be1d narrows the comments and docstrings the first round found broader than their evidence. It also rewords the freeze's two refusal messages, makes check_board read the frozen score with get, adds an assertion to the property test that a filled cost is not negative, and adds a score case to the missing-key test. 7eb2422a narrows four more comments the second round flagged and changes no code. This description is rewritten from both rounds' findings.

Round 4 (job 20261121-145615-pb273-r4, scoped, head 7eb2422a) found round 3's P1 closed and no runtime regression since round 3 (the executable code in analysis.py is unchanged since then): the freeze accepted all 52 frozen costs, and the two filled figures matched main and the frozen payload bit for bit on pandas 2.3.3 and 3.0.1. It approved, with two findings, which the review rule says to fix or file:

  • P2, this description's invocation rule left out malformed data. The trigger tests the model's summed total_cost_usd, and recorded costs of +inf and -inf sum to NaN, so they invoke the fill too. Main has the same trigger. b33ec0ae states the rule that way in the model_cost_latency docstring, "When the fill is invoked" states it here, and Cost fallback follow-ups from the review of #273 #277 lists the handling of non-finite costs.
  • P3, a comment said a count column with an empty cell is a float column. A nullable Int64 column keeps its dtype. The comment now says it can be one (b33ec0ae).

b33ec0ae changes a docstring and a comment only: with docstrings removed, the module's syntax tree is the same as at 7eb2422a (runs/r4-fix-273/trigger_and_ast.out in the review record, which also holds the trigger probe). 05135437 makes the same correction in the require_published_costs docstring and the freeze test's header, and changes no code either (runs/r4-fix-273/ast_same.second-commit.out).

Not changed, reported for a decision

  • Should a board model whose exported row has no cost block a release? That is what the freeze does on this branch. A cost a release driver carries onto the row passes, as Claude Fable 5's does. The alternative is to publish an explained dash. That needs a caption for it in the paper (the paper's cost_latency_caption asserts that the models with a cost dash are its zero-cost models) and a rule for the freeze's report, whose published total sums every board model's cost.
  • The release wrapper scripts/freeze_fable52.py writes the app data pointer, the versions file and the adjudications before it calls freezer.main(), and its --dry-run returns before that call. So the new check runs before the write steps of freeze_snapshot.main, not before the wrapper's. Cost fallback follow-ups from the review of #273 #277 asks the next release driver to call require_published_costs in its export gate and dry run.
  • Cost fallback follow-ups from the review of #273 #277 also lists what the review rounds, the read-only verification pass and my own checks found outside this change: Grok 4.5's published cost leaves out one household's usage (16 rows record usage and no cost); the synchronous extractor reads a 0 reported as prompt_tokens or completion_tokens as no count when the usage carries no input_tokens or output_tokens value; effort_sensitivity.summarize indexes frozen scores; grok-build-0.1's 132,096 cached prompt tokens price at the full input rate; the Claude Haiku 5.5 entry lists rates only up to 100,000 prompt tokens; an input-and-output fill prices a row that holds both counts even when they come from two responses that each reported one; and non-finite recorded costs (+inf and -inf together invoke the fill, and a lone +inf gives model_cost_latency a costUsd of inf; require_published_costs refuses a costUsd of inf, as tests/test_freeze_published_costs.py checks).

Tests

  • tests/test_cost_latency.py: the invariants header and the property test state the one-response rule as the two count pairs, and the property test asserts a filled cost is not negative. Tests added since round 2 cover the impossible count pairs, float counts, a fully cached prompt at a zero cache-read rate over fractional shares, and the merged batch row above.
  • tests/test_effort_sensitivity_run_stats.py (new in this PR): check_board on the committed snapshot with a frozen key removed; untimed rows; rows without the call-count columns; unrecorded cache hits and output tokens; a household whose run raised; the null lockfile pin; the committed run figures.
  • tests/test_freeze_published_costs.py (new since round 2): the refusal names each model without a cost and each with an invalid one, and ignores the tools arm; a zero cost passes; the committed snapshot passes; Fable 5.2 rows that sum two responses go through model_cost_latency and stop regenerate_analysis by name; a recorded negative cost is refused as invalid.
  • tests/test_model_response_window.py: the freeze refuses a board model without a cost before any of the four write steps the test stubs in main. Its fixture's dashboard now carries one costed model.
  • Locally the four files pass at 7eb2422a: 362 passed (litellm 1.95.0, pandas 3.0.1, Python 3.12). Against the round-2 head's code, the tests as they stood at 314c2738 (357 tests) had 32 failures: 31 tests added or changed for round 2 (one was later replaced by the failed-household test), and test_the_real_manifest_builder_publishes_the_window_it_is_given, which failed because that copy was an extracted archive with no git history. b33ec0ae and 05135437 change only docstrings and a comment; I did not rerun the files on them, and CI runs them.
  • A random check outside pytest, run at b7cb1c68, after which no commit changes the fill's arithmetic (2,000 cases per part; the 13 cache-rate entries that are not Responses-API GPT models; counts above the prompt count included): rows sharing one response priced to what the reconstruction returns for it within 6e-16 relative, a partial frame priced to its share, several one-response households priced to the sum, and no case of rows summing two responses got a cost.
  • I ran the locked environment of another local checkout read-only, with this branch first on the import path, and no other test file. CI runs its configured suite (pytest -m "not slow").

Verified and inferred

  • I checked by running code: the three frozen models the fill is invoked for and the two unchanged costs; that b33ec0ae and 05135437 leave each file's syntax tree unchanged apart from docstrings, and that recorded costs of +inf and -inf invoke the fill while a lone +inf or finite costs do not; the observations under "When the fill is invoked"; the four test files, including that main() refuses a board model without a cost before any of its four stubbed write steps runs, and that 31 of the tests added or changed for round 2 fail on the round-2 head's code; the random check; Grok 4.5's 16 costless rows; the counts of entries by kind.
  • I checked by reading code: the release wrapper's write order; the paper's cost-dash assertion; the rates _reconstruct_token_cost and _gpt_responses_cache_rates apply; main's fill.
  • Reviewers, a read-only verification pass and the two claims checks ran or read code that I did not rerun: their reproductions of each round-2 finding; that git reads --before=None as now; that the extractor keeps a 0 reported as input_tokens or output_tokens; what main's run_stats did in each case above; what the runners write on a row, including for a household whose run raised; that main and this branch give bit-identical costs for the 18 input-and-output entries in 400 random frames under pandas 3.
  • My judgment, not a source: no cost for GPT models; no cost for a cache-rate model whose rows lack a response count or a cache count its entry's pricing needs; refusing a freeze for a board model without a cost. The first two trade a possibly recoverable price for not publishing a wrong one. The third stops a release rather than publish a board with a cost missing.
  • Not established: why the two published models' rows record no cost. I ran the test files under pandas 3.0.1 only. The pandas 2 evidence is the round-1 probe on pandas 2.3.1 and 2.3.3 and the round-3 reviewer's comparison of the published figures on pandas 2.3.3. The categorical-column test runs under pandas 3 with pandas 2's observed=False groupby default set by a fixture.

Review

This changes what the analysis computes, so it needs an independent Subfleet review before merge. That review has finished: round 4, scoped to round 3's P1 and the commits that answer it (9135560f, 3797be1d, 7eb2422a), found the P1 closed and raised no P0 or P1, and b33ec0ae and 05135437 answer its P2 and P3. Merging waits on Max's decision, because published costs pass through this path.

🤖 Generated with Claude Code

When a model records no cost, model_cost_latency filled one from its price
override with the base input and output rates on the summed token counts.
That ignored cached and cache-write tokens for overrides that price them
apart, and the Responses-API GPT models' long-context rate, which turns on
one request's prompt length (above 272k prompt tokens OpenAI charges the
whole request 2x input and cache and 1.5x output).

A row records its share of its household's usage, summed over the
household's requests, so no row holds one request's prompt length. The
fallback now gives a GPT Responses model no cost, prices cached and
cache-write tokens at an override's cache rates the way
eval_no_tools._reconstruct_token_cost does, and gives no cost when the
frame lacks the column of a cache rate the override lists. Overrides with
only input and output rates keep the old arithmetic, so the two published
fallback costs (Gemini 3.6 Flash, Grok Build 0.1) do not move by a bit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Nov 21, 2026 2:30pm UTC

Request Review

A batch run splits one chunk's request among the chunk's outputs, not a
household's summed requests. The rows still record neither a request's
prompt length nor how many rows share it, so the conclusion stands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 3 commits November 20, 2026 20:43
…review r1)

Review round 1 of #273 found three ways the fill gave a wrong cost:

- A categorical model column: under pandas 2's groupby default the grouped
  sum has a row for every category, and .iloc[0] took the first one, so a
  model that was not the first category was priced from zeros. Select the
  model's own group, with observed=True.
- Rows that sum several responses: bounding the summed cache counts by the
  summed prompt is not what the reconstruction does response by response.
  A cache-rate fill now needs every priced row to record one response
  (live_calls + cache_hit_calls == 1).
- Empty cache counts read as zero: concatenating per-model frames pads a
  column one frame lacks, so a model whose rows never recorded cache usage
  got a cache-blind cost. An empty count for a rate the override lists now
  gives no cost.

GPT Responses models still get no cost: a household's row sum bounds a
request's prompt only on a frame known to hold every row, which the rows
cannot show.

The two published fallback costs (Gemini 3.6 Flash, Grok Build 0.1) use
input-and-output overrides and are unchanged to the bit.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
run_stats indexed costUsd, costPerHousehold and latencySeconds directly,
so a model whose cost model_cost_latency leaves out raised a KeyError.
It now reports null, and --check-board counts a null figure as a mismatch.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
MaxGhenis and others added 8 commits November 21, 2026 01:32
The dashboard leaves out a cost or latency it cannot establish, so a frozen
modelStats row can lack the key (the committed gpt-5.5 row has no
latencySeconds). check_board indexed both and raised KeyError before
comparing. It now reads them with get; a figure only one side records fails
the check and a figure neither side records passes. The tests go through
check_board on the committed snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… write (review r2)

model_cost_latency now leaves costUsd out for rows it cannot price (a
cache-rate model whose rows sum several responses or lack a count, a
Responses-API GPT model), where main priced them at the flat input and output
rates. regenerate_analysis indexed costUsd on every board row, so such a
board raised KeyError('costUsd') after main() had already deleted and
rewritten the run directory.

require_published_costs names the models without a finite cost and exits.
main() calls it on the published dashboard before remove_stale_artifacts, and
regenerate_analysis calls it where it built the dict. The committed board
passes with the same costs in the same order.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…(review r2)

The cache-rate fill checked that live_calls + cache_hit_calls is 1, which
also passed (-1, 2) and (0.5, 0.5). It now needs (1, 0) or (0, 1), compared
as floats so counts read back as 1.0 and 0.0 still pass.

It also subtracted the summed cache counts from the grouped prompt sum. The
two sums round differently, so a fully cached prompt split into fractional
shares could price below zero at a zero cache-read rate (-8.9e-22 for seven
tokens over six rows). Each row's uncached count is now taken after the
bounds and then summed. The input-and-output return, which prices the two
published fallback costs, comes before both lines and is unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
run_stats raised ValueError (NaT) on rows with no request timestamps and
KeyError on rows written before the call-count columns, and reported zero
output tokens or cache hits for rows that record none. Output tokens, each
call count and each end of the request window are now null unless every row
records them. lockfile_litellm(None) returns a null pin: git reads
--before=None as now and would return today's lockfile.

The frozen board rows still reproduce the committed summary's run figures.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ve (review r2)

model_cost_latency's docstring said the fallback reaches rows from a run
that had no price for the model. It also reaches a model none of whose
responses recorded both token counts: a batch cell answered with a prompt
count only, then repaired with a completion count only, merges to a row with
both counts and no cost. A test builds that row with the batch runner and
pins what the fill does with it (an input-and-output override prices the
sums; a cache-rate override gives no cost).

The GPT bullet said the rows cannot show whether a request crossed 272k. A
one-response row above 272k does show it; the fill prices neither case. No
computed result changes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ew r2)

The dashboard also drops exact and within1pct when it cannot compute them,
so check_board read them the way it read costUsd. A score the frozen row
lacks now fails the check instead of raising KeyError.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A failed household's rows record no usage and no request times (and call
counts of zero). Requiring every row to record completion tokens and request
times would have nulled the whole run's output tokens, request window and
lockfile pin for one failed household, where they came from the households
that answered. Those three figures now use the rows that record them and are
null only when none does. The call counts keep the every-row rule.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s one

require_published_costs accepted -1.0 as a cost. A cost is now a finite
number, zero or more. The refusal told the reader to carry a released cost,
which a new model does not have; it now says to rerun or replay the model so
its rows record their costs, or to carry a cost a release decision supplies.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
MaxGhenis and others added 2 commits November 21, 2026 13:58
The model_cost_latency docstring said no row records a cost when no response
recorded both token counts. A synchronous response without both extracted
counts still records the provider's reported charge when it carries one, and
the extractor drops a count reported as 0. The docstring now says only that
the fill can be invoked for a model whose run had a price, and names the
test that builds such a row. The batch test's docstring is scoped to
batch_eval.rows_from_unit, and the frozen-cost test's docstring names Claude
Fable 5, for which the fill is invoked and returns nothing.

The freeze's refusal reported a recorded cost of -1.0 as a cost no row
records. It now reports a missing costUsd and an invalid one separately.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A claim-by-claim check of the comments this branch adds found several that
were broader than their evidence. Narrowed here, with no change to what any
function computes for a row the committed data holds:

- _override_fallback_cost: how a household run, a batch run and a row
  repair put usage on a row, as each does it; the reconstruction's length
  tiers "for an override".
- model_cost_latency: the fill runs "in the cases that function lists";
  the batch example names which count each response reported.
- effort_sensitivity: the model is left out only with no cost, no latency
  and no total tokens; a household "none of whose requests was answered"
  records no usage; no statement about what the dashboard displays.
- freeze_snapshot: a cost is absent when no row records one and the model
  has no override or the override cannot price its rows; an invalid cost
  can come from the rows' token counts as well as their costs.
- Test docstrings: "returns" for _reconstruct_token_cost, not "records";
  no statement of what a case raised before.

check_board now reads the frozen score with get as well, and the property
test asserts a filled cost is not negative.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
- _override_fallback_cost: say only that a row's counts can sum several
  responses and can be an equal share. The sentence about what a household
  run adds up had a counterexample (a household whose first round raises
  after a truncated response was answered).
- check_board: a cost or latency neither side has passes; the comment said
  a missing figure fails.
- Test header: the recorded-cost invariant is for finite recorded costs.
- _two_response_rows: name the household runner.

Comments only; no code changes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@MaxGhenis MaxGhenis changed the title Price the analysis cost fallback only where row usage supports it Cost fallback: apply cache rates or give no cost, and none for GPT models Nov 21, 2026
model_cost_latency invokes the fill when a model's summed total_cost_usd
is empty. That covers recorded costs of +inf and -inf, whose sum is NaN,
as well as a model with no recorded cost; the docstring named only the
second. The response-count comment now says a count column with an empty
cell can be a float column: a nullable Int64 column keeps its dtype.

Docstring and comment only; the module's AST with docstrings removed is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The require_published_costs docstring and the freeze test's header said
model_cost_latency leaves costUsd out when no row of a model records a
cost. The trigger is the model's summed cost being empty, which recorded
costs of +inf and -inf also produce. Same correction as b33ec0a.

Docstrings only; each file's AST with docstrings removed is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 05135437 Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant