Skip to content

Give each repeat of eval-no-tools-repeated its own cache namespace, so repeats ask the models again - #276

Draft
MaxGhenis wants to merge 12 commits into
mainfrom
fix/repeated-runs-fresh-sample
Draft

MaxGhenis wants to merge 12 commits into
mainfrom
fix/repeated-runs-fresh-sample

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Draft, waiting on Max (d3294). Independent review round 4 (the last scoped round) left P1 findings open, all in the directory check described below or in this description's account of it; none is in the per-repeat namespace. The PR rule allows no round 5, so d3294 asks whether to cut the check from this pull request (recommended), keep a rows-only best-effort check, or rework it to record provenance at run time. Nothing merges while a P1 is open. The paper wording below waits on a separate ruling (d3291) and is not part of this pull request. Follow-ups from round 2 that this pull request leaves: #279.

Round 4 found these statements below wrong or too strong at head 9e7b11b (source traces, not runs). They stand corrected here, and the code and README follow under whichever option d3294 picks:

  • The check can refuse independent files. A display name in one file can match another model's id from a ledger. A lost ledger record can make a false match through MODELS, so it does not "only narrow" what the check sees. And a deleted file's old ledger records can refuse a new, independent file at the same path.
  • The ledger reader skips a line that JSON cannot parse and keeps the rest. Only a failure that aborts the whole read discards the ledger.
  • A file written while a check reads the directory may be missed; it is not certain to be.
  • The precondition must also rule out setting or removing litellm.cache.namespace directly while a repeat runs. After the end-of-repeat error, a rerun into the same directory may keep the file's rows.
  • The count of shared responses in the refusal message counts bare response ids, so two models sharing one id count once.

What was wrong

policybench eval-no-tools-repeated runs the benchmark N times on the same households so that policybench analyze --runs-dir can report run-to-run variation. The CLI turns the LiteLLM disk cache on for this command (policybench/cli.py, the enable_cache block), and every repeat builds the same first-round requests from the same inputs (a household's first request, or each explanation chunk). LiteLLM keys its disk cache on the request, so repeats 2..N got repeat 1's responses back from the cache for every request whose stored answer a rebuilt request finds: 42 of the 48 model cards on ab896ddc (table below).

analyze --runs-dir then reports no run-to-run variation for those cards. I ran run_stability_by_model on three identical run files: every *_run_std it returns is 0 to floating-point rounding (largest 8.7e-15), which the report prints as 0.000. Nothing in its output tells a replay from a deterministic model, because policybench/analysis.py never reads the live_calls and cache_hit_calls columns.

Reading the code at each commit: the command has been in the cached set since its first commit (6cd9bce on v2, 2026-03-27; 1f0f7cb on main, 2026-05-01, #5), and nothing has put per-repeat material in the request or the cache key since. run_id reaches only the output rows, the resume metadata and the spend ledger. I ran the replay itself on ab896ddc only.

Offline confirmation

I ran the harness's run_repeated_no_tools_eval offline, through LiteLLM's real request path and the harness's real disk cache (enable_cache in a temporary directory), with a stub in place of httpx.Client.send and sockets blocked: every model card (48), two repeats, one household, two outputs. The stub answers each request with a value and a response id that name the call, so a replayed answer differs from a second answer.

Provider path / answer contract Cards Main (ab896ddc): repeat 2 got repeat 1's answers This change (5fb50e32): repeat 2 asked again
Anthropic / JSON 4 4 4
Anthropic / tool 3 3 3
DeepSeek / JSON 3 3 3
Gemini / JSON 5 5 5
Gemini / tool 6 0 6
OpenAI Responses / tool 11 11 11
OpenRouter / JSON 7 7 7
OpenRouter / tool 6 6 6
xAI / tool 3 3 3
All 48 42 48

On main with the cache off (a control, one card per row), repeat 2 sent as many requests as repeat 1 in all nine rows.

The six gemini/ tool cards ask again on main only because of a separate defect: LiteLLM rewrites their tool schema between its cache lookup and its store, so a rebuilt request never finds its answer. #275 repairs that. On a local merge of this branch (8da0ea99) with #275 at c708e707, repeat 2 got repeat 1's answers for all 48 cards with the namespace switch off, and every one of those replayed rows carried repeat 1's provider_response_id; with the switch on, repeat 2 asked again for all 48 and no repeat shared a response id. The directory check below rests on that.

The namespace switch has not changed since 5fb50e32; the later commits add the directory check (which, since round 3, also reads each file's spend ledger), the end-of-repeat check and the process-wide lock that keeps two repeats from overlapping, and change the per-repeat line, the tests and the wording.

What was published from this command

Nothing that I found.

The paper's run-to-run sentence

paper/index.qmd line 1191 ends: "the public leaderboard and app report point estimates only, because the benchmark sample is the dominant source of uncertainty here, ahead of run-to-run model variation."

I found no measurement behind it in the records I examined: the repository's history, the local results and the review records on this machine. A comparison kept nowhere I looked cannot be ruled out.

Proposed wording (recommended):

These intervals are a manuscript artifact: the public leaderboard and app report point estimates only. Each score rests on one set of answers per model, so the bootstrap cannot separate household sampling from run-to-run variation.

"One set of answers per model" rests on the paper's own snapshot: its predictions file holds 103,168 rows for 52 models and 100 households, and no model, household and output appears twice. The bootstrap resamples households, each carrying the answers its model gave once, so its intervals mix household sampling with the answers' own variation and cannot tell the two apart.

Smallest alternative: delete everything after "point estimates only" and add nothing.

This pull request does not change the paper. The wording is Max's call, filed as d3291. A pull request that applies it also edits the line in docs/site_improvements_scope.md and re-renders the HTML and the PDF with their pins.

The change

Each repeat runs under its own disk-cache namespace: <namespace>:repeat=run_000, <namespace>:repeat=run_001, and so on (_OwnCacheNamespace in policybench/eval_no_tools.py). The namespace goes back to what it was when the repeat ends, whether it returns or raises.

Why a namespace:

  • LiteLLM prefixes every key it computes with the cache's namespace attribute, which it reads on each call, for the lookup and for the store (Cache._add_namespace_to_cache_key, LiteLLM 1.95.0, the version uv.lock pins). A request that names its own namespace (in cache or metadata["redis_namespace"]) or brings its own cache_key bypasses the attribute. The one per-request namespace the harness sets, for an effort override (_effort_cache_namespace), is built from the attribute when the request is built, so it keeps the repeat; a test pins this for the lookup, the store and a rerun's replay. Pin the response-cache key so a rebuilt gemini/ tool request replays #275's pinned key is computed by LiteLLM's own key function at call time, from the same attribute, so it keeps the repeat too. The two branches merge without conflict.
  • So a repeat looks up and stores under its own namespace, which no other repeat of its set uses, and a rerun of the same repeat on the same cache looks up under the namespace that repeat stored under. (A second set on the same cache directory shares the namespaces repeat by repeat, which is how a rerun replays; the README says an independent set needs a new cache directory.) The rerun replays an answer only where LiteLLM computes the same key for the rebuilt request's lookup as it did for the original's store, and the entry is still there: LiteLLM's disk cache drops the oldest entries past 1 GiB. A rebuilt gemini/ tool request misses until Pin the response-cache key so a rebuilt gemini/ tool request replays #275 pins its key, in a repeat or out of one. A household's first-round requests (its first request, or each explanation chunk where the card splits explanations into chunks) are rebuilt from the same inputs every time. A follow-up request is built from the answers before it (a repair round from the outputs they left out, a larger completion budget after a cut-off answer, an explanation-only request from their values), so a rerun rebuilds the same follow-up only where those answers lead to it again.
  • A rerun can rebuild a set without paying again for the answers the cache finds. The task asked for that.
  • It uses the one vetted cache directory. A directory per repeat would mean creating and vetting another directory for each repeat.
  • Stability measurement suite: answer, reasoning, and counterfactual layers (spec + rung-0 implementation) #159 took the cache off the command instead. That stops the replay, but rebuilding a set into a new output directory then pays for every repeat again. (A rerun into the same output directory resumes from the CSVs either way.)

The namespace is an attribute of the process-wide cache, so the command must be the only evaluation in its process, which the CLI always is, and nothing may call enable_cache() or assign litellm.cache while a repeat runs. Under that condition repeat 1's own namespace also keeps it from looking up an eval-no-tools run's answers.

Other changes to the command:

  • A directory where two files hold one response is refused. Before its first request and again after each repeat, the command reads every CSV in the output directory (what analyze --runs-dir reads), whoever wrote it, and raises ValueError (the CLI exits with the message) if two files hold the same model and response id. A response id is the provider's, or the chatcmpl- id LiteLLM generates where the provider gives none (as for Anthropic), and a cache replay returns the stored response with its id, so two files sharing one hold one answer twice. The responses of a file that holds rows are its rows' provider_response_id (the first response each request group got), counted under the row's model name and its model id, and every response its spend ledger (run_000.csv.spend.jsonl, written by the harness since ea67093, 2026-08-09) records for that path, replays included, under the model id the ledger records. The ledger adds the responses the rows do not show: later explanation chunks, follow-ups, and one model under two names. It also keeps the responses of rows a resume replaced or a deleted file held. A ledger that is missing, cannot be read, or holds a line json cannot parse adds nothing, and the check then sees what the rows show. It also refuses a file with answered rows but no response id (written by code from before c7bf2db, 2026-05-08, which added the column, or cut short by an interrupted write), which it cannot check, and a CSV file it cannot read. It reads only files named *.csv and their ledgers, and it reads a response id as written, so an id such as NA stays an id (the resume reader now does the same). This replaces repeated_run.json, the record of which repeats the command started from round 1: round 2 showed a record of who wrote a file loses to an older checkout replacing a listed file's rows, to files written while the command paused after its one-time scan, and to a stale write from a concurrent process. The check reads the directory as it is when it runs.
  • A repeat raises if the cache changes under it. A repeat that ends to find litellm.cache replaced (by another enable_cache() in the process, say), or its namespace changed or removed, puts the namespace back and raises RuntimeError, unless it is already raising. The error names the repeat's file and says to use a new output directory, since a rerun into the same one would keep that file's rows (and the file's spend ledger keeps the responses its requests got, so deleting the file alone does not clear the directory). Two repeats cannot run at once in one process: the second raises RuntimeError on entry, and a repeat that cannot set its namespace releases the lock before it raises.
  • After each repeat it prints one line: the provider responses and cache replays the repeat's rows record (their live_calls and cache_hit_calls, counted once per request group: a model and household, and also an output with --single-output), the request groups that lack either count if there are any, and, when the runner kept rows from an existing file, those rows' own counts. It decides what was kept with the runner's own loader, so a file the runner keeps nothing from (a header only after a fatal error, or rows with infrastructure errors) is not reported as kept.

Known and unchanged:

  • For a file without a readable spend ledger, the check sees only the first response id of each request group, so it does not see a later response replayed from another file (a later explanation chunk, or a follow-up), or one model under two names that MODELS does not map. The harness has written ledgers since ea67093. The ledger is rewritten whole on each update (upsert_spend_ledger reads it, then replaces it), so two writers on one file can lose a record; a lost record only narrows what the check sees.
  • With the command alone in its process (the CLI's case), nothing else uses a repeat's namespace. A library caller that runs another evaluation in the same process while a repeat runs shares the repeat's namespace, and the repeat can replay that evaluation's answers; the README says the caller must not. The end-of-repeat check sees only what is still true when the repeat ends: a replaced cache or a changed or removed namespace, not a change undone before then. When it raises, the repeat's file may already hold rows from the other namespace, which is why the error says to use a new output directory. A repeat that is already raising keeps its own error and does not name its file.
  • The check reads the directory as it is when it runs, before the first request and after each repeat. A file another writer finishes while a check is reading the directory, or after the last check, is not checked; the README says to run nothing else into the output directory while the command runs.
  • A rebuilt gemini/ tool request finds no stored answer until Pin the response-cache key so a rebuilt gemini/ tool request replays #275, so a rerun asks again for those six cards. The README says a rerun asks the model for the answers the cache does not find.
  • A rerun into the same output directory resumes from the CSVs before it consults any cache. The batch runner keeps a household whose rows cover every requested output without an infrastructure error, even if an answer in them is missing; --single-output keeps every row without an infrastructure error from a file with the columns its loader requires (and none otherwise). The README now says this, and that a new, independent set needs a new output directory and a new cache directory.
  • Inside one repeat, a request identical to one that repeat already sent replays where LiteLLM computes the same key for it. A household's second repair round does this when the first repaired nothing (Pin the response-cache key so a rebuilt gemini/ tool request replays #275 describes it). So a first run of a repeat can record cache replays.
  • analyze --runs-dir still cannot tell a replay from a deterministic model, and it reads any CSV directory it is pointed at.

Out of scope: eval-no-tools-chunked and the supervisor (policybench run) have no repeats.

Invariants

For every base namespace (or none), repeat count and failing repeat:

  1. Each repeat that runs sees its own namespace, built on the base, and no two repeats share one.
  2. The cache's namespace is the base again after the run, whether a repeat failed or not.

For every provider path and answer contract on the board, through the test stub:

  1. Every key a repeat looks up or stores starts with its own namespace, for the household's first-round requests (its first request, or each explanation chunk) and for each follow-up request (a repair round, a larger completion budget, the explanation-only request). Across a run and a rerun on the same cache, no stub call and no provider response id comes back under a run id other than the one that first got it.
  2. The namespace scopes the cache and does not decide whether the cache can find an answer: a rerun's repeats replay exactly when a plain rebuilt request for that card does.

For any output directory:

  1. The command sends no request into a directory whose CSV files, as it reads them before the first request, hold one response twice (the same model and response id in two files, by their rows or their ledgers) or an answered row with no id; and after each repeat it reads them again and raises if they do, before any later repeat sends anything.

Tests

tests/test_repeated_runs_sample_fresh.py has 65 tests. Most run the repeated path through LiteLLM's real request path and replace only the wire; all of those but the no-cache control use the real disk cache. The autouse fixture blocks sockets while each test runs, gives every request one attempt, and puts back the process environment afterwards. It cannot reach LiteLLM's model-cost map load, which happens at import, before any fixture; the local runs below set LITELLM_LOCAL_MODEL_COST_MAP=True, and CI does not. Requests per repeat are counted by wrapping the runner; cache keys are recorded, where a test asks for them, by wrapping the disk cache.

  • Repeat 2 sends as many requests as repeat 1 when repeat 1 is cached, records no cache replay and shares no response with repeat 1, and every key each repeat looks up or stores starts with its namespace: one card per provider path and contract. This holds whether or not the cache can find the answer later, so it covers the gemini/ tool group on main.
  • The same for each request path beyond a single first request (explanation chunks, a repair round, a larger completion budget, the explanation-only request), with an Anthropic tool card and a stub that sends each household down that path: the test checks from the wire that the path ran in both repeats, requires two lookups and two stores under each repeat's namespace, and requires a rerun on a new cache object to send nothing and return the same answers.
  • The single-output runner: one request per output (so the batch runner did not run instead), both repeats using the disk cache, each under its namespace.
  • A rerun on a new cache object and into a new output directory, with one more repeat: repeats 1 and 2 send nothing and each replays its own answers with their original response ids, and repeat 3 alone asks. The test measures with two plain runs whether a rebuilt request replays for the card, and expects the rerun to replay exactly then. It lets a card not replay only if it is a gemini/ tool card and the harness passed LiteLLM no cache key, so once Pin the response-cache key so a rebuilt gemini/ tool request replays #275 merges it requires every card to replay, with no edit.
  • Across a run and a rerun, no two repeats share a stub call or a provider response id, for every provider path and contract.
  • The command as the CLI runs it.
  • A rerun into the same output directory sends nothing, and the line gives the kept rows' counts; after a fatal error that left a header-only file, or rows with infrastructure errors, the line names no kept rows; with one household kept and one asked again, the line's overall and kept counts differ as they should.
  • The directory check: two files written as the command used to write them (one namespace, the second replaying the first) are refused before any request; files another writer adds between two repeats, or while the last repeat runs, are refused when that repeat ends, before any later repeat sends anything; a file with no id column, a blank id on an answered row, a file it cannot read, or one response in two files (one of them not named as a repeat) is refused before any request; error rows, one id under two models, one id within one file, and empty and header-only files pass, and a file not named *.csv is not read even when its rows would be refused. With the ledgers, two files where the second replayed both of the first's explanation chunks in the other order, or the first's repair round, and two files naming one model differently, are refused; a ledger that cannot be read, or holds a line json cannot parse, adds nothing. A provider id spelled like a missing value (NA, null) stays an id through a run and a resume.
  • A repeat raises when the cache is replaced, its namespace changes or is removed, or a cache is turned on while it runs, puts the namespace back and lets a later repeat start; a repeat that fails keeps its own error. Following the error's advice works: a rerun into the same directory is refused before any request, and one into a new directory asks again. A repeat that cannot set its namespace leaves the lock free for the next.
  • The response counter: two request groups whose call ids collide are counted apart, and a group missing either count is reported.
  • The namespace's spelling, so that a rerun keeps finding earlier repeats. Without a cache, every repeat asks. An effort override keeps the repeat for the lookup, the store and the replay. The namespace comes back after a repeat raises, and two repeats cannot overlap.
  • A property test (Hypothesis) of invariants 1 and 2.

Twelve test functions send no request: the namespace's spelling, the check of directories the command cannot check (each case), the check's accepted rows, a ledger json cannot parse, the response counter, the overlap guard, a repeat that cannot enter, the three that check a repeat raises when the cache changes under it, and the two that check the namespace comes back.

Local runs at 9e7b11b2 with a clean tree, one pytest process at a time:

  • This module with the two repeated-run tests in tests/test_eval_no_tools.py: 67 passed. With tests/test_eval_no_tools.py, tests/test_call_provenance.py, tests/test_batch_eval.py and tests/test_chunked_eval.py in full (the resume reader changed): 418 passed.
  • The same 67 with the namespace switch off (a pytest plugin that gives every repeat the base namespace): 38 failed, 29 passed.
  • With the directory check off: 11 failed. With only its ledger reading off: 3 failed (the chunks, the repair round, the two names). With the end-of-repeat check off: 5 failed.
  • On a local merge with Pin the response-cache key so a rebuilt gemini/ tool request replays #275 at c708e707 (c4f1d7a8, not pushed): this module with Pin the response-cache key so a rebuilt gemini/ tool request replays #275's tests/test_response_cache_replay.py, 335 passed; this module with the namespace switch off, 41 failed.
  • Round 3's four new findings, written as tests and run against d8c00621 before the fix: all five cases failed (the removed-namespace test runs with a failing and a returning runner).

Checks

CI runs pytest -m "not slow" and the lint job.

🤖 Generated with Claude Code

The command turns the LiteLLM disk cache on, and every repeat builds the
same requests, so repeats 2..N got repeat 1's responses back from the
cache and the run reported no run-to-run variation. Each repeat now runs
under its own namespace (repeat=run_000, ...): it never replays another
repeat's answers, and a rerun on the same cache replays its own. The
command prints, after each repeat, how many responses came from the
provider and how many from the cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Nov 21, 2026 5:07pm UTC

Request Review

- Write the namespace switch as a small class, so this change adds no
  import line where the open cache fix (#275) adds one.
- Test the command as the CLI runs it (cache on, two repeats), and that
  across a run and a rerun no repeat returns another repeat's responses,
  for every provider path and contract.
- README and docstrings: a rerun replays the answers the cache finds
  under the repeat's namespace.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The test kept a hand-written list that left the gemini/ tool group out,
because on main a rebuilt gemini/ tool request never finds its stored
answer. It now measures that per card with two plain runs and asserts
that a rerun's repeats replay exactly when a plain rebuilt request does.
So the gemini/ tool group joins the strict replay check by itself once
the harness pins the cache key (#275), in either merge order. The same
test now pins that a repeat never replays an eval-no-tools run's answers.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… and README exact

From an in-session review of this pull request:

- A directory of run_*.csv files written before this change resumed as
  if its repeats were independent. The command now writes
  repeated_run.json beside the run files and refuses a directory that
  holds repeat files without it.
- The per-repeat line says when a repeat resumed from an existing file,
  whose stored rows its counts include, and counts request groups with
  no record apart from zero.
- README: a rerun asks the model only for answers the cache does not
  find (a rebuilt gemini/ tool request finds none until #275), and a
  new independent set needs a new output directory as well as a new
  cache directory.
- Tests: every key a repeat looks up or stores sits under its own
  namespace, for every provider path and contract, which fails for the
  gemini/ tool group too when the switch is removed; the namespace's
  spelling; a rerun on a new cache object; the resume line; the guard.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
MaxGhenis and others added 2 commits November 21, 2026 15:38
…view r1)

- repeated_run.json now lists the repeats this command started, each added
  before its runner writes. A run_*.csv the list does not name is refused
  before any request: a marker for the directory could not vouch for a file
  an older checkout added beside a recorded one.
- The per-repeat line counts request groups by the runner's own unit (model
  and household, plus output for --single-output), so colons in model names
  or household ids no longer merge groups, and a group missing either count
  is reported, not summed.
- The line names kept rows only when the runner's loader keeps some, and
  then gives their counts separately.
- Repeats refuse to overlap in one process (the namespace is process-wide).
- Docstrings and README say what the namespace guarantees and what it does
  not (requests with their own namespace or key; requests LiteLLM rewrites).
- Tests: repair, chunk, escalation and explanation-only requests ask again
  under the repeat's namespace; single-output path proven; effort override
  checked on lookup, store and replay; answers compared pairwise; resume
  line after a fatal error and after infrastructure rows; marker cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits November 21, 2026 16:04
…eview r2)

Replace repeated_run.json, which recorded which repeats this command
started, with a check of what the files hold. Every answered row records
the id of the provider response behind it, and a cache replay returns the
stored response with its id, so two CSV files in the directory holding the
same model and response id hold one answer twice, whatever wrote them.
The command runs the check over every CSV in the output directory (what
analyze --runs-dir reads) before its first request and again after each
repeat. A file with answered rows but no response id, or one it cannot
read, is refused too.

The record could be defeated by an older checkout replacing a listed
file's rows, by files written while the command paused after its one-time
scan, and by a stale write from a concurrent process; the check reads the
directory as it is each time.

Offline, with the namespace switch off on a merge with #275, all 48 cards
replay repeat 1 into repeat 2 and every replay carries repeat 1's response
id; with it on, no two repeats share an id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…space gives (review r2)

The namespace is an attribute of the process-wide cache, so another
evaluation in the same process while a repeat runs shares it, and an
enable_cache() call during the repeat swaps in a cache with the base
namespace. A repeat now raises RuntimeError when it ends to find
litellm.cache replaced or its namespace changed (unless it is already
raising), after putting the namespace back. The docstrings and README state
the one-evaluation-per-process precondition the CLI meets.

The docstring said a rerun "looks under the keys that repeat stored". It
looks up under the namespace that repeat stored under; it replays only
where LiteLLM computes the same key for the lookup as for the store, which
a gemini/ tool request misses until its key is pinned (#275). A follow-up
request is built from the answers before it, so a rerun rebuilds it only
where those answers lead to the same request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… r2 P2/P3)

- The follow-up-path tests require two lookups and two stores in each
  repeat, and a rerun on a new cache object that sends nothing and returns
  the same answers, so dropping the store on any follow-up request fails.
- The no-shared-answer test compares stub call numbers and provider
  response ids per repeat, so a chunked household sharing one call of
  several is caught.
- A partial resume (one household kept, one asked again) checks the line's
  overall and kept counts apart.
- The offline fixture restores the whole process environment, which
  LiteLLM can extend (TIKTOKEN_CACHE_DIR).
- README: with --single-output the loader keeps rows only from a file with
  model, scenario_id, variable and prediction columns, and an explanation
  column when explanations are on.
- Two helper docstrings now say what they check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The other-writer test now also covers a file written while the last repeat
runs, and pins exact request counts. The rerun test checks that a replay
keeps the original response id on every provider path. The accept test
leaves a CSV that would be refused under a name the check does not read.
README and docstrings say what the check sees (each request group's first
id) and when the cache finds a rebuilt first-round request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits November 21, 2026 17:34
…it (review r3)

- The directory check adds every response each CSV file's spend ledger
  records, replays included, under the model id the ledger records, to the
  rows' first response ids, which it now counts under the row's model name
  and its model id. It finds a file that replayed another's explanation
  chunks in another order, and one model under two names. A missing or
  unreadable ledger adds nothing (review r3: both-chunks and display-name
  regressions, P1 2).
- The check and the resume reader read a response id as written, so an id
  such as "NA" is not taken for a missing value (review r3, false refusal).
- A repeat whose cache lost its namespace attribute puts it back and keeps
  a failing runner's error; the cache-changed error names the repeat's file
  to delete before a rerun (review r3: exit masking, P1 5).
- Docstrings, README: repeats build the same first-round requests (P1 1);
  enable_cache() and assigning litellm.cache join the process precondition
  (P1 5); the check reads the directory as it is when it runs, and a file
  written while it reads or after the last check is not checked (P1 3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n a failed entry

From an in-session adversarial check of the round-3 answers (no P0/P1):
- The cache-changed error says to use a new output directory: deleting the
  named file left its spend ledger, so a rerun paid again and was refused.
- A repeat that cannot set its namespace releases the overlap lock.
- A ledger line json rejects with a plain ValueError (or a RecursionError)
  makes that ledger add nothing, as an unreadable one does.
- Docs: a file's responses are its rows' ids (not only answered rows') and
  its ledger's, for a file that holds rows; the ledger includes requests of
  rows a resume replaced or a deleted file held; the refusal says the files'
  requests got one answer twice; the README notes a failing repeat keeps its
  own error.
- Tests: a replayed repair round is refused through the ledger; the advice
  is followed to a clean run; the lock on a failed entry; a bad ledger line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Status after independent review round 4: waiting on Max (d3294). Not ready to merge.

Head: 9e7b11b2. Draft; CI and mergeability are unaffected.

Round 4 was the last scoped round, and it left P1 findings open. Under the PR rule there is no round 5, and no hub covers this repo, so the approach goes to Max as d3294. Nothing merges while a P1 is open.

What round 4 closed:

  • Round 3 findings 1, 2, 5, 6, 7 and 8: first-round wording, chunk-order and repair-round replays, one model under two names, literal NA/null ids, and the cleanup when the cache namespace goes missing.

What is still open:

  • Round 3 finding 3 (partly closed): a file written while the scan reads the directory "may be missed", not "is not checked".
  • Round 3 finding 4 (partly closed): the precondition must also rule out setting or removing litellm.cache.namespace directly while a repeat runs, and the restart advice should say "may keep".
  • New P1: a display name can match a different model's id from a ledger, so the check refuses independent files (a false refusal).
  • New P1: a lost ledger record can make a false match through MODELS.
  • New P1: a deleted file's old ledger records can refuse a new, independent file at the same path.
  • New P1: the PR text misstated how the check handles a ledger line JSON cannot parse (that line is skipped; the rest of the ledger is kept).
  • New P2: the refusal message's count of shared responses merges responses from different models.

The four new P1s are source traces, not runs. Every open finding is in the directory check or in the description's account of it; none is in the per-repeat cache namespace. The namespace fixes the replay itself: turning it off fails 38 tests. The description now lists the statements round 4 found wrong.

d3294 asks Max to choose one of three options:

  • Cut the check (recommended): land the per-repeat namespace alone, after one scoped review of the reduced diff.
  • Rows-only check: keep a best-effort check that reads only the CSV rows, with its misses documented.
  • Provenance rework: record which run wrote each file at run time, and check against that.

Review records (local to the maintainer's machine), ~/reviews/policybench-repeated-fresh-sample-2026-11/:

  • runs/review-r1/review.md
  • runs/review-r2/review.md
  • runs/review-r3/review.md
  • runs/review-r4/review.md

Follow-ups this pull request leaves are in #279. The paper sentence is a separate ruling, d3291.

This branch was successfully deployed

1 active deployment
Preview — 9e7b11b2 Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant