Repository navigation
Give each repeat of eval-no-tools-repeated its own cache namespace, so repeats ask the models again - #276
Give each repeat of eval-no-tools-repeated its own cache namespace, so repeats ask the models again#276MaxGhenis wants to merge 12 commits into
Conversation
The command turns the LiteLLM disk cache on, and every repeat builds the same requests, so repeats 2..N got repeat 1's responses back from the cache and the run reported no run-to-run variation. Each repeat now runs under its own namespace (repeat=run_000, ...): it never replays another repeat's answers, and a rerun on the same cache replays its own. The command prints, after each repeat, how many responses came from the provider and how many from the cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
- Write the namespace switch as a small class, so this change adds no import line where the open cache fix (#275) adds one. - Test the command as the CLI runs it (cache on, two repeats), and that across a run and a rerun no repeat returns another repeat's responses, for every provider path and contract. - README and docstrings: a rerun replays the answers the cache finds under the repeat's namespace. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The test kept a hand-written list that left the gemini/ tool group out, because on main a rebuilt gemini/ tool request never finds its stored answer. It now measures that per card with two plain runs and asserts that a rerun's repeats replay exactly when a plain rebuilt request does. So the gemini/ tool group joins the strict replay check by itself once the harness pins the cache key (#275), in either merge order. The same test now pins that a repeat never replays an eval-no-tools run's answers. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… and README exact From an in-session review of this pull request: - A directory of run_*.csv files written before this change resumed as if its repeats were independent. The command now writes repeated_run.json beside the run files and refuses a directory that holds repeat files without it. - The per-repeat line says when a repeat resumed from an existing file, whose stored rows its counts include, and counts request groups with no record apart from zero. - README: a rerun asks the model only for answers the cache does not find (a rebuilt gemini/ tool request finds none until #275), and a new independent set needs a new output directory as well as a new cache directory. - Tests: every key a repeat looks up or stores sits under its own namespace, for every provider path and contract, which fails for the gemini/ tool group too when the switch is removed; the namespace's spelling; a rerun on a new cache object; the resume line; the guard. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…view r1) - repeated_run.json now lists the repeats this command started, each added before its runner writes. A run_*.csv the list does not name is refused before any request: a marker for the directory could not vouch for a file an older checkout added beside a recorded one. - The per-repeat line counts request groups by the runner's own unit (model and household, plus output for --single-output), so colons in model names or household ids no longer merge groups, and a group missing either count is reported, not summed. - The line names kept rows only when the runner's loader keeps some, and then gives their counts separately. - Repeats refuse to overlap in one process (the namespace is process-wide). - Docstrings and README say what the namespace guarantees and what it does not (requests with their own namespace or key; requests LiteLLM rewrites). - Tests: repair, chunk, escalation and explanation-only requests ask again under the repeat's namespace; single-output path proven; effort override checked on lookup, store and replay; answers compared pairwise; resume line after a fatal error and after infrastructure rows; marker cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eview r2) Replace repeated_run.json, which recorded which repeats this command started, with a check of what the files hold. Every answered row records the id of the provider response behind it, and a cache replay returns the stored response with its id, so two CSV files in the directory holding the same model and response id hold one answer twice, whatever wrote them. The command runs the check over every CSV in the output directory (what analyze --runs-dir reads) before its first request and again after each repeat. A file with answered rows but no response id, or one it cannot read, is refused too. The record could be defeated by an older checkout replacing a listed file's rows, by files written while the command paused after its one-time scan, and by a stale write from a concurrent process; the check reads the directory as it is each time. Offline, with the namespace switch off on a merge with #275, all 48 cards replay repeat 1 into repeat 2 and every replay carries repeat 1's response id; with it on, no two repeats share an id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…space gives (review r2) The namespace is an attribute of the process-wide cache, so another evaluation in the same process while a repeat runs shares it, and an enable_cache() call during the repeat swaps in a cache with the base namespace. A repeat now raises RuntimeError when it ends to find litellm.cache replaced or its namespace changed (unless it is already raising), after putting the namespace back. The docstrings and README state the one-evaluation-per-process precondition the CLI meets. The docstring said a rerun "looks under the keys that repeat stored". It looks up under the namespace that repeat stored under; it replays only where LiteLLM computes the same key for the lookup as for the store, which a gemini/ tool request misses until its key is pinned (#275). A follow-up request is built from the answers before it, so a rerun rebuilds it only where those answers lead to the same request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… r2 P2/P3) - The follow-up-path tests require two lookups and two stores in each repeat, and a rerun on a new cache object that sends nothing and returns the same answers, so dropping the store on any follow-up request fails. - The no-shared-answer test compares stub call numbers and provider response ids per repeat, so a chunked household sharing one call of several is caught. - A partial resume (one household kept, one asked again) checks the line's overall and kept counts apart. - The offline fixture restores the whole process environment, which LiteLLM can extend (TIKTOKEN_CACHE_DIR). - README: with --single-output the loader keeps rows only from a file with model, scenario_id, variable and prediction columns, and an explanation column when explanations are on. - Two helper docstrings now say what they check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The other-writer test now also covers a file written while the last repeat runs, and pins exact request counts. The rerun test checks that a replay keeps the original response id on every provider path. The accept test leaves a CSV that would be refused under a name the check does not read. README and docstrings say what the check sees (each request group's first id) and when the cache finds a rebuilt first-round request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it (review r3) - The directory check adds every response each CSV file's spend ledger records, replays included, under the model id the ledger records, to the rows' first response ids, which it now counts under the row's model name and its model id. It finds a file that replayed another's explanation chunks in another order, and one model under two names. A missing or unreadable ledger adds nothing (review r3: both-chunks and display-name regressions, P1 2). - The check and the resume reader read a response id as written, so an id such as "NA" is not taken for a missing value (review r3, false refusal). - A repeat whose cache lost its namespace attribute puts it back and keeps a failing runner's error; the cache-changed error names the repeat's file to delete before a rerun (review r3: exit masking, P1 5). - Docstrings, README: repeats build the same first-round requests (P1 1); enable_cache() and assigning litellm.cache join the process precondition (P1 5); the check reads the directory as it is when it runs, and a file written while it reads or after the last check is not checked (P1 3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n a failed entry From an in-session adversarial check of the round-3 answers (no P0/P1): - The cache-changed error says to use a new output directory: deleting the named file left its spend ledger, so a rerun paid again and was refused. - A repeat that cannot set its namespace releases the overlap lock. - A ledger line json rejects with a plain ValueError (or a RecursionError) makes that ledger add nothing, as an unreadable one does. - Docs: a file's responses are its rows' ids (not only answered rows') and its ledger's, for a file that holds rows; the ledger includes requests of rows a resume replaced or a deleted file held; the refusal says the files' requests got one answer twice; the README notes a failing repeat keeps its own error. - Tests: a replayed repair round is refused through the ledger; the advice is followed to a clean run; the lock on a failed entry; a bad ledger line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Status after independent review round 4: waiting on Max (d3294). Not ready to merge. Head: Round 4 was the last scoped round, and it left P1 findings open. Under the PR rule there is no round 5, and no hub covers this repo, so the approach goes to Max as d3294. Nothing merges while a P1 is open. What round 4 closed:
What is still open:
The four new P1s are source traces, not runs. Every open finding is in the directory check or in the description's account of it; none is in the per-repeat cache namespace. The namespace fixes the replay itself: turning it off fails 38 tests. The description now lists the statements round 4 found wrong. d3294 asks Max to choose one of three options:
Review records (local to the maintainer's machine),
Follow-ups this pull request leaves are in #279. The paper sentence is a separate ruling, d3291. |
Draft, waiting on Max (d3294). Independent review round 4 (the last scoped round) left P1 findings open, all in the directory check described below or in this description's account of it; none is in the per-repeat namespace. The PR rule allows no round 5, so d3294 asks whether to cut the check from this pull request (recommended), keep a rows-only best-effort check, or rework it to record provenance at run time. Nothing merges while a P1 is open. The paper wording below waits on a separate ruling (d3291) and is not part of this pull request. Follow-ups from round 2 that this pull request leaves: #279.
Round 4 found these statements below wrong or too strong at head 9e7b11b (source traces, not runs). They stand corrected here, and the code and README follow under whichever option d3294 picks:
MODELS, so it does not "only narrow" what the check sees. And a deleted file's old ledger records can refuse a new, independent file at the same path.litellm.cache.namespacedirectly while a repeat runs. After the end-of-repeat error, a rerun into the same directory may keep the file's rows.What was wrong
policybench eval-no-tools-repeatedruns the benchmark N times on the same households so thatpolicybench analyze --runs-dircan report run-to-run variation. The CLI turns the LiteLLM disk cache on for this command (policybench/cli.py, theenable_cacheblock), and every repeat builds the same first-round requests from the same inputs (a household's first request, or each explanation chunk). LiteLLM keys its disk cache on the request, so repeats 2..N got repeat 1's responses back from the cache for every request whose stored answer a rebuilt request finds: 42 of the 48 model cards onab896ddc(table below).analyze --runs-dirthen reports no run-to-run variation for those cards. I ranrun_stability_by_modelon three identical run files: every*_run_stdit returns is 0 to floating-point rounding (largest 8.7e-15), which the report prints as 0.000. Nothing in its output tells a replay from a deterministic model, becausepolicybench/analysis.pynever reads thelive_callsandcache_hit_callscolumns.Reading the code at each commit: the command has been in the cached set since its first commit (6cd9bce on
v2, 2026-03-27; 1f0f7cb on main, 2026-05-01, #5), and nothing has put per-repeat material in the request or the cache key since.run_idreaches only the output rows, the resume metadata and the spend ledger. I ran the replay itself onab896ddconly.Offline confirmation
I ran the harness's
run_repeated_no_tools_evaloffline, through LiteLLM's real request path and the harness's real disk cache (enable_cachein a temporary directory), with a stub in place ofhttpx.Client.sendand sockets blocked: every model card (48), two repeats, one household, two outputs. The stub answers each request with a value and a response id that name the call, so a replayed answer differs from a second answer.ab896ddc): repeat 2 got repeat 1's answers5fb50e32): repeat 2 asked againOn main with the cache off (a control, one card per row), repeat 2 sent as many requests as repeat 1 in all nine rows.
The six
gemini/tool cards ask again on main only because of a separate defect: LiteLLM rewrites their tool schema between its cache lookup and its store, so a rebuilt request never finds its answer. #275 repairs that. On a local merge of this branch (8da0ea99) with #275 atc708e707, repeat 2 got repeat 1's answers for all 48 cards with the namespace switch off, and every one of those replayed rows carried repeat 1'sprovider_response_id; with the switch on, repeat 2 asked again for all 48 and no repeat shared a response id. The directory check below rests on that.The namespace switch has not changed since
5fb50e32; the later commits add the directory check (which, since round 3, also reads each file's spend ledger), the end-of-repeat check and the process-wide lock that keeps two repeats from overlapping, and change the per-repeat line, the tests and the wording.What was published from this command
Nothing that I found.
run_NNN.csv, norun_model_summary.csvorrun_stability_by_model.csv, no report with a "Run stability" section, and no log of a provider run with the command's=== Starting run_0banner. (This pull request's own offline probe logs carry the banner; they record stub answers, not provider calls.)output/repeat_smoke/, two 2-row CSVs (modelgpt-5.4, householdss1ands2, five columns, no response, usage or timing) with their analysis files. It sat on main from 1f0f7cb (2026-05-01) to 0443a23 (2026-05-05). Its shape marks it as a hand-made fixture for the analysis path, not provider output (my reading; no commit message says so). Off main, a local salvage commit (6e8f1744, underrefs/subfleet-salvage/) holds pytest temporary directories with the existing repeated-run tests'run_000.csvandrun_001.csv; those are test fixtures too.dashboard-data-20261111) has norunCount,scoreRun*,within10pctRun*ormaeRun*key, and neither do 18 older payload files on this machine. I did not download release assets that have no local copy. 0ea540b (Review fixes: drop 0.3-floor metric, add paper bootstrap CIs, harden + clean up #58, 2026-06-04) removed the site's run-stability display: "these were dead anyway (no run data is shipped)".The paper's run-to-run sentence
paper/index.qmdline 1191 ends: "the public leaderboard and app report point estimates only, because the benchmark sample is the dominant source of uncertainty here, ahead of run-to-run model variation."I found no measurement behind it in the records I examined: the repository's history, the local results and the review records on this machine. A comparison kept nowhere I looked cannot be ruled out.
docs/argon/design.mdsays so for Gemini 4 Argon. No text compares two such runs' scores. Those pairs came later and say nothing about what the sentence rested on when it was written. What they show is in the decision record, because the first runs' scores are unpublished.docs/site_improvements_scope.mdlines 65-66 give the same claim as a reason to keep intervals off the site. The site's rendered paper carries the sentence inapp/public/paper/web/index.htmlandapp/public/paper/policybench.pdf, and the snapshot manifest andapp/src/paperSnapshot.jsonpin both files' hashes.Proposed wording (recommended):
"One set of answers per model" rests on the paper's own snapshot: its predictions file holds 103,168 rows for 52 models and 100 households, and no model, household and output appears twice. The bootstrap resamples households, each carrying the answers its model gave once, so its intervals mix household sampling with the answers' own variation and cannot tell the two apart.
Smallest alternative: delete everything after "point estimates only" and add nothing.
This pull request does not change the paper. The wording is Max's call, filed as d3291. A pull request that applies it also edits the line in
docs/site_improvements_scope.mdand re-renders the HTML and the PDF with their pins.The change
Each repeat runs under its own disk-cache namespace:
<namespace>:repeat=run_000,<namespace>:repeat=run_001, and so on (_OwnCacheNamespaceinpolicybench/eval_no_tools.py). The namespace goes back to what it was when the repeat ends, whether it returns or raises.Why a namespace:
namespaceattribute, which it reads on each call, for the lookup and for the store (Cache._add_namespace_to_cache_key, LiteLLM 1.95.0, the versionuv.lockpins). A request that names its own namespace (incacheormetadata["redis_namespace"]) or brings its owncache_keybypasses the attribute. The one per-request namespace the harness sets, for an effort override (_effort_cache_namespace), is built from the attribute when the request is built, so it keeps the repeat; a test pins this for the lookup, the store and a rerun's replay. Pin the response-cache key so a rebuilt gemini/ tool request replays #275's pinned key is computed by LiteLLM's own key function at call time, from the same attribute, so it keeps the repeat too. The two branches merge without conflict.gemini/tool request misses until Pin the response-cache key so a rebuilt gemini/ tool request replays #275 pins its key, in a repeat or out of one. A household's first-round requests (its first request, or each explanation chunk where the card splits explanations into chunks) are rebuilt from the same inputs every time. A follow-up request is built from the answers before it (a repair round from the outputs they left out, a larger completion budget after a cut-off answer, an explanation-only request from their values), so a rerun rebuilds the same follow-up only where those answers lead to it again.The namespace is an attribute of the process-wide cache, so the command must be the only evaluation in its process, which the CLI always is, and nothing may call
enable_cache()or assignlitellm.cachewhile a repeat runs. Under that condition repeat 1's own namespace also keeps it from looking up aneval-no-toolsrun's answers.Other changes to the command:
analyze --runs-dirreads), whoever wrote it, and raisesValueError(the CLI exits with the message) if two files hold the same model and response id. A response id is the provider's, or thechatcmpl-id LiteLLM generates where the provider gives none (as for Anthropic), and a cache replay returns the stored response with its id, so two files sharing one hold one answer twice. The responses of a file that holds rows are its rows'provider_response_id(the first response each request group got), counted under the row's model name and its model id, and every response its spend ledger (run_000.csv.spend.jsonl, written by the harness since ea67093, 2026-08-09) records for that path, replays included, under the model id the ledger records. The ledger adds the responses the rows do not show: later explanation chunks, follow-ups, and one model under two names. It also keeps the responses of rows a resume replaced or a deleted file held. A ledger that is missing, cannot be read, or holds a line json cannot parse adds nothing, and the check then sees what the rows show. It also refuses a file with answered rows but no response id (written by code from before c7bf2db, 2026-05-08, which added the column, or cut short by an interrupted write), which it cannot check, and a CSV file it cannot read. It reads only files named*.csvand their ledgers, and it reads a response id as written, so an id such asNAstays an id (the resume reader now does the same). This replacesrepeated_run.json, the record of which repeats the command started from round 1: round 2 showed a record of who wrote a file loses to an older checkout replacing a listed file's rows, to files written while the command paused after its one-time scan, and to a stale write from a concurrent process. The check reads the directory as it is when it runs.litellm.cachereplaced (by anotherenable_cache()in the process, say), or its namespace changed or removed, puts the namespace back and raisesRuntimeError, unless it is already raising. The error names the repeat's file and says to use a new output directory, since a rerun into the same one would keep that file's rows (and the file's spend ledger keeps the responses its requests got, so deleting the file alone does not clear the directory). Two repeats cannot run at once in one process: the second raisesRuntimeErroron entry, and a repeat that cannot set its namespace releases the lock before it raises.live_callsandcache_hit_calls, counted once per request group: a model and household, and also an output with--single-output), the request groups that lack either count if there are any, and, when the runner kept rows from an existing file, those rows' own counts. It decides what was kept with the runner's own loader, so a file the runner keeps nothing from (a header only after a fatal error, or rows with infrastructure errors) is not reported as kept.Known and unchanged:
MODELSdoes not map. The harness has written ledgers since ea67093. The ledger is rewritten whole on each update (upsert_spend_ledgerreads it, then replaces it), so two writers on one file can lose a record; a lost record only narrows what the check sees.gemini/tool request finds no stored answer until Pin the response-cache key so a rebuilt gemini/ tool request replays #275, so a rerun asks again for those six cards. The README says a rerun asks the model for the answers the cache does not find.--single-outputkeeps every row without an infrastructure error from a file with the columns its loader requires (and none otherwise). The README now says this, and that a new, independent set needs a new output directory and a new cache directory.analyze --runs-dirstill cannot tell a replay from a deterministic model, and it reads any CSV directory it is pointed at.Out of scope:
eval-no-tools-chunkedand the supervisor (policybench run) have no repeats.Invariants
For every base namespace (or none), repeat count and failing repeat:
For every provider path and answer contract on the board, through the test stub:
For any output directory:
Tests
tests/test_repeated_runs_sample_fresh.pyhas 65 tests. Most run the repeated path through LiteLLM's real request path and replace only the wire; all of those but the no-cache control use the real disk cache. The autouse fixture blocks sockets while each test runs, gives every request one attempt, and puts back the process environment afterwards. It cannot reach LiteLLM's model-cost map load, which happens at import, before any fixture; the local runs below setLITELLM_LOCAL_MODEL_COST_MAP=True, and CI does not. Requests per repeat are counted by wrapping the runner; cache keys are recorded, where a test asks for them, by wrapping the disk cache.gemini/tool group on main.gemini/tool card and the harness passed LiteLLM no cache key, so once Pin the response-cache key so a rebuilt gemini/ tool request replays #275 merges it requires every card to replay, with no edit.*.csvis not read even when its rows would be refused. With the ledgers, two files where the second replayed both of the first's explanation chunks in the other order, or the first's repair round, and two files naming one model differently, are refused; a ledger that cannot be read, or holds a line json cannot parse, adds nothing. A provider id spelled like a missing value (NA,null) stays an id through a run and a resume.Twelve test functions send no request: the namespace's spelling, the check of directories the command cannot check (each case), the check's accepted rows, a ledger json cannot parse, the response counter, the overlap guard, a repeat that cannot enter, the three that check a repeat raises when the cache changes under it, and the two that check the namespace comes back.
Local runs at
9e7b11b2with a clean tree, one pytest process at a time:tests/test_eval_no_tools.py: 67 passed. Withtests/test_eval_no_tools.py,tests/test_call_provenance.py,tests/test_batch_eval.pyandtests/test_chunked_eval.pyin full (the resume reader changed): 418 passed.c708e707(c4f1d7a8, not pushed): this module with Pin the response-cache key so a rebuilt gemini/ tool request replays #275'stests/test_response_cache_replay.py, 335 passed; this module with the namespace switch off, 41 failed.d8c00621before the fix: all five cases failed (the removed-namespace test runs with a failing and a returning runner).Checks
CI runs
pytest -m "not slow"and the lint job.🤖 Generated with Claude Code