Repository navigation
Compare a local corpus against a base you name, not the shared snapshot - #888
Merged
Merged
Conversation
snapshot.py --local SOURCE and local_agreement.py psc-individuals each compared against one file per corpus in the cache (snapshot.jsonl, agreement.jsonl), shared by every worktree and decided by whoever recorded last. On 2026-10-25 parallel sessions of #752's phase 2 read a sibling's re-record as their own moves; the silent case is a sibling's record from a branch holding the same change, which reads as no moves. --base PATH reads the base checkout's readings from a record kept by the digest of its nameparser/ files (snapshot.py's baselines/, and a new agreements/ folder for the score), recording one in a child process where none is kept, and never reads or writes the shared file. It prints the base's path, digest prefix and whether it was reused or recorded, and the head's digest, saying so when the two packages are byte-identical. It refuses a base that is the tree being read or holds no nameparser package; the child strips PYTHONPATH and refuses a package imported from anywhere but the base. --sample composes with it. Without --base both tools behave as before. BASELINES_KEPT goes from 4 to 10: four sessions with a base and a head each, plus one, at FEC's 41.7 MB a baseline about 420 MB for the largest corpus. The tests plant a change in a copied BASE tree, so a base read from the head's tree by mistake would show no move, and check a plain copy moves nothing, the shared files are left byte- and mtime-identical, a kept base is reused, and the refusals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of the --base commit. The agreement child wrote straight to the kept path through corpus_build.write_atomic's one `.part` name, so two sessions scoring the same base at once could rename each other's half written file into place, and a record cut on a line boundary passed read_record and was reused by every later session. It also published under the digest the parent took before the child started. The child now scores into a scratch folder of the run's own (snapshot.scratch, also used for the snapshot side's base), and the parent renames the record into place only after reading it back and taking the base's digest again. The record's header counts its entries and read_record refuses a record holding another number (a record written before the count passes). _prune sweeps a scratch folder a killed run left, once stale, and _reused reads a baseline another run pruned meanwhile as none kept rather than damaged. The PYTHONPATH test becomes the unit check it was: --tree's sys.path entry already outranks a decoy in the child, so no end-to-end run can show the strip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second review of d1ee456. Two of its tests passed against the pre-fix code: the changed-base test now asserts the kept path is empty when the after-check runs (it held the record before the fix), and the pruned-baseline test removes the file between the lookup and the open, where the old unlink raised. A header whose entry count is not the record's own, `null` included, is refused rather than raising while its message is formatted. Each of the four rows was run against 47bd62a and fails there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # AGENTS.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
snapshot.py --local SOURCEandlocal_agreement.py psc-individualscompared against one file per corpus in~/.cache/nameparser-tools/local/<source>/(snapshot.jsonl,agreement.jsonl). That file is shared by every worktree and decided by whoever recorded last. On 2026-10-25, parallel sessions of #752's phase 2 read a sibling's re-record as their own moves. The silent case is a sibling's record from a branch holding the same change, which reads as "no moves".--base PATH(both tools)nameparser/files: snapshot.py'sbaselines/, and a newagreements/folder for the score.PYTHONPATHstripped and refuses a package imported from anywhere but the base.nameparserpackage.--sample Ncomposes with it.Unchanged: without
--baseboth tools behave as before; only refusal texts now mention--basetoo.local_agreement.py --recordgains--out, which the child uses. The agreement record's header gains an entry count; a record without one still reads.BASELINES_KEPT4 → 10. That covers four sessions at once, each with a base and a head, plus one spare. FEC's baseline is the largest at 41.7 MB, so its cap is about 420 MB; across the eight corpora it is about 1.2 GB (measured 2026-10-25).Tests (small invented fixture corpora only)
Smoke runs on real corpora, base = a copy of this tree, counts only:
ukec-individualsrecorded, then reused; "matches the base", byte-identical noted, shared snapshot mtime unchanged.psc-individualsscored its base in about 3 minutes; "matches the base", shared record untouched.Docs: tools/differential/README.md "Local corpora" leads with
--base. In AGENTS.md, the Validating paragraph now says--basewhere it said "recorded first … with--tree", and the Gotchas cache paragraph says to use--basewhenever another session may be measuring, and how a kept reading must be published. mechanisms.md#NAME-DATA-JOBS does not describe recording, so it is unchanged. Dated decisions entries that mention--record --treeare history and stay as written.Checks:
uv run pytest(24,448 passed),-n0 -m serial,--python 3.15, mypy, ruff. Two review rounds; the second reviewed the first round's fix commit.🤖 Generated with Claude Code