Repository navigation
Conversation
…rame._adopt with_weights, place and select now hand their result's tables to Frame._adopt instead of the copying constructor. Under pandas copy-on-write the result shares every unchanged column's memory with its source, in DataFrame objects of its own, so a pandas write to either frame never reaches the other; with copy-on-write off, _adopt still deep-copies. place keeps its column insert, on a shallow copy, and ignores pandas' fragmentation warning around it. The checkpoint loader takes its decoded tables without a copy, so reading a checkpoint holds one copy of the data instead of two. calibrate gains return_frame (default True). With return_frame=False the result carries frame=None, and the calibrated weights still pass the checks with_weights makes, now in Frame._checked_weight_change, which both paths call. The memory-sharing test in test_us_post_export_scoring.py now asserts that calibration leaves its input frame unchanged and unreferenced, and that the calibrated frame shares the input's column memory. The release tool's comment on del target_frame and the release build rule document describe why the tables are freed only once both references are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Pin a SHA-256 of each checkpoint's dataset contents (names, dtypes, shapes, bytes) for every operation, recorded with the copying code at 8afea44, so a change to an operation's output, not just to the table layout, fails the test. Add a source frame with named column axes. - Drop the loader test's writability check, which could not fail. - Assert that each result holds its own strata object. - The post-export memory test now follows a column to the array that owns its memory: the calibrated frame keeps it alive, a result built with return_frame=False does not. - refit_l0_selection refuses a selection without a frame with a clear ValueError instead of an AttributeError; L0RefitResult.frame is typed Frame | None. - _checked_weight_change and calibrate's docstrings now say that the frame constructor's table checks are not re-run without a frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- test_place_shares_every_table_it_does_not_change also checks the table the column moves out of; drop a strata assertion that held whatever the operations did. - CalibrationResult docs and the release tool's comment on del target_frame describe which frames share the target tables and that a frameless result carries no frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The sharing tests turn copy-on-write on under pandas 2.x, as the _adopt tests do; packages still allow pandas >= 2.3. - load_frame_checkpoint's docstring says that a pandas fixed-format HDF write of a loaded table can group its columns into different datasets than a write of the same table built in memory, with the same values. - _checked_weight_change's docstring states the skipped length check correctly; the release build rule document names the frames that share the target tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The copy-on-write-off test drops the partial-selection case, which copies rows whatever _adopt does, and also checks the strata. The test frame's strata are object dtype so that memory comparisons of them are meaningful; with_weights' sharing test checks them too. - select's docstring and comment say the selected tables escape a second copy only under pandas copy-on-write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
calibrate's result frame now shares the calibrated frame's tables instead of copying them, so us_annual_static_aging's module docstring, its comment on releasing the calibration result, and docs/us-local-annual-projection.md no longer count a calibration copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebasing onto the Frame._adopt branch's latest commit, which rewrote the same import block, dropped `import warnings`, so Frame.place raised NameError. Restore it and re-record the calibrate and simulate parity pins, whose implementation hash covers bundle.py's source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The calibrate.adam@1 and simulate kernels hash the source of frame/bundle.py, and calibrate's also that of calibrate/solve.py, both of which this branch changes. Re-recorded on top of the Frame._adopt branch's pins with tools/graph_parity_repin.py on arm64/darwin/py3.14: only the implementation hash and node key move; the tool checks that each kernel's direct output bytes did not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
calibrate's return_frame option changes solve.py, whose bytes the seed protocol hashes. Regenerated with tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py in the locked US environment. The US bundle itself does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frame operations now record a SHA-256 digest of each column when they start sharing it. The digest belongs to the array's memory, so a column is hashed once however many frames share it, and it is dropped when the memory is freed. Checkpoint and output writers compare the digests first and raise SharedColumnChangedError naming each changed table and column. A column replaced through pandas moves to new memory and is not compared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compare recorded columns before the Frame constructor and the SPM stage make deep copies, so a changed column cannot pass through them unseen. Hash category values and sparse positions, and leave out values hidden by a nullable column's mask, which pandas itself overwrites. Stop recording tables passed to replace_tables, which callers may keep editing, and correct what the error and docs claim about in-place pandas writes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compare shared columns at the start of uk_release_export_frame, whose copies would otherwise hide a change. Hash block-format sparse positions, and tighten the tests of nullable masks and of the constructor's error reporting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 19, 2026
Brings in 4ee5584 from claude/frame-adopt-replace-tables-b93d80. Both sides changed bundle.py, so the calibrate and simulate parity pins were regenerated with tools/graph_parity_repin.py under Python 3.14 rather than taken from either side. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
5 tasks done
anth-volk
changed the base branch from
claude/frame-adopt-replace-tables-b93d80
to
main
November 19, 2026 19:23
Three files conflicted. test_us_fiscal_refresh_builder.py keeps both sides' new tests. The US spec_sha256 pin in test_us_multispine_pool_tool.py and docs/evidence/spec-engine/us-f0-coverage.json were regenerated with tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py, because main's us/gates.json row and this branch's solve.py change both move them. The calibrate and simulate parity pins regenerate unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…indings reach microcosm.frame.bundle imports the shared-column check, so the module count in test_the_batterys_binding_identity_adds_only_the_leaf_and_its_imports grows by one. engine-free (3.13) and (3.14) failed on this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1682's in-place QRF leaf array in microcosm.fit.qrf and this branch's calibrate return_frame option in solve both move the US seed-protocol digests, so neither side's values were kept. Regenerated with tools/repin_us_spec_engine_contracts.py and tools/spec_engine_coverage.py in the US engine environment; both --check runs pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…K target kernels microcosm.frame.bundle now imports microcosm.frame.shared_columns, so the walk from the UK target kernels' hashed modules meets that edge. engine-free (3.13) failed on test_every_import_a_target_kernel_reaches_is_hashed_or_recorded. The check changes no column, so no target value comes from it; the edge is recorded beside bundle's other imports, as #931 did for receipt_counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oc write describe_uk_export shares the person columns with the frame uk_dataset_year_frame returns (with_weights now shares unchanged columns). Once that frame is freed, the test's .loc write lands in the recorded memory and the shared-column check refuses it before the descriptor comparison runs, so test_export_refuses_population_changed_after_descriptor failed on engine-free (3.14) and fails locally on 3.13 and 3.14. Frame tables are read-only; the test now builds the changed population as a new frame and still expects the descriptor refusal. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…k (red) microcosm.frame.bundle and microcosm.frame.materialize now import microcosm.frame.shared_columns, and engine-free (3.13) and (3.14) failed on test_every_import_a_compile_kernel_reaches_is_hashed_or_recorded. The record expects takeup.compare, takeup.gap and transport.create to hash the module beside the bundle, as they hash its schema and weights, and records the edges for targets.compile and transport.quantile_map@1 and @2, which leave the bundle's imports unhashed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… kernels target_kernels._MATRIX_MODULES (targets.problem, takeup.compare, takeup.gap) and transport.create's hash list name microcosm.frame.bundle with its schema and weights. The bundle now calls the shared-column check when it builds and copies frames, so these lists hash microcosm.frame.shared_columns too. Their hashes already change in this PR because bundle.py changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #1666 (
Frame._adopt,replace_tables,with_columns), which is merged; the diff shows only this PR's work.Summary
Frame.with_weights,Frame.place,Frame.selectandload_frame_checkpointno longer deep-copy every table. They build their result throughFrame._adopt.Frameoperations share the memory of unchanged columns with their source, and the loader keeps the columns it decoded without a second copy. Each frame still holds its own DataFrame objects, so a write made through pandas does not reach another frame. With copy-on-write off,_adoptdeep-copies as before.SharedColumnChangedError, naming each changed table and column, before a checkpoint or output is written.calibrate(..., return_frame=False)returns the weights and diagnostics withframe=None, after the same weight checks. The default is unchanged. The release tool does not use the option yet.The expected memory effect is that reading a checkpoint holds one frame instead of two. That follows from removing the copy. It has not been measured on a real build.
How sharing works
with_weights: shallow-copies every table and passes them to_adopt. Its entity, type, weight-kind and mass-policy checks moved unchanged intoFrame._checked_weight_change.place: inserts the column into a shallow copy of the destination table and drops it from a shallow copy of the owner table. The insert stays a column assignment becausepd.concat(axis=1)changes the dtype of the column labels.PerformanceWarningis suppressed for that one insert.select: passes the tables it already built to_adopt.load_frame_checkpoint: passes its decoded tables to_adopt.calibrate: withreturn_frame=False, calls_checked_weight_changeand skips building the frame.refit_l0_selectionraisesValueErrorif a selection that needs a frame has none.How the column check works
The code is in
microcosm/frame/shared_columns.py.Frame.check_shared_columnshashes the recorded arrays again. It runs inwrite_frame_checkpoint,write_nullable_us_h5, the target-frame checkpoint writer, the legacy US writer,put_frame_table, and before these deep copies: theFrameconstructor,engine_tables,uk_release_export_frame, the SPM role stage and the ACS local structure extraction.cumsum.Cost, from synthetic tables on an arm64 Mac, not a real build: SHA-256 ran at about 2.4 GB/s on one thread. Hashing needs no extra memory for NumPy numeric columns. In a review measurement on a 2.4 GB, 1,500-column frame,
engine_tablestook 3.15 s, against 0.58 s for its copies alone. Modal's CPUs were not measured.Pins
calibrateandsimulatekernel pins are regenerated because they hashbundle.pyandsolve.py.solve.py. This used the documented sequence intools/repin_us_spec_engine_contracts.py. The US bundle does not change.Behaviour to review
.loc,fillna(inplace=True)andupdatewrite into it directly, and the check reports them. Frame tables are documented as read-only, and a scan of production code found no such write, but a future one would stop the build.table.copy()or aselectthat drops rows, carries the changed values into new memory.replace_tablesorwith_columnsare recorded only once a later operation shares them. Neither method has a production caller._dataand_maskof nullable arrays, read-only, because pandas has no public access to them when values are missing. Both are recorded as reviewed intest_shared_buffer_writes.py.tools/build_us_multispine_pool.pyfeeds loaded frames to such a writer, on resume. H5 comparisons should go by variable name.warnings.catch_warningschanges process-wide filters for the duration of theplaceinsert. None of theplacecall sites runs in threads.PerformanceWarning.graph_evidence.result_summary_datareadsresult.frame, so it fails on areturn_frame=Falseresult. No caller passes one.Test plan
Run by pytest node ID. The full suite was not run locally.
return_frameand column-check tests, the checkpoint bit-equivalence tests, the post-export scoring tests, the raw-write scan and the spec-engine digest tests.tools/repin_us_spec_engine_contracts.py --checkandtools/spec_engine_coverage.py --checkpass.test_monolith_and_staged_builds_are_bit_exact_at_every_boundarywas not run. It needs US input files that are not available locally.🤖 Generated with Claude Code