Repository navigation
Frame: share unchanged tables under copy-on-write, and refuse raw writes into shared buffers - #1666
Conversation
…r write scan Frame._adopt stores the tables it is given without copying them when pandas copy-on-write is on, and deep-copies them when it is off (pandas 2.x defaults), with the constructor's full validation. replace_tables builds a frame from shallow copies of every table plus the replacements, and with_columns assigns columns on a shallow copy of one entity's table. The default constructor still deep-copies every table; its body moved into _assemble unchanged. Nothing calls the new methods yet. test_shared_buffer_writes.py reads the source of every workspace package and tools/ and refuses writes that copy-on-write does not protect: making an array writable, writes through .array, pandas' private buffer attributes, np.copyto, and out=, ufunc.at and NumPy's in-place functions into arrays the function did not create. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frame._adopt now raises TypeError for a table that is not a DataFrame. On pandas 2.x, the copy-on-write check also requires the block data manager, as pandas' own check does. The _adopt docstring lists the extension types whose to_numpy() and .values are writable shared views, and says the check is made when a frame is built. The frame tests use object-dtype strata, so the strata memory checks can fail, and turn copy-on-write on when they run under pandas 2.x. The scan now also refuses: - augmented assignment to a name bound to a writable view; - writes through .base and .asi8, astype(copy=False) and fillna(copy=False); - np.put_along_axis, np.nan_to_num(copy=False), .shuffle methods and Generator methods given out=; - NumPy functions reached through a local alias. It follows views through containers, unpacking, loop and with targets, `or` and NumPy view functions. It treats an item of a container or of a .copy() as not new, and treats comprehension targets and decorators in their own scopes. np.diff(x, 0) returns its input, so diff left the list of allocating functions. The per-statement text filter now also matches `flags` and dots split by a backslash continuation. Reviewed exemptions are keyed on a digest of the enclosing function's source, so any change to that function needs a new review. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frame: _pandas_copy_on_write now asks pandas 2.x's own pandas._config.using_copy_on_write(), which emits no deprecation warnings, instead of changing process-wide warning filters around option reads. When that function is missing, as on pandas 3, a version of 3 or later counts as on (an undotted version string no longer raises), and any other version reads mode.copy_on_write. with_columns drops a deep copy that _adopt already made, and the constructor test now checks the strata copy. Scan: - Newness and views are computed as per-scope fixed points, stored on each scope, so long chains of names no longer take exponential time and tables are never reused across functions. - Views are tracked by container depth: an item of a container holding a view is a view, but writing a column into a frame that holds one is not a write into the view. - Views are followed through zip, enumerate, .items(), .values(), .get(), containers filled by append, extend or name[key] = ..., joined lists, np.split and the other functions returning views. - `+` and `*` give a new container whose items may be shared. - np.clip, np.tile and np.round can return a pandas object's memory, so they no longer count as allocating. - SciPy ufuncs count like NumPy's. - Random Generator draws, .flatten(), ufunc reductions, views of new arrays and to_numpy(dtype, True) count as new. - Exemption keys hold the full expression. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frame: fix two docstring claims (the order of _adopt's checks, which weight columns with_columns refuses). Under pandas 2.x the copy-on-write-off tests now turn copy-on-write off for real, and the check is tested with a version string that does not start with a number. Scan: - An item of a view is now an element, which may be a scalar. Writes into views and elements are refused, but augmented assignment only into a view itself, so accumulating scalars taken from a view is accepted. - `+=` adds a value's items to a container, so a running total of view columns no longer counts as a view. - Comprehension elements see their own loop targets, and later generators see earlier ones. - for-targets over zip, enumerate and .items() are paired with their own sources. - Displays and dicts unpacked with * and ** pass on their items. - dict | joins containers; * on an array is arithmetic. - .get(key, default) includes its default. - Unpacking into subscripts counts items. - Views, subscripts and items of new arrays count as new. - np.take is treated as a possible view of a pandas object. - .base counts only on array expressions. - copy=True by keyword or position stops a view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frame: the _adopt docstring now lists which extension arrays .values and to_numpy() share (category columns' to_numpy() is a copy; pyarrow columns share under pandas 2.x). replace_tables and with_columns point to it for the writes copy-on-write does not protect. The tests decide "pandas 3" by the missing pandas._config.using_copy_on_write, as bundle.py does. Scan: - zip pairing nests inside enumerate and reaches through list, sorted and the other item-keeping wrappers. - Comprehension loop targets over a new array, and np.split pieces of one, count as created; comprehension targets are cached. - `+=` on a name bound only to .array is accepted: an extension array builds a new array on `+=`. - Joins of any length are handled without recursion. - dict(...) is followed. - np.clip, np.round and np.take no longer count as views of an ndarray or extension array, and still do not count as created. - The docstring now says the scan follows only the forms it lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frame: the docstrings now say which extension-typed columns share memory through .values and to_numpy(), with examples rather than a list, and the test docstrings describe how the deep-copy path is reached. Scan: - Filling a NumPy array the function created by key copies, so it no longer counts as holding a view. - `*=` on items of a container built only from .array is accepted. - dict.update with a mapping, pairs or keywords is followed. - A positional False copy for nan_to_num and astype is refused. - .base counts only on .values or a library or array-method call. - Literals count as created; assignment expressions are followed. - Finding text no longer re-splits the source for each finding. - itemset (gone in NumPy 2) is dropped. - The four binding-kind self-tests now shadow a new array, so they fail if the binding is ignored. - Every fill method has a self-test. - The comprehension-cache test runs under a 30-second alarm. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frame: correct the docstrings on python-backed strings (their to_numpy() shares memory even with missing values) and on the invariants with_columns can break. Scan: - In-place methods count only on a view itself, so string methods such as label.partition on scalars taken from a view are accepted. - A slice of a view is a view, so `+=` on one is refused. - A call given a created output array returns a created array. - .copy() of an array expression is a new array. - NumPy calls such as np.append are no longer taken for container fills. - insert and setdefault put in only their value. - dict(**mapping) and update(**mapping) pass on the mapping's values. - A positional copy counts only as the constant False. - A module that fails to import no longer stops the scan. - The time-limited test now also covers the newness cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Frame: the _adopt docstring frames its list of unprotected writes as examples and adds .base and .asi8; replace_tables says it can add a declared link table; with_columns says pandas raises TypeError or ValueError for values it cannot assign. Scan: - A container fill is skipped as a library call only when the name is bound to a library object where it is filled. Library alias names come only from names bound directly, so `x[mask] = np.nan` no longer hides later fills of `x`. - Multi-dimensional slices of new containers count as new. - __setitem__ writes like item assignment again. - .copy() of a split list is a list copy. - The extension-array `+=` exemption follows names and slices. - `arrays[1:] += more` only rebinds list items. - New self-tests cover insert and a module that fails to import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven review rounds showed that following arrays through names, containers, comprehensions and loops has no natural end: each fix added new edge cases. The scan is now a check on how code is written. It looks at one expression at a time, and only for out= and ufunc.at follows a name to its own assignments in the same function. It refuses: - setflags(write=...) and flags.writeable / flags["W"] assignments; - item assignment into x.array, and x.array.__setitem__; - ._ndarray, ._mgr, and ._values, ._data, ._mask and ._codes on other objects; - np.copyto; - out= given by keyword to NumPy or an ndarray method, and ufunc.at's first argument, unless the array is created in the function. Its docstring lists what it does not see. Raw writes into extension-typed columns' .values and to_numpy() are among them, and need a run-time check. Frame: three docstring corrections (what the scan refuses, what the constructor does with a table without copy(), which errors with_columns raises). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- np.array(copy=True) and to_numpy(copy=True), the copies the failure message suggests, count as created, and so do arithmetic, comparisons and random Generator draws. - A .copy() counts as created only whole or sliced: an item of a dict or list copy is a shared item. - Annotations without a value are not writes. - Files are read as utf-8-sig, so a byte-order mark cannot stop the scan. - NumPy imports are found by walking statements only. - New self-tests cover a parameter rebound on one path, a lambda parameter, a name only ever None, nonlocal and import bindings. - A new test fails if a function the reviewed exemption's reason rests on changes. Frame: the _adopt docstring says which writes through .values reach the other frame, and names the folders the check scans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Their implementation hashes include the source of microcosm/frame/bundle.py, which this branch changes. Re-pinned with tools/graph_parity_repin.py under Python 3.14 (the pinned arm64/darwin/py3.14 platform): only the implementation hash and node keys moved, and the pinned direct.csv bytes are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- np.copy(x) counts as created; the .copy() method rule no longer hides it. - A name assigned a subscript of a created array (np.sort(x)[::-1]) counts as created. - +, * and | count like .copy(): whole or sliced they are new, but an item is not, since joining lists or dicts shares their items. - The reviewed-helper test finds functions by qualified name and refuses a name defined more than once. - A new test checks that every output in out=(a, b) is reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Unpacking into items of an array (`a[0], b = ...`) no longer counts as rebinding `a`; only the names a target binds are recorded. - `.copy()` of `.index` or `.columns` is not counted as a new array, because `Index.copy()` is shallow by default. - `choice` is no longer counted as a random draw that creates an array, because the standard library's `random.choice` returns an existing object. - New snippets cover non-constant values assigned to the writeable flag, multi-dimensional slices of a copy, and the cases above. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A slice of an index shares the index's memory, and `Index.copy()` is shallow, so `table.index[1:].copy()` is not a new array. The check now strips subscripts before testing whether `.copy()` is called on `.index` or `.columns`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@anth-volk, could you take #1679 once this PR is in? It is the next step after this one, and it is not assigned to anyone. The graph executor's 🤖 Generated with Claude Code |
juaristi22
left a comment
There was a problem hiding this comment.
Reviewed at a467f35b5. No blocking issues: the new methods behave as described, and nothing calls them yet, so no output changes. Two fixes before merge, and two decisions for the next task.
Before merge
- Add a changelog fragment, for example
changelog.d/frame-adopt-replace-tables.added.md. Each of the last 25 PRs merged to main adds one, and this PR addsreplace_tablesandwith_columns. - Reflow the
_adoptdocstring atbundle.py:173: the line is 107 characters, and the repo's line length is 88.
Before the next task
- Decide how
_adoptprotects extension-typed columns before any caller moves onto it. Under pandas 3.0.3, on a frame built withreplace_tables(), a write into the array that.valuesor.to_numpy()returns reaches the source frame forInt64,booleanand python-backed string columns; forcategory, a write through.valuesdoes. The source check does not see these writes. One option:_adoptdeep-copies extension-typed columns only, so NumPy-typed columns keep sharing memory. A hash check would detect such a write but not prevent it. - Decide whether callers outside microcosm-frame use the private
Frame._adoptor a public no-copy constructor. The next two callers are in other packages: the checkpoint loader (frame_checkpoint.pyin microcosm-build,Frame(...)at lines 302 and 395) and the graph executor for #1679.
Checked
- The test plan reproduces under pandas 3.0.3, NumPy 2.4.6 and Python 3.14: 76 sharing, 99 scan and 11 parity-pin tests pass, as do the 63 in
test_bundle.pyandtest_place.py. The scan, pins and sharing tests also pass on a local merge with main at402c3587d, 89 commits ahead, so main's new code adds no scan findings and the re-pinned hashes hold. - Under pandas 2.3.3, the 76 sharing tests pass with FutureWarning and DeprecationWarning as errors.
_pandas_copy_on_write()reads the default and"warn"as off. Awith_columnsthat replaces a column leaves the source frame unchanged with copy-on-write off, warn or on. .locand.ilocwrites on either frame never reach the other, for float,Int64,boolean,category, string and object columns. The reviewed exemption holds: its array comes fromnp.asarray(...).copy().
Review fixes from juaristi22 on PR 1666: - Add changelog.d/frame-adopt-replace-tables.added.md for Frame.replace_tables, Frame.with_columns and the source check. - Reflow the _adopt docstring line that was 107 characters to the repository's 88-character limit. The text is unchanged. The docstring edit changes the bytes of microcosm/frame/bundle.py, which the calibrate and simulate graph parity cases hash, so both are re-pinned with tools/graph_parity_repin.py under Python 3.14 (arm64/darwin/py3.14, their only pinned platform). Only the implementation hash and node keys moved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@juaristi22 thanks for the review. Both fixes before merge are in 4ee5584:
Editing Points 3 and 4, how 🤖 Generated with Claude Code |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five pin files conflicted, as Max listed: #1682's in-place QRF leaf array and this branch's engine refresh each moved the same pins. Each takes this branch's side, and the pins both sides moved are recomputed on the merged tree: - tools/repin_us_spec_engine_contracts.py (twice, so the inventory count is read with the new seed hashes in place) rewrote the seed map and seed protocol digests, the spec-engine loader's golden vector and the pool tool's spec_sha256; tools/spec_engine_coverage.py regenerated the F0 coverage report (50,842 of 50,842 fields, 41 of 41 inventory checks), and both --check runs pass. - APPROVED_UV_LOCK_SHA256 is the merged uv.lock's digest. The lock merged without a conflict (#1682 pins quantile-forest==1.4.2), and uv lock --check resolves it unchanged. The US bundle generator needs policyengine-us, which the engine-free environment lacks; #1666 and #1682 change none of its inputs, so the bundle stands, and CI's engine lane checks it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Frameconstruction deep-copies every table. This PR adds a way to build frames that share the memory of unchanged columns under pandas copy-on-write.Frame(packages/microcosm-frame/src/microcosm/frame/bundle.py)Frame._adopt(tables, schema, weights, strata=None, *, mass_log=(), metadata=None), a classmethod:pandas>=2.3): deep-copies the tables and strata exactly as the constructor does.TypeErrorfor a table that is not a DataFrame.replace_tables(self, /, **changed_tables):_adopt;with_columns(self, entity, /, **columns): assigns each column withtable[name] = valueson a shallow copy of one entity's table, then goes throughreplace_tables._pandas_copy_on_write():pandas._config.using_copy_on_write(), which also requires the block data manager and emits no deprecation warnings.mode.copy_on_write._assemble(..., copy_data=True). Its docstring promise, "the caller's frames are never mutated or aliased", still holds._adoptyet:with_weights,place,select,concatand the checkpoint loader. That is the next task.The source check (
packages/microcosm-frame/tests/engine_free/shared/test_shared_buffer_writes.py)Scope: every Python file under
packages/*/src/, which is the source of all seven workspace packages, including the US and UK frame-building code in microcosm-build, and undertools/. Files innode_modulesor hidden directories are skipped. Tests andtest_support/are not scanned.The check reads the source with
astand refuses the direct forms of five writes. Each was checked with pandas 3.0.3 and NumPy 2.4.6 to change a frame that shares the memory, without any error:makes-array-writable:setflags(write=...)with anything but a false constant, or assigning toflags.writeableorflags["W"].writes-through-series-array: item assignment (plain or augmented) intox.array[...], orx.array.__setitem__.pandas-private-attribute:._ndarrayor._mgron any object;._values,._data,._maskor._codeson any object other thanselforcls.numpy-copyto: any use ofnp.copyto.writes-into-existing-array:out=given by keyword to NumPy or an ndarray method, and the first argument ofnp.<ufunc>.at(which writes even into a read-only array), unless the array is created in the function. Created means:np.arraywithoutcopy=or withcopy=True,.to_numpy(copy=True), or a randomGeneratordraw;.copy()with no arguments (not of.index,.columnsor a slice of them, sinceIndex.copy()is shallow and an index slice shares the index's memory),+,*or|, whole or sliced (not an item of them, because a copy or join of a dict or list shares its items);None).How it works: the check looks at one expression at a time. Only for the last rule does it follow a name, and then only to that name's own assignments in the same function.
Its docstring lists what it does not see. That covers:
np.asarray(x.array),.baseor.asi8;outand other in-place functions;copy=Falseconstructors andnp.asarray(index);.valuesandto_numpy()return for extension-typed columns. For example,Int64,boolean,Float64,categoryand python-backed string columns can return the column's own memory. US frames use several of these dtypes. Source reading cannot tell them apart from NumPy-typed columns, so protecting them needs a check at run time.Today's findings: one reviewed exemption, keyed on a digest of the enclosing function, so any change to that function needs a new review. In
tools/verify_uk_identity_stability.py,np.add.atwrites into an array that a helper function creates with.copy(). A separate test records digests of the two helper functions that reason rests on, and fails if either one changes.Design history: earlier commits on this branch followed arrays through names, containers, comprehensions and loops. Seven adversarial review rounds kept finding new edge cases in that design, so it was replaced by this narrower check.
Graph parity pins
calibrateandsimulateparity cases hash the source ofbundle.pyinto their implementation hashes.tools/graph_parity_repin.pyunder Python 3.14, the pinnedarm64/darwin/py3.14platform.direct.csvoutput bytes are unchanged, and the tool refuses to write if they would change.Findings for the follow-up work
np.<ufunc>.atignores the read-only flag.np.add.at(frame.table(e)[col].to_numpy(), idx, 1)would change every frame sharing that column.to_numpy()and.valuescan return writable arrays that share the column's memory, as described above. How to protect them is open. Two options are a run-time debug check that hashes columns at_adoptand again before outputs are written, or having_adoptdeep-copy only extension-typed columns.with_columnswill fragment the table and can raisePerformanceWarning.Test plan
test_frame_table_sharing.py, each test selected by node ID: 76 tests pass under the locked pandas 3.0.3, and under pandas 2.3.3 with-W error::FutureWarning -W error::DeprecationWarning.test_shared_buffer_writes.py, each test selected by node ID: all pass. That covers the refused and accepted snippets, the workspace scan, and the tests for reviewed exemptions, coverage and skipped directories.test_graph_parity_pins.py::test_the_pinned_implementation_hash_is_the_registered_kernel_sand::test_every_pinned_platform_key_is_the_one_that_platform_would_compute, under Python 3.13 and 3.14.python tools/ci_test_plan.py verifyandruff check.Framemethods, and four rounds on the final form of the source check. The fourth reviewed only the third round's fixes and found one case (index[1:].copy()), fixed as it proposed.🤖 Generated with Claude Code