Skip to content

Resolve parameter nodes at instant on first read - #570

Open
MaxGhenis wants to merge 15 commits into
masterfrom
lazy-parameter-node-at-instant
Open

MaxGhenis wants to merge 15 commits into
masterfrom
lazy-parameter-node-at-instant

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Taken over by Anthony Volk (anth-volk) on 2026-11-17 as part of the work on report runtimes (see the comment above). Max's 4 commits (bcf2094, 98bb55e, c58dede, 31469a4) are unchanged; current master (175e7ee) is merged in and the fixes below are added on top. His original measurements are kept under "Max's measurements".

Problem

ParameterNodeAtInstant.__init__ built every descendant up front. The first time a tax-benefit system was asked for its parameters at a date, it built that date's whole tree of nodes at instant, and ParameterNode._at_instant_cache (plus TaxBenefitSystem._parameters_at_instant_cache) kept it for the system's lifetime. Every reform system, whose caches start empty, paid again.

Change

Max's commits:

  • A node at instant resolves a child the first time it's read (__getattr__, __getitem__) and keeps it. It resolves all of its direct children only when something needs all of them: _children, iteration, repr, dir, vectorial indexing. A child read as an attribute becomes a plain instance attribute, so later reads cost what they did.
  • Which children a node has, and in what order, is fixed when it's built. A child resolves once, under one module-level reentrant lock that later reads don't take. Children named like the node's own attributes shadow them as before, and pickles of up-front nodes still load.

Added on top, each with failing tests first:

  • ParameterNode.add_child clears the cached nodes at instant of the node and of every node above it, which were built without the new child. It walks up the parent links just before Link scale brackets and homogenized nodes to their parents #660's _link_child links the child. The walk doesn't recurse, stops at a node it has already seen, and skips a node with no cache of its own and a parent that isn't a parameter node, so add_child still accepts every tree it accepted before. clear_parent_cache would raise on each of those, so it isn't used here. Since Link scale brackets and homogenized nodes to their parents #660, a child named like one of the node's own attributes (policyengine-us has gov.hhs.medicaid.eligibility.categories.parent) is not set as an attribute, so it can't replace the node's parent or its cache while the node loads.
  • Parent links for scale brackets and homogenized nodes come from Link scale brackets and homogenized nodes to their parents #660, which this branch has merged. Without them, Parameter.update below a bracket or a homogenized node (policyengine-us has 9,811) cleared caches only up to it. On master that left stale values; with lazy nodes, whether a later read saw the update would depend on what had been read before.
  • Copies and pickles are snapshots. A deep copy or pickle of a node at instant resolves every child first. It holds the values the up-front build held and no parameter nodes. Before, a partly read node carried its unread parameter nodes, and through their parents the whole tree: a policyengine-us CTC node pickled to 128 MB, against 2.4 kB on master. A copied cached node keeps the values it was copied with, as on master.
  • Copying and pickling while other threads read. Reads now add to a node's __dict__, its lazy state and the _at_instant_cache of the parameter nodes below it. Copying or pickling those dictionaries directly raised RuntimeError: dictionary changed size during iteration if a child was resolved meanwhile.
    • A node at instant's copy state is a copy of its __dict__.
    • A parameter node's cache is an _AtInstantCache: a dict whose copy or pickle starts from a one-step copy of its items and still keeps it one object for nodes that share it.
    • _get_at_instant replaces a plain dict cache set from outside core (policyengine-us's copy_parameter_nodes sets one) before adding to it.
    • The lazy state now pickles under protocols 0 and 1. That fixes the 8 tests from Let vectorial parameter nodes be copied and pickled #584 that failed in every Test job of this PR's earlier CI run.
  • The up-front build where callers relied on it.
    • A node at an instant that isn't a date, or a date that doesn't exist (parameters(None) reads "None"; "2015-02-29"), is built up front, so it fails, or succeeds, as before. Simulation._calculate reads parameters(period).gov.abolitions[...] with period None and ignores errors; a lazy node there would have read the latest values.
    • Any subclass of ParameterNodeAtInstant gets the up-front build, so its add_child, __getattr__ and __getitem__ see every child as before.
    • A node's children are the ones node.children.items() gives, as the up-front build iterated them.
  • Forked processes get a new resolution lock (os.register_at_fork). A process forked while another thread held the lock could never resolve a child.

Same values as before

  • For any instant and any order of reads, every read returns what the up-front build held, and a node read in full equals it, child order included. Max's Hypothesis property test checks this against a reference that walks the ParameterNode tree directly.
  • Nodes requested after Parameter.update, add_child or a cache reset read the change, as before. With Link scale brackets and homogenized nodes to their parents #660's parent links, that includes changes below scale brackets and homogenized nodes.
  • Intended differences (in the class docstring):
    • A node at instant held across an in-place change to its parameters reads the new value for a child it hasn't read yet; the up-front node held every value from construction. In Core, the long-lived holder is TaxBenefitSystem._parameters_at_instant_cache, which nothing clears after Parameter.update or Simulation.apply_reform (Define how Core's parameter caches are owned, invalidated, copied and shared #621, problem 1). Formulas read through the root's _at_instant_cache, which those clear. Removing this would take either the up-front build or Define how Core's parameter caches are owned, invalidated, copied and shared #621's rule that trees don't change after construction. The changed changelog entry says so, for callers of get_parameters_at_instant such as policyengine-household-api and policyengine-us's spm.py.
    • A child that fails to build at a date (policyengine-uk's NI scale refuses some reforms) fails when it's read, not when the node is built, so the node's other children can still be read. A deep copy or pickle of a tree that holds such a node builds it and fails then.
    • A pickle of a node at instant holds this version's lazy state, which earlier versions of core can't load. A pickled cache is a plain dict.

policyengine-us's spm.py

policyengine_us/spm.py (main e283430b2e) handles these caches directly:

  • it gives a traced request a shallow copy of the root with its own _at_instant_cache;
  • it shares one system's _parameters_at_instant_cache between policies;
  • it path-copies parameter nodes with copy_parameter_nodes;
  • it resets both caches when it detaches a tree.

All of this works unchanged. A lazy node resolves children through the shared child nodes' caches, so it holds the same objects the up-front build did. test_parameter_node_at_instant_systems.py follows each pattern in Core. A script against policyengine-us main read identical values and parameter-tree hashes on master and this branch (20 of 20): isolate_parameter_tracing through a traced simulation, a branch sharing a policy and then a reform that detaches it, pin_spm_policy, clone_spm_system, copy_parameter_nodes with an update, and share_spm_policy.

Interaction with #660

#660 merged on 2026-11-20, and this branch has merged it. The one conflict was in add_child: it keeps #660's _check_child_type(child) and self._link_child(name, child), and clears the caches just before _link_child.

#660 links brackets and homogenized nodes when a tree loads, and stops a child named like a node attribute from replacing that attribute. So this branch's own homogenization parent-link pass is removed, and homogenize_parameters.py is as on master. add_child keeps its own walk rather than clear_parent_cache, for the trees described under "Change"; four new tests in test_parameter_node_add_child_caches.py cover them. The PR's 239 tests and the five policyengine-uk tests from PolicyEngine/policyengine-uk#2926 pass on the result.

Reforms built with Reform(baseline)

Reform.modify_parameters deep-copies the baseline's parameter tree (reforms/reform.py:188), cached nodes at instant included, and a deep copy of a node at instant resolves every child first. So constructing a reform directly on a baseline that has already been read builds every child the baseline hasn't read: about 500,000 nodes for policyengine-us (498,900 in the review's measurement), which the baseline then keeps. Values are unchanged.

This PR leaves that copy as it is:

  • Simulation(reform=...), which reports and .py use, applies a reform to its own system through TaxBenefitSystem.modify_parameters, with no copy. Its node counts are the ones under "Measurements".
  • The YAML test runner builds each reform from baseline.clone(), whose caches are empty, so it builds no extra nodes.
  • Only code that constructs a reform on a baseline it has already read pays the cost: TaxBenefitSystem.apply_reform(path), the --reforms option, and scripts.

Rather than change the copy, Core will build every reform system by applying the reform to a clone of its baseline, deprecate constructing Reform(baseline), and later remove it and its deep copy, which also resolves #618. #645 removes the copy in another way; this PR doesn't need it.

Relation to #621

#621 proposes rules for Core's parameter caches and suggested lazy nodes wait for them. This PR doesn't wait. It keeps every value a node requested after a change reads, and it handles the three problems #621 lists for this PR's earlier version:

  • Mixed stale and fresh values after a missing invalidation. add_child clears now, and brackets and homogenized nodes have parents. The held-node case above is what remains.
  • Copying a tree while another thread read it raised RuntimeError. Fixed, with deterministic tests.
  • Order-dependent copy logic and a per-thread pickle counter. Not used: a copy resolves every child first.

The rest of #621 (freezing trees, removing the system-level cache, not copying caches) is unchanged and still applies.

Measurements

Re-measured on 2026-11-17 against master 175e7ee. Setup:

  • policyengine-us main e283430b2e (2.144.4) on Python 3.13, with its default dataset (populace_us_2024.h5);
  • subsample(n=2862, seed=0), which gives 2,490 households;
  • household_net_income for 2026;
  • one fresh process per run, running two phases:
    • First calculation: Microsimulation(), the subsample, then the calculation.
    • Reform system with empty caches: then, in the same process, Microsimulation(reform={"gov.irs.deductions.standard.amount.SINGLE": {"2026-01-01.2100-12-31": 20000}}) (policyengine-us builds a new system), the same subsample and the same calculation.
master this branch
Nodes at instant built, first calculation 665,933 121,349
Nodes at instant built, reform system 699,744 122,008
Nodes at instant alive after both phases (two systems) 1,399,464 243,997
Calculation CPU time, reform system, median of 5 31.5 s 21.3 s
Calculation CPU time, first calculation, median of 5 27.3 s 28.2 s

Outputs are identical. Every stored array of both simulations and their branches matches master: 428,303 arrays, hashed over dtype, shape and bytes. So do the weighted totals.

Times. Runs were interleaved (master, branch, master, ...) on a shared machine with a load average of about 20 on 10 cores, so single runs vary by several seconds.

  • Reform system: the branch was faster in all 5 pairs, by 6.0 to 12.5 s of CPU time.
  • First calculation: no difference showed. The paired differences ran from 8.3 s slower to 3.3 s faster, median 0.6 s slower.
  • Wall times, from an earlier set of 3 pairs, agree: reform system 28.0 s against 21.5 s, first calculation 31.1 s against 30.7 s.
  • On master, the reform system's calculation runs with the first system's trees still alive, which a fresh process's first calculation doesn't have; that difference may explain why only the reform system got faster here. This wasn't measured separately.

Memory. Peak resident memory couldn't be compared on this machine: the medians were 6.94 GB against 5.54 GB in one set of runs, and 5.43 GB against 5.44 GB in the next.

Node objects were counted by wrapping ParameterNodeAtInstant.__init__ in a separate run, and live objects with gc after each phase.

Max's measurements (2026-10-02)

Measured on policyengine-us (132,396 parameter descendants) on master b78b0ba:

  • each instant cost about 34,000 node objects, 16 to 33 MB, and 4 to 5 s on a loaded host, on every system;
  • a household on a reform system added 349 MB, most of it the reform system's own at-instant trees.

A YAML run touches every year it tests and every month of a monthly variable, so the shared system and each reform system all pay this many times over. It was the third cause of the 118 GB policyengine-core test runs on 2026-10-02, next to the test-runner causes fixed in #569; the two PRs are independent. Combined with #569 on the two policyengine-us CI batches that failed on 16 GB runners, peak memory fell from 16.3 GB to 7.0 GB (NY/OH/OK batch) and from 17.1 GB to 6.4 GB (USDA batch): #569 (comment).

For downstream packages

  • policyengine-uk. Three of its tests on main rely on the up-front build and fail with this change. Update parameter-read tests for children built on first read policyengine-uk#2926 updates them, and they pass there on master and on this branch; it needs to merge before a Core release contains this PR. Simulation outputs are not affected.
    • tests/code_health/test_labour_supply_parameters_are_read.py records parameter reads by patching ParameterNodeAtInstant.__getattribute__. A first read of a child now goes through __getattr__, and object.__getattribute__(node, "_children") raises until the node is read in full. So it misses reads: test_every_labour_supply_parameter_is_read[gov.simulation.labour_supply_responses] reports four labour supply parameters as unread, and test_recorder_counts_each_kind_of_read fails as well.
    • test_uprating_inputs_reach_data.py::test_a_failing_probe_leaves_the_system_and_its_caches_alone expects reading the root at an instant to fill descendants' caches.
  • policyengine-household-api. prewarm_parameter_caches calls get_parameters_at_instant for many instants so the memory snapshot holds built trees. It now builds only each root; the first requests resolve what they read.

Tests

Added or changed, run file by file:

  • tests/core/parameters/test_parameter_node_at_instant_lazy.py
  • test_parameter_node_at_instant_property.py
  • test_parameter_node_add_child_caches.py
  • test_parameter_node_at_instant_copies.py
  • test_parameter_node_at_instant_systems.py

All 239 pass on this branch, which includes #660. The rest of the suite wasn't run locally. Five rounds of independent adversarial review: the first four found bugs, each fixed with a failing test first; the fifth found none in Core.

Since the review on 2026-11-18: the changelog sentence on held nodes, the merge of master with #660, the removal of the homogenization pass #660 made redundant, four tests for trees add_child must accept, and a fixed changelog entry that leaves the parent links to #660's.

Credit: the lazy resolution, its thread safety and its property tests are Max Ghenis's work.

🤖 Generated with Claude Code

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Measurements of this PR combined with #569 on the two policyengine-us CI batches that are failing on 16 GB runners: see #569 (comment). Together they take the NY/OH/OK batch from 16.3 GB (3.32.11) to 7.0 GB and the USDA batch from 17.1 GB to 6.4 GB peak footprint.

@anth-volk

Copy link
Copy Markdown
Collaborator

Rebased onto master (b909d5a6) and force-pushed: de47cde0 → 089761c5. Beyond the rebase:

Details are under "October 14 rebase" in the description.

anth-volk added a commit that referenced this pull request Oct 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 14, 2026
#570 now provides the walk, so modify_parameters uses it instead of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 15, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits October 16, 2026 00:20
Building every descendant in the constructor made each instant a system is
asked about cost a copy of the whole parameter tree, kept for the system's
lifetime. For policyengine-us that is about 34,000 node objects, 16-33 MB and
several seconds per instant, per tax-benefit system.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-coherent

Review of 404a3a3 found five regressions against the up-front build:
- concurrent first reads could raise KeyError/AttributeError or resolve a
  child twice: first resolution now runs under a module RLock;
- children named like the new state or helpers (_node, _resolved, _resolve,
  ...) read as internal objects: state is name-mangled, helpers are module
  functions, and children named like the node's own attributes resolve at
  construction, as before;
- a shallow copy and its original could resolve a scale twice: both now read
  one grow-only resolved map;
- a child with no value at the instant was re-resolved on every read: its
  absence is cached;
- deepcopy/pickle of a traced tree's raw node recursed through
  TracingParameterNodeAtInstant.__getattr__, which now raises AttributeError
  for its own unset fields.
A node also drops its parameter node once every child has been read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ases

Round-2 review of a90b524 found reads that still differed from the up-front
build. A node now keeps all of its own state in one _LazyChildren object under
a single reserved name (a parameter of that name is rejected):
- which children a node has, and their order, are fixed at construction
  (a child deleted from the tree later keeps its place);
- completion counts only those children, so add_child extras can no longer
  send the first read into recursive completion;
- resolution results and add_child entries are kept apart: items are shared
  by shallow copies, attributes belong to each copy, as before;
- a copy installs what it reads as attributes (no lock on warm reads);
- children named like attributes of the node's runtime type (dunders and
  subclass methods included) resolve at construction; a subclass overriding
  add_child gets the up-front build through it;
- a reentrant read keeps the first recorded child;
- nodes pickled by the up-front build load (__setstate__).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk

Copy link
Copy Markdown
Collaborator

I've dropped the commits I added to this branch and restored Max's description.

  • Dropped:
    • 089761c5, which let lazy nodes pickle under protocols 0 and 1;
    • 62d63079 and b26aa0a7, cache-consistency fixes;
    • 5719353e, a merge of master;
    • four review fixes that were never pushed.
  • Now on the branch: only Max's 4 commits, rebased onto master (4adfea70) and force-pushed: 5719353e → 31469a4d. The October 14 rebase's one change to Max's code remains: the _OWN_ATTRIBUTES guard in the tracing wrapper is gone, because master's _wrapped covers that case.
  • Expected CI failures: without 089761c5, four of master's copy-and-pickle tests from Let vectorial parameter nodes be copied and pickled #584 fail on this branch, the protocol 0 and 1 cases of:
    • test_tracing_wrapper_of_a_slotted_node_can_be_pickled;
    • test_copied_parameter_tree_keeps_its_cached_vectorial_node.

Review kept finding edge cases in how Core's parameter caches are filled, invalidated, copied and shared. Some predate this PR; others come from nodes changing their own state as they're read. #621 proposes rules for those caches, which this PR should wait for.

anth-volk and others added 2 commits November 17, 2026 13:47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Finishes the lazy nodes at instant:

- ParameterNode.add_child clears the nodes at instant cached by the node
  and every node above it, which were built without the new child. It
  walks up without recursing and stops at a node it has seen: while a node
  loads, a child named "parent" (policyengine-us's
  gov.hhs.medicaid.eligibility.categories.parent) replaces the node's
  parent attribute and points back at it.
- Copies and pickles take copies of the dictionaries that reads change:
  a node at instant's __dict__, its shared lazy state and a parameter
  node's _at_instant_cache. Copying a tree while another thread resolved a
  child raised "dictionary changed size during iteration". The lazy state
  now also pickles under protocols 0 and 1, which master's copy and pickle
  tests need. A shallow copy of a parameter node still shares its cache,
  and slotted subclasses keep their slots.
- A node at an instant that is not a date, such as parameters(None), is
  built up front, so it fails, or succeeds, as the up-front build did.
  Simulation._calculate reads parameters(None).gov.abolitions for a
  variable calculated without a period and ignores the error.
- A forked process gets a new resolution lock, so a fork while another
  thread resolves a child doesn't leave the lock held for ever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk

Copy link
Copy Markdown
Collaborator

@MaxGhenis, I'm taking this PR over as part of the work on report runtimes, and I'll keep working on this branch. Your 4 commits stay as they are, with your authorship: I'll merge current master in and add commits on top, so there won't be another force-push. I'll also update the description as the work goes, keeping your measurements.

Thanks for the groundwork here!

Fixes from the first review round:

- A deep copy or pickle of a node at instant resolves every child first,
  so it is a snapshot that holds no parameter nodes, as when every child
  was built up front. Before, a partly read node carried its unread
  parameter nodes, and through their parents the whole tree: 128 MB and
  22 s to deep-copy a policyengine-us CTC node, against about nothing on
  master. A copied cached node keeps the values it was copied with, as on
  master.
- The lazy state is copied as it is once complete, so a copy's
  _children and its state stay one mapping and add_child on the copy is
  listed.
- add_child on a fully read node no longer changes what attribute reads
  return on its other copies (the resolved children and the items were
  one dictionary). Items are read from the complete mapping, so direct
  writes to _children are items, as before.
- A parameter node's at-instant cache is an _AtInstantCache, a dict that
  copies and pickles from a one-step copy of its items. Copies stay safe
  while other threads read, and parameter nodes sharing one cache still
  share its copy. ParameterNode's own __getstate__ and __copy__ are gone.
- ParameterScale sets each loaded bracket's parent, as clone already did
  and as #660 does. Parameter.update and add_child below a bracket now
  clear the caches above the scale, so a reform of a bracket reads the
  same whatever was calculated before it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 3 commits November 17, 2026 15:27
Fixes from the second review round:

- homogenize_parameter_structures links the nodes it puts in place of
  missing breakdown values to their parent node. They were set straight
  into children with no parent, so an update below one cleared caches
  only up to it, and whether a later read saw the update depended on
  what had been read before (policyengine-us has 9,811 such nodes).
- A subclass of ParameterNodeAtInstant that overrides add_child gets
  the up-front build as before: every child added into _children, which
  is also the mapping items are read from. An add_child that wrote
  _children directly lost its items.
- A node at a date that does not exist ("2015-02-29", "0000-01-01") is
  built up front, so it fails as before. periods.instant reads these;
  str(Instant), which a scale's build calls, refuses them.
- A node's children are the ones node.children.items() gives, as the
  up-front build iterated them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes from the third review round:

- Any subclass of ParameterNodeAtInstant gets the up-front build, not
  only one that overrides add_child. A subclass whose __getattr__ or
  __getitem__ answers names that are not children returned its fallback
  for real children not read yet.
- ParameterNode._get_at_instant replaces a plain dict cache, with its
  entries, by an _AtInstantCache before adding to it. policyengine-us's
  copy_parameter_nodes sets plain dict caches, as do trees pickled by an
  earlier core, and copying or pickling such a tree while another thread
  resolved a child raised "dictionary changed size during iteration".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes from the fourth review round:

- ParameterNode._get_at_instant stores the node it built into the cache
  the node holds after building it, as before. If code run while the
  node was built replaced the cache (policyengine-us's
  isolate_parameter_tracing does, to drop an earlier request's nodes),
  the old entries came back.
- A complete node's resolved children are no longer the same dictionary
  as its items, so a write into _children changes items only, as when
  every child was built up front, and attributes keep what the
  parameters give.
- A pickled _AtInstantCache is a plain dict, so trees pickled by this
  version load in earlier versions of core when they hold no nodes at
  instant.
- The class docstring says that a child that fails to build fails when
  read, and that a pickle of a node at instant holds this version's lazy
  state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 17, 2026 15:24

@PavelMakarchuk PavelMakarchuk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST_CHANGES (head f8f77d2495)

The lazy resolution gives the same parameter values as master for every leaf of policyengine-us that I read, in every read order. The policyengine-us tests that I ran pass, with lower peak memory. Three policyengine-uk tests on main fail with this core and pass with master core, and a reform that uses modify_parameters removes most of the memory saving.

Critical

None.

Should address

  1. Three policyengine-uk tests fail with this core. The description names two test files. I ran them on policyengine-uk main (origin/main at review time) with each core:

    • master core: 22 passed.
    • this PR: 3 failed, 19 passed. The failures are:
      • tests/code_health/test_labour_supply_parameters_are_read.py::test_every_labour_supply_parameter_is_read[gov.simulation.labour_supply_responses]
      • tests/code_health/test_labour_supply_parameters_are_read.py::test_recorder_counts_each_kind_of_read (the description does not name this one)
      • tests/test_uprating_inputs_reach_data.py::test_a_failing_probe_leaves_the_system_and_its_caches_alone

    Core CI does not show this, because the country job runs only pytest -m smoke (.github/workflows/pr.yaml, "Test Core and country packages"). I found no policyengine-uk issue or PR that tracks the change. Fix: open the policyengine-uk change (read recorder that also hooks __getattr__, and a probe test that does not expect descendant caches to fill) before this core is released, and link it here. Alternatively, give the recorder a supported hook in Core.

Suggestions

  1. Reform.modify_parameters resolves every cached node of the baseline. reforms/reform.py:188 deep-copies baseline.parameters. The deep copy of _AtInstantCache (parameter_node.py:62) deep-copies each cached node at instant, and _LazyChildren.__getstate__ (parameter_node_at_instant.py:102-113) resolves every child first. The baseline keeps these resolved trees. policyengine-us has 58 files that call modify_parameters. Measurement (one household calculation for 2026 on the baseline, then create_limit_salt_deduction_to_property_taxes() applied to it, nodes counted by a wrapper on ParameterNodeAtInstant.__init__):

    master this PR
    Nodes built by the baseline calculation 543,118 11,040
    Nodes built while the reform is constructed 0 498,900
    Live nodes after the reform calculation 1,052,400 778,758

    Values are the same (household_net_income 49,640.60 on both). This is not a regression against master, but the saving that the description claims for reform systems does not apply to these reforms. Fix: let the deep copy of a ParameterNode start with an empty _AtInstantCache (a copy of a parameter tree does not need the source's nodes at instant), or merge #645 first, which removes this deep copy. Add a test that counts the nodes built by a modify_parameters reform.

  2. A held node can mix old and new values in one subtree. The class docstring (parameter_node_at_instant.py:135) documents this. A small tree shows the effect: read held = root("2021-01-01") and held.a.x, then update a.y to 99 and a.x to 98. This PR gives held.a.x == 1 and held.a.y == 99. Master gives 1 and 2 (old, but consistent). In Core, only callers of TaxBenefitSystem.get_parameters_at_instant hold such nodes (formulas read through the root cache, which Parameter.update clears). Fix: name this in the changed changelog fragment, so that downstream users who call get_parameters_at_instant (policyengine-household-api, policyengine-us spm.py) see it.

  3. Coordinate with #660 and #645. #660 makes the same bracket.parent and homogenization changes and changes add_child. #645 removes the deep copy in item 1. #621 asked that this PR wait for its rules. The description gives a merge plan for #660. Fix: agree on the merge order in #621 before merge.

What I checked

  • Diff: read all source changes in parameter_node_at_instant.py, parameter_node.py, parameter_scale.py and homogenize_parameters.py, and the callers of _children (parameter_scale.py, VectorialParameterNodeAtInstant.build_from_node) and of get_parameters_at_instant.

  • Equivalence with policyengine-us (origin/main), both cores, instants 2017-01-01, 2024-01-01, 2026-01-01 and 2026-07-01:

    • 129,748 parameter and scale paths per instant, read by attribute path, in file order and in a shuffled order, and on a reform system. That is 1,556,976 reads per core. All are equal, including the 67,488 reads per system (all four instants) that raise KeyError on both.
    • The reform used Reform.from_dict on gov.irs.deductions.standard.amount.SINGLE, gov.irs.income.bracket.rates.2, gov.irs.income.bracket.thresholds.3.JOINT (breakdown leaf) and gov.irs.credits.ctc.amount.base[0].amount (scale bracket). income_tax and ca_income_tax for a household are equal.
    • A full walk of the tree at 2026-01-01 (112,875 leaves, values and order) is byte-identical with the same PYTHONHASHSEED. Without a fixed seed, the order of homogenized children differs between any two processes, also on master (homogenize_parameters.py:102 iterates a set).
    • Parameter.update after a read: same value on both.
  • Small-tree probes: hasattr on a missing child is False; enum-array indexing of a partly read node is equal; deep copy and pickle work; 8 threads that clear, read, iterate and deep-copy for 200 rounds raise nothing. This PR also fixes two master bugs: a bracket update now reaches the scale ([0.1, 0.5], master [0.1, 0.2]), and add_child now reaches a cached root (master raises ParameterNotFoundError).

  • Core tests: full suite, this PR 4,112 passed, 4 failed; master 3,877 passed, 4 failed. The same 4 tests fail on both (tests/core/test_release_tagging.py, which needs the release Git setup). CI at the head is green (19 of 19 checks).

  • New tests test the new code: with master's source and the PR's 5 new test files, test_parameter_node_at_instant_lazy.py fails to import, and the other 4 files give 78 failed, 88 passed. With the PR's source, 235 passed.

  • policyengine-us YAML tests (policyengine-core test ... -c policyengine_us):

    Folder master core this PR Peak memory, master / PR Wall time, master / PR
    baseline/partners 623 passed 623 passed 9.7 / 8.8 GB 422 / 418 s
    baseline/gov/irs 3,449 passed 3,449 passed 48.6 / 40.1 GB 1,465 / 1,316 s
    baseline/gov/states/ca 1,382 passed 1,382 passed 17.1 / 11.5 GB 361 / 266 s
    contrib (partial, stopped) first 6 files: 101 passed first 16 files: 152 passed not measured not measured

    The two cores ran at the same time on one shared machine, so the times are approximate.

  • Startup and a household calculation (3 runs of each core, interleaved, one process per run): medians, in seconds unless stated. The third pair ran during a load spike (PR import 55 s), so the medians matter more than any single run.

    Step master this PR
    import policyengine_us 15.8 15.9
    CountryTaxBenefitSystem() 10.3 10.3
    First household calculation, 2026 2.9 2.4
    Reform system and its household calculation 13.6 10.1
    Peak resident memory 3,131 MB 2,341 MB

    Import and system construction times do not change. household_net_income is equal (55,152.70) in every run.

  • Merge: the head merges into master (175e7eec) with no conflicts. The merge base is the current master.

  • Changelog: two fragments, lazy-parameter-node-at-instant.changed.md and .fixed.md, at the top level of changelog.d/. The API docs use autoclass, so the new class docstring appears there.

Not verified

  • I did not test forked processes, or threads on Windows.
  • I did not measure a full microsimulation. The description's numbers (median calculation time 31.5 s to 21.3 s for the reform system) are not reproduced here.
  • I did not run the merge of this PR with #660.
  • I did not finish contrib. It was too slow on this machine (6 and 16 of 210 files in about 30 minutes). No case failed before I stopped it.
  • I did not test policyengine-household-api or policyengine-canada.

A node at instant built before an in-place parameter change, such as one
TaxBenefitSystem.get_parameters_at_instant returns, now reads the changed
value for each child it has not read yet. Before, every child kept the
value from when the node was built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk

Copy link
Copy Markdown
Collaborator

Thanks for the review, @PavelMakarchuk. Here's where each point stands.

policyengine-uk tests. PolicyEngine/policyengine-uk#2926 updates all three tests you listed. They pass there with both master Core and this branch. It has to merge before a Core release contains this PR, and the description now names all three tests and links it.

Reform.modify_parameters (suggestion 1). I'm leaving the deep copy out of this PR. Only a reform constructed directly on a baseline that has already been read reaches it: TaxBenefitSystem.apply_reform(path), the --reforms option, and scripts like your measurement. Simulation(reform=...), which reports use, applies the reform to its own system with no copy. The YAML runner builds reforms from baseline.clone(), whose caches are empty. Instead of changing the copy, the plan is for Core to build every reform system by applying the reform to a clone, deprecate constructing Reform(baseline), and later remove it with its deep copy (#618). The description has a new section on this.

Held nodes (suggestion 2). I reproduced your example on this branch: held.a.x stays 1 and held.a.y reads 99, while master gives 1 and 2. The changed changelog entry now says this.

Merge order (suggestion 3). #660 merges first. After it does, this branch takes it, resolves the one conflict in add_child, and drops its own homogenization parent-link pass and _clear_at_instant_caches_upward, which #660 makes redundant. #645 isn't needed for this PR. This PR doesn't wait for #621's rules; its remaining proposals (trees that don't change after construction, and no system-level cache) are planned as later work, and they would also remove the held-node difference.

anth-volk and others added 4 commits November 20, 2026 01:15
#660 moved the body of ParameterNode.add_child into _link_child. add_child
keeps #660's structure and clears the nodes at instant cached above the
node just before _link_child, as this branch did before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#660 links each node homogenization puts in place of a missing breakdown
value, so the pass that linked them after homogenizing found nothing to
link. homogenize_parameters.py is now as on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With #660, a child named parent or _at_instant_cache no longer replaces
the node's own attribute, but clear_parent_cache still can't replace the
walk: it raises on a node with no cache of its own, on a parent that is
not a parameter node, and on a loop in the parent links, all of which
add_child accepts on master. Tests pin those three cases, and the walk's
docstring now says why add_child uses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#660 links loaded scale brackets and homogenized nodes and has its own
changelog entry, so this branch's fixed entry names only the add_child
cache clearing it adds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

modify_parameters reads stale cached values after in-place edits, and Reform.modify_parameters deep-copies a tree the reform already owns

3 participants