Skip to content

Avoid eager Holder allocation in input_variables discovery - #663

Merged
anth-volk merged 3 commits into
PolicyEngine:masterfrom
jindal-git:fix/lazy-holder-input-variables
Nov 19, 2026
Merged

anth-volk merged 3 commits into
PolicyEngine:masterfrom
jindal-git:fix/lazy-holder-input-variables

Conversation

@jindal-git

@jindal-git jindal-git commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #662.

Summary

  • Updated SimulationBuilder.build_from_dict() and Simulation.__init__() to build input_variables from _user_input_keys (using getattr(simulation, "_user_input_keys", ()) in build_from_dict for subclasses that replace __init__, and filtering against tax_benefit_system.variables to preserve variable registration order) instead of calling get_holder(variable.name) on every variable in the system.
  • When populations is passed directly to Simulation.__init__(), existing non-empty holders are included without allocating new holders; variables whose entity changed under a reform are excluded via name in self.populations[variable.entity.key]._holders.
  • Updated existing core tests and fixtures that previously assumed Simulation initialization eagerly allocated empty Holder objects for every variable in the system, and added regression tests covering lazy holder allocation, dataset variable order, pre-populated populations=, clone(), reforms removing or moving input variables, and build_from_dict on a simulation without _user_input_keys.
  • Updated towncrier fragment changelog.d/avoid-eager-holder-allocation.fixed.md (including the effect on memory_config set after setup).

Checks run

  • uv run --extra dev ruff format --check . (386 files already formatted)
  • uv run --extra dev ruff check . (All checks passed!)
  • uv run policyengine-core test policyengine_core/country_template/tests -c policyengine_core.country_template (39 passed)
  • uv run pytest tests --disable-pytest-warnings -q (3887 passed, 4 skipped, 3 xfailed)
  • policyengine-us: uv run --with-editable /Users/deepakjindal/policyengine-core pytest policyengine_us/tests/core/test_social_security_input_state_machine.py -q (1 passed, run prior to adding the getattr guard in build_from_dict)

Checks not run

  • make mypy and make documentation were not run (type-checking and Jupyter Book docs build were outside the scope of this runtime fix).

@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.79%. Comparing base (c94a745) to head (1579c8d).
⚠️ Report is 214 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master     #663       +/-   ##
===========================================
+ Coverage   84.69%   94.79%   +10.09%     
===========================================
  Files         223      354      +131     
  Lines       11830    33228    +21398     
  Branches     1169     2887     +1718     
===========================================
+ Hits        10019    31497    +21478     
+ Misses       1496     1351      -145     
- Partials      315      380       +65     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hua7450
hua7450 requested a review from anth-volk November 18, 2026 16:53
@anth-volk

Copy link
Copy Markdown
Collaborator

Thanks for tracking this down, Deepak. The diagnosis in #662 is right: building input_variables shouldn't create a Holder for every variable, and your tests for that are useful.

I'd like to change how the fix gets there. Core already records inputs as they are stored in Simulation._user_input_keys, so I think we can build input_variables from that record instead of scanning holders.

This would also address two problems with the current version:

  1. It breaks a policyengine-us test helper. policyengine_us/tests/core/_social_security_state_backend.py runs Core's build_from_dict source against its own stand-in Simulation, which has no _collect_input_variables. Calling build_from_dict there raises AttributeError, so I expect test_social_security_input_state_machine.py to fail once this Core is released. Core's CI doesn't run that test.
  2. It changes the order of input_variables for dataset simulations. The list currently follows variable registration order, and this PR changes it to population and holder creation order.

In SimulationBuilder.build_from_dict:

recorded_input_names = {
    name for name, _branch_name, _period in simulation._user_input_keys
}
simulation.input_variables = [
    name
    for name in simulation.tax_benefit_system.variables
    if name in recorded_input_names
]

In Simulation.__init__:

recorded_input_names = {
    name for name, _branch_name, _period in self._user_input_keys
}
if populations is not None:
    # Populations passed in may already hold values this simulation
    # did not record.
    recorded_input_names.update(
        name
        for population in self.populations.values()
        for name, holder in population._holders.items()
        if len(holder.get_known_periods()) > 0
    )
# A reform may have moved a recorded variable to another entity, leaving
# its value in a holder this simulation no longer reads.
self.input_variables = [
    name
    for name, variable in self.tax_benefit_system.variables.items()
    if name in recorded_input_names
    and name in self.populations[variable.entity.key]._holders
]

With this, _collect_input_variables and the condition that skips the scan in __init__ can go.

I tried this on a copy of current master with policyengine-us main:

US household, no reform master This PR Using the record
Holders after setup 7,799 14 14
Setup time 36–38 ms about 5 ms about 6 ms
  • input_variables matched master for 5,158 inputs from the policyengine-us YAML tests. This built the simulations only and did not check expected outputs.
  • Plain, axes and reform households gave the same input lists and the same values for six calculated outputs.
  • The build_from_dict call that fails in the test helper works.

I haven't run the Core or policyengine-us test suites against this, a real microdata dataset, or policyengine-uk.

Could you:

  1. Rework the PR around the record, keeping your tests that check untouched variables get no holders, along with your updates to tests that assumed every holder existed.
  2. Add tests for dataset input order, already-filled populations=, and a reform that moves an input to another entity.
  3. Rerun your warm 501-point household benchmark and post the before and after numbers. I haven't reproduced that one, and it should confirm this still solves the problem you hit.

@jindal-git

Copy link
Copy Markdown
Contributor Author

Thanks for catching the _social_security_state_backend.py stand-in compatibility issue and the dataset variable-order change — building input_variables from _user_input_keys is much cleaner.

I've updated the PR as requested:

  1. Reworked around _user_input_keys:

    • SimulationBuilder.build_from_dict now builds simulation.input_variables from simulation._user_input_keys in simulation.tax_benefit_system.variables order, removing _collect_input_variables.
    • Simulation.__init__ now builds self.input_variables from self._user_input_keys (plus any non-empty existing holders when populations is not None), filtering against self.tax_benefit_system.variables and self.populations[variable.entity.key]._holders without allocating new holders.
  2. Added the requested tests in tests/core/test_simulations.py:

    • test__given_dataset_simulation__then_input_variables_preserve_system_order
    • test__given_prepopulated_populations__then_input_variables_include_their_inputs
    • test__given_reform_moving_input_variable_entity__then_init_excludes_it_without_allocating_holders
  3. Verification:

    • Core formatting/lint: ruff format --check . (386 files already formatted) and ruff check . (All checks passed!)
    • Core country-template tests: 39 passed
    • Full Core pytest suite: 3886 passed, 4 skipped, 3 xfailed
    • policyengine-us stand-in test (test_social_security_input_state_machine.py run against this editable policyengine-core): 1 passed
  4. 501-point axes household simulation:

Metric master (175e7eec) Reworked PR
Holders after Simulation.__init__ 7,799 67
Simulation.__init__ (median / min–max) 46.20 ms (44.97–49.24 ms) 10.29 ms (10.01–11.21 ms)
Total warm simulation (median / min–max) 229.33 ms (222.67–1078.51 ms) 145.07 ms (142.91–146.65 ms)
Numerical parity across all 40 outputs — 0 diffs (np.array_equal)

@anth-volk

Copy link
Copy Markdown
Collaborator

Thanks, Deepak. The rework covers everything I asked for, and the benchmark results are what we were hoping to see.

I'd recommend three changes before merge. The first fixes a problem in the snippet I suggested.

1. Handle simulations that have no input record yet. policyengine_uk.Simulation doesn't call Core's __init__, so _user_input_keys doesn't exist until the first input is stored. With this PR, a UK situation that stores no inputs raises AttributeError: _user_input_keys in build_from_dict. For example:

Simulation(situation={
    "people": {"adult": {}},
    "benunits": {"benunit": {"members": ["adult"]}},
    "households": {"household": {"members": ["adult"]}},
})

The same happens when a person only requests an output, such as income_tax: {"2026": None}. Both work on master. This fixed both cases when I tried it:

# A subclass that replaces ``__init__`` has no record until an input is stored.
recorded_input_names = {
    name
    for name, _branch_name, _period in getattr(simulation, "_user_input_keys", ())
}

We plan to remove this tolerance once country packages use Core's constructor.

2. Add a regression test. Please cover build_from_dict on a simulation that has populations but no _user_input_keys, with a situation that stores no inputs. It should succeed and leave input_variables empty.

3. Update the changelog. Please mention that memory_config set after setup now applies to holders created afterwards. Before this PR, every holder already existed by then, so it had almost no effect. Now values can go to disk, and variables_to_drop values are no longer stored.

The source change here follows code I suggested, so this is partly a self-review. I compared this version with master using targeted scripts, but I did not run the Core, policyengine-us or policyengine-uk test suites.

@jindal-git

Copy link
Copy Markdown
Contributor Author

Thanks for catching the missing _user_input_keys case for subclasses replacing __init__. I've pushed the three updates:

  1. Guarded SimulationBuilder.build_from_dict with getattr(simulation, "_user_input_keys", ()).
  2. Added test__given_simulation_without_user_input_keys__then_build_from_dict_leaves_input_variables_empty in tests/core/test_simulations.py covering a Simulation subclass without _user_input_keys and a situation storing no inputs.
  3. Updated changelog.d/avoid-eager-holder-allocation.fixed.md to note that memory_config set after setup now applies to holders created afterwards (so values can go to disk and variables_to_drop variables are no longer stored).

Re-ran ruff format --check ., ruff check ., the country-template tests (39 passed), and the full Core pytest suite (3887 passed, 4 skipped, 3 xfailed).

@anth-volk
anth-volk merged commit 8360cab into PolicyEngine:master Nov 19, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Simulation.__init__ and SimulationBuilder.build_from_dict eagerly allocate Holders for every variable in tax_benefit_system

2 participants