Skip to content

Return plain arrays to Microsimulation calls made inside a calculation - #661

Merged
anth-volk merged 6 commits into
masterfrom
unweighted-internal-calls
Nov 19, 2026
Merged

anth-volk merged 6 commits into
masterfrom
unweighted-internal-calls

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator
  • make format && make documentation has been run. (ruff format and ruff check were run on the changed files only; the documentation was not built, since no documentation page changed.)

Bug fix

  • Regression test added
  • Regression test passing

Fixes #658.

What changes

Microsimulation.calculate, calculate_add and calculate_divide return a weighted MicroSeries, and calculate decodes enums to member names. Core also calls these methods for its own reads while it calculates: each term of adds and subtracts, a defined_for mask, each month of a monthly variable read over a year, the year of a yearly variable read over a month, and the terms of ADD and DIVIDE. Each such call computed household_weight, projected it to the variable's entity and built a MicroSeries, and core then discarded the weights before storing the value (_cast_formula_result). Decoding broke enum reads with ADD and DIVIDE (#658).

This PR changes only policyengine_core/simulations/microsimulation.py:

  • Detection. A private ContextVar holds a record of the innermost Microsimulation calculate, calculate_add or calculate_divide call running in the current context; each record links to the one for the call that encloses it. A call that arrives while one of these calls is running is a call from inside a calculation. When a call ends, its record is marked as finished. A context copied during a call (for a worker thread, say) is therefore inside while that call, or one enclosing it, is still running, or while a call of its own runs there, and outside afterwards. A new thread starts with an empty context, so its calls count as outside (unless Python runs with -X thread_inherit_context=1, the default on free-threaded builds, which gives the thread a copy). Core's _calculation_frames and the tracer stack are not used.
  • Defaults. All three methods now default to use_weights=None and decode_enums=None.
    • Inside a calculation, None gives a plain NumPy array with enums left as codes, as a Simulation returns.
    • Outside, None gives the previous defaults: calculate is weighted and decoded; calculate_add and calculate_divide are weighted and not decoded (Fix formula reads with ADD or DIVIDE in a Microsimulation #647).
    • An explicit argument wins in both cases.
  • ADD and DIVIDE. Their term reads arrive while the call is running, so they get codes, and the result is decoded once when decode_enums=True. A twelfth of a code from a month DIVIDE is returned undecoded, as a Simulation returns it.
  • get_weights reads household_weight with decode_enums=False.
  • simulation.py's call sites are unchanged. Calling the base class from there would skip subclass overrides of calculate.

Effect

One weighted series is built per top-level call. Before this change, for example, a top-level calculate of a yearly variable with a monthly adds term built 13; a 500-household policyengine-us household_net_income built 35,368. On 100 policyengine-us households (warm household_net_income, 3 fresh processes), a variant of this change that detected internal calls through _calculation_frames took 3.57 s, against 6.84 s on master. It classifies every consumer call the same way as this change; this change's own timing was not measured on that setup. On 500 households, an earlier prototype of this change's detection took the baseline from 10.2 s to 6.3 s, measured on a loaded machine.

Checked against every consumer

Each consumer was run on master 175e7ee and with this change (or a variant that classifies every consumer call the same way), comparing every stored array's values, dtype, writability and keys, and every top-level value and weight. Consumers checked: policyengine-us 2.144.1, policyengine-uk 2.179.1, policyengine-canada 0.99.2, policyengine-il 0.2.2, policyengine-ng 0.6.2, policyengine-italy 0.16.3, and .py (policyengine package) at main 47d1d8bf (code read, not run).

  • No value that computes on master changes. A copy of this change's detection gave 2,898,199 stored arrays bit-identical to the _calculation_frames variant, which in turn differed from master only as listed below. Runs covered a targeted 1,140-household policyengine-us file for 2024 and 2027, current law and a CTC reform; 121 policyengine-us YAML cases (402 outputs) replayed inside a Microsimulation, 42 more at 2027, and the WI WHEAP file; and one synthetic household each in Canada, Israel, Nigeria and Italy. Separately, 500 policyengine-us households gave 508,321 stored arrays bit-identical to master.
  • policyengine-uk never reaches core's Microsimulation: it does not subclass it, and a counter recorded 0 calls.
  • Direct calculate calls in consumer formulas. 22 calls in 20 policyengine-us formula files call calculate on a simulation, branch or baseline. Examples: tax_liability_if_itemizing / _not_itemizing, tax_unit_itemizes, the NY and NYC EITC, the VA, DE, HI, ID and MO pairs, claim of right, co_adult_financial_medicaid_enrolled, medicaid_enrolled_for_ssi_state_supplement, spm_unit_allocated_housing_subsidy and marginal_tax_rate_helpers. They now get an ndarray rather than a MicroSeries. Each reads a float or bool variable and uses the result element by element, returns it, or passes it to map_result or set_input. None uses .values, .weights, .sum() or pandas indexing on the result, and every one ran in a Microsimulation in the comparisons above.
  • Calls that raised on master now compute.
    • wi_wheap_categorically_eligible (policyengine-us) raised microdf's "Cannot propagate weights unambiguously" in a Microsimulation on master (8 of 8 YAML cases); all 8 now pass.
    • Canada, Israel, Nigeria and Italy define no household_weight, so most internal reads raised there. Now 294 of 405 calls compute in Canada, 27 of 40 in Israel, 27 of 36 in Nigeria and 38 of 60 in Italy. Weighted calls from outside a calculation still raise there.
  • Stored arrays that were read-only pandas views are now writable NumPy arrays with the same bytes: about 1,000 to 3,300 per policyengine-us run, from period conversion and adds.
  • Some branch household_weight arrays are no longer calculated or copied, because a formula's branch.calculate is no longer weighted. Where they still exist, the values are identical.
  • Enums. No consumer has an adds/subtracts list, a defined_for or a period-converted read that involves an Enum variable, and no consumer formula reads with options=[ADD] or [DIVIDE], so no stored enum code changes. The Enum reads with ADD or DIVIDE raise or return names in a Microsimulation #658 reads now match Simulation. One top-level result changes: calculate_divide(enum_variable, year) left at decode_enums=None now returns codes, as Fix formula reads with ADD or DIVIDE in a Microsimulation #647's False default intends; master returned names. No consumer makes that call.

Not covered by those checks: full national datasets; .py at runtime; this change's exact detection on policyengine-uk (which cannot reach it) and its timing; Simulation.calculate_output has no caller in any consumer.

Other behaviour to note

  • Formula code that calls these methods directly now gets a NumPy array with enum codes, not a weighted MicroSeries with member names. A formula that took .sum() of a direct simulation.calculate(...) read, compared a direct enum read with a member name, or used .values or .weights on it would now get a different result or an error. The consumer check above found no such code; policyengine-us's direct calls return the result or use it element by element. The same applies to the inner reads of to_input_dataframe and derivative when they are called inside a formula.
  • Detection follows contextvars contexts, not threads. A call run through a context captured before the calculation started (for example Context.run on a context copied at import time, or an asyncio.Runner first used outside any calculation) counts as outside, even in the same thread, and gets weighted values, as on master.
  • Explicit use_weights=False from outside a calculation is now honoured everywhere. On master, without map_to, calculate_add, calculate_divide and a calculate of a FLOW variable over a period of another length (a monthly variable over a year, a yearly one over a month) returned a weighted MicroSeries even with use_weights=False, because their inner reads were weighted. They now return a plain array, so .sum() is the unweighted sum (for example 624 rather than 10,140 in the new tests' fixture). A review of policyengine-us, policyengine-uk, .py and the Public API found no caller that depends on the old result.
  • Reads inside a calculation return what a Simulation returns, including the stored array itself. On master, some of these reads in a Microsimulation returned a copy as a side effect of building a MicroSeries: a period-converted read, a read with options=[ADD] or [DIVIDE] over a year, and a formula's direct calculate. Values stored from period conversion (ADD from months to a year, DIVIDE from a year to a month) were stored as read-only arrays built from a pandas Series on master; they are now stored writable, with the same bytes. A formula that writes in place into such a read now changes the stored value, as it already does in a Simulation and for plain same-period reads. A scan of policyengine-us found no formula that writes into these reads.
  • Passing use_weights=None or decode_enums=None explicitly used to mean unweighted and not decoded; it now means the default for the caller (weighted and, for calculate, decoded from outside a calculation). No caller was found that passes None.
  • calculate_dataframe returns the same frame wherever it is called from: weighted unless use_weights=False, with enum member names. It now reads each column unweighted and decoded, so weights are built once per frame rather than once per column plus once for the frame (33 builds on master for 3 columns in the new tests' fixture). For 18 requests covering mixed entities, map_to, enum columns, period conversion and use_weights=False, the frames and error messages were identical to master. A column read that raised only because weights could not be built (no household_weight) now computes when use_weights=False.
  • A Microsimulation used inside a formula of a plain Simulation (not a Microsimulation) still gets weighted results, as on master: only a running Microsimulation call marks a call as internal. No consumer does this.

Tests

New tests/core/test_microsimulation_internal_calls.py, with a fixture system in tests/fixtures/microsimulation_internal_calls.py:

  • one test for each way a call can arrive: a formula reading its own simulation (population read, direct calculate, fast-cache re-read, calculate_add, year and month calculate_divide), trace off and on; a branch, trace off and on; a reform's baseline; a Microsimulation built in a Microsimulation's formula and in a plain Simulation's formula; a worker thread with and without a copied context; a context copied in a formula and used after the calculation ends; a call in a copied context that keeps running after the call it was copied from has ended; a context copied during a nested call and used while an enclosing call still runs; the fast-cache re-read; explicit arguments inside and at the top level;
  • calculate_dataframe called from a formula keeps enum names and weights, builds weights only for the frame, maps columns to the first variable's entity, and rejects unknown variables;
  • top-level calculate, calculate_add, calculate_divide, calculate_output and calculate_dataframe are still weighted, and each top-level call builds exactly one weighted series (for adds/subtracts, defined_for, a monthly variable over a year, a yearly variable over a month, ADD and DIVIDE), with household_weight calculated once;
  • after a formula raises (RuntimeError, a BaseException, trace on), an ADD or DIVIDE term raises, a formula catches a branch's error, or a branch input set during a calculation makes core run it a second time, calls from outside a calculation are weighted again;
  • a Microsimulation stores and returns what a Simulation does: same dtype, bytes and writability for every stored array;
  • the Enum reads with ADD or DIVIDE raise or return names in a Microsimulation #658 enum reads (ADD over a year, DIVIDE over a year and a month, from the top level and from formulas) match a Simulation, and top-level enum calculate_add / calculate_divide decode when asked.

test_enum_read_with_divide_over_a_year_gives_member_names_in_a_microsimulation recorded the #658 bug; it now checks that the codes match a Simulation. test_microsimulation_calculate_signatures.py checks the new None defaults.

Run: each new and modified test, selected individually by node ID (all pass; the thread and copied-context tests also with -X thread_inherit_context=1), and ruff format / ruff check on the changed files. The rest of the suite was not run locally; CI runs it.

🤖 Generated with Claude Code

anth-volk and others added 4 commits November 17, 2026 13:35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ile its own call runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runs

Fix tests that did not reach the fast cache or the rerun after an input
change, wait for the worker thread's call to start, and narrow the
changelog fragments to what changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rectly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@PavelMakarchuk PavelMakarchuk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST_CHANGES (head 7a41e25494)

The detection logic is correct in every case I tried. The new tests fail when the source change is reverted, and a policyengine-us run gives bit-identical stored arrays with about 5,000 times fewer weight builds. One documented side effect has no test, and the changelog describes it incorrectly: calculate_dataframe in a formula now returns a weighted frame with enum codes.

Critical

None.

Should address

  1. calculate_dataframe in a formula returns a mixed result, and no test covers it.
    • Microsimulation.calculate_dataframe keeps use_weights: bool = True (policyengine_core/simulations/microsimulation.py:263) and does not record a running call. Its columns come from self.calculate(variable_name, period, map_to) (policyengine_core/simulations/simulation.py:1677).
    • In a formula, these column reads now count as inside, so enum columns hold codes. The frame still gets weights (line 269 onward).
    • A probe formula that calls calculate_dataframe(["yearly_grant", "yearly_tenure"], period) on the #647 fixture gave this result:
      master  inside formula: MicroDataFrame ['OWNER', 'RENTER', 'OWNER'] weighted sum 15600.0
      PR      inside formula: MicroDataFrame [0, 1, 0]                    weighted sum 15600.0
      PR      top level:      MicroDataFrame ['OWNER', 'RENTER', 'OWNER'] weighted sum 15600.0
      
    • The result is weighted like a top-level call, but it has codes like an internal call. A filter such as df[df.yearly_tenure == "OWNER"] now silently matches no rows. This is the same class of error that #658 fixes.
    • The .changed fragment groups calculate_dataframe with the calls that "get a plain NumPy array with enum codes instead of a weighted MicroSeries". The frame is not plain. It is a weighted MicroDataFrame.
    • No test in tests/core/test_microsimulation_internal_calls.py calls calculate_dataframe from a formula. Only the top-level case is tested (line 270).
    • Fix, option A: in Microsimulation.calculate_dataframe, build each column with self.calculate(name, period, map_to, use_weights=False, decode_enums=True) instead of calling the base method. The frame then has names everywhere, and the per-column weights go away too.
    • Fix, option B: keep the behavior. Add a test that pins it, and reword the changelog so that it says what a formula gets from calculate_dataframe.

Suggestions

  1. Consider a minor or breaking version bump. Both fragments are changed and fixed, so this release is a patch. The PR changes public defaults:

    • A formula's direct simulation.calculate(...) now returns an ndarray, not a MicroSeries. In my probe, .sum() changed from 15600.0 (weighted) to 960.0.
    • A top-level calculate_divide(enum_variable, year) now returns codes [0, 1, 0], not names.
    • calculate_add(..., use_weights=False) now returns a plain array. .sum() changed from 1560.0 to 96.0.

    The consumer review in the PR body is thorough, but code outside the PolicyEngine organization can depend on these defaults. Rename the .changed fragment to .breaking, or add an .added fragment if the maintainers want a minor bump.

  2. Make the long changelog sentence shorter. The .changed fragment is one sentence of about 75 words. Split it into two or three sentences: one for internal calls, and one for direct formula calls.

What I checked

  • Diff. I read all 8 files and the functions they call in simulation.py, for example calculate, calculate_add, calculate_divide, _run_formula (adds and subtracts), calculate_dataframe and the fast cache.
    • The state is reset in finally in all three methods. get_weights runs after _end_call and passes both flags explicitly.
    • _decode is applied once, after the record ends.
  • Full core suite at the head: 3,940 passed, 4 failed, 4 skipped, 3 xfailed.
  • Full core suite on master 175e7eecdb: 3,876 passed, 4 failed, 4 skipped, 3 xfailed.
    • The same 4 tests fail in both runs. All are in tests/core/test_release_tagging.py, and they fail because the release script runs a system Python that has no tomllib. This is a local environment problem, and the PR does not cause it.
    • The head has 64 more passing tests. These are the new tests.
  • New and changed tests at the head, one file at a time:
    • test_microsimulation_internal_calls.py: 61 passed.
    • test_microsimulation_option_reads.py: 29 passed.
    • test_microsimulation_option_reads_property.py: 12 passed.
    • test_microsimulation_calculate_signatures.py: 24 passed.
  • Thread context on Python 3.14.2: test_microsimulation_internal_calls.py gave 61 passed with -X thread_inherit_context=1 (sys.flags.thread_inherit_context == 1) and 61 passed without it.
  • Revert test. I restored microsimulation.py from master and kept the PR's tests:
    • test_microsimulation_internal_calls.py: 44 failed, 17 passed.
    • test_microsimulation_option_reads.py: 1 failed.
    • test_microsimulation_calculate_signatures.py: 3 failed.
    • The property file still passed (12), as expected, because it leaves enums out.
  • Behavior probe on the #647 fixture, master compared with the PR:
    • Inside a formula, calculate changed from MicroSeries (sum 15600) to ndarray (sum 960).
    • An enum read inside a formula changed from names to EnumArray [0, 1, 0].
    • calculate_add inside a formula changed from MicroSeries (1560) to ndarray (96).
    • A branch calculate inside a formula changed from MicroSeries to ndarray.
    • A worker thread without a copied context still gets a MicroSeries. A worker thread with ctx.run now gets an ndarray.
    • Top-level calculate, calculate_add and the enum calculate are unchanged (weighted, names).
    • The #658 reads now match Simulation. ADD over a year changed from TypeError to [0, 1, 0]. DIVIDE over a year changed from names that compare as [False, False, False] to codes that compare as [True, False, True]. DIVIDE over a month changed from TypeError to [0.0, 0.0833, 0.0].
    • After an inner calculation, a top-level call is weighted again.
  • policyengine-us main 2574529521 with the PR's core installed:
    • Setup: the default populace dataset, subsample(n=300, seed=1), and 7 top-level outputs for 2026 (household_net_income, income_tax, state_income_tax, snap, tax_unit_itemizes, spm_unit_net_income, household_benefits).
    • The values and weights were identical to master. All 22,240 stored arrays had the same dtype and the same bytes.
    • get_weights ran 7 times with the PR and 35,843 times on master.
    • The time to calculate was 21.9 s with the PR and 25.3 s on master, from one run each.
  • policyengine-us callers. I searched for .calculate(, calculate_add, calculate_divide and calculate_dataframe outside the tests and found 22 formula call sites. Each one returns the result, uses it element by element, or passes it to map_result. None uses .sum(), .weights or .values.
    • TobCalculationMixin.calculate (tools/tob.py:39) runs its pre-calculations before the core call record starts, so they stay weighted, as on master.
  • Related issues and PRs: #658 (fixed by this PR), #659 (a separate unweighted= keyword issue, not touched here), #647 and #657. No other open PR changes microsimulation.py.
  • Merge and CI. The merge base equals the master tip 175e7eecdb, and GitHub reports the PR as MERGEABLE and CLEAN. All 19 checks passed at the head. The last check finished at 5:57am EST on 2026-11-17.
  • Docs. docs/python_api/simulations.md autodocs Microsimulation with :members:, so the new class and method docstrings appear there.

Not verified

  • policyengine-uk, Canada, Israel, Nigeria and Italy, and the policyengine package. I did not run them.
  • A full national dataset. I used only a 300-household subsample.
  • Timing on more than one run, or on a quiet machine.
  • A KeyboardInterrupt that arrives between _start_call() and the try that follows it. In theory it leaves the record marked as running. I did not reproduce it.

Read each column unweighted and decoded, so a frame built in a formula
matches one built at the top level, and weights are built once per frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The None defaults of use_weights and decode_enums are a new way to call
these methods, and the change in what formulas get from direct calls
warrants more than a patch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk merged commit 4122aab into master Nov 19, 2026
19 checks passed
@anth-volk
anth-volk deleted the unweighted-internal-calls branch November 19, 2026 19:31

@PavelMakarchuk PavelMakarchuk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE (reviewed at 986a83b874; the merged head afcc7fd9fd adds only a changelog fragment)

The new commit fixes the calculate_dataframe issue from the last review. A frame read in a formula now has enum names and weights, as on master, and it builds weights only once. The new tests fail without the fix, the full suite passes, and a policyengine-us run still gives bit-identical stored arrays.

Resolved since the last review

Item Status
Should address 1: calculate_dataframe in a formula returned enum codes, with no test Resolved. Microsimulation.calculate_dataframe now reads each column with use_weights=False, decode_enums=True (option A). The same probe now gives names in a formula: MicroDataFrame ['OWNER', 'RENTER', 'OWNER'], weighted sum 15600.0, weights [10, 10, 20]. df[df.yearly_tenure == "OWNER"] gives 2 rows again, as on master. The previous head gave [0, 1, 0] and 0 rows. The top-level result is unchanged. test_calculate_dataframe_in_a_formula_has_names_and_weights covers it.
Should address 1, changelog part Resolved. The .changed fragment no longer groups calculate_dataframe with the plain-array calls. It now says that calculate_dataframe "returns the same frame everywhere, building weights once". The probe confirms this.
Suggestion 1: minor or breaking version bump Resolved in afcc7fd9fd. The new .added fragment makes the release a minor version, and it describes the new None defaults.
Suggestion 2: shorten the long changelog sentence Resolved. The .changed fragment now has two sentences (49 words), not one sentence of about 75 words.

What I checked

  • Diff since 7a41e25494. I read all 4 changed files line by line.
    • The new calculate_dataframe copies the checks of Simulation.calculate_dataframe: period parsing, the unknown-variable ValueError, and the mapping of other entities to the first variable's entity (or to map_to). column_map_to matches the base logic.
    • Each column read has explicit use_weights=False and decode_enums=True, so the result does not depend on the running-call record. The weights still come from get_weights(variable_names[0], period, map_to), as before.
    • The method keeps no state of its own, so an exception in a column read leaves nothing to reset. Each calculate call still resets its record in finally.
  • Edge cases, master compared with the new head. These requests gave identical frames, weights and errors:
    • map_to="person" and map_to="household" with an enum column.
    • A household variable first, then a person enum column, over a year and over a month.
    • A monthly variable with an enum column, over a year and over a month.
    • The person-to-household enum cases raise TypeError: Cannot cast array data from dtype('<U7') to dtype('float64') on master and at the head. This error is older than the PR.
  • Weight builds per frame (3 columns: yearly, monthly, enum). With use_weights=True, get_weights ran 17 times on master and 1 time at the head. With use_weights=False, it ran 16 times on master and 0 times at the head.
  • Revert test. I restored microsimulation.py from 7a41e25494 and kept the new tests. test_microsimulation_internal_calls.py gave 3 failed, 63 passed. The failures are test_calculate_dataframe_in_a_formula_has_names_and_weights and both cases of test_calculate_dataframe_builds_weights_only_for_the_frame. The other two new tests pass on the old code, because they pin behavior that did not change (column mapping and the unknown-variable error).
  • PR test files at the head: 131 passed in the four Microsimulation test files.
  • Full core suite at the head: 3,945 passed, 4 failed, 4 skipped, 3 xfailed.
    • The last review had 3,940 passed. The 5 more are the new tests.
    • The 4 failures are the same test_release_tagging.py tests as before. They fail because the release script runs a local system Python that has no tomllib. The PR does not cause them.
    • A first parallel run also had one failure in test_branch_shared_arrays.py, from an HDF5 file lock (BlockingIOError: Errno 35). That file passed 3 times alone on master, the previous head and the new head, and it passed in the second full run. It is a local parallel-run problem.
  • policyengine-us main 2574529521, the default populace dataset, subsample(n=300, seed=1), 7 top-level outputs for 2026 (as in the last review):
    • Values and weights were identical to master. All 22,240 stored arrays had the same dtype and the same bytes.
    • get_weights ran 7 times at the head and 35,843 times on master.
    • policyengine-us has no calculate_dataframe call outside its tests.
  • Thread. No new comments or reviews since the last review.
  • Merge and CI. The merge base is still the master tip 175e7eecdb, and the PR merges without conflicts. GitHub reports MERGEABLE and CLEAN. All 19 checks passed at the head. The last check finished at 8:40am EST on 2026-11-18.

Not verified

  • The PR body says that a column read that failed only because no household_weight exists now computes with use_weights=False. I did not test a system without household_weight.
  • policyengine-uk, Canada, Israel, Nigeria and Italy, and the policyengine package. I did not run them.
  • A full national dataset. I used only a 300-household subsample.
  • Timing. I ran each case one time only, on a machine that was not quiet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enum reads with ADD or DIVIDE raise or return names in a Microsimulation

2 participants