Repository navigation
Return plain arrays to Microsimulation calls made inside a calculation - #661
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ile its own call runs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runs Fix tests that did not reach the fast cache or the rerun after an input change, wait for the worker thread's call to start, and narrow the changelog fragments to what changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rectly Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PavelMakarchuk
left a comment
There was a problem hiding this comment.
Verdict: REQUEST_CHANGES (head 7a41e25494)
The detection logic is correct in every case I tried. The new tests fail when the source change is reverted, and a policyengine-us run gives bit-identical stored arrays with about 5,000 times fewer weight builds. One documented side effect has no test, and the changelog describes it incorrectly: calculate_dataframe in a formula now returns a weighted frame with enum codes.
Critical
None.
Should address
calculate_dataframein a formula returns a mixed result, and no test covers it.Microsimulation.calculate_dataframekeepsuse_weights: bool = True(policyengine_core/simulations/microsimulation.py:263) and does not record a running call. Its columns come fromself.calculate(variable_name, period, map_to)(policyengine_core/simulations/simulation.py:1677).- In a formula, these column reads now count as inside, so enum columns hold codes. The frame still gets weights (line 269 onward).
- A probe formula that calls
calculate_dataframe(["yearly_grant", "yearly_tenure"], period)on the #647 fixture gave this result:master inside formula: MicroDataFrame ['OWNER', 'RENTER', 'OWNER'] weighted sum 15600.0 PR inside formula: MicroDataFrame [0, 1, 0] weighted sum 15600.0 PR top level: MicroDataFrame ['OWNER', 'RENTER', 'OWNER'] weighted sum 15600.0 - The result is weighted like a top-level call, but it has codes like an internal call. A filter such as
df[df.yearly_tenure == "OWNER"]now silently matches no rows. This is the same class of error that #658 fixes. - The
.changedfragment groupscalculate_dataframewith the calls that "get a plain NumPy array with enum codes instead of a weightedMicroSeries". The frame is not plain. It is a weightedMicroDataFrame. - No test in
tests/core/test_microsimulation_internal_calls.pycallscalculate_dataframefrom a formula. Only the top-level case is tested (line 270). - Fix, option A: in
Microsimulation.calculate_dataframe, build each column withself.calculate(name, period, map_to, use_weights=False, decode_enums=True)instead of calling the base method. The frame then has names everywhere, and the per-column weights go away too. - Fix, option B: keep the behavior. Add a test that pins it, and reword the changelog so that it says what a formula gets from
calculate_dataframe.
Suggestions
-
Consider a minor or breaking version bump. Both fragments are
changedandfixed, so this release is a patch. The PR changes public defaults:- A formula's direct
simulation.calculate(...)now returns anndarray, not aMicroSeries. In my probe,.sum()changed from 15600.0 (weighted) to 960.0. - A top-level
calculate_divide(enum_variable, year)now returns codes[0, 1, 0], not names. calculate_add(..., use_weights=False)now returns a plain array..sum()changed from 1560.0 to 96.0.
The consumer review in the PR body is thorough, but code outside the PolicyEngine organization can depend on these defaults. Rename the
.changedfragment to.breaking, or add an.addedfragment if the maintainers want a minor bump. - A formula's direct
-
Make the long changelog sentence shorter. The
.changedfragment is one sentence of about 75 words. Split it into two or three sentences: one for internal calls, and one for direct formula calls.
What I checked
- Diff. I read all 8 files and the functions they call in
simulation.py, for examplecalculate,calculate_add,calculate_divide,_run_formula(addsandsubtracts),calculate_dataframeand the fast cache.- The state is reset in
finallyin all three methods.get_weightsruns after_end_calland passes both flags explicitly. _decodeis applied once, after the record ends.
- The state is reset in
- Full core suite at the head: 3,940 passed, 4 failed, 4 skipped, 3 xfailed.
- Full core suite on master
175e7eecdb: 3,876 passed, 4 failed, 4 skipped, 3 xfailed.- The same 4 tests fail in both runs. All are in
tests/core/test_release_tagging.py, and they fail because the release script runs a system Python that has notomllib. This is a local environment problem, and the PR does not cause it. - The head has 64 more passing tests. These are the new tests.
- The same 4 tests fail in both runs. All are in
- New and changed tests at the head, one file at a time:
test_microsimulation_internal_calls.py: 61 passed.test_microsimulation_option_reads.py: 29 passed.test_microsimulation_option_reads_property.py: 12 passed.test_microsimulation_calculate_signatures.py: 24 passed.
- Thread context on Python 3.14.2:
test_microsimulation_internal_calls.pygave 61 passed with-X thread_inherit_context=1(sys.flags.thread_inherit_context == 1) and 61 passed without it. - Revert test. I restored
microsimulation.pyfrom master and kept the PR's tests:test_microsimulation_internal_calls.py: 44 failed, 17 passed.test_microsimulation_option_reads.py: 1 failed.test_microsimulation_calculate_signatures.py: 3 failed.- The property file still passed (12), as expected, because it leaves enums out.
- Behavior probe on the #647 fixture, master compared with the PR:
- Inside a formula,
calculatechanged fromMicroSeries(sum 15600) tondarray(sum 960). - An enum read inside a formula changed from names to
EnumArray [0, 1, 0]. calculate_addinside a formula changed fromMicroSeries(1560) tondarray(96).- A branch
calculateinside a formula changed fromMicroSeriestondarray. - A worker thread without a copied context still gets a
MicroSeries. A worker thread withctx.runnow gets anndarray. - Top-level
calculate,calculate_addand the enumcalculateare unchanged (weighted, names). - The #658 reads now match
Simulation. ADD over a year changed fromTypeErrorto[0, 1, 0]. DIVIDE over a year changed from names that compare as[False, False, False]to codes that compare as[True, False, True]. DIVIDE over a month changed fromTypeErrorto[0.0, 0.0833, 0.0]. - After an inner calculation, a top-level call is weighted again.
- Inside a formula,
- policyengine-us
main2574529521with the PR's core installed:- Setup: the default populace dataset,
subsample(n=300, seed=1), and 7 top-level outputs for 2026 (household_net_income,income_tax,state_income_tax,snap,tax_unit_itemizes,spm_unit_net_income,household_benefits). - The values and weights were identical to master. All 22,240 stored arrays had the same dtype and the same bytes.
get_weightsran 7 times with the PR and 35,843 times on master.- The time to calculate was 21.9 s with the PR and 25.3 s on master, from one run each.
- Setup: the default populace dataset,
- policyengine-us callers. I searched for
.calculate(,calculate_add,calculate_divideandcalculate_dataframeoutside the tests and found 22 formula call sites. Each one returns the result, uses it element by element, or passes it tomap_result. None uses.sum(),.weightsor.values.TobCalculationMixin.calculate(tools/tob.py:39) runs its pre-calculations before the core call record starts, so they stay weighted, as on master.
- Related issues and PRs: #658 (fixed by this PR), #659 (a separate
unweighted=keyword issue, not touched here), #647 and #657. No other open PR changesmicrosimulation.py. - Merge and CI. The merge base equals the master tip
175e7eecdb, and GitHub reports the PR asMERGEABLEandCLEAN. All 19 checks passed at the head. The last check finished at 5:57am EST on 2026-11-17. - Docs.
docs/python_api/simulations.mdautodocsMicrosimulationwith:members:, so the new class and method docstrings appear there.
Not verified
- policyengine-uk, Canada, Israel, Nigeria and Italy, and the
policyenginepackage. I did not run them. - A full national dataset. I used only a 300-household subsample.
- Timing on more than one run, or on a quiet machine.
- A
KeyboardInterruptthat arrives between_start_call()and thetrythat follows it. In theory it leaves the record marked as running. I did not reproduce it.
Read each column unweighted and decoded, so a frame built in a formula matches one built at the top level, and weights are built once per frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The None defaults of use_weights and decode_enums are a new way to call these methods, and the change in what formulas get from direct calls warrants more than a patch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PavelMakarchuk
left a comment
There was a problem hiding this comment.
Verdict: APPROVE (reviewed at 986a83b874; the merged head afcc7fd9fd adds only a changelog fragment)
The new commit fixes the calculate_dataframe issue from the last review. A frame read in a formula now has enum names and weights, as on master, and it builds weights only once. The new tests fail without the fix, the full suite passes, and a policyengine-us run still gives bit-identical stored arrays.
Resolved since the last review
| Item | Status |
|---|---|
Should address 1: calculate_dataframe in a formula returned enum codes, with no test |
Resolved. Microsimulation.calculate_dataframe now reads each column with use_weights=False, decode_enums=True (option A). The same probe now gives names in a formula: MicroDataFrame ['OWNER', 'RENTER', 'OWNER'], weighted sum 15600.0, weights [10, 10, 20]. df[df.yearly_tenure == "OWNER"] gives 2 rows again, as on master. The previous head gave [0, 1, 0] and 0 rows. The top-level result is unchanged. test_calculate_dataframe_in_a_formula_has_names_and_weights covers it. |
| Should address 1, changelog part | Resolved. The .changed fragment no longer groups calculate_dataframe with the plain-array calls. It now says that calculate_dataframe "returns the same frame everywhere, building weights once". The probe confirms this. |
| Suggestion 1: minor or breaking version bump | Resolved in afcc7fd9fd. The new .added fragment makes the release a minor version, and it describes the new None defaults. |
| Suggestion 2: shorten the long changelog sentence | Resolved. The .changed fragment now has two sentences (49 words), not one sentence of about 75 words. |
What I checked
- Diff since
7a41e25494. I read all 4 changed files line by line.- The new
calculate_dataframecopies the checks ofSimulation.calculate_dataframe: period parsing, the unknown-variableValueError, and the mapping of other entities to the first variable's entity (or tomap_to).column_map_tomatches the base logic. - Each column read has explicit
use_weights=Falseanddecode_enums=True, so the result does not depend on the running-call record. The weights still come fromget_weights(variable_names[0], period, map_to), as before. - The method keeps no state of its own, so an exception in a column read leaves nothing to reset. Each
calculatecall still resets its record infinally.
- The new
- Edge cases, master compared with the new head. These requests gave identical frames, weights and errors:
map_to="person"andmap_to="household"with an enum column.- A household variable first, then a person enum column, over a year and over a month.
- A monthly variable with an enum column, over a year and over a month.
- The person-to-household enum cases raise
TypeError: Cannot cast array data from dtype('<U7') to dtype('float64')on master and at the head. This error is older than the PR.
- Weight builds per frame (3 columns: yearly, monthly, enum). With
use_weights=True,get_weightsran 17 times on master and 1 time at the head. Withuse_weights=False, it ran 16 times on master and 0 times at the head. - Revert test. I restored
microsimulation.pyfrom7a41e25494and kept the new tests.test_microsimulation_internal_calls.pygave 3 failed, 63 passed. The failures aretest_calculate_dataframe_in_a_formula_has_names_and_weightsand both cases oftest_calculate_dataframe_builds_weights_only_for_the_frame. The other two new tests pass on the old code, because they pin behavior that did not change (column mapping and the unknown-variable error). - PR test files at the head: 131 passed in the four Microsimulation test files.
- Full core suite at the head: 3,945 passed, 4 failed, 4 skipped, 3 xfailed.
- The last review had 3,940 passed. The 5 more are the new tests.
- The 4 failures are the same
test_release_tagging.pytests as before. They fail because the release script runs a local system Python that has notomllib. The PR does not cause them. - A first parallel run also had one failure in
test_branch_shared_arrays.py, from an HDF5 file lock (BlockingIOError: Errno 35). That file passed 3 times alone on master, the previous head and the new head, and it passed in the second full run. It is a local parallel-run problem.
- policyengine-us
main2574529521, the default populace dataset,subsample(n=300, seed=1), 7 top-level outputs for 2026 (as in the last review):- Values and weights were identical to master. All 22,240 stored arrays had the same dtype and the same bytes.
get_weightsran 7 times at the head and 35,843 times on master.- policyengine-us has no
calculate_dataframecall outside its tests.
- Thread. No new comments or reviews since the last review.
- Merge and CI. The merge base is still the master tip
175e7eecdb, and the PR merges without conflicts. GitHub reportsMERGEABLEandCLEAN. All 19 checks passed at the head. The last check finished at 8:40am EST on 2026-11-18.
Not verified
- The PR body says that a column read that failed only because no
household_weightexists now computes withuse_weights=False. I did not test a system withouthousehold_weight. - policyengine-uk, Canada, Israel, Nigeria and Italy, and the
policyenginepackage. I did not run them. - A full national dataset. I used only a 300-household subsample.
- Timing. I ran each case one time only, on a machine that was not quiet.
make format && make documentationhas been run. (ruff formatandruff checkwere run on the changed files only; the documentation was not built, since no documentation page changed.)Bug fix
Fixes #658.
What changes
Microsimulation.calculate,calculate_addandcalculate_dividereturn a weightedMicroSeries, andcalculatedecodes enums to member names. Core also calls these methods for its own reads while it calculates: each term ofaddsandsubtracts, adefined_formask, each month of a monthly variable read over a year, the year of a yearly variable read over a month, and the terms of ADD and DIVIDE. Each such call computedhousehold_weight, projected it to the variable's entity and built aMicroSeries, and core then discarded the weights before storing the value (_cast_formula_result). Decoding broke enum reads with ADD and DIVIDE (#658).This PR changes only
policyengine_core/simulations/microsimulation.py:ContextVarholds a record of the innermostMicrosimulationcalculate,calculate_addorcalculate_dividecall running in the current context; each record links to the one for the call that encloses it. A call that arrives while one of these calls is running is a call from inside a calculation. When a call ends, its record is marked as finished. A context copied during a call (for a worker thread, say) is therefore inside while that call, or one enclosing it, is still running, or while a call of its own runs there, and outside afterwards. A new thread starts with an empty context, so its calls count as outside (unless Python runs with-X thread_inherit_context=1, the default on free-threaded builds, which gives the thread a copy). Core's_calculation_framesand the tracer stack are not used.use_weights=Noneanddecode_enums=None.Nonegives a plain NumPy array with enums left as codes, as aSimulationreturns.Nonegives the previous defaults:calculateis weighted and decoded;calculate_addandcalculate_divideare weighted and not decoded (Fix formula reads with ADD or DIVIDE in a Microsimulation #647).decode_enums=True. A twelfth of a code from a month DIVIDE is returned undecoded, as aSimulationreturns it.get_weightsreadshousehold_weightwithdecode_enums=False.simulation.py's call sites are unchanged. Calling the base class from there would skip subclass overrides ofcalculate.Effect
One weighted series is built per top-level call. Before this change, for example, a top-level
calculateof a yearly variable with a monthlyaddsterm built 13; a 500-household policyengine-ushousehold_net_incomebuilt 35,368. On 100 policyengine-us households (warmhousehold_net_income, 3 fresh processes), a variant of this change that detected internal calls through_calculation_framestook 3.57 s, against 6.84 s on master. It classifies every consumer call the same way as this change; this change's own timing was not measured on that setup. On 500 households, an earlier prototype of this change's detection took the baseline from 10.2 s to 6.3 s, measured on a loaded machine.Checked against every consumer
Each consumer was run on master 175e7ee and with this change (or a variant that classifies every consumer call the same way), comparing every stored array's values, dtype, writability and keys, and every top-level value and weight. Consumers checked: policyengine-us 2.144.1, policyengine-uk 2.179.1, policyengine-canada 0.99.2, policyengine-il 0.2.2, policyengine-ng 0.6.2, policyengine-italy 0.16.3, and .py (
policyenginepackage) atmain47d1d8bf (code read, not run)._calculation_framesvariant, which in turn differed from master only as listed below. Runs covered a targeted 1,140-household policyengine-us file for 2024 and 2027, current law and a CTC reform; 121 policyengine-us YAML cases (402 outputs) replayed inside aMicrosimulation, 42 more at 2027, and the WI WHEAP file; and one synthetic household each in Canada, Israel, Nigeria and Italy. Separately, 500 policyengine-us households gave 508,321 stored arrays bit-identical to master.Microsimulation: it does not subclass it, and a counter recorded 0 calls.calculatecalls in consumer formulas. 22 calls in 20 policyengine-us formula files callcalculateon a simulation, branch or baseline. Examples:tax_liability_if_itemizing/_not_itemizing,tax_unit_itemizes, the NY and NYC EITC, the VA, DE, HI, ID and MO pairs, claim of right,co_adult_financial_medicaid_enrolled,medicaid_enrolled_for_ssi_state_supplement,spm_unit_allocated_housing_subsidyandmarginal_tax_rate_helpers. They now get anndarrayrather than aMicroSeries. Each reads a float or bool variable and uses the result element by element, returns it, or passes it tomap_resultorset_input. None uses.values,.weights,.sum()or pandas indexing on the result, and every one ran in aMicrosimulationin the comparisons above.wi_wheap_categorically_eligible(policyengine-us) raised microdf's "Cannot propagate weights unambiguously" in aMicrosimulationon master (8 of 8 YAML cases); all 8 now pass.household_weight, so most internal reads raised there. Now 294 of 405 calls compute in Canada, 27 of 40 in Israel, 27 of 36 in Nigeria and 38 of 60 in Italy. Weighted calls from outside a calculation still raise there.adds.household_weightarrays are no longer calculated or copied, because a formula'sbranch.calculateis no longer weighted. Where they still exist, the values are identical.adds/subtractslist, adefined_foror a period-converted read that involves an Enum variable, and no consumer formula reads withoptions=[ADD]or[DIVIDE], so no stored enum code changes. The Enum reads with ADD or DIVIDE raise or return names in a Microsimulation #658 reads now matchSimulation. One top-level result changes:calculate_divide(enum_variable, year)left atdecode_enums=Nonenow returns codes, as Fix formula reads with ADD or DIVIDE in a Microsimulation #647'sFalsedefault intends; master returned names. No consumer makes that call.Not covered by those checks: full national datasets; .py at runtime; this change's exact detection on policyengine-uk (which cannot reach it) and its timing;
Simulation.calculate_outputhas no caller in any consumer.Other behaviour to note
MicroSerieswith member names. A formula that took.sum()of a directsimulation.calculate(...)read, compared a direct enum read with a member name, or used.valuesor.weightson it would now get a different result or an error. The consumer check above found no such code; policyengine-us's direct calls return the result or use it element by element. The same applies to the inner reads ofto_input_dataframeandderivativewhen they are called inside a formula.contextvarscontexts, not threads. A call run through a context captured before the calculation started (for exampleContext.runon a context copied at import time, or anasyncio.Runnerfirst used outside any calculation) counts as outside, even in the same thread, and gets weighted values, as on master.use_weights=Falsefrom outside a calculation is now honoured everywhere. On master, withoutmap_to,calculate_add,calculate_divideand acalculateof a FLOW variable over a period of another length (a monthly variable over a year, a yearly one over a month) returned a weightedMicroSerieseven withuse_weights=False, because their inner reads were weighted. They now return a plain array, so.sum()is the unweighted sum (for example 624 rather than 10,140 in the new tests' fixture). A review of policyengine-us, policyengine-uk, .py and the Public API found no caller that depends on the old result.Simulationreturns, including the stored array itself. On master, some of these reads in aMicrosimulationreturned a copy as a side effect of building aMicroSeries: a period-converted read, a read withoptions=[ADD]or[DIVIDE]over a year, and a formula's directcalculate. Values stored from period conversion (ADD from months to a year, DIVIDE from a year to a month) were stored as read-only arrays built from a pandas Series on master; they are now stored writable, with the same bytes. A formula that writes in place into such a read now changes the stored value, as it already does in aSimulationand for plain same-period reads. A scan of policyengine-us found no formula that writes into these reads.use_weights=Noneordecode_enums=Noneexplicitly used to mean unweighted and not decoded; it now means the default for the caller (weighted and, forcalculate, decoded from outside a calculation). No caller was found that passesNone.calculate_dataframereturns the same frame wherever it is called from: weighted unlessuse_weights=False, with enum member names. It now reads each column unweighted and decoded, so weights are built once per frame rather than once per column plus once for the frame (33 builds on master for 3 columns in the new tests' fixture). For 18 requests covering mixed entities,map_to, enum columns, period conversion anduse_weights=False, the frames and error messages were identical to master. A column read that raised only because weights could not be built (nohousehold_weight) now computes whenuse_weights=False.Microsimulationused inside a formula of a plainSimulation(not aMicrosimulation) still gets weighted results, as on master: only a runningMicrosimulationcall marks a call as internal. No consumer does this.Tests
New
tests/core/test_microsimulation_internal_calls.py, with a fixture system intests/fixtures/microsimulation_internal_calls.py:calculate, fast-cache re-read,calculate_add, year and monthcalculate_divide), trace off and on; a branch, trace off and on; a reform's baseline; aMicrosimulationbuilt in aMicrosimulation's formula and in a plainSimulation's formula; a worker thread with and without a copied context; a context copied in a formula and used after the calculation ends; a call in a copied context that keeps running after the call it was copied from has ended; a context copied during a nested call and used while an enclosing call still runs; the fast-cache re-read; explicit arguments inside and at the top level;calculate_dataframecalled from a formula keeps enum names and weights, builds weights only for the frame, maps columns to the first variable's entity, and rejects unknown variables;calculate,calculate_add,calculate_divide,calculate_outputandcalculate_dataframeare still weighted, and each top-level call builds exactly one weighted series (foradds/subtracts,defined_for, a monthly variable over a year, a yearly variable over a month, ADD and DIVIDE), withhousehold_weightcalculated once;RuntimeError, aBaseException, trace on), an ADD or DIVIDE term raises, a formula catches a branch's error, or a branch input set during a calculation makes core run it a second time, calls from outside a calculation are weighted again;Microsimulationstores and returns what aSimulationdoes: same dtype, bytes and writability for every stored array;Simulation, and top-level enumcalculate_add/calculate_dividedecode when asked.test_enum_read_with_divide_over_a_year_gives_member_names_in_a_microsimulationrecorded the #658 bug; it now checks that the codes match aSimulation.test_microsimulation_calculate_signatures.pychecks the newNonedefaults.Run: each new and modified test, selected individually by node ID (all pass; the thread and copied-context tests also with
-X thread_inherit_context=1), andruff format/ruff checkon the changed files. The rest of the suite was not run locally; CI runs it.🤖 Generated with Claude Code