Repository navigation
UK firm data: paper-parity port and policyengine-uk rules seam - #1706
Draft
vahid-ahmadi wants to merge 9 commits into
Draft
vahid-ahmadi wants to merge 9 commits into
vahid-ahmadi wants to merge 9 commits into
Conversation
Port sector-conditional employment, two-universe calibration with per-band registration propensity, stratified thinning with base weights and in_frame/vat_scope flags from firm-microsim-paper. Solve with the public solve_constraint_system (fixes #636) and use the paper's input-ratio imputation (fixes #258). Add the UKFirmRules seam keyed by firm_id, the firm universe helpers, and a policyengine-uk-named firm table export. Name policyengine-uk as the firm rules home in docs/firm-support-plan.md. Refs #1705, PolicyEngine/policyengine-uk#2939. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Commit the paper's processed 2024-25 tables with provenance and golden numbers from paper runs at aa1ce07 (full, fast and CI modes, seed 42). The engine-free test runs a thinned build and checks exact targets and propensities plus distributional tolerances; tools/firm_paper_parity.py prints the full comparison table. Refs #1705. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Share the HMRC band names between firm_universe and firm_generation, remove the unused assign_vat_flags and calculate_vat_liability_values, add UKFirmGenerationConfig.with_vintage, prove thinned solves start from base weights, feed a generated population through the policyengine-uk firm table, rename the parity tool to tools/uk_firm_paper_parity.py, and correct the changelog and docs. Refs #1705. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hash the reference JSON in the fixture provenance with the commands that produced it, assert the golden full run reproduces the paper's published 2024-25 headline, reuse the shared HMRC band names, and note the parity test in the firm support plan. Refs #1705. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
15 of 16 tasks
…ates Draw region (ONS UK Business 2025 Table 8) and legal status (Table 13) per firm after calibration, with Table 15 held out, and export them as policyengine-uk's firm_region, firm_in_england, firm_legal_form and firm_is_public_body. Record the status of all 52 policyengine-uk firm inputs, check the export against an ast snapshot of policyengine-uk's firm variables, and declare value-free M2 target candidates. The 8-target uk_firms_targets.json contract is unchanged. Refs #1705, PolicyEngine/policyengine-uk#2940. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refresh the policyengine-uk firm snapshot from a clean head (efeb6f6de) and refuse dirty trees by default; mark Table 3 pending (it is SIC x employment x region, not extracted); correct publication titles, the corporation tax band dimension (liability, not profit) and capital allowances (Corporation Tax statistics Table 12A); treat firm_in_england as assumed; reuse UK_REGION_TIER codes in the fixture test; tie Table 8 band edges to ONS_TURNOVER_BANDS; pin the attribute seed offset, the fallback levels and the unfed policyengine-uk inputs. Refs #1705. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tools/build_uk_firms.py runs generation, attribute draws and the policyengine-uk table export to uk_firms.parquet with a JSON receipt (no publish path, required --rules). PolicyEngineUKFirmRules runs policyengine-uk's firm entity on the candidate table and reads firm_vat_liability_if_registered; it imports policyengine-uk lazily and is tested engine-free with a fake simulation, and was checked locally against PolicyEngine/policyengine-uk#2940. Refs #1705. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compare receipts without the parquet hash and with floats rounded, so platform float reductions cannot fail the reproducibility test; use the 2024 rule period in the adapter tests and correct the tool docstring. Refs #1705. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #258, fixes #636. Refs #1705 and PolicyEngine/policyengine-uk#2939.
Draft, work in progress. This is the first step of moving UK firm data generation from
PolicyEngine/firm-microsim-paperinto microcosm, as the data side of a firm rules engine whose tax rules live in policyengine-uk. Axiom stays an optional cross-check.What moved
In
uk_runtime/firm_generation.py, the existing port now follows the paper:UKFirmGenerationConfig.fast()sets a window fraction of 0.30 and a tail fraction of 0.05.in_frame,vat_scope,vat_registeredare assigned after calibration by seeded weighted selection per band.solve_constraint_systemon a float32 CSR, never a dense matrix. On the small fixture the weights are bit-identical to the old_optimizepath, checked locally across three configs.0.2 + Beta(4,2)·0.6, clamped to [0.1, 0.95]. The 0.20 VAT rate stays in the rules engine.New modules:
uk_runtime/firm_rules.py: the rules seam.UKFirmRules.evaluate(firms, *, period)protocol returnsUKFirmRuleOutputskeyed byfirm_id.align_rule_outputsrefuses a missing, duplicate or extra id.LegacyVATEvaluatorRules.PolicyEngineUKFirmRules(added later in this PR; see below) readsfirm_vat_liability_if_registered. Reconcilingfirm_vat_registeredis still to come.uk_runtime/firm_universe.py: thinning, registration propensity, and scope and registration selection. Pure numpy.uk_runtime/firm_table.py: the firm table in policyengine-uk column names, in pounds.firm_turnover,firm_input_costs,firm_employees,firm_sic_section,firm_standard_rated_supplies,firm_exempt_supplies,firm_vat_voluntarily_registeredandfirm_input_vat.firm_input_vat = 0.20 × inputsis a named data assumption, to be replaced in M3.docs/firm-support-plan.mdnames policyengine-uk as the rules home, with Axiom as an optional cross-check.M2/M3 additions (overnight, 22 Nov)
uk_runtime/firm_attributes.py):tools/extract_uk_firm_ons_attribute_tables.py --check.firm_region,firm_in_england,firm_legal_formandfirm_is_public_body, and never falls back to policyengine-uk's defaults (LONDON,COMPANY). Three mappings are assumptions:OTHER, which takes them out of CT.firm_in_englandcomes from the enterprise's region, where policyengine-uk means the property's location.UK_FIRM_POLICYENGINE_UK_INPUT_STATUSrecords each of policyengine-uk's 52 formula-less firm inputs as set, derived, assumed or not set.tools/snapshot_policyengine_uk_firm_inputs.pytakes anastsnapshot of Add firm entity and firm-side tax rules policyengine-uk#2940 at the clean commitefeb6f6de. An engine-free test checks that every exported column is a formula-less policyengine-uk firm input with valid enum values.uk_runtime/firm_target_candidates.py): value-free declarations.uk_firms_targets.jsonand the solve are unchanged.Local build and policyengine-uk adapter (M1 build, M4 start)
tools/build_uk_firms.py(uk_runtime/firm_build.py) runs generation, attribute draws and the policyengine-uk export.uk_firms.parquetand a JSON receipt holding inputs and outputs with sha256s, the config, rule period, accuracy, target diagnostics, fallback counts and rules provenance.--rulesis required. There is no publish or staging path.PolicyEngineUKFirmRules(uk_runtime/firm_policyengine_uk_rules.py) builds a policyengine-uk firm dataset from the candidate table and readsfirm_vat_liability_if_registered.efeb6f6de, overlay environment, 295,271-row thinned 2024-25 table, about 6 s): the planned gate of within £1 per firm of 0.20 × supplies − input VAT failed literally.convert_to_fiscal_year_parameters, and by running pe-uk: £85k at 2023, £90k at 2024.Parity with the paper (vintage 2024-25, £90k threshold, seed 42)
Golden numbers come from paper runs at aa1ce07 (
generate(write=False)). The full run reproduces the paper's publishedresults/calibration_accuracy.txtheadline. Microcosm runs use a parity-only0.20 × (turnover − inputs)rules double that lives in the tool and the test, not insrc.Full build, local (
tools/uk_firm_paper_parity.py --full, 1,000 epochs, 500 s, 3.4 GB peak):All checks pass, and the per-band registration propensities equal the paper's to six decimals.
Thinned build, as run in CI (window 0.05, tail 0.01, 1,000 epochs, about 295k rows, about 30 s locally):
An earlier thinned run missed the net-VAT bar (0.938). The cause was 300 epochs against the paper's 1,000, not the loss.
CI runs
test_uk_firm_paper_parity.py(engine-free) on the committed fixture: the paper's six processed 2024-25 tables, with sha256 provenance, andpaper_reference_2024_25.json.Left for later
firm_vat_registered; run it in CI once pe-uk releases the firm entity.populace-uk-firmsrelease.Frame(EntitySchemaneeds person and group entities);source_stages.jsonbuild stage (a local build tool exists instead);Decisions (made by a Fable 5.2 planning agent while Vahid was away)
test_uk_firms_targets.pypins the 8-target contract.solve_constraint_system, with no change tocalibrate/solve.py, which would move spec pins.fastpreset. CI always thins.🤖 Generated with Claude Code