Skip to content

Add firm entity and firm-side tax rules - #2940

Draft
vahid-ahmadi wants to merge 9 commits into
mainfrom
firm-entity-engine
Draft

vahid-ahmadi wants to merge 9 commits into
mainfrom
firm-entity-engine

Conversation

@vahid-ahmadi

@vahid-ahmadi vahid-ahmadi commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2939

What and why

Adds a firm entity to policyengine-uk beside person, benefit unit and household, and firm-level rules for the main business taxes. Today the model only spreads national totals of firm taxes over households (business rates, corporate SDLT, corporate_incident_tax_change). Corporation tax and the apprenticeship levy are listed under uncovered_receipts, and the Employment Allowance is not modelled.

Part of #1320 (firm-level VAT and a Firm entity): this PR adds the entity and firm-side VAT, but not the sector-driven supply split or a calibrated firm dataset, so #1320 stays open.

The PR is a draft and still in progress. The design notes and the decisions behind them are below. The firm dataset itself is being built in microcosm (PolicyEngine/microcosm#1706).

Entity design

  • Firm is a root population, not a group of persons. Firms have their own ids and their own weights (firm_weight), which come from business data rather than FRS households. No person is a member of a firm.
    • Existing household situations and datasets work unchanged.
    • A situation or dataset without firms has a firm count of 0.
  • How Firm is built. It is a plain core Entity listed after Person ([Person, BenUnit, Household, Firm]), so core treats it as neither the person entity nor a group entity.
    • A guarded replacement of core TaxBenefitSystem.instantiate_entities (in tax_benefit_system.py) gives it a RootPopulation. This also covers core Reform systems built by the YAML runner.
    • RootPopulation raises on cross-entity shortcuts such as firm.household.
  • Simulation changes.
    • The firm population is kept through _bind_to_tax_benefit_system (reforms and baselines), clone and get_branch.
    • map_result raises when asked to map firm values to or from another entity.
    • Firm variables are left out of the per-person exports.
    • build_from_situation accepts an optional firms: {name: {variable: value}} block.
    • build_from_ids takes an optional firm_id.
    • Both dataset loaders read an optional firm table.
  • Dataset schema. UKSingleYearDataset and UKMultiYearDataset take an optional firm= table. tables stays three long, so uprating and subsampling are unchanged. subsample raises NotImplementedError when firms are present.
  • Multiple jobs per person are left for a later job-link table. Without it, firm employee data is held at firm level (paybill, employees, zero-rated earnings).

Programs

All new variables use entity = Firm. Every parameter has a legislation or GOV.UK reference. Values that have been announced but not yet enacted are marked in their YAML files.

Program Variables Notes
Employer NICs firm_employer_ni, firm_employer_ni_before_allowance Reuses the gov.hmrc.national_insurance.class_1 rate and threshold nodes. Earnings above the secondary threshold can be given as an input; by default they are approximated from paybill and employees.
Employment Allowance employment_allowance, employment_allowance_eligible Covers the amount history from 2014 to £10,500, the single-director exclusion, the public-body exclusion and the £100k prior-year limit for 2020-21 to 2024-25.
Apprenticeship levy apprenticeship_levy 0.5% of paybill less the £15,000 allowance, with a connected-group share.
Auto-enrolment firm_auto_enrolment_employer_contribution 3% of qualifying earnings (£6,240–£50,270).
Corporation tax corporation_tax, firm_taxable_profit, firm_marginal_relief, firm_losses_used, … Main and small profits rates, marginal relief (3/200) with the associated-companies divisor, and loss relief with the £5m deductions allowance. Only companies are liable.
Capital allowances firm_capital_allowances + components Full expensing, 50% special-rate FYA, AIA, new main-rate FYA (40% from 1 January 2026), WDAs (main 18% → 14%), SBA, and balancing charges.
R&D relief firm_rd_expenditure_credit, firm_rd_additional_deduction, firm_rd_payable_credit Merged-scheme RDEC (20%) and ERIS for R&D-intensive loss-making SMEs.
Bank surcharge bank_surcharge 8% above £25m, then 3% above £100m. The announced 6% (1 January 2027 to 31 March 2028) is marked as not yet enacted.
Business rates (England) firm_business_rates + reliefs 2023-24 to 2027-28 multipliers, including the 2026 RHL and high-value multipliers. Small business rate relief (RHL £18k/£21k from 2027), charitable relief, RHL relief to 2025-26, pubs and live music relief, and empty property relief.
VAT (firm side) firm_vat, firm_vat_registered, firm_output_vat, … Registration threshold history (£85k, then £90k), output less recoverable input VAT with partial exemption, and the flat rate scheme with the limited cost trader rate (16.5%). Reuses the existing VAT rate nodes.

Aggregates and incidence (not wired into totals)

  • firm_tax: total statutory liability, less R&D tax credits paid out to the firm (RDEC in excess of corporation tax, and the ERIS payable credit).
  • firm_only_tax: the liability set by rules only the firm side models: CT, bank surcharge, levy and business rates, less R&D credits paid out. Employer NIC and VAT rates are left out, because they already reach households through ni_employer and vat. The Employment Allowance is left out too, because it moves with employer NIC liability.
  • firm_tax_change_total: the reform less baseline change in firm_only_tax, weighted by firm_weight, less the change in the Employment Allowance from its own rules (amount and eligibility). Each firm's employer NIC liabilities are held at their baseline values: this year's, and last year's for the 2020-21 to 2024-25 prior-year limit. Without this, an employer NIC rate reform would count twice; tests cover a 2026 rate reform and a 2024 eligibility flip.
  • firm_tax_incidence: splits firm_tax_change_total between households by capital, consumption and labour shares. The shares come from new gov.contrib.policyengine.firm_incidence parameters, which default to capital 1.
  • Not wired into totals yet. None of these is in GOV_TAX_VARIABLES, household_tax, programs.yaml budget lists or fiscal_partition.yaml. That waits for a calibrated firm dataset from microcosm. Household outputs are therefore unchanged by construction. A regression gate checks that adding firms, or a firm-only reform, leaves household results exactly unchanged.

Downstream changes needed before release

Core marks firm as is_person=True, because a plain Entity is the only non-group entity core accepts. Downstream code that reads is_person sees two person entities. Read-only research found these consequences:

  • policyengine.py. tests/test_model_entities.py:90 asserts that the person entities are exactly ["person"], so the UK release certification run would fail. The fix is in src/policyengine/tax_benefit_models/common/model_version.py:213: derive is_person by comparing against the system's person entity key, then update the pinned entities.
  • policyengine-api.
    • policyengine_api/data/v2/catalog/extraction.py:281 raises unless there is exactly one person entity.
    • policyengine_api/country.py:413 reports is_person: true for firm in v1 metadata.
    • The fix is the same key comparison.
  • app-v2. VariableResolver.ts:55 would offer firm inputs as person inputs. This PR sets hidden_input = True on all 58 firm inputs, so they stay out of the household builder until a firm builder exists.

Household calculations without firms work unchanged in the API. These downstream PRs are not yet opened: they wait for a maintainer's go-ahead. Patches are prepared locally for policyengine.py (key-based is_person and entity pins; 76 tests pass on the pinned UK version and on this branch) and policyengine-api (v1 metadata). This PR should not be released until the policyengine.py fix is released.

Compatibility with policyengine-core

  • The PR patches core TaxBenefitSystem.instantiate_entities and Simulation.clone so they keep root populations. Both patches are guarded and idempotent. The clone patch is what lets the YAML runner, and formulas that clone the simulation (such as marginal tax rates), see firms.
  • It also imports four core private names.
  • Tested on core 3.34.9 (locked) and 3.37.1 (latest) with no failures, so no upper bound is set.
  • A draft issue proposing a first-class root entity in core is ready but not filed.

Known simplifications

  • Every firm uses an April–March financial year.
  • Yearly parameters are read at 30 April, so changes dated 1 January are not apportioned. This affects the 40%/50% main-rate FYA and the 6% surcharge for January–March 2027.
  • Employer NICs and auto-enrolment use firm-level approximations, since there is no per-employee pay yet. Optional inputs take out staff below the secondary threshold, and staff below the auto-enrolment trigger or above the upper limit; without them, the default assumes average pay. The default also does not apply the auto-enrolment £10,000 trigger.
  • Business rates:
    • one aggregate property per firm
    • England only
    • no transitional relief
  • Corporation tax:
    • no group relief
    • no Patent Box
    • no quarterly instalment payments
  • VAT:
    • registration is tested on annual turnover, not a rolling 12 months
    • no cash or annual accounting
  • Freeport employer NIC relief goes in through firm_zero_rated_class_1_earnings. firm_in_freeport is recorded but no rule reads it yet. Freeport capital allowances and business rates relief are not modelled.
  • In a firms-only simulation, household variables cannot be computed (there are 0 benefit units).
  • Firm tables are not uprated between years. A multi-year dataset must carry the same firms in every year, or loading raises an error.
  • The labour incidence share is normalised over the simulated households. The default labour share is 0.
  • Business rates parameters start in 2023-24. Earlier years reuse the 2023-24 values.
  • The bank surcharge allowance is not shared across a group.
  • Float32 storage. Core stores every float variable as float32, so firm money values for the largest firms carry rounding: about £1 on £10m–£50m of turnover, against a float64 calculation by microcosm's adapter. firm_id comes back as int32.
  • Announced values not yet in law. Finance Bill 2026-27 had not received Royal Assent on 22 Nov 2026, so the 12% main WDA (April 2027), the 50% main-rate FYA (January 2027) and the 6% bank surcharge (1 Jan 2027 to 31 Mar 2028) are announced values. Each is marked in its YAML. Whether to keep them in the baseline or move them to a reform is an open decision.

Test plan

  • For policy changes: YAML tests with values hand-computed from legislation and GOV.UK guidance, under policyengine_uk/tests/policy/baseline/gov/hmrc/{corporation_tax,bank_surcharge,apprenticeship_levy,vat/firm,business_rates/firm,national_insurance/employer,national_insurance/employment_allowance} and .../gov/dwp/auto_enrolment.
  • test_firm_entity.py, test_firm_incidence.py, test_supplied_inputs.py: 119 passed.
  • Focused existing tests: income tax and pension credit YAML, test_entity_ids, test_input_dataframe_export, test_dataframe_dataset, test_build_metadata, test_variable_units, test_parameter_metadata, test_subsample, test_household_subset, test_supplied_inputs, and the code_health tests.
  • Household separation (test_firm_regression_gate.py, 34 passed):
    • No firm variable is a GOV_TAX_VARIABLES member or a budget term.
    • For 8 household situations in 2025 and 2026, adding firms leaves every household and government total exactly unchanged, and so does a firm-only reform.
    • Before review, a frozen fixture also matched base 70014b3 exactly (35 passed). It was dropped so that later changes to household rules don't break this test.
  • Incidence and aggregate tests (test_firm_incidence.py), the firms: guard for YAML (test_firm_yaml_guard.py), and 3 integration cases (tests/policy/integration/firm/).
  • Full YAML suite locally at 1b320d8: 6,946 passed, 0 failed.
  • Firm, registry and code_health Python tests locally at 42939b5: 4,284 passed.
  • Toy microsimulation (test_firm_microsimulation.py): a seeded 200-firm frame checks weighted CT, levy, employer NIC, business rates and VAT against an independent numpy calculation. It also checks the firm_tax identity, that every firm variable is finite, and that household totals are unchanged whether or not the dataset has a firm table. The totals are toy numbers, not estimates.
  • Microcosm contract (test_firm_microcosm_contract.py): checks the 10 firm columns and outputs the microcosm frame (UK firm data: paper-parity port and policyengine-uk rules seam microcosm#1706) reads or writes.
  • Worked reform examples page (docs/book/usage/firm-reforms.md): every figure on it is asserted in test_firm_docs_examples.py.
  • Parameter values checked independently against legislation.gov.uk, GOV.UK and HMRC manuals (about 150 of 173 dated values fetched and confirmed). No UNVERIFIED notes remain.
  • Firm tests, marginal-rate YAMLs, income tax and UC YAMLs, and the export, subsample and branch tests also pass on policyengine-core 3.37.1.
  • programs.yaml registers 8 firm programmes as partial (test_program_registry and the budget tests pass). New docs page: docs/book/usage/firms.md.
  • Full Python suite in CI. At 42939b5, every job passed except shards 2/8 and 7/8. Both were cancelled by the 90-minute limit at 99%, with no test failing. On main, the same shards took 82 to 90 minutes in the last successful run, and a local benchmark shows no slowdown in household simulations from this branch. Rerunning at 1b320d8.
  • make format (ruff format + ruff check).
  • Changelog fragment changelog.d/firm-entity-engine.added.md.

Reproducer

Reproducer: policyengine_uk/tests/test_firm_entity.py

axiom: needed (new firm-side programmes. The rulespec-uk parity issues are not yet opened and are left for a maintainer's decision.)

🤖 Generated with Claude Code

vahid-ahmadi and others added 2 commits November 21, 2026 23:25
Adds a Firm entity beside person, benunit and household as a root
population (no person membership), with firm-level rules for employer
NICs and the Employment Allowance, the apprenticeship levy, corporation
tax with capital allowances, R&D relief and the bank surcharge, England
business rates, firm-side VAT and auto-enrolment. Incidence variables are
added but not wired into gov_tax or household_tax.

Refs #2939

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yment Allowance out of firm-only tax

Employer NI rate reforms raised the Employment Allowance of firms under
the cap, which leaked into firm_tax_change_total and double counted with
ni_employer. firm_only_tax now leaves the allowance out, and
firm_tax_change_total adds only the change from the allowance's own
rules at baseline liabilities.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Aggregate impact on certified data: FAIL

The report did not run, so this change has no report. See the workflow run.

vahid-ahmadi and others added 5 commits November 22, 2026 00:03
…redits from firm aggregates

- Core Simulation.clone (used by the YAML runner and by formulas that
  clone, such as marginal tax rates) now keeps root populations; this fixes
  KeyError: 'firm' in 26 YAML tests.
- Later-year firm tables must hold the same firms as the first year;
  tables without firm_id are numbered; filter_dataset keeps the firm table;
  validate rejects non-firm columns in the firm table.
- firm_tax and firm_only_tax subtract R&D credits paid out (excess RDEC and
  the ERIS payable credit).
- Sentence-case labels on corporation tax variables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Core marks firm as a person-like entity, so the app would otherwise offer
the 52 firm inputs as person variables in the household builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the remaining UNVERIFIED notes, backfills auto-enrolment
thresholds for 2012-2018, adds missing references, and records the
Finance Bill 2026-27 stage for announced-not-enacted values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d gate, single firm clone

- firm_tax_change_total holds last year's employer NI liabilities at
  baseline too, so a rate reform cannot flip eligibility under the
  2020-21 to 2024-25 prior-year limit; mismatched baseline firm tables
  raise.
- Replace the frozen regression fixture with structural checks (no firm
  variable in gov_tax or the budget totals) and live checks that firms and
  firm-only reforms leave household results unchanged.
- The UK Simulation.clone reuses the root population the patched core
  clone already made.
- Corporation tax verified_start_year 2026; docs wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vahid-ahmadi added a commit to PolicyEngine/microcosm that referenced this pull request Nov 22, 2026
…ates

Draw region (ONS UK Business 2025 Table 8) and legal status (Table 13)
per firm after calibration, with Table 15 held out, and export them as
policyengine-uk's firm_region, firm_in_england, firm_legal_form and
firm_is_public_body. Record the status of all 52 policyengine-uk firm
inputs, check the export against an ast snapshot of policyengine-uk's
firm variables, and declare value-free M2 target candidates. The
8-target uk_firms_targets.json contract is unchanged.

Refs #1705, PolicyEngine/policyengine-uk#2940.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vahid-ahmadi and others added 2 commits November 22, 2026 00:53
…; pay-distribution inputs; source checks

- Toy-frame microsimulation test (200 firms) checking each programme
  against an independent calculation, and a contract test for the columns
  microcosm reads and writes.
- Worked firm reform examples page, every figure asserted in a test.
- Optional inputs for staff below the secondary threshold and outside the
  auto-enrolment band, identical results when absent.
- Remaining parameter values confirmed against sources; SBRR boundary
  cases; firm_in_freeport documented as not yet read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Clip the optional pay-distribution inputs to the firm's headcount and
  paybill, and document it.
- A baseline without the same firm table raises in firm_tax_change_total.
- Parity test between the Employment Allowance rule-change helper and
  employment_allowance; keep-in-step notes on both sides.
- Docs describe the optional inputs and the earnings trigger; wording
  'at or below' the threshold and trigger.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vahid-ahmadi added a commit to PolicyEngine/microcosm that referenced this pull request Nov 22, 2026
tools/build_uk_firms.py runs generation, attribute draws and the
policyengine-uk table export to uk_firms.parquet with a JSON receipt
(no publish path, required --rules). PolicyEngineUKFirmRules runs
policyengine-uk's firm entity on the candidate table and reads
firm_vat_liability_if_registered; it imports policyengine-uk lazily
and is tested engine-free with a fake simulation, and was checked
locally against PolicyEngine/policyengine-uk#2940.

Refs #1705.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a firm entity and firm-side rules engine (employer NICs, corporation tax, business rates, VAT)

1 participant