diff --git a/changelog.d/c1-42-drop-report-columns.removed.md b/changelog.d/c1-42-drop-report-columns.removed.md new file mode 100644 index 000000000..b073ebe67 --- /dev/null +++ b/changelog.d/c1-42-drop-report-columns.removed.md @@ -0,0 +1 @@ +The simulation service and the v2 Modal runtime no longer map `reports.label`, `description` and `user_id`, `report_runs.trigger` and its four cliff columns, or `budget_window_runs.trigger`, which API v1 drops. They work on a database with or without those columns. diff --git a/docs/migration/stage-12-modal-worker-foundation.md b/docs/migration/stage-12-modal-worker-foundation.md index e267e15d1..0f6a97408 100644 --- a/docs/migration/stage-12-modal-worker-foundation.md +++ b/docs/migration/stage-12-modal-worker-foundation.md @@ -351,7 +351,7 @@ report's run on the current versions the same way, or else creates the report's new run naming each side's simulation run on those versions, found or inserted (C1 · 24's answers 3 and 8); a forced re-run, and every re-run of a report created before keys, creates an extra run on a new run of each -side, all marked forced. No new run names a trigger (answer 6). The new runs record the +side, all marked forced. The new runs record the catalog version, dataset and region of the current .py version's catalog; the `simulations` and `reports` rows are unchanged. A budget-window report's re-run joins or creates a `budget_window_runs` row with a report run per @@ -414,8 +414,7 @@ output reads from either side (`self_employment_income`, its region-group split. Cliffs are a calculation of each simulation run of their own (above). A report run asking for them (`include_cliffs: true`) still fails before any worker starts, with `cliff_impacts_unavailable`, and touches -none of its simulation runs, and its completion never writes `report_runs`' -four cliff columns. The +none of its simulation runs. The coordinator gives each worker its simulation's stored plan, and keeps the report's own plan, from its pair, for what the aggregation reads. A reused or waited-for file must hold what the report reads diff --git a/libs/policyengine-simulation-persistence/README.md b/libs/policyengine-simulation-persistence/README.md index 528fe58b0..b3d2511a6 100644 --- a/libs/policyengine-simulation-persistence/README.md +++ b/libs/policyengine-simulation-persistence/README.md @@ -65,8 +65,8 @@ or `report_runs.spm_provenance` is stored as SQL `NULL`, not JSON `null`, so a query for `IS NULL` finds a run with no result. Each service writes only its own tables, and the foreign keys between them -stay: 20 keys name rows the Public API writes in its users, policies, -dynamics, households, datasets, regions and model catalog tables. PostgreSQL +stay: 19 keys name rows API v1 writes in its policies, dynamics, households, +datasets, regions and model catalog tables. PostgreSQL checks each of them on insert, as the referenced table's owner, so the service's role needs no privilege on those tables for it, and a row naming one that doesn't exist fails with SQLSTATE `23503`. `referenced_tables.py` @@ -242,9 +242,7 @@ report and run rows (C1 · 24). These modules do it: runs, found or inserted in simulation ID order, and each year's report run (`insert_window_years`, which the window re-runs share). `EconomyRunStore.find_or_create_budget_window` runs it in a transaction - of its own. No keyed create writes a label, description, user or trigger - (C1 · 24's answer 6): each user's saved report and report-run record hold - them, in the Public API. + of its own. ## Household reports @@ -284,8 +282,8 @@ version, dataset and region it calculates on. each simulation's policy and dynamic. - `insert_report_rerun` joins the report's run on the re-run's versions that hasn't failed and wasn't forced (pending, running or succeeded), - creating nothing, or else creates the report's new run, naming no - trigger (answer 6), each side joining its simulation's run on those versions that hasn't + creating nothing, or else creates the report's new run, each side joining + its simulation's run on those versions that hasn't failed and wasn't forced, in any state, or inserting one (`find_or_insert_side_runs`, as a keyed create does). A report run another request inserts after the lookup is joined, and a savepoint takes back @@ -310,8 +308,8 @@ re-run's versions that hasn't failed and isn't a recalculation, with its year runs, or else inserts the window run and each year's report run on its sides' runs, found or inserted. One that recalculates is a forced re-run, never refused, an extra window run whose year runs and simulation -runs are marked `forced`. Either records the window run's .py version and -no trigger, and `EconomyRunStore.create_window_rerun` returns the window +runs are marked `forced`. Either records the window run's .py version, and +`EconomyRunStore.create_window_rerun` returns the window run and years it ended on. A household report's definition also names its household (`household_id`). @@ -320,7 +318,7 @@ A household report's definition also names its household (`household_id`). report's re-run is, it takes the same lock and joins the report's run on the re-run's versions, or else each side joins its own household simulation's run on those versions (no dataset or region, the report's -year) or inserts one, under a new report run (no trigger). A forced +year) or inserts one, under a new report run. A forced one, as every re-run of a household report without keys is (interim answer B), takes no lock, is never refused, and marks its report run and a new run of each side `forced`. `HouseholdReportStore.create_report_rerun` runs it @@ -392,8 +390,8 @@ the service writes, as the service's role; `test_modal_worker_writes`, as the Modal worker role, checks the role's privileges, records a run's state in each run table, inserts a row in every result table and reads back that a `None` JSON result is SQL `NULL`. Another test, outside the rule, requires the modelled -tables' foreign keys to the Public API's users, policies, dynamics, -households, datasets, regions and model catalog tables to be exactly the 20 +tables' foreign keys to API v1's policies, dynamics, households, datasets, +regions and model catalog tables to be exactly the 19 in `tests/public_api_keys.py`, each by its own table, its name and the table it names, so a key dropped, added, renamed, moved or pointed at another table shows even if the models follow it. The job then gives the Modal worker role @@ -464,9 +462,8 @@ run's and cliff calculation's state (`status`, `modal_call_id`, run has no call of its own) and of the results its own row holds: a report run's `spm_provenance` and `data_version`, a simulation run's output file and `household_result`, a household job's `result`, and a cliff calculation's -`cliff_gap` and `cliff_share`. It no longer writes a report run's four cliff -columns, which API v1 is to drop: a report run's cliff impact is its -two simulation runs' cliff results. The same privilege manifest grants them; +`cliff_gap` and `cliff_share`. A report run's cliff impact is its two +simulation runs' cliff results. API v1's manifest grants these privileges; `test_modal_worker_writes` checks that in CI. `missing_table_privileges` returns each one the connected role lacks, ordered diff --git a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/__init__.py b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/__init__.py index 0c0ca9823..f79b36b7c 100644 --- a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/__init__.py +++ b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/__init__.py @@ -209,7 +209,6 @@ Report, ReportRun, ReportRunStatus, - ReportRunTrigger, Simulation, SimulationRun, SimulationStatus, @@ -306,7 +305,6 @@ "ReportRun", "ReportRunState", "ReportRunStatus", - "ReportRunTrigger", "ReportSide", "ReportSideError", "RerunRecordDeletedError", diff --git a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/access.py b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/access.py index dd50b889c..c310d27e9 100644 --- a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/access.py +++ b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/access.py @@ -112,8 +112,6 @@ # failure and times. MODAL_WORKER_UPDATABLE_COLUMNS: Final[Mapping[str, frozenset[str]]] = MappingProxyType( { - # A report run's cliff impact is its simulation runs' cliff results, - # so the role never writes its four cliff columns. "report_runs": _RUN_STATE | {"spm_provenance", "data_version"}, "simulation_runs": _RUN_STATE | {"output_uri", "output_sha256", "household_result"}, diff --git a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/economy_rerun_records.py b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/economy_rerun_records.py index 7bb49df2e..a4d115781 100644 --- a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/economy_rerun_records.py +++ b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/economy_rerun_records.py @@ -10,8 +10,7 @@ - creates, in one transaction, the new run and each report side's run of its own simulation, all ``pending`` with no Modal call ID; an unforced economy or household report re-run's side joins its simulation's run on - the current versions instead, when there is one. No new run names a - trigger (answer 6): each user's report-run record holds it. A budget + the current versions instead, when there is one. A budget window's new run is a ``budget_window_runs`` row with a report run per year, each year on the simulations its report run in the report's newest window run names. A household report's new @@ -348,10 +347,7 @@ def insert_report_rerun(session: Session, rerun: NewReportRerun) -> FoundReportR def _rerun_report_run_row( *, run_id: UUID, report_id: UUID, versions: RunVersions ) -> dict[str, Any]: - """Return a re-run's report run as inserted, without its sides' runs. - - It names no trigger: each user's report-run record holds it (answer 6). - """ + """Return a re-run's report run as inserted, without its sides' runs.""" return { "id": run_id, "report_id": report_id, @@ -404,8 +400,8 @@ def insert_window_rerun(session: Session, rerun: NewWindowRerun) -> FoundBudgetW window's create does (C1 · 24's answer 8). A forced re-run (``recalculate``) takes no lock, is never refused, and inserts an extra window run on new runs of every year, all marked forced. Every new run - is pending with no Modal call; the window run records its .py version - and no trigger (answer 6). Return the window run and its years, which + is pending with no Modal call; the window run records its .py version. + Return the window run and its years, which only when this call inserted the window run need their first years started. Raise as ``insert_report_rerun`` does. """ @@ -495,7 +491,7 @@ def insert_household_report_rerun( ) -> FoundReportRun: """Join the household report's run on the re-run's versions, or create it. - The report run names no budget-window run and no trigger (answer 6); + The report run names no budget-window run; each side's run is of the report's own household simulation, with no Modal call: the service then starts one household function call for a report run it created. Unforced (a keyed report's re-run), take the diff --git a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_economy_records.py b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_economy_records.py index 9e7b73741..329ffc35a 100644 --- a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_economy_records.py +++ b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_economy_records.py @@ -88,8 +88,6 @@ class KeyedEconomyReport: current catalog version's rows with the definition's dataset name and region code. ``inputs`` is what ``reports.inputs`` stores: the SPM selection when the request gave one, and no other option (answer 4). - No label, description, user or trigger is stored on the shared rows - (answer 6): each user's saved report and report-run record hold them. """ definition: EconomyReportDefinition diff --git a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_household_records.py b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_household_records.py index fb3eb8dab..8975a8866 100644 --- a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_household_records.py +++ b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_household_records.py @@ -73,8 +73,7 @@ class KeyedHouseholdReport: ``reform`` is None exactly when the definition has no reform (a report of the baseline alone). The runs calculate on the catalog version - ``tax_benefit_model_version_id``. No label, description, user or - trigger is stored on the shared rows (answer 6). + ``tax_benefit_model_version_id``. """ definition: HouseholdReportDefinition diff --git a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_window_records.py b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_window_records.py index 85dc19a6d..6368f0ab3 100644 --- a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_window_records.py +++ b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/keyed_window_records.py @@ -24,8 +24,7 @@ A year's simulation run may be a single-year report's, or another window's, pending, running or succeeded (answers 4 and 5). Rows keep random IDs: the -IDs a request proposes are used only for the rows it inserts. No per-user -column is written (answer 6). +IDs a request proposes are used only for the rows it inserts. """ from __future__ import annotations @@ -247,8 +246,7 @@ def window_run_row( ) -> dict[str, Any]: """Return a window's new run as inserted, pending, with its .py version. - ``recalculate`` marks a forced re-run's, which nothing joins. It names - no trigger: each user's report-run record holds it (answer 6). + ``recalculate`` marks a forced re-run's, which nothing joins. """ return { "id": run_id, diff --git a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/referenced_tables.py b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/referenced_tables.py index 14f44eac7..681e93c52 100644 --- a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/referenced_tables.py +++ b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/referenced_tables.py @@ -1,7 +1,7 @@ """API v1's own tables that the simulation service's rows name, by ``id`` alone. -Some keys of the service's tables name rows API v1 writes: users, policies, -dynamics, households, datasets, regions and the model catalog. PostgreSQL checks +Some keys of the service's tables name rows API v1 writes: policies, dynamics, +households, datasets, regions and the model catalog. PostgreSQL checks each of those keys when a row is inserted, as the referenced table's owner, so the service's role needs no privilege for it, and a row naming one that doesn't exist fails with SQLSTATE 23503. These classes give SQLAlchemy each key's @@ -86,13 +86,4 @@ class ReferencedTaxBenefitModel(ReferencedBase): id: Mapped[UUID] = mapped_column(Uuid) -class ReferencedUser(ReferencedBase): - """The ``id`` of a row of ``users``.""" - - __tablename__ = "users" - __table_args__: ClassVar = (PrimaryKeyConstraint("id", name="pk_users"),) - - id: Mapped[UUID] = mapped_column(Uuid) - - referenced_metadata = ReferencedBase.metadata diff --git a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/tables.py b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/tables.py index 1c4fc22c0..603d0b9f8 100644 --- a/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/tables.py +++ b/libs/policyengine-simulation-persistence/src/policyengine_simulation_persistence/tables.py @@ -49,7 +49,6 @@ ReferencedRegion, ReferencedTaxBenefitModel, ReferencedTaxBenefitModelVersion, - ReferencedUser, ) if TYPE_CHECKING: @@ -91,14 +90,6 @@ class ReportRunStatus(StrEnum): FAILED = "failed" -class ReportRunTrigger(StrEnum): - """What started a report run or a budget window's calculation.""" - - INITIAL = "initial" - MANUAL = "manual" - SYSTEM = "system" - - class HouseholdJobStatus(StrEnum): """Lifecycle state of a household job.""" @@ -276,12 +267,6 @@ class Report(ServiceBase): __tablename__ = "reports" __table_args__: ClassVar = ( PrimaryKeyConstraint("id", name="pk_reports"), - _api_v1_key( - "user_id", - ReferencedUser.id, - ondelete="SET NULL", - name="fk_reports_user_id_users", - ), _api_v1_key( "tax_benefit_model_id", ReferencedTaxBenefitModel.id, @@ -348,20 +333,13 @@ class Report(ServiceBase): Index("ix_reports_dataset_independent_key", "dataset_independent_key"), Index("ix_reports_country_type_created_at", "country", "type", "created_at"), Index("ix_reports_tax_benefit_model_id", "tax_benefit_model_id"), - # Deleting a user sets its reports' user_id to null. - Index("ix_reports_user_id", "user_id"), ) id: Mapped[UUID] = identifier_column() created_at: Mapped[datetime] = server_timestamp_column() updated_at: Mapped[datetime] = server_timestamp_column() - # Per-user columns the service no longer writes (C1 · 24's answer 6): a - # report's name and description live on each user's saved report. - label: Mapped[str | None] = mapped_column(String(255)) - description: Mapped[str | None] = mapped_column(String) country: Mapped[str] = mapped_column(String(16)) type: Mapped[str | None] = mapped_column(String(128)) - user_id: Mapped[UUID | None] = mapped_column(Uuid) tax_benefit_model_id: Mapped[UUID] = mapped_column(Uuid) policy_id: Mapped[UUID | None] = mapped_column(Uuid) baseline_simulation_id: Mapped[UUID | None] = mapped_column(Uuid) @@ -528,10 +506,6 @@ class BudgetWindowRun(ServiceBase): # The .py version the window calculates on, so a window run is found by # its report and version set (answer 8); null on runs before C1 · 27. policyengine_version: Mapped[str | None] = mapped_column(String(128)) - # Per-request, so the service no longer writes it (answer 6). - trigger: Mapped[ReportRunTrigger | None] = mapped_column( - enum_type(ReportRunTrigger, "v2_report_run_trigger") - ) # The window's forced marker: a recalculating run is never joined. recalculate: Mapped[bool] = mapped_column(Boolean) status: Mapped[ReportRunStatus] = mapped_column( @@ -591,12 +565,6 @@ class ReportRun(ServiceBase): "budget_window_run_id IS NULL OR year IS NOT NULL", name="ck_report_runs_budget_window_year", ), - CheckConstraint( - "(baseline_cliff_gap IS NULL) = (baseline_cliff_share IS NULL) AND " - "(baseline_cliff_gap IS NULL) = (reform_cliff_gap IS NULL) AND " - "(baseline_cliff_gap IS NULL) = (reform_cliff_share IS NULL)", - name="ck_report_runs_cliff_impact_complete", - ), _completion_required("report_runs"), _year_between("report_runs", nullable=True), Index( @@ -638,10 +606,6 @@ class ReportRun(ServiceBase): status: Mapped[ReportRunStatus] = mapped_column( enum_type(ReportRunStatus, "v2_report_run_status") ) - # Per-request, so the service no longer writes it (answer 6). - trigger: Mapped[ReportRunTrigger | None] = mapped_column( - enum_type(ReportRunTrigger, "v2_report_run_trigger") - ) idempotency_key: Mapped[UUID | None] = mapped_column(Uuid) started_at: Mapped[datetime | None] = optional_timestamp_column() completed_at: Mapped[datetime | None] = optional_timestamp_column() @@ -655,11 +619,6 @@ class ReportRun(ServiceBase): budget_window_run_id: Mapped[UUID | None] = mapped_column(Uuid) year: Mapped[int | None] = mapped_column(Integer) modal_call_id: Mapped[str | None] = mapped_column(String(MODAL_CALL_ID_LENGTH)) - # Cliff impact: all four values or none. - baseline_cliff_gap: Mapped[float | None] = mapped_column(Float) - baseline_cliff_share: Mapped[float | None] = mapped_column(Float) - reform_cliff_gap: Mapped[float | None] = mapped_column(Float) - reform_cliff_share: Mapped[float | None] = mapped_column(Float) spm_provenance: Mapped[dict[str, Any] | None] = mapped_column( JSONB(none_as_null=True) ) diff --git a/libs/policyengine-simulation-persistence/tests/economy_rows.py b/libs/policyengine-simulation-persistence/tests/economy_rows.py index c49e4ab65..18b8e500b 100644 --- a/libs/policyengine-simulation-persistence/tests/economy_rows.py +++ b/libs/policyengine-simulation-persistence/tests/economy_rows.py @@ -19,7 +19,6 @@ Report, ReportRun, ReportRunStatus, - ReportRunTrigger, RunVersions, Simulation, SimulationChoice, @@ -243,13 +242,12 @@ def insert_run_of( @dataclass(frozen=True) class ReportRunOutcome: - """How a stored report run ended, its versions and trigger, and its options.""" + """How a stored report run ended, its versions, and its options.""" status: ReportRunStatus = ReportRunStatus.SUCCEEDED inputs: dict[str, Any] | None = None error_message: str | None = None versions: RunVersions = VERSIONS - trigger: ReportRunTrigger = ReportRunTrigger.INITIAL def insert_report_run_naming( @@ -287,7 +285,6 @@ def insert_report_of( # noqa: PLR0913 # the report's sides and what else it ma connection.execute( insert(Report).values( id=report_id, - label="An earlier report", country=economy.catalog.country, type="economy", tax_benefit_model_id=economy.catalog.model_id, @@ -327,7 +324,6 @@ def insert_report_run_of( country_package_version=outcome.versions.country_package_version, policyengine_version=outcome.versions.policyengine_version, status=outcome.status, - trigger=outcome.trigger, completed_at=COMPLETED_AT if finished else None, error_message=outcome.error_message, ) diff --git a/libs/policyengine-simulation-persistence/tests/household_report_rows.py b/libs/policyengine-simulation-persistence/tests/household_report_rows.py index 4564b6f71..7957d78e4 100644 --- a/libs/policyengine-simulation-persistence/tests/household_report_rows.py +++ b/libs/policyengine-simulation-persistence/tests/household_report_rows.py @@ -3,10 +3,9 @@ The service now finds or creates a household report by definition key (``keyed_household_records``). A report it created before then has no key: each side is a household simulation of its own with one run, the report -names the household, both simulations and the reform's policy, and its -first run has trigger ``initial``. Every run is ``pending`` with no Modal -call ID. ``insert_household_report_rows`` inserts such a report directly, -as whatever role the connection acts as. +names the household, both simulations and the reform's policy. Every run +is ``pending`` with no Modal call ID. ``insert_household_report_rows`` +inserts such a report directly, as whatever role the connection acts as. """ from __future__ import annotations @@ -20,7 +19,6 @@ Report, ReportRun, ReportRunStatus, - ReportRunTrigger, Simulation, SimulationRun, SimulationStatus, @@ -129,11 +127,8 @@ def insert_household_report_rows( # noqa: PLR0913 # the report's catalog, hous connection.execute( insert(Report).values( id=report.id, - label="A larger child tax credit", - description="For one family", country=catalog.country, type=HOUSEHOLD_REPORT_TYPE, - user_id=None, tax_benefit_model_id=catalog.model_id, policy_id=None if reform is None else reform.policy_id, baseline_simulation_id=report.baseline.simulation_id, @@ -159,7 +154,6 @@ def insert_household_report_rows( # noqa: PLR0913 # the report's catalog, hous country_package_version=versions.country_package_version, policyengine_version=versions.policyengine_version, status=ReportRunStatus.PENDING, - trigger=ReportRunTrigger.INITIAL, modal_call_id=None, ) ) diff --git a/libs/policyengine-simulation-persistence/tests/public_api_keys.py b/libs/policyengine-simulation-persistence/tests/public_api_keys.py index 8c35acadd..a5b8d629c 100644 --- a/libs/policyengine-simulation-persistence/tests/public_api_keys.py +++ b/libs/policyengine-simulation-persistence/tests/public_api_keys.py @@ -42,7 +42,6 @@ "regions(id)", "SET NULL", ), - ("reports", "fk_reports_user_id_users"): (("user_id",), "users(id)", "SET NULL"), ("reports", "fk_reports_tax_benefit_model_id_tax_benefit_models"): ( ("tax_benefit_model_id",), "tax_benefit_models(id)", diff --git a/libs/policyengine-simulation-persistence/tests/service_rows.py b/libs/policyengine-simulation-persistence/tests/service_rows.py index 02d4b8f14..d19acf936 100644 --- a/libs/policyengine-simulation-persistence/tests/service_rows.py +++ b/libs/policyengine-simulation-persistence/tests/service_rows.py @@ -19,7 +19,6 @@ Report, ReportRun, ReportRunStatus, - ReportRunTrigger, Simulation, SimulationRun, SimulationStatus, @@ -163,7 +162,6 @@ def insert_service_rows( report_id = connection.execute( insert(Report) .values( - label="Test report", country="us", tax_benefit_model_id=public_api_rows.tax_benefit_model_id, baseline_simulation_id=baseline_simulation_id, @@ -186,7 +184,6 @@ def insert_service_rows( .values( report_id=report_id, tax_benefit_model_version_id=public_api_rows.model_version_id, - trigger=ReportRunTrigger.INITIAL, recalculate=False, status=ReportRunStatus.PENDING, ) @@ -203,7 +200,6 @@ def insert_service_rows( country_package_version=COUNTRY_PACKAGE_VERSION, policyengine_version=POLICYENGINE_VERSION, status=ReportRunStatus.PENDING, - trigger=ReportRunTrigger.INITIAL, ) .returning(ReportRun.id) ).scalar_one() diff --git a/libs/policyengine-simulation-persistence/tests/test_access.py b/libs/policyengine-simulation-persistence/tests/test_access.py index 37bdf43a3..197c6c9c3 100644 --- a/libs/policyengine-simulation-persistence/tests/test_access.py +++ b/libs/policyengine-simulation-persistence/tests/test_access.py @@ -136,8 +136,6 @@ def test_the_modal_worker_role_updates_its_runs_state_and_results() -> None: its status, failure and times. """ assert dict(MODAL_WORKER_UPDATABLE_COLUMNS) == { - # A report run's cliff impact is its simulation runs' cliff results, - # so its own four cliff columns are never written. "report_runs": RUN_STATE_COLUMNS | {"spm_provenance", "data_version"}, "simulation_runs": RUN_STATE_COLUMNS | {"output_uri", "output_sha256", "household_result"}, diff --git a/libs/policyengine-simulation-persistence/tests/test_economy_reruns.py b/libs/policyengine-simulation-persistence/tests/test_economy_reruns.py index 7429997a4..9b96294e0 100644 --- a/libs/policyengine-simulation-persistence/tests/test_economy_reruns.py +++ b/libs/policyengine-simulation-persistence/tests/test_economy_reruns.py @@ -50,7 +50,6 @@ NewSimulationRerun, ReportDefinition, ReportRunStatus, - ReportRunTrigger, ReportSideError, RerunRecordDeletedError, RerunSide, @@ -247,8 +246,8 @@ def _used_on_later_catalog( The other report, created on ``economy`` beside the stored one, compares the stored report's baseline simulation (naming its run) with a reform - simulation of its own; its initial run succeeded, and its manual re-run - on the later catalog created both sides' later runs and succeeded. The + simulation of its own; its first run succeeded, and its re-run on the + later catalog created both sides' later runs and succeeded. The stored report has no run on the later versions, so a re-run of it has none to join, and its reform simulation has no later run. """ @@ -266,7 +265,7 @@ def _used_on_later_catalog( connection, other_report_id, (later_baseline, later_other_reform), - ReportRunOutcome(versions=LATER_VERSIONS, trigger=ReportRunTrigger.MANUAL), + ReportRunOutcome(versions=LATER_VERSIONS), ) return later_baseline @@ -319,7 +318,6 @@ def test_a_report_rerun_joins_or_creates_runs_of_the_reports_own_simulations( "country_package_version", "policyengine_version", "status", - "trigger", "modal_call_id", "budget_window_run_id", "year", @@ -331,8 +329,6 @@ def test_a_report_rerun_joins_or_creates_runs_of_the_reports_own_simulations( "country_package_version": LATER_VERSIONS.country_package_version, "policyengine_version": LATER_VERSIONS.policyengine_version, "status": ReportRunStatus.PENDING, - # Answer 6: the trigger is each user's record's, not the shared run's. - "trigger": None, "modal_call_id": None, "budget_window_run_id": None, "year": None, @@ -693,7 +689,6 @@ def test_a_rerun_and_its_runs_are_created_through_the_store( joined = store.create_report_rerun(_rerun_of(stored, later)) assert (joined.id, joined.inserted) == (rerun.run_id, False) - assert _row(connection, "report_runs", rerun.run_id)["trigger"] is None assert _row(connection, "simulation_runs", simulation_rerun.run_id)[ "status" ] == (SimulationStatus.PENDING) @@ -881,9 +876,8 @@ def test_a_forced_report_rerun_is_an_extra_run_beside_any_run_on_its_versions( ] earlier_run = _row(connection, "report_runs", stored.report_run_id) - assert (report_run["forced"], report_run["trigger"], report_run["status"]) == ( + assert (report_run["forced"], report_run["status"]) == ( True, - None, ReportRunStatus.PENDING, ) assert len(simulation_runs) == 2 diff --git a/libs/policyengine-simulation-persistence/tests/test_economy_run_reads.py b/libs/policyengine-simulation-persistence/tests/test_economy_run_reads.py index 90909c485..3e2442492 100644 --- a/libs/policyengine-simulation-persistence/tests/test_economy_run_reads.py +++ b/libs/policyengine-simulation-persistence/tests/test_economy_run_reads.py @@ -32,13 +32,20 @@ read_economy_report, read_economy_simulation, ) -from sqlalchemy import update +from sqlalchemy import select, update pytestmark = pytest.mark.public_api_schema EARLIER = datetime(2026, 11, 1, tzinfo=UTC) +def _report_id_of(connection, report_run_id: UUID) -> UUID: + """Return the ID of the report the stored report run belongs to.""" + return connection.execute( + select(ReportRun.report_id).where(ReportRun.id == report_run_id) + ).scalar_one() + + def _report_with_runs(connection, economy: EconomyCatalog) -> tuple[UUID, list[UUID]]: """Insert a report and three runs of it; return the report and its runs. @@ -60,13 +67,7 @@ def _report_with_runs(connection, economy: EconomyCatalog) -> tuple[UUID, list[U ReportRunOutcome(status=ReportRunStatus.FAILED, error_message="first"), ) ] - report_id = ( - connection.execute( - ReportRun.__table__.select().where(ReportRun.id == run_ids[0]) - ) - .one() - .report_id - ) + report_id = _report_id_of(connection, run_ids[0]) forced_shape = RunShape(status=SimulationStatus.PENDING, forced=True) for _ in range(2): run_id = uuid4() @@ -83,7 +84,6 @@ def _report_with_runs(connection, economy: EconomyCatalog) -> tuple[UUID, list[U country_package_version=COUNTRY_PACKAGE_VERSION, policyengine_version=POLICYENGINE_VERSION, status=ReportRunStatus.PENDING, - trigger="manual", forced=True, ) ) @@ -141,13 +141,7 @@ def test_a_failed_newest_run_is_read_with_its_error(public_api_database_url) -> error_message="simulation_failed: the reform failed", ), ) - report_id = ( - connection.execute( - ReportRun.__table__.select().where(ReportRun.id == run_id) - ) - .one() - .report_id - ) + report_id = _report_id_of(connection, run_id) act_as_the_service(connection) report = in_session(connection, read_economy_report, report_id) diff --git a/libs/policyengine-simulation-persistence/tests/test_economy_window_reruns.py b/libs/policyengine-simulation-persistence/tests/test_economy_window_reruns.py index e3845ef85..b271f1247 100644 --- a/libs/policyengine-simulation-persistence/tests/test_economy_window_reruns.py +++ b/libs/policyengine-simulation-persistence/tests/test_economy_window_reruns.py @@ -48,7 +48,6 @@ ProposedSide, ReportRun, ReportRunStatus, - ReportRunTrigger, RerunRecordDeletedError, RerunSide, SimulationStatus, @@ -101,7 +100,7 @@ def _insert_window_rows( Each year has its own current-law simulations, with a pending run each; the report names its first year's; its window run and each year's report - run are pending, with trigger ``initial``. + run are pending. """ pending = RunShape(status=SimulationStatus.PENDING) years = tuple( @@ -130,7 +129,6 @@ def _insert_window_rows( id=window_run_id, report_id=report_id, tax_benefit_model_version_id=economy.catalog.version_id, - trigger=ReportRunTrigger.INITIAL, recalculate=False, status=ReportRunStatus.PENDING, ) @@ -148,7 +146,6 @@ def _insert_window_rows( "country_package_version": VERSIONS.country_package_version, "policyengine_version": VERSIONS.policyengine_version, "status": ReportRunStatus.PENDING, - "trigger": ReportRunTrigger.INITIAL, "modal_call_id": None, "forced": False, } @@ -302,7 +299,7 @@ def test_a_definitions_years_come_from_the_newest_window_run( def test_a_forced_window_rerun_creates_a_window_run_and_each_years_new_runs( public_api_database_url, ) -> None: - """Answers 3 and 6: no run is reused or joined, and no trigger is written.""" + """Answer 3: no run is reused or joined.""" with rolled_back_transaction(public_api_database_url) as connection: economy = insert_economy_catalog(connection) later = insert_later_catalog(connection, economy) @@ -326,7 +323,6 @@ def test_a_forced_window_rerun_creates_a_window_run_and_each_years_new_runs( "report_id", "tax_benefit_model_version_id", "policyengine_version", - "trigger", "recalculate", "status", ) @@ -334,7 +330,6 @@ def test_a_forced_window_rerun_creates_a_window_run_and_each_years_new_runs( "report_id": stored.report_id, "tax_benefit_model_version_id": later.catalog.version_id, "policyengine_version": LATER_POLICYENGINE_VERSION, - "trigger": None, "recalculate": True, "status": ReportRunStatus.PENDING, } @@ -347,7 +342,6 @@ def test_a_forced_window_rerun_creates_a_window_run_and_each_years_new_runs( ( run["year"], run["budget_window_run_id"], - run["trigger"], run["status"], run["policyengine_version"], ) @@ -356,7 +350,6 @@ def test_a_forced_window_rerun_creates_a_window_run_and_each_years_new_runs( ( year, rerun.window_run_id, - None, ReportRunStatus.PENDING, LATER_POLICYENGINE_VERSION, ) @@ -495,7 +488,7 @@ def test_an_unforced_window_rerun_after_a_failed_run_names_its_years_runs( rerun.window_run_id, True, ) - assert (window_run["recalculate"], window_run["trigger"]) == (False, None) + assert window_run["recalculate"] is False for first, again in zip(found.years, created.years, strict=True): assert again.baseline.simulation_run_id == first.baseline.simulation_run_id assert again.baseline.run_inserted is False diff --git a/libs/policyengine-simulation-persistence/tests/test_household_report_reruns.py b/libs/policyengine-simulation-persistence/tests/test_household_report_reruns.py index 702efa88c..e6b2efc61 100644 --- a/libs/policyengine-simulation-persistence/tests/test_household_report_reruns.py +++ b/libs/policyengine-simulation-persistence/tests/test_household_report_reruns.py @@ -1,9 +1,9 @@ """Re-running a stored household report: what the re-run reads, and what it creates. A household report's re-run gives each side a new run of its own household -simulation, never reusing one, under a new report run with trigger -``manual``. These run against a database API v1's own v2 migration -chain built (``PUBLIC_API_SCHEMA_DATABASE_URL``). The stored report and its +simulation, never reusing one, under a new report run. These run against a +database API v1's own v2 migration chain built +(``PUBLIC_API_SCHEMA_DATABASE_URL``). The stored report and its first runs are written as the superuser; the re-run's reads and writes run as the simulation service's role, in a transaction the test rolls back. """ @@ -235,7 +235,7 @@ def test_a_household_report_is_read_with_its_household_and_each_sides_policy( assert alone_definition.reform is None -def test_a_household_reports_rerun_is_a_manual_run_of_its_own_simulations( +def test_a_household_reports_rerun_is_a_new_run_of_its_own_simulations( public_api_database_url, ) -> None: with rolled_back_transaction(public_api_database_url) as connection: @@ -259,8 +259,6 @@ def test_a_household_reports_rerun_is_a_manual_run_of_its_own_simulations( side.simulation_run_id for side in rerun.sides ) assert report_run["report_id"] == report.id - # Answer 6: the trigger is each user's record's, not the shared run's. - assert report_run["trigger"] is None assert report_run["status"] == ReportRunStatus.PENDING assert report_run["modal_call_id"] is None assert ( diff --git a/libs/policyengine-simulation-persistence/tests/test_keyed_economy_records.py b/libs/policyengine-simulation-persistence/tests/test_keyed_economy_records.py index 7d577ba38..07d9d9a88 100644 --- a/libs/policyengine-simulation-persistence/tests/test_keyed_economy_records.py +++ b/libs/policyengine-simulation-persistence/tests/test_keyed_economy_records.py @@ -189,9 +189,6 @@ def test_a_keyed_report_is_created_with_its_simulations_and_first_run( "inputs", "type", "budget_window_size", - "label", - "description", - "user_id", ) } == { "key_version": KEY_VERSION, @@ -203,22 +200,13 @@ def test_a_keyed_report_is_created_with_its_simulations_and_first_run( "inputs": {"spm": NATIONAL_SPM}, "type": "economy", "budget_window_size": None, - # Answer 6: a shared report holds no request's label or user. - "label": None, - "description": None, - "user_id": None, } assert (baseline["key_version"], baseline["definition_key"]) == ( KEY_VERSION, definition.simulation("baseline").definition_key, ) assert reform["definition_key"] == definition.simulation("reform").definition_key - # Answer 6: the trigger is each user's record's, not the shared run's. - assert (run["status"], run["forced"], run["trigger"]) == ( - ReportRunStatus.PENDING, - False, - None, - ) + assert (run["status"], run["forced"]) == (ReportRunStatus.PENDING, False) assert ( reform_run["status"], reform_run["forced"], @@ -360,7 +348,6 @@ def test_a_forced_run_is_never_joined_by_a_create(public_api_database_url) -> No country_package_version=VERSIONS.country_package_version, policyengine_version=VERSIONS.policyengine_version, status=ReportRunStatus.PENDING, - trigger="manual", forced=True, ) ) @@ -549,7 +536,6 @@ def create_a_competing_run_first( country_package_version=VERSIONS.country_package_version, policyengine_version=VERSIONS.policyengine_version, status=ReportRunStatus.PENDING, - trigger="initial", ) ) competing.append(run_id) diff --git a/libs/policyengine-simulation-persistence/tests/test_keyed_household_records.py b/libs/policyengine-simulation-persistence/tests/test_keyed_household_records.py index bda88b6f1..9f6942df4 100644 --- a/libs/policyengine-simulation-persistence/tests/test_keyed_household_records.py +++ b/libs/policyengine-simulation-persistence/tests/test_keyed_household_records.py @@ -207,9 +207,6 @@ def test_a_keyed_household_report_is_created_with_its_simulations_and_first_run( "key_version", "definition_key", "dataset_independent_key", - "label", - "description", - "user_id", ) } == { "type": "household", @@ -224,16 +221,11 @@ def test_a_keyed_household_report_is_created_with_its_simulations_and_first_run( "key_version": KEY_VERSION, "definition_key": definition.definition_key, "dataset_independent_key": definition.dataset_independent_key, - # Answer 6: a shared report holds no request's label or user. - "label": None, - "description": None, - "user_id": None, } assert (report["baseline_simulation_id"], report["reform_simulation_id"]) == ( request.baseline.simulation_id, request.reform.simulation_id, ) - assert report_run["trigger"] is None assert report_run["forced"] is False assert ( report_run["baseline_simulation_run_id"], @@ -505,7 +497,7 @@ def test_a_keyed_household_rerun_after_a_failed_run_names_its_simulations_runs( created.baseline_simulation_run_id, created.reform_simulation_run_id, ) == (found.baseline.simulation_run_id, found.reform.simulation_run_id) - assert (report_run["trigger"], report_run["forced"]) == (None, False) + assert report_run["forced"] is False def test_a_forced_keyed_household_rerun_is_an_extra_run_beside_the_one_under_way( diff --git a/libs/policyengine-simulation-persistence/tests/test_keyed_window_records.py b/libs/policyengine-simulation-persistence/tests/test_keyed_window_records.py index 33bc7e5fe..ca0a2e4d1 100644 --- a/libs/policyengine-simulation-persistence/tests/test_keyed_window_records.py +++ b/libs/policyengine-simulation-persistence/tests/test_keyed_window_records.py @@ -183,9 +183,6 @@ def test_a_keyed_window_is_created_with_each_years_simulations_and_runs( "key_version", "definition_key", "dataset_independent_key", - "label", - "description", - "user_id", ) } == { "budget_window_size": 3, @@ -193,19 +190,15 @@ def test_a_keyed_window_is_created_with_each_years_simulations_and_runs( "key_version": KEY_VERSION, "definition_key": definition.definition_key, "dataset_independent_key": definition.dataset_independent_key, - "label": None, - "description": None, - "user_id": None, } # The report names its first year's simulations. assert report["reform_simulation_id"] == found.years[0].reform.simulation_id assert { name: window_run[name] - for name in ("policyengine_version", "recalculate", "trigger", "status") + for name in ("policyengine_version", "recalculate", "status") } == { "policyengine_version": VERSIONS.policyengine_version, "recalculate": False, - "trigger": None, "status": ReportRunStatus.PENDING, } assert [year.year for year in found.years] == list(WINDOW_YEARS) @@ -219,7 +212,7 @@ def test_a_keyed_window_is_created_with_each_years_simulations_and_runs( request.window_run_id, year.year, ) - assert (year_run["trigger"], year_run["forced"]) == (None, False) + assert year_run["forced"] is False assert year_run["reform_simulation_run_id"] == year.reform.simulation_run_id # Each year's simulations are that year's, keyed by it. assert simulation["year"] == year.year diff --git a/libs/policyengine-simulation-persistence/tests/test_mapped_classes.py b/libs/policyengine-simulation-persistence/tests/test_mapped_classes.py index 8441a2242..00733235e 100644 --- a/libs/policyengine-simulation-persistence/tests/test_mapped_classes.py +++ b/libs/policyengine-simulation-persistence/tests/test_mapped_classes.py @@ -59,7 +59,6 @@ service_metadata, tables, ) -from public_api_keys import KEYS_TO_PUBLIC_API_TABLES from schema_comparison import describe_tables from sqlalchemy import create_engine, inspect, select, text, update from sqlalchemy.dialects import postgresql @@ -83,7 +82,6 @@ "regions": referenced_tables.ReferencedRegion, "tax_benefit_model_versions": referenced_tables.ReferencedTaxBenefitModelVersion, "tax_benefit_models": referenced_tables.ReferencedTaxBenefitModel, - "users": referenced_tables.ReferencedUser, } INPUT_CLASSES = { "tax_benefit_models": TaxBenefitModel, @@ -209,7 +207,6 @@ def test_the_keys_to_api_v1s_tables_name_the_referenced_classes() -> None: } assert named_tables == referenced - assert len(KEYS_TO_PUBLIC_API_TABLES) == 20 def test_every_mapper_configures() -> None: diff --git a/libs/policyengine-simulation-persistence/tests/test_public_api_schema.py b/libs/policyengine-simulation-persistence/tests/test_public_api_schema.py index b4e5a399b..098dfcc52 100644 --- a/libs/policyengine-simulation-persistence/tests/test_public_api_schema.py +++ b/libs/policyengine-simulation-persistence/tests/test_public_api_schema.py @@ -199,13 +199,13 @@ def test_rule_proof_models_match_public_api() -> None: def test_the_public_api_keeps_its_keys_from_the_services_tables( public_api_database_url, ) -> None: - """The keys from the service's tables to the Public API's own are the 20 expected. + """The keys from the service's tables to API v1's own are the 19 expected. Each is compared by its table, its name and the table it names; the comparison with the models checks the rest of its definition. - The Public API's own tables are those ``referenced_metadata`` models: users, - policies, dynamics, households, datasets, regions and the model catalog. + API v1's own tables are those ``referenced_metadata`` models: policies, + dynamics, households, datasets, regions and the model catalog. """ table_names = [table.name for table in service_metadata.tables.values()] public_api_tables = {table.name for table in referenced_metadata.tables.values()} diff --git a/libs/policyengine-simulation-persistence/tests/test_shared_rows.py b/libs/policyengine-simulation-persistence/tests/test_shared_rows.py index d83038f1c..03f21bcd0 100644 --- a/libs/policyengine-simulation-persistence/tests/test_shared_rows.py +++ b/libs/policyengine-simulation-persistence/tests/test_shared_rows.py @@ -30,7 +30,6 @@ from policyengine_simulation_persistence import ( KEY_VERSION, ReportRunStatus, - ReportRunTrigger, RunVersions, SimulationRun, SimulationStatus, @@ -73,7 +72,6 @@ def _report_values( ) -> dict[str, Any]: return { "id": uuid4(), - "label": "Shared report", "country": economy.catalog.country, "type": "economy", "tax_benefit_model_id": economy.catalog.model_id, @@ -116,7 +114,6 @@ def _report_run_values( "country_package_version": versions.country_package_version, "policyengine_version": versions.policyengine_version, "status": ReportRunStatus.PENDING, - "trigger": ReportRunTrigger.INITIAL, } diff --git a/libs/policyengine-simulation-persistence/tests/test_tables.py b/libs/policyengine-simulation-persistence/tests/test_tables.py index 09b73d8e7..06adea473 100644 --- a/libs/policyengine-simulation-persistence/tests/test_tables.py +++ b/libs/policyengine-simulation-persistence/tests/test_tables.py @@ -2,6 +2,7 @@ from __future__ import annotations +import pytest from policyengine_simulation_persistence import ( SERVICE_TABLE_PRIVILEGES, BudgetWindowRun, @@ -16,9 +17,35 @@ referenced_metadata, service_metadata, ) -from policyengine_simulation_persistence.orm_base import enum_type +from policyengine_simulation_persistence.orm_base import ServiceBase, enum_type from public_api_keys import KEYS_TO_PUBLIC_API_TABLES -from sqlalchemy import Enum, ForeignKeyConstraint, Index, Table, UniqueConstraint +from sqlalchemy import ( + Enum, + ForeignKeyConstraint, + Index, + Table, + UniqueConstraint, + inspect, +) + +# The columns API v1's revision 4e30076d4b9f drops. The models name none of +# them, so a release works on a database that has them and on one that doesn't. +COLUMNS_API_V1_DROPS = { + Report: {"label", "description", "user_id"}, + ReportRun: { + "trigger", + "baseline_cliff_gap", + "baseline_cliff_share", + "reform_cliff_gap", + "reform_cliff_share", + }, + BudgetWindowRun: {"trigger"}, +} +# The foreign key, index and check that revision drops with those columns. +CONSTRAINTS_API_V1_DROPS = { + Report: {"fk_reports_user_id_users", "ix_reports_user_id"}, + ReportRun: {"ck_report_runs_cliff_impact_complete"}, +} def _key(constraint: ForeignKeyConstraint) -> tuple[tuple[str, ...], str, str]: @@ -51,6 +78,32 @@ def _indexed_columns(table: Table, index_name: str) -> tuple[str, ...]: return tuple(column.name for column in index.columns) +@pytest.mark.parametrize( + ("model", "dropped_columns"), + COLUMNS_API_V1_DROPS.items(), + ids=lambda value: getattr(value, "__name__", None), +) +def test_a_model_maps_none_of_the_columns_api_v1_drops( + model: type[ServiceBase], dropped_columns: set[str] +) -> None: + mapped_columns = {column.name for column in inspect(model).columns} + + assert mapped_columns.isdisjoint(dropped_columns) + + +@pytest.mark.parametrize( + ("model", "dropped_names"), + CONSTRAINTS_API_V1_DROPS.items(), + ids=lambda value: getattr(value, "__name__", None), +) +def test_a_model_declares_no_constraint_or_index_api_v1_drops( + model: type[ServiceBase], dropped_names: set[str] +) -> None: + declared_names = {getattr(item, "name", None) for item in model.__table_args__} + + assert declared_names.isdisjoint(dropped_names) + + def test_the_models_describe_every_table_the_service_creates_rows_in() -> None: created_tables = { table_name @@ -74,7 +127,7 @@ def test_the_models_declare_every_key_to_the_public_apis_own_tables() -> None: } assert keys_to_public_api_tables == KEYS_TO_PUBLIC_API_TABLES - assert len(KEYS_TO_PUBLIC_API_TABLES) == 20 + assert len(KEYS_TO_PUBLIC_API_TABLES) == 19 def test_the_public_apis_tables_are_modelled_by_their_id_alone() -> None: @@ -94,7 +147,6 @@ def test_the_public_apis_tables_are_modelled_by_their_id_alone() -> None: "regions", "tax_benefit_model_versions", "tax_benefit_models", - "users", } assert set(referenced_metadata.tables.values()) == named_tables for table in named_tables: @@ -103,8 +155,7 @@ def test_the_public_apis_tables_are_modelled_by_their_id_alone() -> None: def test_the_columns_the_public_apis_deletes_search_are_indexed() -> None: - """Deleting a user or model version searches these columns for its rows.""" - assert _indexed_columns(Report.__table__, "ix_reports_user_id") == ("user_id",) + """Deleting a model version searches these columns for its rows.""" assert _indexed_columns( SimulationRun.__table__, "ix_simulation_runs_tax_benefit_model_version_id" ) == ("tax_benefit_model_version_id",) @@ -134,7 +185,7 @@ def test_every_enum_type_names_the_public_schema() -> None: ] # Including cliff_impacts.status, which reuses the simulation runs' type. - assert len(enum_columns) == 8 + assert len(enum_columns) == 6 for column in enum_columns: assert column.type.schema == "public", f"{column.table.name}.{column.name}" # The helper every enum column uses, before any table adopts its type. @@ -312,15 +363,6 @@ def test_a_run_is_unforced_unless_its_insert_marks_it() -> None: assert str(table.c.forced.server_default.arg) == "false", table.name -def test_the_per_user_columns_are_nullable() -> None: - """Answer 6: the service writes no request's label, description or trigger.""" - assert Report.label.nullable is True - assert Report.description.nullable is True - assert Report.user_id.nullable is True - assert ReportRun.trigger.nullable is True - assert BudgetWindowRun.trigger.nullable is True - - def test_a_window_run_is_one_per_report_and_version_set_unless_recalculated() -> None: """Answer 8: a window run on the versions is joined; a forced one is extra.""" assert BudgetWindowRun.policyengine_version.nullable is True diff --git a/libs/policyengine-stage12-persistence/src/policyengine_stage12_persistence/report_run_results.py b/libs/policyengine-stage12-persistence/src/policyengine_stage12_persistence/report_run_results.py index 1252711b3..d3ce95106 100644 --- a/libs/policyengine-stage12-persistence/src/policyengine_stage12_persistence/report_run_results.py +++ b/libs/policyengine-stage12-persistence/src/policyengine_stage12_persistence/report_run_results.py @@ -232,9 +232,8 @@ def report_run_result( def _report_run_columns(economy: SingleYearMacroOutput) -> dict[str, Any]: """Return the SPM receipts and data version, as report run columns. - No cliff column: a cliff impact is each simulation run's own cliff - result, so the report run's four cliff columns stay null, and its - completion names none of them, which lets API v1 drop them. + A cliff impact is each simulation run's own cliff result, not a column of + the report run. """ receipts = economy.spm_provenance return { diff --git a/libs/policyengine-stage12-persistence/tests/budget_window_database.py b/libs/policyengine-stage12-persistence/tests/budget_window_database.py index 83b5858a5..11e7ecb7b 100644 --- a/libs/policyengine-stage12-persistence/tests/budget_window_database.py +++ b/libs/policyengine-stage12-persistence/tests/budget_window_database.py @@ -88,7 +88,6 @@ def seed_window(self, first_year: int = 2026, size: int = 7) -> SeededWindow: connection.execute( insert(table("reports")).values( id=report_id, - label="Budget window test", country="us", tax_benefit_model_id=model_id, baseline_simulation_id=simulation_ids[0], @@ -103,7 +102,6 @@ def seed_window(self, first_year: int = 2026, size: int = 7) -> SeededWindow: id=window_run_id, report_id=report_id, tax_benefit_model_version_id=version_id, - trigger="initial", recalculate=False, status="pending", **SEEDED_AT, @@ -135,7 +133,6 @@ def seed_window(self, first_year: int = 2026, size: int = 7) -> SeededWindow: budget_window_run_id=window_run_id, year=seeded.year, status="pending", - trigger="initial", **SEEDED_AT, **VERSIONS, ) @@ -182,7 +179,6 @@ def seed_window_rerun(self, window: SeededWindow) -> SeededWindow: id=window_run_id, report_id=window.report_id, tax_benefit_model_version_id=first_run.tax_benefit_model_version_id, - trigger="manual", recalculate=True, status="pending", **later, @@ -219,7 +215,6 @@ def seed_window_rerun(self, window: SeededWindow) -> SeededWindow: budget_window_run_id=window_run_id, year=seeded.year, status="pending", - trigger="manual", forced=True, **later, **VERSIONS, diff --git a/libs/policyengine-stage12-persistence/tests/household_report_database.py b/libs/policyengine-stage12-persistence/tests/household_report_database.py index 837b314e0..93968b2b6 100644 --- a/libs/policyengine-stage12-persistence/tests/household_report_database.py +++ b/libs/policyengine-stage12-persistence/tests/household_report_database.py @@ -79,7 +79,6 @@ def seed_household_report_run( connection.execute( insert(seeder.table("reports")).values( id=report_id, - label="Household report run test", country="us", type="household", tax_benefit_model_id=model_id, @@ -98,7 +97,6 @@ def seed_household_report_run( baseline_simulation_run_id=simulation_run_ids[0], reform_simulation_run_id=reform_run_id, status="pending", - trigger="initial", **VERSIONS, ) ) diff --git a/libs/policyengine-stage12-persistence/tests/report_run_database.py b/libs/policyengine-stage12-persistence/tests/report_run_database.py index 42fb5c18b..4baf04a31 100644 --- a/libs/policyengine-stage12-persistence/tests/report_run_database.py +++ b/libs/policyengine-stage12-persistence/tests/report_run_database.py @@ -126,7 +126,6 @@ def seed_report_run(self) -> SeededReportRun: connection.execute( insert(self.table("reports")).values( id=report_id, - label="Report run test", country="us", tax_benefit_model_id=model_id, baseline_simulation_id=simulation_ids[0], @@ -157,7 +156,6 @@ def seed_report_run(self) -> SeededReportRun: baseline_simulation_run_id=run_ids.baseline_simulation_run_id, reform_simulation_run_id=run_ids.reform_simulation_run_id, status="pending", - trigger="initial", **VERSIONS, ) ) diff --git a/libs/policyengine-stage12-persistence/tests/test_household_report_run_store.py b/libs/policyengine-stage12-persistence/tests/test_household_report_run_store.py index cac44fc4c..df8b027c1 100644 --- a/libs/policyengine-stage12-persistence/tests/test_household_report_run_store.py +++ b/libs/policyengine-stage12-persistence/tests/test_household_report_run_store.py @@ -277,7 +277,6 @@ def test_a_household_report_run_completes_only_while_its_call_holds_it( row = seeder.row("report_runs", report_run_id) assert row["completed_at"] == LATER assert row["error_message"] is None - assert row["baseline_cliff_gap"] is None assert row["spm_provenance"] is None diff --git a/libs/policyengine-stage12-persistence/tests/test_report_run_results.py b/libs/policyengine-stage12-persistence/tests/test_report_run_results.py index 226ebe0bd..f7abcfe52 100644 --- a/libs/policyengine-stage12-persistence/tests/test_report_run_results.py +++ b/libs/policyengine-stage12-persistence/tests/test_report_run_results.py @@ -5,6 +5,7 @@ from copy import deepcopy from dataclasses import replace from datetime import timedelta +from typing import Any from uuid import uuid4 import pytest @@ -18,6 +19,7 @@ us_output_with_every_part, ) from policyengine_simulation_contract.stage12_execution import ReportRunIds +from policyengine_simulation_persistence import CliffImpact from policyengine_simulation_persistence.result_tables import BudgetSummary from policyengine_stage12_persistence import ( DistrictCodes, @@ -50,12 +52,6 @@ "race": "poverty_by_race", } POVERTY_PART_NAMES = ("poverty", "poverty_by_gender", "poverty_by_race") -CLIFF_COLUMNS = ( - "baseline_cliff_gap", - "baseline_cliff_share", - "reform_cliff_gap", - "reform_cliff_share", -) LABOR_SUPPLY_TABLES = ("labor_supply_responses", "labor_supply_decile_responses") @@ -271,11 +267,7 @@ def test_a_report_run_stores_its_economy_output_as_rows( output["inequality"]["top_10_pct_share"][side], output["inequality"]["top_1_pct_share"][side], ) - # A cliff impact is each simulation run's own cliff result, which the - # report run's completion never writes. report_run = seeder.row("report_runs", run_ids.report_run_id) - assert output["cliff_impact"] is not None - assert {report_run[column] for column in CLIFF_COLUMNS} == {None} assert report_run["data_version"] == output["data_version"] assert ( _rebuilt_labor_supply(seeder, run_ids.report_run_id) @@ -326,9 +318,41 @@ def test_a_report_runs_result_columns_never_name_its_cliff_columns() -> None: assert set(result.report_run_columns) == {"spm_provenance", "data_version"} -def test_a_completed_report_run_leaves_its_cliff_columns_null( +def _insert_pending_cliff_results(seeder, run_ids: ReportRunIds) -> None: + """Insert a pending cliff result of each of the report run's simulation runs.""" + with seeder.engine.begin() as connection: + for simulation_run_id in ( + run_ids.baseline_simulation_run_id, + run_ids.reform_simulation_run_id, + ): + connection.execute( + insert(CliffImpact.__table__).values( + id=uuid4(), simulation_run_id=simulation_run_id, status="pending" + ) + ) + + +def _cliff_results_of(seeder, run_ids: ReportRunIds) -> list[dict[str, Any]]: + """Return the cliff results of the report run's two simulation runs, by ID.""" + statement = ( + select(CliffImpact.__table__) + .where( + CliffImpact.simulation_run_id.in_( + (run_ids.baseline_simulation_run_id, run_ids.reform_simulation_run_id) + ) + ) + .order_by(CliffImpact.id) + ) + with seeder.engine.connect() as connection: + return [dict(row._mapping) for row in connection.execute(statement)] + + +def test_completing_a_report_run_with_a_cliff_impact_leaves_cliff_results_alone( store, seeder, seeded_run: SeededReportRun ) -> None: + """Only a simulation run's own cliff calculation writes its cliff result.""" + _insert_pending_cliff_results(seeder, seeded_run.run_ids) + cliff_results = _cliff_results_of(seeder, seeded_run.run_ids) output = deepcopy(US_ECONOMY_OUTPUT) output["cliff_impact"] = { "baseline": {"cliff_gap": 120.5, "cliff_share": 0.04}, @@ -338,15 +362,15 @@ def test_a_completed_report_run_leaves_its_cliff_columns_null( assert _complete(store, seeded_run, output=output) == RunWrite( changed=True, status=RunStatus.SUCCEEDED ) - - report_run = seeder.row("report_runs", seeded_run.run_ids.report_run_id) - assert {report_run[column] for column in CLIFF_COLUMNS} == {None} + assert _cliff_results_of(seeder, seeded_run.run_ids) == cliff_results def test_an_output_without_labour_supply_or_cliff_impact_stores_neither( store, seeder, seeded_run: SeededReportRun ) -> None: report_run_id = seeded_run.run_ids.report_run_id + _insert_pending_cliff_results(seeder, seeded_run.run_ids) + cliff_results = _cliff_results_of(seeder, seeded_run.run_ids) output = deepcopy(US_ECONOMY_OUTPUT) output["labor_supply_response"] = None output["cliff_impact"] = None @@ -357,8 +381,8 @@ def test_an_output_without_labour_supply_or_cliff_impact_stores_neither( for table_name in LABOR_SUPPLY_TABLES: assert seeder.rows(table_name, report_run_id=report_run_id) == [] + assert _cliff_results_of(seeder, seeded_run.run_ids) == cliff_results report_run = seeder.row("report_runs", report_run_id) - assert {report_run[column] for column in CLIFF_COLUMNS} == {None} assert report_run["data_version"] == output["data_version"] # No receipt is SQL NULL, not JSON null, which reads back as None too. report_runs = seeder.table("report_runs") @@ -477,7 +501,6 @@ def test_a_result_that_cannot_be_stored_leaves_the_report_run_running( report_run = seeder.row("report_runs", run_ids.report_run_id) assert report_run["status"] == "running" # The report run's own result columns roll back with its success. - assert {report_run[column] for column in CLIFF_COLUMNS} == {None} assert report_run["data_version"] is None for table_name in ( *LABOR_SUPPLY_TABLES, diff --git a/libs/policyengine-stage12-persistence/tests/test_report_run_sweep_store.py b/libs/policyengine-stage12-persistence/tests/test_report_run_sweep_store.py index 60dec9b00..00e5bbfe5 100644 --- a/libs/policyengine-stage12-persistence/tests/test_report_run_sweep_store.py +++ b/libs/policyengine-stage12-persistence/tests/test_report_run_sweep_store.py @@ -349,7 +349,6 @@ def _second_report_run( # noqa: PLR0913 # the run, then what it names and whos else reform_simulation_run_id ), status=status, - trigger="manual", country_package_version=first["country_package_version"], policyengine_version=first["policyengine_version"], **( diff --git a/projects/policyengine-simulation-entry/README.md b/projects/policyengine-simulation-entry/README.md index db161ed07..59d4e6ae5 100644 --- a/projects/policyengine-simulation-entry/README.md +++ b/projects/policyengine-simulation-entry/README.md @@ -317,9 +317,7 @@ them and the Public API's one-unfailed-run indexes leave them out, and a forced re-run is never refused while a run is under way. A report created before keys (C1 · 27), whose simulations have none, is always re-run that way, forced or not: its runs may have been calculated with plans that -depended on that report (C1 · 24's interim answer B). The new report run -names no trigger (answer 6): the Public API's per-user report-run record -holds it. Such a +depended on that report (C1 · 24's interim answer B). Such a report stored asking for cliff impacts (`include_cliffs: true` in its options) is refused with 422 `cliffs_not_available`, located at `report.inputs.include_cliffs`, until @@ -350,7 +348,7 @@ unforced re-run joins its run on the current versions, or each side joins its own household simulation's run on them or gets a new one; `force`, and every re-run of a household report created before keys, makes an extra run on a new run of each side, all marked `forced`. A report run it created -(naming no trigger: answer 6) is started by one household function call, as a new +is started by one household function call, as a new household report's is, and the re-run answers 201 with the household report create's fields. `POST /simulations/{simulation_id}/runs`, with no body, joins a stored diff --git a/projects/policyengine-simulation-entry/src/policyengine_simulation_entry/economy_reruns.py b/projects/policyengine-simulation-entry/src/policyengine_simulation_entry/economy_reruns.py index c95772da3..9d9856e83 100644 --- a/projects/policyengine-simulation-entry/src/policyengine_simulation_entry/economy_reruns.py +++ b/projects/policyengine-simulation-entry/src/policyengine_simulation_entry/economy_reruns.py @@ -17,7 +17,7 @@ 3. for a simulation or a keyed economy report, joins, in one transaction, the record's run on the current versions that hasn't failed and wasn't forced, pending, running or succeeded (C1 · 24's answers 2 and 3), or - else creates the new run (naming no trigger, answer 6); a keyed + else creates the new run; a keyed report's sides join their simulations' runs on those versions the same way, or create them. A report whose simulations have no key, created before C1 · 27, may have runs whose plans depended on the report that diff --git a/projects/policyengine-simulation-entry/src/policyengine_simulation_entry/household_report_reruns.py b/projects/policyengine-simulation-entry/src/policyengine_simulation_entry/household_report_reruns.py index 2a16e73dc..235b378f4 100644 --- a/projects/policyengine-simulation-entry/src/policyengine_simulation_entry/household_report_reruns.py +++ b/projects/policyengine-simulation-entry/src/policyengine_simulation_entry/household_report_reruns.py @@ -13,7 +13,7 @@ policy values, and checks each side's household contract request; 3. for a keyed report's unforced re-run, joins, in one transaction, the report's run on the current versions that hasn't failed and wasn't - forced, or else creates the report's new run (naming no trigger) + forced, or else creates the report's new run naming each side's household simulation run on those versions, found or inserted, as an economy report's re-run does (C1 · 24's answer 8). A forced re-run, and every re-run of a report created before keys, whose