Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
cc24bb4
Merge #1072's branch (shared simulation runs) into C1 · 32
anth-volk Nov 16, 2026
c53e901
Test that the Stage 12 stores use the service library's run-table mod…
anth-volk Nov 16, 2026
8921122
Make the Stage 12 stores use the service library's run-table models
anth-volk Nov 16, 2026
d3b98aa
Test that the result-table models are the service library's, and matc…
anth-volk Nov 16, 2026
66e0ae9
Model the twelve result tables completely in the service library
anth-volk Nov 16, 2026
e7672c6
Test that one engine and URL helper serve both libraries (failing)
anth-volk Nov 16, 2026
3f655bd
Use the service library's one engine and URL helper in the Stage 12 l…
anth-volk Nov 16, 2026
04c03b9
Test the Modal worker role's privileges and writes (failing)
anth-volk Nov 16, 2026
9789b21
List the Modal worker role's privileges and check them against the gr…
anth-volk Nov 16, 2026
8fd1b18
Test that the v2 Modal images carry the service library and v1 code d…
anth-volk Nov 16, 2026
f566202
Copy the service library into every v2 Modal image
anth-volk Nov 16, 2026
44d8945
Test that the image smoke runs when the service library changes (fail…
anth-volk Nov 16, 2026
0f36f49
Run the image smoke when the service library changes
anth-volk Nov 16, 2026
7044bfa
Test that the Public API schema job runs the Stage 12 tests as the wo…
anth-volk Nov 16, 2026
ac0e723
Run the Stage 12 library's tests in the Public API schema job
anth-volk Nov 16, 2026
a6afe39
Describe the one table-model library in the rules, docs and changelog
anth-volk Nov 16, 2026
826b4b6
Test that the v1 import walk counts a module's packages and sim's lib…
anth-volk Nov 16, 2026
e9fcb60
Follow every parent package in the v1 import walk
anth-volk Nov 16, 2026
262a176
Pin RunStatus to each run table's status values
anth-volk Nov 16, 2026
98853c5
Say which tables the comparison leaves out, and what the schema job runs
anth-volk Nov 16, 2026
969fad8
Test that the Stage 12 library and the Modal image group use the shar…
anth-volk Nov 16, 2026
282a838
Narrow the Stage 12 library and the Modal image group to the shared r…
anth-volk Nov 16, 2026
20ff954
Expect the shared SQLAlchemy range in the Modal image group's test
anth-volk Nov 16, 2026
42b2071
Test that the v1 import walk counts the entry module's own package (f…
anth-volk Nov 16, 2026
aafa274
Start the v1 import walk from the entry module's own packages too
anth-volk Nov 16, 2026
79ab78c
Leave the database ranges to their comparison test in the image-group…
anth-volk Nov 16, 2026
4ebeee6
Merge #1072's branch again, at 95a93e16
anth-volk Nov 16, 2026
7f55b22
Merge main into C1 · 32, at 2d95c268
anth-volk Nov 16, 2026
0335ec7
Merge main into C1 · 32, at 28d15d90
anth-volk Nov 17, 2026
f8d2bb8
Merge main into C1 · 32, after #1072 landed
anth-volk Nov 17, 2026
f550430
Merge main into C1 · 32, at 5d4f81a6
anth-volk Nov 17, 2026
4fe9218
Merge main into C1 · 32, after #1074 and #1070 landed
anth-volk Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/scripts/public-api-schema-worker-login.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Give the Public API's Modal worker role a login on the disposable database
# the Public API schema job built, so the Stage 12 library's tests connect as
# that role (rule sim.persistence.models-match-public-api).
#
# Usage: public-api-schema-worker-login.sh
#
# PUBLIC_API_SCHEMA_DATABASE_URL names that database, as the superuser's
# postgresql:// URL; the Public API's privilege manifest creates its roles
# without a login. MODAL_WORKER_TEST_PASSWORD is the password the tests' URL
# gives, which exists only in the job's database.
set -euo pipefail

database_url="${PUBLIC_API_SCHEMA_DATABASE_URL:?PUBLIC_API_SCHEMA_DATABASE_URL must name the database the job built}"
password="${MODAL_WORKER_TEST_PASSWORD:?MODAL_WORKER_TEST_PASSWORD must give the password the Modal worker role logs in with}"

# psql quotes the password variable as a literal (:'password'), so it never
# appears in the statement itself.
psql "${database_url}" --no-psqlrc --set=ON_ERROR_STOP=1 "--set=password=${password}" <<'SQL'
ALTER ROLE policyengine_v2_modal_worker LOGIN PASSWORD :'password';
SQL
1 change: 1 addition & 0 deletions .github/workflows/pr-image-smoke.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ on:
- 'libs/policyengine-simulation-contract/**'
- 'libs/policyengine-simulation-observability/**'
- 'libs/policyengine-fastapi/**'
- 'libs/policyengine-simulation-persistence/**'
- 'libs/policyengine-stage12-persistence/**'
- '.github/workflows/pr-image-smoke.yml'
- '.github/scripts/modal-image-smoke.sh'
Expand Down
27 changes: 25 additions & 2 deletions .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,10 @@ jobs:
# commit pinned in libs/policyengine-simulation-persistence/public-api-commit,
# grants the privileges its manifest declares, and runs the persistence
# tests marked public_api_schema against them: sim's table models must
# match those tables, and the simulation service's role must write them
# (rule sim.persistence.models-match-public-api). The pinned commit must
# match those tables, and the simulation service's role and the Modal
# worker role must write the tables each writes (rule
# sim.persistence.models-match-public-api). It then runs the Stage 12
# library's tests there, as the Modal worker role. The pinned commit must
# also be on the Public API's default branch. The Public API's repository
# is public, so the job reads no secret. Full CI runs only on the lowest
# unmerged pull request in a stack.
Expand Down Expand Up @@ -213,6 +215,27 @@ jobs:
working-directory: libs/policyengine-simulation-persistence
run: uv run --frozen pytest tests/ -v -m public_api_schema

# The Stage 12 library's stores write the same tables as the Modal worker
# role, which the manifest creates without a login: the password exists
# only in this job's database.
- name: Give the Modal worker role a login on the job's database
env:
MODAL_WORKER_TEST_PASSWORD: modal_worker_ci
run: .github/scripts/public-api-schema-worker-login.sh

- name: Install the Stage 12 library's dependencies
working-directory: libs/policyengine-stage12-persistence
run: uv sync --extra test --frozen

- name: Run the Stage 12 library's tests as the Modal worker role
working-directory: libs/policyengine-stage12-persistence
env:
REPORT_RUN_TEST_DATABASE_URL: postgresql://policyengine_v2_modal_worker:modal_worker_ci@127.0.0.1:5432/policyengine_v2_alembic_test
REPORT_RUN_TEST_SEED_DATABASE_URL: postgresql+psycopg://postgres:simulation_ci@127.0.0.1:5432/policyengine_v2_alembic_test
# A missing database fails the tests instead of skipping them.
REPORT_RUN_REQUIRE_DATABASE_TESTS: "1"
run: uv run --frozen pytest tests/ -v

- name: Require the pinned commit on the Public API's default branch
run: .github/scripts/public-api-commit-on-default-branch.sh public-api master

Expand Down
1 change: 1 addition & 0 deletions changelog.d/c1-32-one-table-model-library.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The simulation service and the v2 Modal runtime use one library of the Public API's table models, `policyengine-simulation-persistence`. The Stage 12 library's stores, which the v2 coordinators, workers, household functions and stuck-run sweep use, now import its models of `report_runs`, `simulation_runs`, `budget_window_runs` and `household_jobs`, and its new complete models of the twelve result tables, instead of partial models of their own, and connect with its engine and URL check. They write the same columns as before. The v2 Modal images copy the library. CI's comparison with the Public API's tables now covers the result tables, a new test checks the Modal worker role's grants and writes against the Public API's, and the Stage 12 library's database tests, which skipped in CI, run in the `Public API schema` job as that role. A malformed Modal worker database URL now raises the service's "Simulation database URL" error.
15 changes: 8 additions & 7 deletions docs/engineering/skills/database-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,14 @@ and this repository has no migration chain of its own
Their read-only models in that library's `input_tables.py` name only the
columns the service reads, apart from the models of the tables it writes.
- the run tables the v2 Modal coordinator, workers, household functions and
stuck-run sweep write as the Modal worker role: `report_runs`, `simulation_runs`,
`budget_window_runs`, the result tables and `household_jobs`. Their mappings in
`policyengine_stage12_persistence.report_run_tables` and
`report_run_results`, separate from the simulation service's models,
declare only the columns the coordinator's, workers', household functions'
and stuck-run sweep's statements read or write; deployment validation
doesn't compare them with the live schema.
stuck-run sweep write as the Modal worker role: `report_runs`,
`simulation_runs`, `budget_window_runs`, the result tables and
`household_jobs`. They use the same models, in
`libs/policyengine-simulation-persistence` (`tables.py` and
`result_tables.py`, which `sim.persistence.models-match-public-api`
compares with the Public API's tables), through the stores in
`libs/policyengine-stage12-persistence`. The v1 `run_simulation` code
imports neither library.

This repository's code reads and writes their rows; nothing here changes
them. A change to one of those tables starts in policyengine-api, and the
Expand Down
12 changes: 6 additions & 6 deletions docs/generated/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,23 +142,23 @@ flowchart LR

| Rule | Statement | Tier | Status | Blocking | Enforced by | CI | Proof |
| --- | --- | --- | --- | --- | --- | --- | --- |
| `sim.persistence.missing-privileges-reported` | The simulation service's privilege check, which its readiness check uses, names every table privilege and every column UPDATE privilege its role needs on the Public API's tables and lacks, as PostgreSQL reports it, counts a table missing from the public schema as lacking all of them and a missing column as lacking its own, and names a missing USAGE privilege on that schema. | mechanical | enforced | yes | pytest `libs/policyengine-simulation-persistence/tests/test_access.py` | `pr.yml:simulation-database` | yes |
| `sim.persistence.models-match-public-api` | Sim's models of the Public API's tables match the tables the Public API's v2 migration chain builds at the pinned Public API commit, which is on the Public API's default branch: every column's type, nullability and default, every constraint and index by name and definition, and every enum's values; every column its read-only models of the Public API's input tables name exists there with the same type and nullability; and the simulation service's role, with the privileges the Public API's manifest declares, writes every modelled table. | mechanical | enforced | yes | pytest `libs/policyengine-simulation-persistence/tests/test_public_api_schema.py`, `libs/policyengine-simulation-persistence/tests/test_service_writes.py` | `pr.yml:public-api-schema`, `public-api-schema-drift.yml:public-api-schema-drift` | yes |
| `sim.persistence.missing-privileges-reported` | The privilege check, which the simulation service's readiness check uses and CI runs for the Modal worker role too, names every table privilege and every column UPDATE privilege the connected role needs on the Public API's tables and lacks, as PostgreSQL reports it, counts a table missing from the public schema as lacking all of them and a missing column as lacking its own, and names a missing USAGE privilege on that schema. | mechanical | enforced | yes | pytest `libs/policyengine-simulation-persistence/tests/test_access.py` | `pr.yml:simulation-database` | yes |
| `sim.persistence.models-match-public-api` | Sim's one set of models of the Public API's tables, those the simulation service writes and the result tables the v2 Modal calls write, match the tables the Public API's v2 migration chain builds at the pinned Public API commit, which is on the Public API's default branch: every column's type, nullability and default, every constraint and index by name and definition, and every enum's values; every column its read-only models of the Public API's input tables name exists there with the same type and nullability; and, with the privileges the Public API's manifest declares, the simulation service's role writes every table the service writes, and the Modal worker role records its runs and inserts a row in every result table. | mechanical | enforced | yes | pytest `libs/policyengine-simulation-persistence/tests/test_modal_worker_writes.py`, `libs/policyengine-simulation-persistence/tests/test_public_api_schema.py`, `libs/policyengine-simulation-persistence/tests/test_service_writes.py` | `pr.yml:public-api-schema`, `public-api-schema-drift.yml:public-api-schema-drift` | yes |

```mermaid
flowchart LR
r_sim_persistence_missing_privileges_reported("sim.persistence.missing-privileges-reported (enforced)")
e_pytest_libs_policyengine_simulation_persistence_tests_test_access_py["pytest libs/policyengine-simulation-persistence/tests/test_access.py"]
j_CI_pr_yml_simulation_database["CI pr.yml:simulation-database"]
r_sim_persistence_models_match_public_api("sim.persistence.models-match-public-api (enforced)")
e_pytest_libs_policyengine_simulation_persistence_tests_test_public_api_schema_py__libs_policyengine_simulation_persistence_tests_test_service_writes_py["pytest libs/policyengine-simulation-persistence/tests/test_public_api_schema.py, libs/policyengine-simulation-persistence/tests/test_service_writes.py"]
e_pytest_libs_policyengine_simulation_persistence_tests_test_modal_worker_writes_py__libs_policyengine_simulation_persistence_tests_test_public_api_schema_py__libs_policyengine_simulation_persistence_tests_test_service_writes_py["pytest libs/policyengine-simulation-persistence/tests/test_modal_worker_writes.py, libs/policyengine-simulation-persistence/tests/test_public_api_schema.py, libs/policyengine-simulation-persistence/tests/test_service_writes.py"]
j_CI_pr_yml_public_api_schema["CI pr.yml:public-api-schema"]
j_CI_public_api_schema_drift_yml_public_api_schema_drift["CI public-api-schema-drift.yml:public-api-schema-drift"]
r_sim_persistence_missing_privileges_reported --> e_pytest_libs_policyengine_simulation_persistence_tests_test_access_py
e_pytest_libs_policyengine_simulation_persistence_tests_test_access_py --> j_CI_pr_yml_simulation_database
r_sim_persistence_models_match_public_api --> e_pytest_libs_policyengine_simulation_persistence_tests_test_public_api_schema_py__libs_policyengine_simulation_persistence_tests_test_service_writes_py
e_pytest_libs_policyengine_simulation_persistence_tests_test_public_api_schema_py__libs_policyengine_simulation_persistence_tests_test_service_writes_py --> j_CI_pr_yml_public_api_schema
e_pytest_libs_policyengine_simulation_persistence_tests_test_public_api_schema_py__libs_policyengine_simulation_persistence_tests_test_service_writes_py --> j_CI_public_api_schema_drift_yml_public_api_schema_drift
r_sim_persistence_models_match_public_api --> e_pytest_libs_policyengine_simulation_persistence_tests_test_modal_worker_writes_py__libs_policyengine_simulation_persistence_tests_test_public_api_schema_py__libs_policyengine_simulation_persistence_tests_test_service_writes_py
e_pytest_libs_policyengine_simulation_persistence_tests_test_modal_worker_writes_py__libs_policyengine_simulation_persistence_tests_test_public_api_schema_py__libs_policyengine_simulation_persistence_tests_test_service_writes_py --> j_CI_pr_yml_public_api_schema
e_pytest_libs_policyengine_simulation_persistence_tests_test_modal_worker_writes_py__libs_policyengine_simulation_persistence_tests_test_public_api_schema_py__libs_policyengine_simulation_persistence_tests_test_service_writes_py --> j_CI_public_api_schema_drift_yml_public_api_schema_drift
```

## Deployment pipeline
Expand Down
Loading
Loading