Skip to content

add explicit_chunk to park-and-ride lot choice - #1139

Open
bwentl wants to merge 1 commit into
ActivitySim:mainfrom
TransLinkForecasting:pnr-lot-choice-chunked-slices-v2
Open

bwentl wants to merge 1 commit into
ActivitySim:mainfrom
TransLinkForecasting:pnr-lot-choice-chunked-slices-v2

Conversation

@bwentl

@bwentl bwentl commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Summary

  • run_park_and_ride_lot_choice now processes choosers in slices sized by the park-and-ride
    explicit_chunk (same sizing rule as activitysim.core.chunk), instead of copying and
    preprocessing the whole chooser table before interaction_simulate chunks it.
  • The unique index used for random draws is still built once for the whole table, and no
    longer needs a full copy of the chooser table, so choices are identical to the unsliced run.
  • Estimation mode and explicit_chunk: 0 (or any non-explicit chunk mode) run as one slice,
    as before.
  • Adds unit tests that check sliced and unsliced results are identical (unique and
    repeated index) and that check the slice size rules.

Changes

All code changes are in activitysim/abm/models/park_and_ride_lot_choice.py, tests in
activitysim/abm/models/util/test/test_park_and_ride_models.py.

_choosers_per_slice(state, num_choosers, explicit_chunk_size) (new)

Returns how many choosers to process at a time. It mirrors the explicit chunking rules in
activitysim/core/chunk.py:

  • chunk mode is not explicit, or explicit_chunk is 0/negative: all choosers (no slicing),
  • explicit_chunk < 1: that fraction of the choosers (ceil(n * fraction)),
  • otherwise: explicit_chunk / num_processes (total choosers per chunk across all
    processes), where num_processes is 1 when not multiprocessing,
  • the result is at least 1 and at most the number of choosers.

_choose_lots_for_choosers(...) (new, code moved)

The body that used to be inline in run_park_and_ride_lot_choice (transit eligibility
filter, skim setup, preprocessor TABLES clean up, chooser preprocessor, optional
CHOOSER_FILTER_EXPR, alternatives preprocessor and column trimming, estimation writes,
and interaction_simulate) moved into this helper unchanged, with two small additions:

  • it works on a copy of pnr_alts and locals_dict, because preprocessing mutates them
    and that must not leak from one slice into the next,
  • it returns None when no chooser in the slice is eligible, instead of returning from
    the outer function.

run_park_and_ride_lot_choice (changed)

  • Unique index for repeated index choosers. Previously: choosers.reset_index() (full
    table copy), then a new index original_index * idx_multiplier + cumcount, then
    add_channel("pnr_lot_choice", choosers). Now the same index values are computed from the
    index alone (groupby(level=0).size().max() for the multiplier and .cumcount()), and
    the rng channel is registered with an index-only frame (the channel only uses the index).
    The index is computed once for the whole table so random draws do not depend on how it
    is sliced. The reset_index(drop=False) that adds the original index as a column is now
    done per slice.
  • Slicing loop. Choosers are sliced with iloc and passed to
    _choose_lots_for_choosers. Slice results are concatenated and reindexed to the full
    index with -1 for choosers without a lot, as before. The original index is restored and the rng
    channel dropped at the end, as before.
  • Single slice keeps old behavior. When everything fits in one slice and the index is
    unique, the caller's choosers frame is used directly, so in-place column additions by the
    preprocessor are preserved exactly as before.
  • Estimation always uses one slice, because the estimator writes the choosers once.
  • Early return behavior is unchanged (all lots full returns -1; no eligible choosers
    returns -1 through return_no_choices).

Tests (test_park_and_ride_models.py)

  • test_run_pnr_slicing_does_not_change_choices (parametrized for unique and repeated
    index): 50 choosers, two lots with equal utility so every choice is a random draw. Runs
    once unsliced and once with explicit_chunk=8 (7 slices of 8,8,8,8,8,8,2) and asserts
    the choices and index are identical. It also asserts the unsliced run picks both lots,
    so a difference in random draws would be visible.
  • test_choosers_per_slice_follows_explicit_chunk_rules: chunk mode disabled, 0, larger
    than the number of choosers, fraction, and the division by num_processes.

Design decisions

Why slice inside run_park_and_ride_lot_choice rather than in compute_location_choice_logsums?

The expensive copies happen inside run_park_and_ride_lot_choice, and it is also called
from tour mode choice, tour scheduling, at-work subtour mode choice and disaggregate
accessibility. Slicing at this level fixes all callers with one setting (the existing
park-and-ride explicit_chunk) and keeps logsums.py unchanged. For the unique index
callers (for example tours) nothing changes unless explicit_chunk is smaller than the
number of choosers.

Why is the output identical?

Random numbers in ActivitySim are drawn per row from a seed based on the row index and a
per-row offset. They do not depend on which other rows are processed together. The only
chooser-set dependent value is idx_multiplier (the maximum number of rows per
person), which is computed over the whole table before slicing, so the unique index
values are exactly the same as before. The channel is registered once with all of them.

Why reuse explicit_chunk instead of a new setting?

The setting already documents "total choosers per chunk across all processes", and the
inner interaction_simulate chunking uses the same value, so each slice is exactly one
chunk of the inner step. A new setting would be one more number to tune that has to
agree with it.

Why not reduce the workplace location SAMPLE_SIZE instead?

That would also reduce memory (rows scale with SAMPLE_SIZE) but it changes the model:
different sampled alternatives, more Monte Carlo noise and different random draws, so
results would no longer be comparable with earlier runs or calibration. This change
keeps results identical.

Trade-offs

  • The chooser preprocessor, skim setup and interaction_simulate set up now run once per
    slice instead of once. The per-slice overhead is small compared with the memory saved,
    but very small explicit_chunk values (for example 20000 with 6 processes is about
    3,300 rows per slice, about 1,600 slices for work_veryhigh) also produce large
    DEBUG logs. The default of 200000 (about 33,000 rows per slice with 6 processes, about
    160 slices) is recommended.
  • Non-explicit chunk modes (adaptive, production, training, disabled) are not
    sliced, because they do not use explicit_chunk.
  • The whole chooser table (location_sample_df joined to persons_merged) still exists in
    memory once per process; only the multiple extra copies are removed.

Expected memory effect

Peak memory for this step should follow the slice size (about 33,000 rows x 7 KB, a few
hundred MB per process) rather than the table size. For the work_veryhigh segment of the example model the rough
estimate is about 100 to 120 GB peak (base plus one copy of the chooser table in each
process), compared with over 251 GB before. This is an estimate from the earlier logs;
it has not been measured on the 256 GB machine yet.

Validation

All run from .py_envs\activitysim with .\.venv\Scripts\python.exe:

Command Result
-m pytest activitysim/abm/models/util/test/test_park_and_ride_models.py -q -x -p no:cacheprovider passed (20 passed)
-m pytest activitysim/abm/models/util/test -q -p no:cacheprovider -k "pnr or park or tour_mode or logsum" passed (67 passed, 29 deselected)
-m black on the two changed files (black 22.12.0) reformatted, no unrelated changes

Equivalence with the previous implementation was also checked with a temporary test
(not committed) that imported the original module from git show HEAD:... and ran both
versions on the same fixture with 37-row slices, for a unique index (tours) and for a
repeated index (location logsums). Both produced identical Series (assert_series_equal).

TransLink has ran this code through our full regional model on the 256 GB machine, and the
ActivitySim example and estimation test suites.

We will provide memory profile in subsequent comments to this pull request.

Corresponding issue - #1138

Location choice logsums run lot choice on one row per person and sampled
destination. Preprocessing, skim setup and filtering copied that whole table
several times before interaction_simulate chunking applied, exhausting memory.
Run the lot choice in slices sized by explicit_chunk. The unique index used for
random draws is still built once for the whole table, so choices are identical
to the unsliced run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bwentl

bwentl commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Hi @dhensle, per our discussion, here are a bit more information about our tests:

We ran our model to confirm the model result does not change with this update.

We used chunk size of 200,000 for park and ride, and 40,000 for workplace location. Introduction of this change is needed to have our model run successfully in 2053 scenario under 256GB of memory. Reducing chunk size in workplace location without adding chunking for park and ride does not seem to resolve the memory issues we were encountering.

Here are the impact on run time and memory on the 2023 scenario (~1.3 million HHs & 3.2 million Persons):

  • Peak memory before: ~186GiB
  • Peak memory after: ~145GiB
  • Run time before: 37.1 mins
  • Run time after: 38.0 mins

Memory Profile before -
image

Memory Profile after -
image

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant