Skip to content

Slower OMX skim loading in 2.16 #90

Open

Description

@aber-sandag

When testing Sharrow implementation for SANDAG ABM3 with ActivitySim 1.5.1, we found that runtime for mp_setup_skims was greatly increased when running with Sharrow 2.16. Downgrading to Sharrow 2.15 brought runtime back down, comparable to non-sharrow runs.

mp_setup_skims
Sharrow 2.16: 29.8 minutes (1789.4 seconds)
Sharrow 2.15: 7.4 minutes (446.4 seconds)

ActivitySim log files for both runs attached:

activitysim_sharrow_mp_test_2.log
activitysim_sharrow_mp_test_3.log

Activity

  1. billyc commented on Oct 29, 2026

    @billyc

    @janzill -

    per today's engr meeting, openmatrix v0.4.0 with h5py internals is now released on pypi.

  2. janzill commented on Oct 31, 2026

    @janzill
    Contributor

    2. @janzill -
    per today's engr meeting, openmatrix v0.4.0 with h5py internals is now released on pypi.

    Nice, thanks Billy! @jpn-- this would be a good new baseline for comparisons in #87

  3. janzill commented on Nov 1, 2026

    @janzill
    Contributor

    @aber-sandag local tests reproduced this issue for me when running with the SANDAG example model, although the factor was smaller, about 1.5x instead of the 4x you report. However, setting BLOSC_NTHREADS=1 made sharrow 2.16.2 load skims faster than v2.15. Can you add this to wherever you set your thread count restrictions and try sharrow 2.16 again and see if it resolves the issue, please?

  4. aber-sandag commented on Nov 13, 2026

    @aber-sandag
    Author

    @aber-sandag local tests reproduced this issue for me when running with the SANDAG example model, although the factor was smaller, about 1.5x instead of the 4x you report. However, setting BLOSC_NTHREADS=1 made sharrow 2.16.2 load skims faster than v2.15. Can you add this to wherever you set your thread count restrictions and try sharrow 2.16 again and see if it resolves the issue, please?

    Setting BLOSC_NTHREADS=1 appears to have largely reduced the increased runtime, though it is still a bit longer than Sharrow 2.15

    mp_setup_skims
    Sharrow 2.16, BLOSC_NTHREADS=1: 9.3 minutes (560.1 seconds)

  5. jpn-- commented on Nov 13, 2026

    @jpn--
    Member

    @aber-sandag That's a good improvement. Are you able to also check using the version of sharrow in PR #92? That PR includes several other changes that I hope will speed the data loading.

  6. aber-sandag commented on Nov 13, 2026

    @aber-sandag
    Author

    @jpn-- Cannot test this PR in our environment due to loss of support for Python 3.10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions