Skip to content

from_omx_3d and from_omx fail with openmatrix 0.4 #91

Open

Description

@janzill

openmatrix 0.4.0 (29 Oct) is built on h5py. openmatrix.File now subclasses h5py.File, and its __getitem__ treats any key without a leading / as a matrix name in /data. Sharrow's h5py.File branches index with relative keys (handle["lookup"], handle["data"]), so these lookups fail.

Sharrow 2.16.2 with openmatrix 0.4.0:

Call Result
from_omx_3d([openmatrix.open_file(p)], ...) LookupError: Key lookup not found (dataset.py:542)
from_omx_3d([p], ...) (a path, which Sharrow opens with openmatrix) same
from_omx(openmatrix.open_file(p)) LookupError: Key data not found (dataset.py:351)
from_omx_3d([h5py.File(p)], ...) works
reload_from_omx_3d(ds, [p]) works

ActivitySim calls from_omx_3d with openmatrix handles, so Sharrow mode fails on new installs that don't use uv.lock, e.g., from pypi. #87 has the same pattern: it passes h5py.File handles through unchanged and indexes ["lookup"] and ["data"] (dataset.py:462 and 957, among others).

Reproduce:

import numpy as np, openmatrix, sharrow as sh
with openmatrix.open_file("skims.omx", "w") as f:
    f["SOV_TIME__AM"] = np.ones((5, 5), "f4")
    f.create_mapping("TAZ", np.arange(1, 6))
h = openmatrix.open_file("skims.omx", "r")
sh.dataset.from_omx_3d([h], index_names=("otaz", "dtaz", "time_period"), time_periods=["AM"])

Suggested fix (2.16.x and #87):

  • In the h5py.File branches, index with absolute paths: handle["/lookup"] and handle["/data"]. These work for both plain h5py and openmatrix 0.4. openmatrix 0.3.5 handles are PyTables objects and keep using the .root branch.
  • Alternatively, reopen any handle that isn't a plain h5py.File by its filename.
  • Add a test that uses openmatrix 0.4 handles.

Re #90: with a shim for this bug, load times under 0.4 are the same as under 0.3.5. The slow step, reload_from_omx_3d, opens files by path through h5py, so 0.4 doesn't change it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions