Skip to content

perf(model): prefetch vtm pieces in parallel before reading a model - #207

Open
BotellaA wants to merge 5 commits into
nextfrom
perf/prefetch_vtm_pieces
Open

BotellaA wants to merge 5 commits into
nextfrom
perf/prefetch_vtm_pieces

Conversation

@BotellaA

Copy link
Copy Markdown
Member

Why

On Windows, register_model took about 6 s to read the .vtm of a large model (3_meshed_300m.og_strm, 764 pieces, 260 MB), while VTK needs only 0.6 s to parse it. The rest was the first opening of freshly written files. Windows Defender scans each new file when it is opened, and vtkXMLMultiBlockDataReader opens the pieces one by one, so the scans happen one after the other.

What

Before reader.Update(), register_model reads every piece listed in the .vtm once, using a 32-thread pool (prefetch_vtm_pieces, in vtm_prefetch.py). Plain Python file reads release the GIL, so the scans run in parallel. VTK then reads files that are already scanned and in memory.

  • The file format and the C++ writer don't change.
  • Missing or unreadable pieces are ignored; the VTK reader still reports them.
  • On Linux, or anywhere without on-access scanning, the cost is one cached read (~0.2 s on this model).

Measured

Vease, 3_meshed_300m.og_strm, mean of 3 runs (with Geode-solutions/OpenGeodeWeb-Back#298):

Before After
Reading viewable.vtm 6.1 s 1.8 s
Viewer register 8.0 s 3.9 s
First image 16.5 s 10.7 s

Alternatives tried:

  • several VTK readers in parallel: no gain, because VTK holds the GIL during each read;
  • a single-file VTKHDF output: 1.2 s, but the file grows from 260 MB to 1.4 GB.

Tests

  • New tests/test_vtm_prefetch.py (every piece is read, missing pieces are ignored, a .vtm without pieces works).
  • tests/model and the new tests: 109 passed. ruff and mypy are clean.

🤖 Generated with Claude Code

"""Read every piece of a .vtm file once, in parallel, and return how many were read.

The XML multiblock reader opens its pieces one by one. Right after an import they
are new files, and an on-access antivirus scan (Windows Defender) makes each first

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be done only on windows ?

The XML multiblock reader opens the pieces of a .vtm one by one. Right
after an import they are new files, and an on-access antivirus scan
(Windows Defender) makes each first open slow: about 6 s for a model of
764 pieces, against 0.6 s of VTK parsing.

register_model now reads every piece once from a thread pool before
reader.Update(): the scans run in parallel and VTK reads files already
scanned and cached. On a 284 MB structural model, the read drops from
6.1 s to 1.8 s and the first image arrives about 6 s earlier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BotellaA
BotellaA force-pushed the perf/prefetch_vtm_pieces branch from 24573b6 to 79e5116 Compare November 20, 2026 18:32
@BotellaA BotellaA changed the title perf: prefetch vtm pieces in parallel before reading a model perf(model): prefetch vtm pieces in parallel before reading a model Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants