Repository navigation
Conversation
BotellaA
commented
Nov 20, 2026
| """Read every piece of a .vtm file once, in parallel, and return how many were read. | ||
|
|
||
| The XML multiblock reader opens its pieces one by one. Right after an import they | ||
| are new files, and an on-access antivirus scan (Windows Defender) makes each first |
Member
Author
There was a problem hiding this comment.
should this be done only on windows ?
The XML multiblock reader opens the pieces of a .vtm one by one. Right after an import they are new files, and an on-access antivirus scan (Windows Defender) makes each first open slow: about 6 s for a model of 764 pieces, against 0.6 s of VTK parsing. register_model now reads every piece once from a thread pool before reader.Update(): the scans run in parallel and VTK reads files already scanned and cached. On a 284 MB structural model, the read drops from 6.1 s to 1.8 s and the first image arrives about 6 s earlier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
BotellaA
force-pushed
the
perf/prefetch_vtm_pieces
branch
from
November 20, 2026 18:32
24573b6 to
79e5116
Compare
# Conflicts: # opengeodeweb_viewer_typed_schemas.d.ts # requirements.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On Windows,
register_modeltook about 6 s to read the.vtmof a large model (3_meshed_300m.og_strm, 764 pieces, 260 MB), while VTK needs only 0.6 s to parse it. The rest was the first opening of freshly written files. Windows Defender scans each new file when it is opened, andvtkXMLMultiBlockDataReaderopens the pieces one by one, so the scans happen one after the other.What
Before
reader.Update(),register_modelreads every piece listed in the.vtmonce, using a 32-thread pool (prefetch_vtm_pieces, invtm_prefetch.py). Plain Python file reads release the GIL, so the scans run in parallel. VTK then reads files that are already scanned and in memory.Measured
Vease,
3_meshed_300m.og_strm, mean of 3 runs (with Geode-solutions/OpenGeodeWeb-Back#298):viewable.vtmregisterAlternatives tried:
Tests
tests/test_vtm_prefetch.py(every piece is read, missing pieces are ignored, a.vtmwithout pieces works).tests/modeland the new tests: 109 passed.ruffandmypyare clean.🤖 Generated with Claude Code