Skip to content

perf: save native file in background after import - #298

Open
BotellaA wants to merge 6 commits into
nextfrom
perf/background_native_save
Open

BotellaA wants to merge 6 commits into
nextfrom
perf/background_native_save

Conversation

@BotellaA

@BotellaA BotellaA commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Why

Importing a large model (284 MB .og_strm) waited for the native re-save before answering, about 4.5 s that the viewer does not need: it only reads the viewables.

What

  • save_viewable_file answers once the viewables are written. The native re-save still runs for every input, native or not, so files stay in the latest format. It now runs on a background executor, one entry per data id, and saves of the same data are chained.
  • Everything that reads, writes or deletes the native file waits for the save first:
    • load_geode_object (covers most routes)
    • load_crs_object (CRS conversion and assignment)
    • apply_time_series
    • export_project (waits for every pending save)
    • the import rollback and the data folder reset
  • A failed save is retried once. If it fails again, the request that needs the file returns a 500.

Measured

Vease, 3_meshed_300m.og_strm, mean of 3 runs:

Before After
Server answers the import 10.07 s 6.28 s
Viewer starts loading 10.68 s 7.19 s
First image 18.03 s 16.04 s

Tests

  • New tests/test_background_native_save.py (12 tests): the route answers while the save is blocked, readers wait, failed saves, parallel imports, export, and a CRS assignment made during the save.
  • Full suite: 184 passed. ruff and mypy are clean.

Limits

  • The registry is in memory, which assumes one back process.
  • If the back restarts during a save, the native file is missing and the data must be imported again.

🤖 Generated with Claude Code

save_viewable_file now answers once the viewables are written. The
native re-save (kept so files are always written in the latest format)
runs on a background executor, one entry per data id, saves of the same
data chained.

Every reader, writer or deleter of the native file waits for it first:
load_geode_object, load_crs_object, apply_time_series, export_project,
the import rollback and the data folder reset. A failed save is retried
once, then the request needing the file fails with a 500.

On a 284 MB structural model, the server answers in 6.3 s instead of
10.1 s and the first image arrives 2 s earlier.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants