Repository navigation
Release netCDF memory-map pages after each frame - #5463
erikfransson wants to merge 12 commits into
Conversation
NCDFReader._read_frame now drops the pages of scipy's memory map once the frame has been copied into the Timestep. Nothing else unmaps them, so without this the resident memory of the process grows towards the size of the whole trajectory file while it is read. The map is read only, so dropping is safe: pages are faulted back in if the data are read again. MADV_DONTNEED does not exist on Windows, where the call is skipped and the reader is unchanged.
Documentation build overview
199 files changed ·
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #5463 +/- ##
===========================================
- Coverage 93.93% 93.93% -0.01%
===========================================
Files 184 184
Lines 22675 22680 +5
Branches 3227 3228 +1
===========================================
+ Hits 21300 21304 +4
- Misses 915 916 +1
Partials 460 460 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| SimpleNamespace(madvise=advice.append), | ||
| ) | ||
| universe.trajectory[1] | ||
| assert advice == [mmap.MADV_DONTNEED] |
There was a problem hiding this comment.
The test is a bit convoluted with monkeypatch--could we avoid that?
Also, am I understanding correctly that on this branch something like https://github.com/pythonprofilers/memory_profiler should show a constant level of physical memory consumption, while it should increase linearly when reading in/iterating over a trajecotry of the appropriate type on the develop branch?
We could probably capture that with https://asv.readthedocs.io/en/stable/writing_benchmarks.html#peak-memory as well, assuming you're saying that physical memory is being used excessively on develop in these scenarios.
There was a problem hiding this comment.
Yea I agree, I removed the test.
There was a problem hiding this comment.
But yes I ran memory profilers locally on 4GB sized trajectory.
Reading frame by frame with develop leads to memory usage climbing steadily and reaching peak memory of 4GB since nothing unmaps the scipys memory map.
With the fix in place the memory didnt climb at all and behaved as expected when reading frame by frame.
Review feedback: a versionchanged directive is not needed for a bug fix with no user-facing behaviour change, and the test only asserted that madvise is called with a particular flag, which is an implementation detail. The values read from the trajectory are unchanged and already covered by the existing NCDF reader tests.
|
@tylerjereddy @orbeckst Anything else you would like me to do? |
|
@tylerjereddy would you be comfortable giving an "approve" or "request changes" review? |
orbeckst
left a comment
There was a problem hiding this comment.
Overall this looks sensible to me. However, we don't know if the problem also exists on Windows (or macOS). I'd like to be able to state clearly which OS is affected and on which the fix solves the problem.
| that provides `MADV_DONTNEED`, i.e. not Windows (Discussion #4792, | ||
| PR #5463). |
There was a problem hiding this comment.
| that provides `MADV_DONTNEED`, i.e. not Windows (Discussion #4792, | |
| PR #5463). | |
| that provides `MADV_DONTNEED`, i.e. not Windows (#5541, PR #5463) |
I raised a proper issue from the discussion #5541
I'm a bit slow getting to this, but I'll make a calendar reminder to look more closely. I may still be slow (few days) though. |
|
It would be really useful if someone could confirm if the memory leak also exists on Windows. |
orbeckst
left a comment
There was a problem hiding this comment.
In principle I am fine with the fix. It does NOT fix the issue on macOS (despite the fact that macOS can use mmap.advise(MADV_DONTNEED)).
I would like to know if the problem exists on Windows (I suspect yes but don't have a machine to test on).
I updated CHANGELOG and comments to be a bit more specific, especially since we should eventually fix the problem for other OS, too.
|
All tests pass except macOS — apparently, this runner does not exist (?!), see #5548 |
|
I've been doing some exploring on macOS with AI support. The preliminary conclusion is that on macOS, there's not really an issue because the kernel can reclaim the memory when it needs to. It is reported as part of RSS (resident size, which is the mem reported by I haven't properly checked if the Linux version is slowed down; from my previous tests in #5463 (comment) I see that the time for mainloop is almost the same (~12 s) so it might not matter. The LLM (Claude/Opus 5.5) did suggest the code change below, which calls
The code, however, looks pretty hacky (do we really want to directly do libc calls??) and it also leads to some measured slow-downs (~3x) , so I am currently inclined to not do anything on macOS until it's confirmed to be a problem. Given the slow-down of MADV_DONTNEED and the fact that it does not do anything useful on macOS we should not use MADV_DONTNEED on macOS. timings for different approachesmacOS 15.7 (Apple Silicon), NCDFReader iteration time (mean wall-clock time, range in brackets; fresh process per run, page cache warmed with an untimed develop pass)
The last row comes from a separate run; its slowdown is relative to the develop runs paired with it in that session (0.196 s for LLM-generated timing script
|
- does not reduce RSS but slows down reading - updated CHANGELOG - added extensive comment
|
My plan is to merge this PR when the CI is happy (and when I look at it the next time...). If anyone has any blocking comments, please jump in. |
|
Not sure why the package check fails one test (RMSD) with a time out. I don't think any code touched here affects that test. |







fixes #5541
Description
NCDFReader._read_frame now drops the pages of scipy's memory map once the frame has been copied into the Timestep. Nothing else unmaps them, so without this the resident memory of the process grows towards the size of the whole trajectory file while it is read, as documented in discussion here. (ping @orbeckst )
MADV_DONTNEED does not exist on Windows, so the call is guarded by hasattr(mmap, "MADV_DONTNEED") and the reader is unchanged there.
Changes made in this Pull Request:
NCDFReader._read_framenow drops the pages of scipy's memory map once theframe has been read into the
Timestep.Added a test that the pages are dropped when a frame is read.LLM / AI generated code disclosure
LLMs or other AI-powered tools (beyond simple IDE use cases) were used in this contribution: yes test written by AI
PR Checklist
package/CHANGELOGfile updated?package/AUTHORS?Developers Certificate of Origin
I certify that I can submit this code contribution as described in the Developer Certificate of Origin, under the MDAnalysis LICENSE.