Skip to content

Release netCDF memory-map pages after each frame - #5463

Open
erikfransson wants to merge 12 commits into
MDAnalysis:developfrom
erikfransson:ncdf-release-mmap-pages
Open

erikfransson wants to merge 12 commits into
MDAnalysis:developfrom
erikfransson:ncdf-release-mmap-pages

Conversation

@erikfransson

@erikfransson erikfransson commented Aug 16, 2026 •

Copy link
Copy Markdown

fixes #5541

Description

NCDFReader._read_frame now drops the pages of scipy's memory map once the frame has been copied into the Timestep. Nothing else unmaps them, so without this the resident memory of the process grows towards the size of the whole trajectory file while it is read, as documented in discussion here. (ping @orbeckst )

MADV_DONTNEED does not exist on Windows, so the call is guarded by hasattr(mmap, "MADV_DONTNEED") and the reader is unchanged there.

Changes made in this Pull Request:

  • NCDFReader._read_frame now drops the pages of scipy's memory map once the
    frame has been read into the Timestep.
  • Added a test that the pages are dropped when a frame is read.

LLM / AI generated code disclosure

LLMs or other AI-powered tools (beyond simple IDE use cases) were used in this contribution: yes test written by AI

PR Checklist

  • Issue raised/referenced?
  • Tests updated/added?
  • Documentation updated/added?
  • package/CHANGELOG file updated?
  • Is your name in package/AUTHORS?
  • I have read and understand the current AI Policy
  • LLM/AI disclosure was updated.

Developers Certificate of Origin

I certify that I can submit this code contribution as described in the Developer Certificate of Origin, under the MDAnalysis LICENSE.

NCDFReader._read_frame now drops the pages of scipy's memory map once
the frame has been copied into the Timestep. Nothing else unmaps them,
so without this the resident memory of the process grows towards the
size of the whole trajectory file while it is read.

The map is read only, so dropping is safe: pages are faulted back in if
the data are read again. MADV_DONTNEED does not exist on Windows, where
the call is skipped and the reader is unchanged.
@codecov

codecov Bot commented Aug 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.93%. Comparing base (951f305) to head (5adbad9).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #5463      +/-   ##
===========================================
- Coverage    93.93%   93.93%   -0.01%     
===========================================
  Files          184      184              
  Lines        22675    22680       +5     
  Branches      3227     3228       +1     
===========================================
+ Hits         21300    21304       +4     
- Misses         915      916       +1     
  Partials       460      460              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread package/MDAnalysis/coordinates/TRJ.py Resolved
SimpleNamespace(madvise=advice.append),
)
universe.trajectory[1]
assert advice == [mmap.MADV_DONTNEED]

@tylerjereddy tylerjereddy Aug 31, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is a bit convoluted with monkeypatch--could we avoid that?

Also, am I understanding correctly that on this branch something like https://github.com/pythonprofilers/memory_profiler should show a constant level of physical memory consumption, while it should increase linearly when reading in/iterating over a trajecotry of the appropriate type on the develop branch?

We could probably capture that with https://asv.readthedocs.io/en/stable/writing_benchmarks.html#peak-memory as well, assuming you're saying that physical memory is being used excessively on develop in these scenarios.

@erikfransson erikfransson Sep 5, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea I agree, I removed the test.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But yes I ran memory profilers locally on 4GB sized trajectory.
Reading frame by frame with develop leads to memory usage climbing steadily and reaching peak memory of 4GB since nothing unmaps the scipys memory map.
With the fix in place the memory didnt climb at all and behaved as expected when reading frame by frame.

@orbeckst orbeckst added the AI-assisted Generated with AI/LLM assistance label Sep 4, 2026
Review feedback: a versionchanged directive is not needed for a bug fix
with no user-facing behaviour change, and the test only asserted that
madvise is called with a particular flag, which is an implementation
detail. The values read from the trajectory are unchanged and already
covered by the existing NCDF reader tests.
@erikfransson

Copy link
Copy Markdown
Author

@tylerjereddy @orbeckst Anything else you would like me to do?

Comment thread package/CHANGELOG Resolved
@orbeckst

orbeckst commented Oct 2, 2026

Copy link
Copy Markdown
Member

@tylerjereddy would you be comfortable giving an "approve" or "request changes" review?

@orbeckst orbeckst self-assigned this Oct 2, 2026
@orbeckst orbeckst linked an issue Nov 1, 2026 that may be closed by this pull request

@orbeckst orbeckst left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks sensible to me. However, we don't know if the problem also exists on Windows (or macOS). I'd like to be able to state clearly which OS is affected and on which the fix solves the problem.

Comment thread package/CHANGELOG Outdated
Comment on lines +26 to +27
that provides `MADV_DONTNEED`, i.e. not Windows (Discussion #4792,
PR #5463).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
that provides `MADV_DONTNEED`, i.e. not Windows (Discussion #4792,
PR #5463).
that provides `MADV_DONTNEED`, i.e. not Windows (#5541, PR #5463)

I raised a proper issue from the discussion #5541

@tylerjereddy

Copy link
Copy Markdown
Member

@tylerjereddy would you be comfortable giving an "approve" or "request changes" review?

I'm a bit slow getting to this, but I'll make a calendar reminder to look more closely. I may still be slow (few days) though.

@orbeckst

orbeckst commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

ncdf test on macOS

I tested develop and the code from this branch on macOS 15.7.9 M1 (arm64) Python 3.14. The mmap module appears to have the necessary advising machinery

>>> import mmap
>>> mmap.MADV_DONTNEED
4

In the script below I am creating longer test trajectories from trajectories included in MDAnalysisTests

2.2G long.trr
184M long.nc

mem_ncdf_mmap.py

I used the following script

import os.path
import MDAnalysis as mda
#from memory_profiler import profile

from MDAnalysisTests import datafiles as data

files = {
    "trajectory": "long.nc",
    "basetraj": data.NCDF,
    "topology": data.PRMncdf,
}

def create_sample_traj(repeat=200, filename=files["trajectory"]):
    if os.path.exists(filename):
        return filename
    print("creating sample trajectory...")
    u = mda.Universe(files["topology"], repeat*[files["basetraj"]])
    u.atoms.write(filename, frames="all")
    return filename

@profile
def loaduniverse(topol, trj):
    print(f"load universe: {topol}, {trj}")
    u = mda.Universe(files["topology"], trj)
    return u
    

@profile
def mainloop(u):
    print("loop over frames")
    for ts in u.trajectory:
        _ = sum(u.atoms.positions) # access data
    print("done")
    
if __name__ == "__main__":
    trj = create_sample_traj()
    u = loaduniverse(files["topology"], trj)
    mainloop(u)

with

mprof run --python ../mem_ncdf_mmap.py
mprof plot

I don't see any difference in the memory consumption as shown in the plots below:

develop

Figure_1_develop

Memory increases during trajectory iteration.

ncdf-release-mmap-pages

Figure_1_fix

Memory increases during trajectory iteration.

trr test (control)

Modify code with

files = {
    "trajectory": "long.trr",
    "basetraj": data.TRR,
    "topology": data.TPR,
}

Same output for both versions of the code

Figure_1_trr

Memory does not increase during trajectory iteration.

@orbeckst

Copy link
Copy Markdown
Member

I can confirm that on Linux the fix removes the memory leak

develop

ncdf_mem_linux_develop

Memory increases during trajectory iteration (bug)

ncdf-release-mmap-pages

ncdf_mem_linux_fixed

Memory does not increase during trajectory iteration and the fix is working.

trr test (control)

Same output for both versions of the code

ncdf_mem_linux_trr

Memory does not increase during trajectory iteration.

@orbeckst

Copy link
Copy Markdown
Member

It would be really useful if someone could confirm if the memory leak also exists on Windows.

@orbeckst orbeckst left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principle I am fine with the fix. It does NOT fix the issue on macOS (despite the fact that macOS can use mmap.advise(MADV_DONTNEED)).

I would like to know if the problem exists on Windows (I suspect yes but don't have a machine to test on).

I updated CHANGELOG and comments to be a bit more specific, especially since we should eventually fix the problem for other OS, too.

@orbeckst

Copy link
Copy Markdown
Member

All tests pass except macOS — apparently, this runner does not exist (?!), see #5548

@IAlibay

IAlibay commented Nov 13, 2026

Copy link
Copy Markdown
Member

All tests pass except macOS — apparently, this runner does not exist (?!), see #5548

Fixed in #5549

@orbeckst

Copy link
Copy Markdown
Member

I've been doing some exploring on macOS with AI support. The preliminary conclusion is that on macOS, there's not really an issue because the kernel can reclaim the memory when it needs to. It is reported as part of RSS (resident size, which is the mem reported by mprof) but not in the phys_footprint (footprint (1) was helpful in learning more about memory in macOS). There's some indication that mmap.advise(MADV_DONTNEED) may slow down the reader in macOS so we might actually want to not use it for darwin. On macOS (and POSIX) MADV_DONTNEED is only advisory while on Linux it apparently leads to immediate freeing of the memory.

I haven't properly checked if the Linux version is slowed down; from my previous tests in #5463 (comment) I see that the time for mainloop is almost the same (~12 s) so it might not matter.

The LLM (Claude/Opus 5.5) did suggest the code change below, which calls msync(..., MS_INVALIDATE) (directly from libc) and indeed that one leads to flat measured RSS memory with mprof as seen in the figure.

image

The code, however, looks pretty hacky (do we really want to directly do libc calls??) and it also leads to some measured slow-downs (~3x) , so I am currently inclined to not do anything on macOS until it's confirmed to be a problem.

Given the slow-down of MADV_DONTNEED and the fact that it does not do anything useful on macOS we should not use MADV_DONTNEED on macOS.

timings for different approaches

macOS 15.7 (Apple Silicon), NCDFReader iteration time (mean wall-clock time, range in brackets; fresh process per run, page cache warmed with an untimed develop pass)

Release call after each frame Runs long.nc (192 MB, 6000 frames) Slowdown vs develop longtrr.nc (1.1 GB, 2000 frames) Slowdown vs develop
none (develop) 8 0.181 s (0.180–0.182) 1.0× 0.895 s (0.893–0.899) 1.0×
madvise(MADV_DONTNEED) (original PR) 5 2.540 s (2.527–2.549) 14.1× 8.288 s (7.790–8.697) 9.3×
msync(MS_INVALIDATE) 5 0.715 s (0.690–0.735) 4.0× 2.999 s (2.988–3.016) 3.3×
none, run right after an MS_INVALIDATE run 5 0.448 s (0.437–0.482) 2.5× 2.306 s (2.271–2.399) 2.6×
MADV_DONTNEED skipped on macOS 3 0.194 s (0.181–0.220) 1.0× 0.919 s (0.890–0.965) 1.0×

The last row comes from a separate run; its slowdown is relative to the develop runs paired with it in that session (0.196 s for long.nc, 0.913 s for longtrr.nc).

LLM-generated timing script time_release.py (needs the mmap_strategies.py module
"""Time NCDF iteration with different page-release calls (macOS).

usage: python time_release.py {develop,madv_dontneed,branch} TRAJ

develop        no release call (the reader as on develop)
madv_dontneed  mm.madvise(MADV_DONTNEED) after each frame (original PR)
branch         _release_mmap_pages() as currently in TRJ.py
"""
import mmap
import sys
import time

import MDAnalysis as mda
from MDAnalysis.coordinates import TRJ
from MDAnalysisTests import datafiles as data


def madv_dontneed(ncfile):
    ncfile._mm.madvise(mmap.MADV_DONTNEED)


RELEASE = {
    "develop": lambda ncfile: None,
    "madv_dontneed": madv_dontneed,
    "branch": TRJ._release_mmap_pages,
}

if len(sys.argv) > 3 or "remap" in sys.argv:
    from mmap_strategies import s_remap
    RELEASE["remap_fixed"] = s_remap

name, traj = sys.argv[1], sys.argv[2]
TRJ._release_mmap_pages = RELEASE[name]

top = data.GRO if "trr" in traj else data.PRMncdf
u = mda.Universe(top, traj)
ag = u.atoms
t0 = time.perf_counter()
for ts in u.trajectory:
    ag.positions.sum()
dt = time.perf_counter() - t0
n = u.trajectory.n_frames
print(f"{name} {traj} {dt:.3f} s {1e6 * dt / n:.1f} us/frame")
LLM-generated mmap_strategies.py module. Mocks different ways to release mmap'ed pages. Requires a patched version of the current PR that uses the function TRJ._release_mmap_pages(ncfile)
"""Compare page-release strategies for the NCDFReader mmap on macOS.

usage: python mmap_strategies.py STRATEGY [trajectory]

Prints RSS and phys_footprint (what Activity Monitor shows) every N frames.
"""
import ctypes
import ctypes.util
import mmap
import os
import sys
import time

import psutil

import MDAnalysis as mda
from MDAnalysis.coordinates import TRJ
from MDAnalysisTests import datafiles as data

libc = ctypes.CDLL(ctypes.util.find_library("c"), use_errno=True)
libc.msync.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int]
libc.madvise.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int]
libc.mmap.restype = ctypes.c_void_p
libc.mmap.argtypes = [ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int,
                      ctypes.c_int, ctypes.c_int, ctypes.c_longlong]

# <sys/mman.h> on Darwin
MS_ASYNC, MS_INVALIDATE, MS_SYNC = 0x1, 0x2, 0x10
MS_KILLPAGES, MS_DEACTIVATE = 0x4, 0x8
MAP_SHARED, MAP_FIXED, PROT_READ = 0x1, 0x10, 0x1


class rusage_info_v2(ctypes.Structure):
    _fields_ = [("uuid", ctypes.c_uint8 * 16)] + [
        (n, ctypes.c_uint64) for n in (
            "user_time system_time pkg_idle_wkups interrupt_wkups pageins "
            "wired_size resident_size phys_footprint proc_start_abstime "
            "proc_exit_abstime child_user_time child_system_time "
            "child_pkg_idle_wkups child_interrupt_wkups child_pageins "
            "child_elapsed_abstime diskio_bytesread diskio_byteswritten"
        ).split()]


libproc = ctypes.CDLL("/usr/lib/libproc.dylib")


def mem():
    ri = rusage_info_v2()
    libproc.proc_pid_rusage(os.getpid(), 2, ctypes.byref(ri))
    return ri.resident_size / 2**20, ri.phys_footprint / 2**20


def pageins():
    ri = rusage_info_v2()
    libproc.proc_pid_rusage(os.getpid(), 2, ctypes.byref(ri))
    return ri.pageins


def region(nc):
    buf = nc._mm_buf
    return buf.ctypes.data, buf.nbytes


def check(rc, what):
    if rc != 0:
        e = ctypes.get_errno()
        raise OSError(e, f"{what}: {os.strerror(e)}")


def s_none(nc):
    pass


MADV_DONTNEED = mmap.MADV_DONTNEED


def s_madv_dontneed(nc):
    nc._mm.madvise(MADV_DONTNEED)


def s_madv_free(nc):
    nc._mm.madvise(mmap.MADV_FREE)


def s_madv_free_reusable(nc):
    nc._mm.madvise(mmap.MADV_FREE_REUSABLE)


def _msync(flags):
    def f(nc):
        addr, n = region(nc)
        check(libc.msync(addr, n, flags), f"msync({flags:#x})")
    return f


def s_remap(nc):
    addr, n = region(nc)
    r = libc.mmap(addr, n, PROT_READ, MAP_SHARED | MAP_FIXED,
                  nc.fp.fileno(), 0)
    if r != addr:
        raise OSError(ctypes.get_errno(), "mmap MAP_FIXED failed")


STRATEGIES = {
    "none": s_none,
    "madv_dontneed": s_madv_dontneed,
    "madv_free": s_madv_free,
    "madv_free_reusable": s_madv_free_reusable,
    "ms_invalidate": _msync(MS_INVALIDATE),
    "ms_deactivate": _msync(MS_DEACTIVATE),
    "ms_killpages": _msync(MS_KILLPAGES),
    "remap_fixed": s_remap,
}

if __name__ == "__main__":
    name = sys.argv[1]
    traj = sys.argv[2] if len(sys.argv) > 2 else "../long.nc"
    release = STRATEGIES[name]

    # disable the madvise in the branch so that only `release` is active
    orig_read_frame = TRJ.NCDFReader._read_frame
    madv = mmap.MADV_DONTNEED
    del mmap.MADV_DONTNEED
    try:
        def _read_frame(self, frame):
            ts = orig_read_frame(self, frame)
            release(self.trjfile)
            return ts
        TRJ.NCDFReader._read_frame = _read_frame

        top = data.GRO if "trr" in traj else data.PRMncdf
        u = mda.Universe(top, traj)
        n = u.trajectory.n_frames
        r0, f0 = mem()
        t0, p0 = time.perf_counter(), pageins()
        print(f"{name}: n_frames={n} start rss={r0:.0f} MiB footprint={f0:.0f} MiB")
        ref = None
        rmax = fmax = 0
        for ts in u.trajectory:
            s = u.atoms.positions.sum()
            if ts.frame == 0:
                ref = s
            r, f = mem()
            rmax, fmax = max(rmax, r), max(fmax, f)
            if ts.frame % (n // 5) == 0 or ts.frame == n - 1:
                print(f"  frame {ts.frame:6d} rss={r:7.1f} MiB footprint={f:7.1f} MiB")
        dt, dp = time.perf_counter() - t0, pageins() - p0
        print(f"  loop time={dt:.2f} s, pageins={dp}")
        # data still correct after release?
        u.trajectory[0]
        assert u.atoms.positions.sum() == ref, "data corrupted"
        print(f"  RESULT {name}: rss growth={rmax - r0:.1f} MiB, "
              f"footprint growth={fmax - f0:.1f} MiB, data OK")
        u.trajectory.close()
    finally:
        mmap.MADV_DONTNEED = madv

libc.msync(MS_INVALIDATE) patch

Only for illustration – don't use:

diff --git a/package/MDAnalysis/coordinates/TRJ.py b/package/MDAnalysis/coordinates/TRJ.py
index 7e67e9c1d..cbf1909a8 100644
--- a/package/MDAnalysis/coordinates/TRJ.py
+++ b/package/MDAnalysis/coordinates/TRJ.py
@@ -135,6 +135,9 @@ import warnings
 import errno
 import logging
 import mmap
+import sys
+import ctypes
+import ctypes.util
 from math import isclose
 
 import MDAnalysis
@@ -144,6 +147,33 @@ from ..lib import util
 from ..lib.util import store_init_arguments
 logger = logging.getLogger("MDAnalysis.coordinates.AMBER")
 
+if sys.platform == "darwin":
+    _libc = ctypes.CDLL(ctypes.util.find_library("c"), use_errno=True)
+    _libc.msync.argtypes = (ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int)
+    _MS_INVALIDATE = 0x0002  # <sys/mman.h>
+
+
+def _release_mmap_pages(ncfile):
+    """Drop the resident pages of a read-only, mmapped netcdf file.
+
+    The data stay in the file, so pages are faulted in again on the next
+    access. Without this, the resident memory of the process grows towards
+    the size of the file while iterating (Issue #5541).
+
+    On Linux, ``madvise(MADV_DONTNEED)`` unmaps the pages. On macOS,
+    ``MADV_DONTNEED`` only moves the pages to the inactive queue and leaves
+    them mapped, so ``msync(MS_INVALIDATE)`` is used instead. On Windows,
+    nothing is done.
+    """
+    mm = getattr(ncfile, "_mm", None)
+    buf = getattr(ncfile, "_mm_buf", None)
+    if mm is None or buf is None:
+        return
+    if sys.platform == "darwin":
+        _libc.msync(buf.ctypes.data, buf.nbytes, _MS_INVALIDATE)
+    elif hasattr(mmap, "MADV_DONTNEED"):
+        mm.madvise(mmap.MADV_DONTNEED)
+
 
 try:
     import netCDF4
@@ -677,11 +707,7 @@ class NCDFReader(base.ReaderBase):
         ts.frame = frame  # frame labels are 0-based
         self._current_frame = frame
         # the frame is now copied into ts, so its pages can be dropped again
-        # (see #5541 and PR #5463 - only works on Linux; macOS has MADV_DONTNEED
-        # but does not free the memory, Windows does not have it)
-        mm = getattr(self.trjfile, "_mm", None)
-        if mm is not None and hasattr(mmap, "MADV_DONTNEED"):
-            mm.madvise(mmap.MADV_DONTNEED)
+        _release_mmap_pages(self.trjfile)
         return ts
 
     def _reopen(self):

- does not reduce RSS but slows down reading
- updated CHANGELOG
- added extensive comment
@orbeckst

Copy link
Copy Markdown
Member

My plan is to merge this PR when the CI is happy (and when I look at it the next time...). If anyone has any blocking comments, please jump in.

@orbeckst

Copy link
Copy Markdown
Member

Not sure why the package check fails one test (RMSD) with a time out. I don't think any code touched here affects that test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-assisted Generated with AI/LLM assistance Component-Readers Format-Amber

Projects

None yet

Development

Successfully merging this pull request may close these issues.

memory leak in NCDF reader

4 participants