Native wheel: a tiny database costs about 80 MiB of resident memory over a bare Python process (study)

Observation

I run the native wheel (arcadedb==26.11.1.post2) inside a small host daemon, and a tiny database costs more resident memory than I expected. One process per engine at nice 19: open a new database, insert 1,000 rows of about 100 bytes, run one count(*), close. Python 3.12, Linux 7.0 on an Intel Core Ultra X9 388H, engine defaults except the last row. This is one machine and one run, and it is not a benchmark of speed or of larger data: it asks what an embedded engine costs in resident memory before it holds anything.

engine	peak resident (MiB)	over a bare Python process (MiB)	after the work: anonymous / file-backed (MiB)
`python`	12.5	0.0	4.0 / 8.6
`sqlite`	13.9	1.3	4.4 / 9.4
`duckdb`	83.6	71.0	23.2 / 44.4
`arcadedb-default`	93.0	80.5	33.2 / 59.8
`arcadedb-log-tuned`	89.9	77.3	29.3 / 60.5

The last row sets ARCADEDB_NATIVE_ISOLATE_ARGS="-Xmx64m -Darcadedb.maxPageRAM=8". It moved the total by about 3 MiB, so I think the floor is the runtime and not the caches, but I have not shown that.

Where it sits

/proc/self/smaps of a separate process doing the same kind of work (create a vertex type, insert 1,000 rows in one transaction, close), largest mappings first:

resident (MiB)	mapping
46.5	`libarcadedb.so`, the wheel's native library
22.0	anonymous memory
10.0	the Python interpreter
3.9	`[heap]`
1.9	`libc.so.6`
0.7	`libm.so.6`
0.4	`locale-archive`

libarcadedb.so is 160.5 MiB on disk, and 46.5 MiB of it is resident after this tiny workload. In real use inside the daemon the process stays at about 100 MiB resident on two hosts, because the library stays mapped after the database is closed.

What I do not know

Which pages of the library are touched at import, at create_database and at the first query, and how much of the resident part is never used again.
Whether the image heap or class initialization at startup decides the anonymous part.
How the JVM wheel (arcadedb-jvm) and the previous native wheel compare on the same workload.

Plan

Page-level breakdown of the library's mappings at three points: after import arcadedb, after create_database, after the workload.
The same workload on arcadedb-jvm and on the previous native wheel.
Check the native-image build options for the image heap and the GC and what each costs in resident memory.
If part of it is owned by the engine (for example what class initialization pulls in at startup), file a separate issue in ArcadeData/arcadedb with a reproducer on the engine and link it here.

Reproduce

"""One process, one engine: open a database, insert 1,000 rows of about 100 bytes, run one query, close. Prints one JSON line with the process memory (kB) from
/proc/self/status before the engine is imported and after the work (VmHWM is the peak). Not a benchmark: it answers 'what does an embedded engine cost in resident memory
for a tiny database'."""
import json
import sys
import time


def status():
    d = {}
    for line in open("/proc/self/status"):
        k, _, v = line.partition(":")
        if k in ("VmRSS", "VmHWM", "RssAnon", "RssFile"):
            d[k] = int(v.split()[0])
    return d


engine, path = sys.argv[1], sys.argv[2]
out = {"engine": engine, "before": status()}
t0 = time.perf_counter()
n = None
if engine == "python":
    n = 0
elif engine == "sqlite":
    import sqlite3

    c = sqlite3.connect(path + "/t.sqlite")
    c.execute("create table t(k integer primary key, v text)")
    c.executemany("insert into t values(?,?)", [(i, "x" * 100) for i in range(1000)])
    c.commit()
    n = c.execute("select count(*) from t").fetchone()[0]
    c.close()
elif engine == "duckdb":
    import duckdb

    c = duckdb.connect(path + "/t.duckdb")
    c.execute("create table t(k integer primary key, v varchar)")
    c.executemany("insert into t values(?,?)", [(i, "x" * 100) for i in range(1000)])
    n = c.execute("select count(*) from t").fetchone()[0]
    c.close()
elif engine.startswith("arcadedb"):
    import arcadedb

    raw = arcadedb.create_database(path + "/adb")
    raw.command("sql", "CREATE VERTEX TYPE T")
    raw.command("sql", "CREATE PROPERTY T.k INTEGER")
    with raw.transaction():
        for i in range(1000):
            raw.command("sql", "INSERT INTO T SET k = ?, v = ?", i, "x" * 100)
    n = raw.query("sql", "SELECT count(*) AS n FROM T").to_list()[0]["n"]
    raw.close()
else:
    sys.exit("unknown engine " + engine)
out["after"] = status()
out["rows"] = n
out["seconds"] = round(time.perf_counter() - t0, 3)
print(json.dumps(out))

Run it once per engine, for example uv run --python 3.12 --with arcadedb==26.11.1.post2 python -I memcmp.py arcadedb-default /some/empty/dir, with an empty directory per run. If someone already knows a build or runtime option that lowers the floor, tell me and I will measure it on the same workload.

Summary. After the 1,000-row workload 48.2 MiB of the 160.5 MiB native library is resident, almost all of it clean file-backed pages: 26.6 MiB in .text and 21.1 MiB in .svm_heap (the image heap); the library is not mapped at all until the first engine call. The JVM wheel peaks at 178.4 MiB against 92.8 MiB for the native one (1.9 times in peak resident, 2.1 times over a bare Python process), and the previous stable native wheel, 26.11.1, peaks 4.7 MiB lower than 26.11.1.post2, for a reason I did not find. The native build sets an optimization level and a CPU baseline but no GC or heap option, and the image holds the full server stack. The candidate changes at the end are hypotheses to measure; nothing was built or tried.

Results for the first three checklist items, all on the laptop from the issue (Intel Core Ultra X9 388H, Linux 7.0, Python 3.12, 4 KiB pages), every process at nice -n 19, one process at a time, each one fresh and a few seconds long, databases on disk. The workload is the one of memcmp.py: create a vertex type, insert 1,000 rows of about 100 bytes in one transaction, one count(*), close. I did not drop the page cache, so the library file was cached in every run. This is still one machine, and I read nothing here as a statement about larger data or about speed.

1. Where the native library's resident pages are

I wrote a probe in standard-library Python that runs the workload and, at five points, reads /proc/self/smaps and /proc/self/pagemap (the present bit of every page of every mapping of libarcadedb.so, readable without privileges). It parses the ELF section headers of the library with struct and maps each page to the section that holds it by its file offset, so the tables below are by ELF section. The library has the sections of a native-image shared library: .text (code, 110.3 MiB on disk), .svm_heap (the image heap, 49.6 MiB) and little else; there is no .symtab. The numbers are medians of 3 runs. The probe itself adds about 0.9 MiB of anonymous memory over memcmp.py (34.1 against 33.2 MiB at the end).

The library is not mapped at all after import arcadedb: the wheel loads it with ctypes.CDLL at the first call that needs the engine, so the "after import" column is zero and the 10.7 MiB that import arcadedb adds (8.2 MiB anonymous, 2.5 MiB file-backed, 205 modules in sys.modules) is the Python side only. I added a column for the point after the library and the isolate are loaded but before any database exists, to separate that step from create_database.

ELF section	size on disk, MiB	after the library and isolate load, MiB	after `create_database`, MiB	after the workload, MiB	of which private copies (copy-on-write) after the workload, MiB
`.text`	110.33	3.38	16.23	26.64	0.00
`.rodata`	0.10	0.10	0.10	0.10	0.00
`.svm_heap`	49.62	3.76	17.18	21.14	0.57
alignment gaps between sections	0.00	0.16	0.16	0.16	0.00
21 other small sections (.note.gnu.build-id, .gnu.hash, .dynsym, .dynstr, .gnu.version, .gnu.version_r, .rela.dyn, .rela.plt, .init, .plt, .fini, .eh_frame_hdr, .eh_frame, .init_array, .fini_array, .data.rel.ro, .dynamic, .got, .got.plt, .data, .bss)	0.13	0.14	0.14	0.14	0.02
all pages of the library (the file is 160.5 MiB)	160.50	7.54	33.80	48.19	0.59
cross-check: `Rss` summed over the library's mappings in `smaps`		7.54	33.80	48.19

After the workload 48.2 MiB of the library is resident (the issue had 46.5 MiB with a slightly different workload). .text holds 26.6 MiB of it (24% of its 110.3 MiB on disk, 55% of the resident library) and .svm_heap holds 21.1 MiB (43% of its 49.6 MiB); everything else together is 0.4 MiB. Only 0.59 MiB of the library's resident pages are private copies (copy-on-write), so what is resident is almost entirely clean file-backed pages, which the kernel can drop and re-read from the file. By phase the library's resident pages grow by 7.54 MiB when it is loaded, by 26.26 MiB in create_database and by 14.39 MiB in the workload, so creating the database costs more than the 1,000 inserts.

Were those pages touched by the engine, or only mapped ahead? I can bound it and I can say what the pattern looks like, but I cannot read per-page access information without root. First, Referenced equals Rss in every mapping at every checkpoint (true), so on its own it cannot separate the two. Second, I cleared the referenced bits at each checkpoint (/proc/self/clear_refs) in separate runs and read Referenced again at the next one. In the load phase all 7.54 MiB gained are referenced, in create_database 28.85 MiB are referenced for 26.26 MiB gained, and in the workload 22.25 MiB for 14.39 MiB gained, so in those phases only 2.6 and 7.9 MiB of the pages that were already resident were touched again; the rest of what was resident stayed untouched during the phase. Third, the present pages come in whole 64 KiB windows: 12144 of the library's 12337 present pages (98%) sit in windows that are completely present (759 of 779 windows), for .text 99% (423 of 431 windows). A completely present 64 KiB window is what the kernel's fault-around produces on a read fault when the pages are in the page cache (the kernel default is 64 KiB; the setting is in debugfs, which needs root, so I did not read it here); sparse single-page touches cannot produce it. So my reading is that most of the resident library is mapped by fault-around around a much smaller set of pages the engine really executes or reads. That set is at least one page per window with a present page, 3.0 MiB, and at most all 48.2 MiB; I cannot narrow it further from here.

The library's mappings by phase, with the referenced bits per phase

mapping	load: resident gained, MiB	load: referenced in the phase, MiB	create: resident gained, MiB	create: referenced in the phase, MiB	workload: resident gained, MiB	workload: referenced in the phase, MiB
`r--p` .svm_heap at 110.9 MiB	2.25	2.25	7.38	8.03	2.12	4.87
`rw-p` .svm_heap at 127.9 MiB	0.05	0.05	0.06	0.06	0.12	0.15
`rw-p` .svm_heap at 128.1 MiB	1.44	1.44	3.68	4.20	0.91	1.83
`r--p` .svm_heap at 138.0 MiB	0.00	0.00	2.30	2.30	0.81	0.95
`r--p` .rela.dyn	0.09	0.09	0.00	0.00	0.00	0.00
`r-xp` .text at 0.1 MiB	3.45	3.45	12.84	14.20	10.42	14.40
`r--p` .rodata at 110.6 MiB	0.12	0.12	0.00	0.04	0.00	0.03
`r--p` .data.rel.ro at 110.7 MiB	0.00	0.00	0.00	0.00	0.00	0.00
`rw-p` .svm_heap at 110.8 MiB	0.15	0.15	0.00	0.01	0.00	0.01
`rw-p` .bss (anonymous tail)	0.00	0.00	0.00	0.00	0.00	0.00
all mappings	7.54	7.54	26.26	28.85	14.39	22.25

The library's mappings at the end of the workload, with the 64 KiB window counts

mapping (permissions, ELF section, file offset)	size, MiB	`Rss`, MiB	`Anonymous`, MiB	`Referenced`, MiB	present pages (pagemap)	64 KiB windows with a present page	windows fully present	present pages inside fully present windows
`r--p` .svm_heap at 110.9 MiB	17.0	11.74	0.00	11.74	3006	188	186	99%
`rw-p` .svm_heap at 127.9 MiB	0.2	0.23	0.00	0.23	60	4	3	80%
`rw-p` .svm_heap at 128.1 MiB	9.8	6.02	0.54	6.02	1542	97	95	99%
`r--p` .svm_heap at 138.0 MiB	22.5	3.12	0.00	3.12	798	50	48	96%
`r--p` .rela.dyn	0.1	0.09	0.00	0.09	22	2	1	73%
`r-xp` .text at 0.1 MiB	110.4	26.71	0.00	26.71	6837	431	423	99%
`r--p` .rodata at 110.6 MiB	0.1	0.12	0.00	0.12	32	2	2	100%
`r--p` .data.rel.ro at 110.7 MiB	0.0	0.00	0.00	0.00	1	1	0	0%
`rw-p` .svm_heap at 110.8 MiB	49.8	0.15	0.04	0.15	38	4	1	42%
`rw-p` .bss (anonymous tail)	0.0	0.00	0.00	0.00	1	1	0	0%

The anonymous part is not in the library. Process-wide, by checkpoint:

checkpoint (native `26.11.1.post2`, median of 3)	resident total, MiB	anonymous, MiB	file-backed, MiB	of which `libarcadedb.so` pages, MiB
interpreter started	13.2	4.6	8.6	0.0
after `import arcadedb`	23.9	12.9	11.0	0.0
after the library and isolate load	32.3	13.2	19.0	7.5
after `create_database`	60.9	15.5	45.4	33.8
after the workload	93.8	34.1	59.6	48.2

The anonymous memory grows by 18.6 MiB during the workload (from 15.5 to 34.1 MiB), after create_database had added only 2.3 MiB. One anonymous mapping becomes the largest one in that phase:

checkpoint	three largest anonymous mappings: size (resident), MiB
isolate	7.0 (6.2), 1.4 (1.0), 0.4 (0.4)
create	7.0 (6.5), 1.5 (1.2), 1.4 (1.0)
work	17.5 (17.3), 7.0 (6.9), 1.5 (1.5)

With the default isolate arguments -XX:+PrintGC logs no collection at all during the workload, so the allocation of those 1,000 inserts stays in memory uncollected. With ARCADEDB_NATIVE_ISOLATE_ARGS="-Xmx64m -XX:+PrintGC" (the issue's last row) one collection runs, 13.00M->3.00M, and the anonymous memory at the end is 29.4 MiB instead of 33.2 MiB. I have not shown that the large anonymous mapping is the Java heap; the PrintGC result is consistent with it and not proof.

2. The same workload on the other builds

Each arm ran once as a warm-up that I do not count (it fills the uv environment and covers any first-start effect, and it is the last column), then three counted runs, interleaved arm by arm, each in a fresh process with an empty database directory. The newest earlier release of arcadedb on PyPI is 26.11.1 (uploaded 2026-11-05). 26.12.1.dev2 (2026-11-07) was also uploaded before 26.11.1.post2 but sorts above it as a version, so I measured it too. Values are the median with the minimum and maximum of the three runs; "over a bare Python process" subtracts the median of the bare Python runs.

engine	`libarcadedb.so` on disk, MiB	peak resident, MiB (median, min to max of 3)	over a bare Python process, MiB	after the work: anonymous, MiB	after the work: file-backed, MiB	first (warm-up) run, peak MiB
bare Python 3.12		12.5 (12.5 to 12.5)	0.0 (0.0 to 0.0)	4.0 (4.0 to 4.0)	8.5 (8.5 to 8.5)	12.5
`arcadedb==26.11.1.post2` (native, control)	160.5	92.8 (92.7 to 92.8)	80.3 (80.2 to 80.3)	33.2 (33.2 to 33.2)	59.5 (59.5 to 59.6)	92.6
`arcadedb==26.11.1` (native, previous stable release)	160.3	88.0 (88.0 to 88.1)	75.6 (75.5 to 75.6)	33.1 (33.1 to 33.1)	54.9 (54.9 to 55.0)	88.1
`arcadedb==26.12.1.dev2` (native, dev release of 2026-11-07)	165.6	90.1 (90.0 to 90.1)	77.6 (77.6 to 77.6)	33.8 (33.8 to 33.8)	56.3 (56.2 to 56.3)	90.0
`arcadedb-jvm==26.11.1.post2` (JVM through JPype)		178.4 (177.5 to 182.0)	165.9 (165.0 to 169.5)	137.0 (136.0 to 140.5)	41.5 (41.4 to 41.5)	173.4

The control reproduces the issue (80.3 MiB over bare Python, the issue had 80.5). The JVM wheel is 178.4 MiB peak, 165.9 MiB over bare Python, 2.1 times the native control; its anonymous memory is 137.0 MiB (native 33.2) and its file-backed memory 41.5 MiB (native 59.5). 26.11.1 is 4.7 MiB lower than 26.11.1.post2, and 26.12.1.dev2 is 2.7 MiB lower. All three native builds were built with the same GraalVM CE, -O3 and -march=compatibility and carry engine 26.11.1 (the dev build reports 26.12.1-SNAPSHOT), but from different commits of the bindings (2f28a32 for 26.11.1, 1454809 for post2).

I ran the same section probe on 26.11.1. The libraries are almost the same size on disk (160.3 and 160.5 MiB), but after the workload 26.11.1 has 43.6 MiB of the library resident and post2 has 48.2 MiB: post2 has 3.2 MiB more in .text and 1.4 MiB more in .svm_heap. I did not find the cause. It is not the engine release (both carry 26.11.1); it could be the Java glue, what the Python side calls, or the layout of a different build.

ELF section	`26.11.1`: size on disk, MiB	`26.11.1`: resident after the workload, MiB	`26.11.1.post2`: size on disk, MiB	`26.11.1.post2`: resident after the workload, MiB	difference, MiB
`.text`	110.17	23.42	110.33	26.64	+3.22
`.rodata`	0.10	0.10	0.10	0.10	+0.00
`.svm_heap`	49.62	19.75	49.62	21.14	+1.39
alignment gaps	0.00	0.16	0.00	0.16	+0.01
all pages of the library	160.31	43.56	160.50	48.19	+4.63

How the JVM wheel starts its JVM: it loads the JRE that is bundled in the wheel through JPype (jpype.startJVM) with the options the wheel builds in _build_jvm_args; I read them from the running JVM (RuntimeMXBean and the HotSpot diagnostic bean) after create_database. The two lists agree, and the wheel adds no -Xms, no -Xmx and no collector option, so the collector and the initial heap are the JVM defaults for this host (cores and RAM as in the table).

JVM setting in effect	value
JVM	OpenJDK 64-Bit Server VM, 25.0.5 (the JRE bundled in the wheel)
collector	G1 (`UseG1GC=true`, `UseSerialGC=false`; MXBeans: G1 Young Generation, G1 Concurrent GC, G1 Old Generation)
maximum heap	`-XX:MaxRAMPercentage=50` added by the wheel; `MaxHeapSize` = 15544 MiB on this host (half of its 30 GiB of RAM)
initial heap	no `-Xms`; `InitialHeapSize` = 488 MiB (JVM default, 1/64 of RAM); `Runtime.totalMemory()` = 32 MiB when probed
object headers	`-XX:+UseCompactObjectHeaders` added by the wheel (`UseCompactObjectHeaders=true`)
compressed oops / class pointers	true / true
G1 region size	8 MiB
code cache, compiler threads	`ReservedCodeCacheSize` = 240 MiB, `CICompilerCount` = 12, tiered up to level 4
AOT cache	off: `AOTCache` is empty in the running JVM (the wheel enables it only with `ARCADEDB_AOT` or `aot=True`, default off)
class-data sharing	the bundled JRE ships `classes_coh.jsa` (14.4 MiB) in `lib/server`; the JVM uses its default archive unless `-Xshare:off` (not verified in this run)
JVM threads when probed	10
cores seen	16

The JVM arguments in effect

The arguments, with the per-user log directory replaced by <home>:

--add-modules=jdk.incubator.vector
-Djava.awt.headless=true
--enable-native-access=ALL-UNNAMED
--sun-misc-unsafe-memory-access=allow
-Dfile.encoding=UTF8
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/java.nio.channels.spi=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
-Darcadedb.polyglotEngineEnabled=false
-XX:+UseCompactObjectHeaders
-XX:MaxRAMPercentage=50
-Darcadedb.server.logsDirectory=<home>/.local/state/arcadedb/log
-Djava.util.logging.config.file=<home>/.local/state/arcadedb/log/logging.properties
-XX:ErrorFile=<home>/.local/state/arcadedb/log/hs_err_pid%p.log

3. How the native library is built

I read the build files of the bindings repository at commit 749c804 (the checkout I had; nothing was built or changed). The library in the 26.11.1.post2 wheel records commit 1454809, which is not in that checkout, so I compared git blob IDs of the build files at that commit through the GitHub API: pom.xml, build-image.sh, native-image.properties, reachability-metadata.json, select_native_matrix.py are identical, and lib.py differs, which I therefore took from the installed wheel instead. The table lists the memory-relevant options; each value is extracted by script from the saved files, and the script also asserts that no --gc, -R:, MaxHeapSize, MinHeapSize, quickBuild or PGO option appears anywhere in them.

option	set today to	where
optimization level	`-O3`	`build-image.sh` passes it; the `pom.xml` default is `-Os`
target CPU	`-march=compatibility`	`build-image.sh` (baseline x86-64); the wheel records `GraalVM CE 25.4.4.1.1, -O3, -march=compatibility`
garbage collector	no `--gc=` option anywhere	GraalVM CE then uses its default collector, the generational one (`GC(0) Pause Incremental GC` appears in the PrintGC log of the `-Xmx64m` run in section 1)
heap limits at build time	no `-R:` option, no `MaxHeapSize` or `MinHeapSize`	searched `pom.xml`, `build-image.sh`, `native-image.properties`, the wheel loader
maximum heap at run time	`-XX:MaximumHeapSizePercent=50`	added by the wheel loader unless the user gives `-Xmx`, `-XX:MaximumHeapSizePercent=` or `-XX:MaxRAM=`
other isolate arguments	log directory and logging config only	`ARCADEDB_NATIVE_ISOLATE_ARGS` is appended in front (`-Xmx64m` of the issue's last row came this way)
image kind	shared library (`<sharedLibrary>true</sharedLibrary>`)	`native-maven-plugin` 1.1.14, `--no-fallback`
builder options (Linux x86-64 release build)	`-J-Dfastops.mode=1 -J-Xmx12g -H:-VectorAPISupport -H:+PrintAnalysisCallTree -march=compatibility`	`NATIVE_IMAGE_OPTIONS` in `build-image.sh` with the values of the Linux x86-64 entry of `select_native_matrix.py` (`NI_EXTRA` is empty there); `-J-Xmx` is the builder's heap, not the library's
vector API	`--add-modules=jdk.incubator.vector`, `-H:-VectorAPISupport`	jvector scalar provider; Highway library shipped beside the image
initialized at build time	`io.github.jbellis.jvector.vector.cnative.FastMode, io.github.jbellis.jvector.vector.cnative.FfmKernels`, `org.slf4j, org.slf4j.helpers, org.slf4j.spi, org.slf4j.event, org.slf4j.jul`	`pom.xml`; everything else follows the GraalVM default
initialized at run time	`org.wildfly.common.net.HostName`	`pom.xml`
force-inlined methods	9 jvector methods (`-H:PriorityForceInline`)	`native-image.properties`; resource bundle `XMLMessages` included
reachability metadata	819 reflection entries, 58 resource entries, 0 bundles	`reachability-metadata.json` (120791 bytes); `-H:IncludeResources=.*` is not used
URL protocols	`http,https`	`pom.xml`
engine modules in the image	10: `arcadedb-bolt`, `arcadedb-console`, `arcadedb-engine`, `arcadedb-graphql`, `arcadedb-integration`, `arcadedb-mcp`, `arcadedb-network`, `arcadedb-postgresw`, `arcadedb-redisw`, `arcadedb-server`	`pom.xml` dependencies (plus jvector 4.0.2, jboss-threads, smallrye-common, slf4j-jdk14)
engine modules left out	`com.arcadedb:arcadedb-gremlin`, `com.arcadedb:arcadedb-grpcw`, `com.arcadedb:arcadedb-ha-raft`, `com.arcadedb:arcadedb-metrics`, `com.arcadedb:arcadedb-mongodbw`, `com.arcadedb:arcadedb-tracing`, `com.fasterxml.jackson.core:`, `com.fasterxml.jackson.dataformat:`, `com.fasterxml.jackson.datatype:`, `org.apache.commons:commons-math3`, `org.graalvm.js:`, `org.graalvm.polyglot:js`, `org.graalvm.regex:`, `org.graalvm.shadowed:icu4j`, `org.graalvm.truffle:`, `org.jline:`, `org.xerial.snappy:`	`pom.xml` exclusions on every engine module (artifact ids; `*` = whole group)
symbols	no `.symtab` section in the shipped library	section headers of the shipped `libarcadedb.so`

In short: the build sets the optimization level (-O3) and the CPU baseline, no garbage collector and no heap option; the only heap setting is the run-time -XX:MaximumHeapSizePercent=50 that the Python loader adds. The image contains the full server stack (the Bolt, GraphQL, MCP, PostgreSQL and Redis wire modules, the console and the integration module), so most of .text is code that an embedded import arcadedb does not run.

Candidate changes, as hypotheses to measure, not findings

Drop the library's clean pages when they are not needed: call madvise (MADV_DONTNEED, MADV_COLD or MADV_PAGEOUT) on the library's file-backed mappings after create_database or when the last database closes. The read-only mappings (r-xp and r--p) hold 41.8 MiB of the 48.2 MiB resident, and those pages are clean, so an idle process could shrink by up to about that much. Only the read-only mappings are safe for MADV_DONTNEED: on the writable rw-p image-heap mappings it would discard pages the engine has modified. It needs no rebuild (it can live in the Python loader). Trade-off: the next call takes minor page faults again, and it does nothing for the peak while a query runs. To measure: resident size after close, and the latency of the first query afterwards.
-Os instead of -O3: .text is the larger resident part (26.6 of 48.2 MiB), and a pom.xml comment records a timed comparison for -O3 ("warm geomean 0.736 at 1.62x the library size", referenced as Native image: -O3 is faster than -O2 on GraalVM CE (warm geomean 0.736), at 1.62x the library size #159), which I read as faster code at 1.62 times the size. Trade-off: going back to -Os gives up that speed for a smaller image; whether the resident part shrinks as much as the file is unknown. To measure: one build each, the same workload, and the existing timed A/B.
A smaller image for embedded use: leave the wire-protocol and server modules (arcadedb-bolt, -graphql, -mcp, -postgresw, -redisw, and possibly -server, -console, -integration) out of a second image. Trade-off: two images to build and test, and the server features of arcadedb.server would need the full one. Resident pages count only what is touched, so the gain on resident memory could be much smaller than on disk; this needs a build to find out.
Start collecting earlier: smaller young generation or heap limit at run time through ARCADEDB_NATIVE_ISOLATE_ARGS (for example -XX:MaximumYoungGenerationSizePercent, or the -Xmx64m of the issue, which gave 3.9 MiB less anonymous memory in my PrintGC run). No rebuild needed. Trade-off: more frequent collections and lower throughput in bulk loads, and a hard heap cap that can fail a large import. To measure: anonymous memory and load throughput over a range of values, on the same workload and on a bulk load.
Shrink the image heap (.svm_heap, 21.1 MiB resident, 49.6 MiB on disk) by moving the build-time initialization of large static state to run time (--initialize-at-run-time for selected packages), guided by a per-class breakdown of the image heap. I have not checked which diagnostic flags GraalVM CE 25.4 offers for that breakdown. Trade-off: slower start and slower first use, and the risk of initialization order bugs at run time. To measure: build with the candidates, then repeat the section table above.

What is still unknown

Which resident pages the engine touches individually and which are only mapped by fault-around. The window pattern points to fault-around, but a per-page answer needs /proc/kpageflags or idle page tracking (root), or a run on a kernel with a smaller fault-around window.
Which anonymous mappings are the Java heap and which are other native memory (thread stacks, malloc arenas). The PrintGC runs fit the heap, they do not prove it.
Why 26.11.1.post2 has about 4.6 MiB more of the library resident than 26.11.1 when both carry engine 26.11.1 and the library has almost the same size.
The effect of any candidate above: nothing was built, and the madvise and isolate-argument ideas were not tried either.
Whether any of this holds on other hosts, larger data or another kernel. The JVM numbers also depend on the host: its maximum heap is half of the RAM it sees (15544 MiB here) and its initial heap a 1/64 of it (488 MiB).

I can attach the probe script, the report script and the raw JSON lines if that is useful.

Navigation Menu

Repository navigation

Native wheel: a tiny database costs about 80 MiB of resident memory over a bare Python process (study) #370

Observation

Where it sits

What I do not know

Plan

Reproduce

1. Where the native library's resident pages are

2. The same workload on the other builds

3. How the native library is built

Candidate changes, as hypotheses to measure, not findings

What is still unknown

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Native wheel: a tiny database costs about 80 MiB of resident memory over a bare Python process (study) #370

Description

Observation

Where it sits

What I do not know

Plan

Reproduce

Activity

tae898 commented on Nov 11, 2026

1. Where the native library's resident pages are

2. The same workload on the other builds

3. How the native library is built

Candidate changes, as hypotheses to measure, not findings

What is still unknown

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions