Repository navigation
Native wheels of one commit differ in speed: the published 26.11.1.post3 runs the single-record UPDATE about 32 percent slower than the CI build of the same commit #404
Description
In #400 I reported that the single-record UPDATE through command() is about 26 percent slower in the paper cell on the published 26.11.1.post3 wheel than on 26.11.1. I have now measured the cause, and it is not the source: two native wheels built from the same commit run the statement at different speeds.
The setup is a timed A/B on one host (an i9-12900HK, turbo off, two P cores, 8g heap, ABBA rounds, an A/A control that came out clean at 1.004 [0.988, 1.010]), in the process state of the cell: the async parallel load of LineItem, the Part load, the indexes and the warm-up, then begin, UPDATE Crud SET qty = 2 WHERE ckey=:c with a parameter, commit, 105 kept runs. The ratio is the median of the per-round ratios against the published 26.11.1 wheel, so below 1 is faster:
arm (native, command() UPDATE in a begin and commit pair) |
UPDATE against 26.11.1 (ratio, 95% interval) | clean pairs | verdict |
|---|---|---|---|
26.11.1.post3, built by CI from 55b427a (run 43520464492) |
0.976 [0.971, 0.981] | 20 | same |
| 26.11.1.post3, the wheel on PyPI (the same commit) | 1.323 [1.313, 1.331] | 20 | slower |
| 26.11.1.post2, the wheel on PyPI | 1.390 [1.385, 1.408] | 20 | slower |
The controls in the same runs (read and delete) moved much less than the update: read 1.034 (CI post3), 1.026 (PyPI post3), 1.046 (PyPI post2); delete 0.975 (CI post3), 1.032 (PyPI post3), 1.074 (PyPI post2).
The two 26.11.1.post3 wheels have the same SOURCE_COMMIT, the same recipe line in _build_info (GraalVM CE 25.4.4.1.1, -O3, -march=compatibility), the same engine jars (tools/post1_identity_check.py says the engine part is identical) and a libarcadedb.so of the same size (168,167,840 bytes). The binaries still differ: 147,260,377 of those bytes are not equal and the .text section is 17,472 bytes larger in the CI build than in the PyPI build. The Java sources of the native glue did not change between 26.11.1.post1 and 26.11.1.post2, so source changes do not explain the step from the first to the second wheel either.
What this means for the post3 release: the timed A/B and the gate run of #315 measured the CI-built wheels, and the wheel on PyPI is a different binary. The figures of that A/B describe the CI build, and a user who installs 26.11.1.post3 gets the slower build, at least on this path and on the host I measured. For post4 it also means a candidate's A/B against a published baseline can show a gain or a loss that is only the build.
I am reading the build logs of the two runs to find what differs. Two changes to the release process look right whatever the cause: publish the artifact that the gate measured instead of building a second one, and add a performance smoke on the wheel that is published (the single-record update, insert and read figures against the previous release, with a threshold).
Activity
tae898 commentedon Nov 14, 2026
I compared the logs of the runs that built the four wheels in the table of this issue. The "measured UPDATE" column is the ratio against the published
26.11.1from the table above.libarcadedb.so(bytes)All four builds use GraalVM CE 25.4.4.1.1 on
ubuntu-latest, the26.11.1engine and-O3 -march=compatibility -J-Xmx12g, with no--parallelism, no-H:NumberOfThreadsand no PGO (profile-guided optimization is not in the Community Edition). The two26.11.1.post3builds share the commit, the container digest and the options. The runner image does not separate fast from slow either:26.11.1(fast) and26.11.1.post2(slow) ran on the same image in about the same time. The analysis reports of the twopost3builds differ (611120 against 611308 KB) at the same number of used classes.The binaries differ everywhere, not in one place: 147,260,377 bytes of the two
post3libraries are not equal, the first difference is in the first function, and the code is laid out in a different order. As far as I read it, oracle/graal#291 describes the cause: a stock native-image build is not reproducible (the analysis and class initialization run in parallel, and identity hashes end up in the image heap), and I found no option that makes the Community Edition 25.4 reproducible. So I think every build of one commit is a different binary, and the speed of one hot path is a draw from a spread that I have not measured yet. In the study the draws move in both directions per figure (the publishedpost3wheel is faster than26.11.1on the insert and slower on the update), which fits inlining and layout, not a uniform slowdown.The published wheel is its own build:
release-native.ymlcallstest-native.ymlwithrelease: true, builds the library again, parity-tests that wheel and publishes it, and no earlier artifact is reused. The release gate checks correctness only; there is no speed figure in it.I do not think chasing bit-identical builds is the way. My proposal is a speed smoke in the release job, between
testandverify, on the wheel that job built: an interleaved ABBA run against the previously published wheel, the single-recordUPDATE ... WHERE ckey=:cin a begin and commit pair as the figure and insert, select and delete as controls, with limits set from an A/A run on the same runner (my first guess is 1.15 for the update and 1.20 for the controls). When the smoke fails the job builds again, up to three times, and publishes the first wheel that passes, with the library sha256 in the release notes. Pinning themanylinux_2_28image by digest andubuntu-latestto a named image removes drift between releases, but not this variance.Before I change the process I am building the
post3commit several times locally and measuring the spread of the update ratio, so that the limits come from data. If five builds are all fast, the cause is the runner or the time of day and not the build.tae898 commentedon Nov 14, 2026
Progress on the build spread. I ran four more builds of the
26.11.1.post3commit throughTest native (arcadedb)onrelease/26.11.1.post3: 43608193215, 43615069156, 43621985829 and 43628914957. The workflow cancels a run in progress on the same ref, so my first attempt (four dispatched at once) lost three runs and only 43608193215 survived; the other three then ran one at a time.All four succeeded, and the four
libarcadedb.sofiles differ from each other:libarcadedb.sosha256 (first 12)bf29ff9f5a8f91546beba8394c46877fd3f34691a6fce6183 of the four have 168,233,376 bytes and 1 has 168,167,840 bytes, which is the size of the published
post3library and of the first CI build. So builds of one commit differ in size as well as in content, which fits the non-reproducibility described above.Next is the speed. I am writing down the study before I run it: the single-record update figure for all six builds of this commit (the published one, the first CI build and these four) beside
26.11.1, on the host that measured the cell, and the release smoke limit then comes from that spread. I will post the table here when it is done.tae898 commentedon Nov 15, 2026
The laptop result for the same cell-like unit (the process state of the paper cell, E cores 6 and 7, turbo on, 65 kept runs, ABBA, an A/A of 0.996 [0.985, 1.001]) does not show the slowdown. The single-record update through
command()against26.11.1, as the ratio of the median per-round ratios (below 1 is faster):26.11.1.post1, the CI build of its tag26.11.1.post2, published26.11.1.post3, publishedSo the published
post3build, which ran at 1.323 on the mini, reads faster than26.11.1here. So whether a build is slow depends on the host as well. That is consistent with a layout or inlining effect that interacts with the core type (the laptop lane runs on E cores, the mini on P cores), but I have not shown that.This changes one part of my proposal above. A speed smoke in the release job would run on a GitHub runner, whose CPU is not the one that measured the cell, so a build that is slow on the mini can pass it. The limits and the host of the smoke should come from the spread study on the mini (the four extra builds are measured there next), and the smoke should at least run on the class of host that the published numbers come from.
tae898 commentedon Nov 15, 2026
The spread study is done. It measured the single-record update through
command()(the cell of the paper) for six builds of the26.11.1.post3commit55b427a51d2ebeside26.11.1, on the mini: solo lane, one pair of P-core threads, turbo off, ABBA order, 168 kept runs over rounds 0 to 20, and an A/A of 1.001 [0.996, 1.007]. The ratio is the median of the per-round ratios, below 1 is faster:3 of the six builds of one commit are about a third slower on this cell (P, E, F) and three are not, so the slowdown is a property of the build and not of the code. It does not follow the size of
libarcadedb.soeither: the published build and the CI builds B and G have the same size, and only the published one is slow; of the three builds with the other size, two are slow and one is not.Two things I did not show. The laptop (E cores, turbo on) did not show the slowdown for the published build, so which builds are slow depends on the host as well as on the build. And one control figure of arm B (the insert cell) did not resolve within the cap (its interval is wider than a tenth of its median), which is why the gate of the study says incomplete; the update figure above is not affected.
For the release smoke this means a limit has to be set from this spread on a host of the class that measured the cell, and a single build that reads fast is not evidence that the next one will. I will propose a limit and its host in a separate comment once I have the measured spread of the other figures of the cell, and I am happy to run the smoke on the mini first if you want it checked against real builds.