NM Dev benchmark charts
+NM Dev benchmark charts
Historical harness issue: ojAlgo scaling returned a deferred expression. Its scaling timings do not measure completed work. See the fairness audit. Raw measurements remain unchanged.
Hover, tap or keyboard-focus any point to inspect its measurements.
Lower time ratios are faster. NM Dev is the 1× reference. Confidence intervals describe timing uncertainty; the displayed ratio is a ratio of means. Results are from a shared VM and different library API paths.
diff --git a/benchmark/results/2026-10-18-fairness-audit/README.md b/benchmark/results/2026-10-18-fairness-audit/README.md new file mode 100644 index 0000000..bd293a0 --- /dev/null +++ b/benchmark/results/2026-10-18-fairness-audit/README.md @@ -0,0 +1,76 @@ +# Benchmark fairness audit — 18 October 2026 + +This audit repairs the measurement harness and expands coverage. It does not change nmdev's numerical implementations. [Current latest-version baseline](../2026-10-18-latest/README.md) · [Earlier fairness checkpoint](../2026-10-18-fairness-corrected/README.md) · [Harness source](../../src/main/java/dev/nm/benchmark) + +## Corrected deferred scaling + +The historical expanded ojAlgo adapter returned `a.multiply(1.5)`. In ojAlgo 57.5.0 that call returns a deferred scalar-operation store. Validation outside timing evaluated its entries, so passing numerical checks did not prove the timed call completed scaling. Returning the expression to a JMH blackhole did not force its entries to be calculated. + +The corrected adapter materializes nonphysical scalar/addition/inverse/solve outputs inside the timed call. Already physical stores are returned directly, avoiding an unnecessary extra copy. Every ojAlgo decomposition factor requiring materialization remains materialized. The trial setup rejects nonphysical ojAlgo matrix outputs, checks correctness twice and verifies exact native-input preservation for all adapters. A regression check explicitly rejects the old deferred expression. + +Historical JSON and numeric data are preserved. Their report/chart annotations identify the invalid scaling comparison. The corrected baseline reruns all 286 cases rather than mixing a few new measurements into the historical run. Its shorter timing settings are recorded in the new archive, so differences from the historical run cannot all be attributed to the adapter fix. + +## Verified SPD inverse path in the measured JAR + +The exact combined obfuscated JAR matches the historical artifact record: + +```text +SHA-256 a3e2b2492e79530b8e12920b48f48bfc295786b138aa26300a16b53bafe1386c +Bytes 4300123 +``` + +Bytecode confirms that `Inverse` checks symmetry and attempts Cholesky, with general LU inversion as a runtime-exception fallback. Separate JDI constructor-breakpoint runs observe the actual constructors without modifying or replacing any JAR classes: + +| Fixture | Chol constructors | LU constructors | Inverse residual | +|---|---:|---:|---:| +| Seeded SPD, 64×64 | 1 | 0 | 1.5543122344752192E-15 | +| Seeded SPD, 256×256 | 1 | 0 | 2.55351295663786E-15 | +| Seeded nonsymmetric control, 64×64 | 0 | 1 | 1.5543122344752192E-15 | + +The previous statement that nmdev takes a general inverse path for these SPD inputs was incorrect. Reports and the report generator now describe symmetry detection, Cholesky and fallback correctly. Debugger observation is outside all timing runs. + +Evidence: [bytecode](inverse-bytecode.txt), [64 SPD trace](inverse-spd-64-trace.txt), [256 SPD trace](inverse-spd-256-trace.txt), [nonsymmetric control](inverse-general-trace.txt). Reproduction commands are in the [guide](../../site/reproduce.md). + +## Expanded coverage and validation + +- Baseline: eleven libraries, thirteen operations, sizes 64/256; all 429 baseline correctness cases pass twice, with native-input checks. All 66 targeted pivot cases pass. +- Rectangular multiplication: four adapters (nmdev, EJML, ojAlgo, native jblas), four shapes including odd dimensions; all 16 independently checked cases pass twice. [Correctness record](rectangular-correctness.csv) and [latest timed track](../2026-10-18-latest-rectangular/README.md). +- Large square and controlled internal-thread profiles use sizes 512/1024 and explicit processor budgets. These tracks are not pooled with the baseline. A budget is a limit, not proof that an API uses all processors: the selected EJML DDRM routines remain sequential; native jblas worker counts are not independently audited. +- Exact JAR hashes, source snapshots, affinity requests, JMH overrides, GC/allocation metrics and raw measurements accompany each timed track. Shared-host interference remains possible; affinity is not exclusive core reservation. Native jblas worker counts are not independently audited. + +## Difficult-input accuracy findings + +The independent diagnostic has 165 cases across eleven libraries, with repeated execution and input-preservation checks. **162 pass and 3 fail.** All failures are retained in [the CSV](difficult-correctness.csv); the diagnostic exits nonzero. + +| Library | Fixture / operation | Result | +|---|---|---| +| nmdev | Row-scaled pivoting matrix, default MR3 SVD | Reconstruction/orthogonality check fails; combined residual 0.184660650255 | +| nmdev | Tightly clustered SPD eigenvalues, default MR3 SVD | Reconstruction/orthogonality check fails; combined residual 2.95076969792e-5 | +| EJML | Exactly rank-deficient matrix, QR | Selected decomposition API returns failure | + +The alternative nmdev Golub–Kahan SVD passes both difficult fixtures at residuals approximately 4.53e-14 and 4.29e-13. [Method comparison](svd-method-accuracy.csv). This identifies a follow-up accuracy investigation in the default MR3 path; no default algorithm was changed in this audit. + +Hilbert solves/inverses use normalized Frobenius backward error, not an unscaled absolute residual. All those cases pass that test. A small backward error does not guarantee accurate solutions on an ill-conditioned problem. Other diagnostics retain reconstruction and orthogonality checks with a common 1e-8 threshold. See [methodology](../../site/methodology.md) and [diagnostic log](difficult-correctness.log). + +## Latest-version rerun + +The [version audit](../../dependency-audit/2026-10-18/README.md) identified newer Commons Math and Parallel Colt distributions. The current run uses Commons Math 4.0-SNAPSHOT (timestamp pinned), Parallel Colt 0.11.4 (unmodified upstream release source), and today's validated NM Dev 1.2.6 obfuscated customer JAR. Other libraries match their latest official releases found. The primary Commons adapter imports math4 classes. + +The new NM Dev JAR is pinned by [CI provenance and SHA-256](../../dependency-audit/2026-10-18/nmdev-validated.json). Separate [64 SPD](latest-inverse-spd-64-trace.txt), [256 SPD](latest-inverse-spd-256-trace.txt), [nonsymmetric](latest-inverse-general-trace.txt) and [bytecode](latest-inverse-bytecode.txt) checks reproduce the same Cholesky/LU dispatch conclusion in the release binary. The original traces above remain evidence for the original published snapshot binary. + +The latest difficult-input diagnostic again records **162 passes and 3 failures out of 165 cases**, with the same two default MR3 SVD failures and one rank-deficient EJML QR rejection. See [latest diagnostics](latest-difficult-correctness.csv) and [SVD method comparison](latest-svd-method-accuracy.csv). The library's default algorithm has not been changed. + +A separate class-origin check confirms that all eleven primary implementations load from their selected JARs, including Math4 and Parallel Colt 0.11.4. See [loaded-class origins](latest-class-origins.csv). This untimed check also guards future runs against an older bundled implementation shadowing a chosen dependency. + +The [latest diagnostic source snapshot](latest-diagnostic-source.zip) and [checksums](latest-diagnostic-source-checksums.json) preserve the exact validation harness used with the updated dependencies. + +## Completed latest-version timing tracks + +| Track | Cases | Computational budget | Report | +|---|---:|---|---| +| Baseline, 64/256, thirteen operations, eleven libraries | 286 | 1; CPU 0 | [Results](../2026-10-18-latest/README.md) | +| Rectangular multiplication, four shapes, four libraries | 16 | 1; CPU 0 | [Results](../2026-10-18-latest-rectangular/README.md) | +| Large square, 512/1024, five operations, four libraries | 40 | 1; CPU 0 | [Results](../2026-10-18-latest-large-1thread/README.md) | +| Large square, same tasks and libraries | 40 | 4; CPUs 0–3 | [Results](../2026-10-18-latest-large-4threads/README.md) | + +All **382 timed cases** contain two complete forks with three measured iterations per fork and GC/allocation profiler data. Every profile passed the 429-case baseline and 66-case pivot guards before timing. Each timed fork also checked its selected task twice and verified native-input preservation. The large and rectangular runs used two 100 ms warmups and three 200 ms measurements; the baseline used two 200 ms warmups and three 200 ms measurements. These short shared-host experiments are diagnostic. Processor budgets do not guarantee parallel execution or exclusive cores, and means from separate tracks should not be pooled into a universal ranking. diff --git a/benchmark/results/2026-10-18-fairness-audit/diagnostic-source-checksums.json b/benchmark/results/2026-10-18-fairness-audit/diagnostic-source-checksums.json new file mode 100644 index 0000000..e5d7c5c --- /dev/null +++ b/benchmark/results/2026-10-18-fairness-audit/diagnostic-source-checksums.json @@ -0,0 +1,27 @@ +{ + "benchmark/src/main/java/dev/nm/benchmark/Validation.java": "7eb17188c9bf22932a94c59394354b31ce505f370a1634ea13a182ba12fe3c3d", + "benchmark/src/main/java/dev/nm/benchmark/JblasAdapter.java": "938f0adb605168e961bad80d96bd4deb5d90bbc9037d0807019c98acb331ab85", + "benchmark/src/main/java/dev/nm/benchmark/SvdAccuracyProbe.java": "a4cd82130e081f2350f966712a09853dcf1cd3e6d5b09105e4b5fc7f3c1369fa", + "benchmark/src/main/java/dev/nm/benchmark/ThreadSettings.java": "1b79e9a7cef2959d55361bbbdd9fdc22e1f3259b6e0db481bbb186b9f03978ab", + "benchmark/src/main/java/dev/nm/benchmark/FairnessValidationMain.java": "2b058d18841a0e745310a252b7594b82e289fa42bc1c919a1e1af033d2f3f74a", + "benchmark/src/main/java/dev/nm/benchmark/MatrixBenchmark.java": "d675226f8d4cb3403a2a62ec30ba463d011adc8c7fb6a105479706578eb1ce34", + "benchmark/src/main/java/dev/nm/benchmark/CommonsAdapter.java": "6dfba066a276b70ffd98a9cda8dab7d792b8e5f7c58c3942a49568f22b4d5e71", + "benchmark/src/main/java/dev/nm/benchmark/CorrectnessMain.java": "41fd00dc5e94666bd63aa640f5599d34273816dfbf53cafe94a7fc3093cc45bb", + "benchmark/src/main/java/dev/nm/benchmark/Inputs.java": "f4422edfb0014234cc475d2631c5164cc786f5204bd4e058025d6ca25f63df05", + "benchmark/src/main/java/dev/nm/benchmark/PivotValidationMain.java": "0afecbf48bf341ca4ca86c9dfcbc767b64d492ff2cac0411fdd5b41686c4b866", + "benchmark/src/main/java/dev/nm/benchmark/Adapter.java": "91143bb099e9bd412c7a6315a67c956c944aa7cff8cfc934c3255c3de74e072a", + "benchmark/src/main/java/dev/nm/benchmark/RectangularBenchmark.java": "9868a8b94321aebbc772b423e996374b5cdee6aec924a7a22d527c3bbe45ebff", + "benchmark/src/main/java/dev/nm/benchmark/UjmpAdapter.java": "9f4a4426ddfbf2ba600599e7956115c5a8922cea6e6dcc8cd66698383cb8b691", + "benchmark/src/main/java/dev/nm/benchmark/RectangularValidationMain.java": "55a4574473d258e27987aecad5a3e40de01776a64ac8246ad15f877073b63ff7", + "benchmark/src/main/java/dev/nm/benchmark/MtjAdapter.java": "bc61b8ba046e7d96b9ebea5a38f2d034004d6fea117a06fe9bda53bdc3a6baf6", + "benchmark/src/main/java/dev/nm/benchmark/ColtAdapter.java": "aa847065f5ddcc984beef44aa291d76d1b64a9f479905bff06cd170098b04c64", + "benchmark/src/main/java/dev/nm/benchmark/JamaAdapter.java": "00f781cef788645708fe630cf8a324f547db18559e6cdba51cb35fbbe8983c33", + "benchmark/src/main/java/dev/nm/benchmark/F2jLapackAdapter.java": "c9c5b20fa94fa09cea7fd7164d8cdf463384e30f5e17a760916b2362edeed084", + "benchmark/src/main/java/dev/nm/benchmark/ParallelColtAdapter.java": "ce923267ed70f7ffc0ca78d4864f18e03f8cb0f858e18bf653ee6249869ebbe4", + "benchmark/src/main/java/dev/nm/benchmark/InversePathProbe.java": "38a91b35dc44ba7a19e40b8e9cba4c53d43f08f77e0b3b00366e87e560377208", + "benchmark/src/main/java/dev/nm/benchmark/EjmlAdapter.java": "a8e97b5f25119e987a4c254526e2f4f4ac16b07508af2478c863c2203afdb3e9", + "benchmark/src/main/java/dev/nm/benchmark/NmAdapter.java": "9d56aec69b2b4479b6e50929fbd5a06317b1224b82f8535544204b223257e4f8", + "benchmark/src/main/java/dev/nm/benchmark/DifficultValidationMain.java": "98b0579d2b0c441057db1ed399d1318a38db672bf655a69051efe2403b1e29e2", + "benchmark/src/main/java/dev/nm/benchmark/OjalgoAdapter.java": "caddc0e8806086771c4000fe12a94059ec9336b62bd80f7f312ddaf2bb9066db", + "benchmark/scripts/InversePathTrace.java": "0a55ab2122833be223744da94e7e85539ef3e6157913769a9ae187f43632f40e" +} diff --git a/benchmark/results/2026-10-18-fairness-audit/diagnostic-source.zip b/benchmark/results/2026-10-18-fairness-audit/diagnostic-source.zip new file mode 100644 index 0000000..c82f643 Binary files /dev/null and b/benchmark/results/2026-10-18-fairness-audit/diagnostic-source.zip differ diff --git a/benchmark/results/2026-10-18-fairness-audit/difficult-correctness.csv b/benchmark/results/2026-10-18-fairness-audit/difficult-correctness.csv new file mode 100644 index 0000000..ad51a4e --- /dev/null +++ b/benchmark/results/2026-10-18-fairness-audit/difficult-correctness.csv @@ -0,0 +1,166 @@ +library,fixture,size,operation,residual,status +NMDEV,scaled-pivot,8,lu,3.00894310416e-18,PASS +NMDEV,scaled-pivot,8,solve,1.54937410247e-18,PASS +NMDEV,scaled-pivot,8,inverse,4.46965765689e-18,PASS +NMDEV,scaled-pivot,8,qr,8.88178419700e-16,PASS +NMDEV,scaled-pivot,8,svd,0.184660650255,FAIL +NMDEV,clustered-spd,8,eigenSymmetric,1.50017886028e-12,PASS +NMDEV,clustered-spd,8,svd,2.95076969792e-05,FAIL +NMDEV,clustered-spd,8,cholesky,1.26882631385e-16,PASS +NMDEV,clustered-spd,8,inverseSPD,3.13978251106e-17,PASS +NMDEV,hilbert,8,lu,2.77555756156e-17,PASS +NMDEV,hilbert,8,solve,1.27670477040e-17,PASS +NMDEV,hilbert,8,cholesky,1.38777878078e-17,PASS +NMDEV,hilbert,8,inverseSPD,1.55073347071e-16,PASS +NMDEV,rank-deficient,8,qr,0.00000000000,PASS +NMDEV,rank-deficient,8,svd,0.00000000000,PASS +EJML,scaled-pivot,8,lu,7.52235776040e-19,PASS +EJML,scaled-pivot,8,solve,1.44054707717e-18,PASS +EJML,scaled-pivot,8,inverse,2.95204769133e-18,PASS +EJML,scaled-pivot,8,qr,4.44089209850e-16,PASS +EJML,scaled-pivot,8,svd,1.33226762955e-15,PASS +EJML,clustered-spd,8,eigenSymmetric,6.66133814775e-16,PASS +EJML,clustered-spd,8,svd,1.99840144433e-15,PASS +EJML,clustered-spd,8,cholesky,1.26882631385e-16,PASS +EJML,clustered-spd,8,inverseSPD,2.95805940499e-17,PASS +EJML,hilbert,8,lu,0.00000000000,PASS +EJML,hilbert,8,solve,1.60825795562e-17,PASS +EJML,hilbert,8,cholesky,1.38777878078e-17,PASS +EJML,hilbert,8,inverseSPD,1.01146651580e-16,PASS +EJML,rank-deficient,8,qr,,FAIL +EJML,rank-deficient,8,svd,0.00000000000,PASS +COMMONS,scaled-pivot,8,lu,7.52235776040e-19,PASS +COMMONS,scaled-pivot,8,solve,1.23739683992e-19,PASS +COMMONS,scaled-pivot,8,inverse,1.34729841086e-19,PASS +COMMONS,scaled-pivot,8,qr,6.66133814775e-16,PASS +COMMONS,scaled-pivot,8,svd,1.11022302463e-15,PASS +COMMONS,clustered-spd,8,eigenSymmetric,8.88178419700e-16,PASS +COMMONS,clustered-spd,8,svd,1.11022302463e-15,PASS +COMMONS,clustered-spd,8,cholesky,1.26882631385e-16,PASS +COMMONS,clustered-spd,8,inverseSPD,3.00321198882e-17,PASS +COMMONS,hilbert,8,lu,2.77555756156e-17,PASS +COMMONS,hilbert,8,solve,1.19468788279e-17,PASS +COMMONS,hilbert,8,cholesky,1.38777878078e-17,PASS +COMMONS,hilbert,8,inverseSPD,2.34502964626e-16,PASS +COMMONS,rank-deficient,8,qr,0.00000000000,PASS +COMMONS,rank-deficient,8,svd,0.00000000000,PASS +JAMA,scaled-pivot,8,lu,7.52235776040e-19,PASS +JAMA,scaled-pivot,8,solve,1.44054708328e-18,PASS +JAMA,scaled-pivot,8,inverse,1.48197792994e-18,PASS +JAMA,scaled-pivot,8,qr,7.70289434665e-16,PASS +JAMA,scaled-pivot,8,svd,1.55431223448e-15,PASS +JAMA,clustered-spd,8,eigenSymmetric,8.88178419700e-16,PASS +JAMA,clustered-spd,8,svd,1.33226762955e-15,PASS +JAMA,clustered-spd,8,cholesky,1.26882631385e-16,PASS +JAMA,clustered-spd,8,inverseSPD,3.07195860224e-17,PASS +JAMA,hilbert,8,lu,0.00000000000,PASS +JAMA,hilbert,8,solve,1.62617287389e-17,PASS +JAMA,hilbert,8,cholesky,0.00000000000,PASS +JAMA,hilbert,8,inverseSPD,2.18612692870e-16,PASS +JAMA,rank-deficient,8,qr,0.00000000000,PASS +JAMA,rank-deficient,8,svd,0.00000000000,PASS +COLT,scaled-pivot,8,lu,7.52235776040e-19,PASS +COLT,scaled-pivot,8,solve,1.52949288483e-18,PASS +COLT,scaled-pivot,8,inverse,1.48037337753e-18,PASS +COLT,scaled-pivot,8,qr,6.66133814775e-16,PASS +COLT,scaled-pivot,8,svd,1.55431223448e-15,PASS +COLT,clustered-spd,8,eigenSymmetric,8.88178419700e-16,PASS +COLT,clustered-spd,8,svd,1.33226762955e-15,PASS +COLT,clustered-spd,8,cholesky,1.26882631385e-16,PASS +COLT,clustered-spd,8,inverseSPD,3.07195860224e-17,PASS +COLT,hilbert,8,lu,0.00000000000,PASS +COLT,hilbert,8,solve,2.20627614633e-17,PASS +COLT,hilbert,8,cholesky,0.00000000000,PASS +COLT,hilbert,8,inverseSPD,1.77068760994e-16,PASS +COLT,rank-deficient,8,qr,0.00000000000,PASS +COLT,rank-deficient,8,svd,0.00000000000,PASS +PARALLEL_COLT,scaled-pivot,8,lu,7.52235776040e-19,PASS +PARALLEL_COLT,scaled-pivot,8,solve,1.52949288483e-18,PASS +PARALLEL_COLT,scaled-pivot,8,inverse,1.48037337753e-18,PASS +PARALLEL_COLT,scaled-pivot,8,qr,8.88178419700e-16,PASS +PARALLEL_COLT,scaled-pivot,8,svd,2.40113659712e-15,PASS +PARALLEL_COLT,clustered-spd,8,eigenSymmetric,8.88178419700e-16,PASS +PARALLEL_COLT,clustered-spd,8,svd,1.14194368247e-15,PASS +PARALLEL_COLT,clustered-spd,8,cholesky,1.26882631385e-16,PASS +PARALLEL_COLT,clustered-spd,8,inverseSPD,3.00321198882e-17,PASS +PARALLEL_COLT,hilbert,8,lu,0.00000000000,PASS +PARALLEL_COLT,hilbert,8,solve,2.20627614633e-17,PASS +PARALLEL_COLT,hilbert,8,cholesky,0.00000000000,PASS +PARALLEL_COLT,hilbert,8,inverseSPD,1.98027392953e-16,PASS +PARALLEL_COLT,rank-deficient,8,qr,0.00000000000,PASS +PARALLEL_COLT,rank-deficient,8,svd,0.00000000000,PASS +JBLAS,scaled-pivot,8,lu,7.52235776040e-19,PASS +JBLAS,scaled-pivot,8,solve,2.55164696431e-19,PASS +JBLAS,scaled-pivot,8,inverse,2.96687364596e-18,PASS +JBLAS,scaled-pivot,8,qr,6.66133814775e-16,PASS +JBLAS,scaled-pivot,8,svd,2.39511871091e-15,PASS +JBLAS,clustered-spd,8,eigenSymmetric,5.55049778132e-16,PASS +JBLAS,clustered-spd,8,svd,2.37904933848e-15,PASS +JBLAS,clustered-spd,8,cholesky,1.26882631385e-16,PASS +JBLAS,clustered-spd,8,inverseSPD,5.17966298827e-17,PASS +JBLAS,hilbert,8,lu,0.00000000000,PASS +JBLAS,hilbert,8,solve,1.62617287389e-17,PASS +JBLAS,hilbert,8,cholesky,1.38777878078e-17,PASS +JBLAS,hilbert,8,inverseSPD,6.84624324520e-17,PASS +JBLAS,rank-deficient,8,qr,0.00000000000,PASS +JBLAS,rank-deficient,8,svd,0.00000000000,PASS +F2J_LAPACK,scaled-pivot,8,lu,7.52235776040e-19,PASS +F2J_LAPACK,scaled-pivot,8,solve,2.60226769427e-19,PASS +F2J_LAPACK,scaled-pivot,8,inverse,1.58354158137e-18,PASS +F2J_LAPACK,scaled-pivot,8,qr,8.88178419700e-16,PASS +F2J_LAPACK,scaled-pivot,8,svd,2.39662318246e-15,PASS +F2J_LAPACK,clustered-spd,8,eigenSymmetric,8.88178419700e-16,PASS +F2J_LAPACK,clustered-spd,8,svd,1.33226762955e-15,PASS +F2J_LAPACK,clustered-spd,8,cholesky,1.26882631385e-16,PASS +F2J_LAPACK,clustered-spd,8,inverseSPD,5.08438636519e-17,PASS +F2J_LAPACK,hilbert,8,lu,2.77555756156e-17,PASS +F2J_LAPACK,hilbert,8,solve,1.26662250155e-17,PASS +F2J_LAPACK,hilbert,8,cholesky,1.38777878078e-17,PASS +F2J_LAPACK,hilbert,8,inverseSPD,1.70135566233e-17,PASS +F2J_LAPACK,rank-deficient,8,qr,0.00000000000,PASS +F2J_LAPACK,rank-deficient,8,svd,0.00000000000,PASS +MTJ,scaled-pivot,8,lu,7.52235776040e-19,PASS +MTJ,scaled-pivot,8,solve,2.60226769427e-19,PASS +MTJ,scaled-pivot,8,inverse,6.15390840533e-20,PASS +MTJ,scaled-pivot,8,qr,8.88178419700e-16,PASS +MTJ,scaled-pivot,8,svd,2.40113659712e-15,PASS +MTJ,clustered-spd,8,eigenSymmetric,6.66133814775e-16,PASS +MTJ,clustered-spd,8,svd,1.14194368247e-15,PASS +MTJ,clustered-spd,8,cholesky,1.26882631385e-16,PASS +MTJ,clustered-spd,8,inverseSPD,3.00321198882e-17,PASS +MTJ,hilbert,8,lu,2.77555756156e-17,PASS +MTJ,hilbert,8,solve,1.26662250155e-17,PASS +MTJ,hilbert,8,cholesky,0.00000000000,PASS +MTJ,hilbert,8,inverseSPD,1.98027392953e-16,PASS +MTJ,rank-deficient,8,qr,0.00000000000,PASS +MTJ,rank-deficient,8,svd,0.00000000000,PASS +OJALGO,scaled-pivot,8,lu,7.52235776040e-19,PASS +OJALGO,scaled-pivot,8,solve,1.44007113238e-18,PASS +OJALGO,scaled-pivot,8,inverse,2.95150739385e-18,PASS +OJALGO,scaled-pivot,8,qr,5.55111512313e-16,PASS +OJALGO,scaled-pivot,8,svd,1.77635683940e-15,PASS +OJALGO,clustered-spd,8,eigenSymmetric,5.17992793668e-16,PASS +OJALGO,clustered-spd,8,svd,9.99200722163e-16,PASS +OJALGO,clustered-spd,8,cholesky,1.26882631385e-16,PASS +OJALGO,clustered-spd,8,inverseSPD,3.14455448527e-17,PASS +OJALGO,hilbert,8,lu,2.77555756156e-17,PASS +OJALGO,hilbert,8,solve,1.33525765322e-17,PASS +OJALGO,hilbert,8,cholesky,1.38777878078e-17,PASS +OJALGO,hilbert,8,inverseSPD,9.73897142825e-17,PASS +OJALGO,rank-deficient,8,qr,0.00000000000,PASS +OJALGO,rank-deficient,8,svd,0.00000000000,PASS +UJMP,scaled-pivot,8,lu,7.52235776040e-19,PASS +UJMP,scaled-pivot,8,solve,1.44054708328e-18,PASS +UJMP,scaled-pivot,8,inverse,1.48197792994e-18,PASS +UJMP,scaled-pivot,8,qr,7.70289434665e-16,PASS +UJMP,scaled-pivot,8,svd,1.55431223448e-15,PASS +UJMP,clustered-spd,8,eigenSymmetric,8.88178419700e-16,PASS +UJMP,clustered-spd,8,svd,1.33226762955e-15,PASS +UJMP,clustered-spd,8,cholesky,1.26882631385e-16,PASS +UJMP,clustered-spd,8,inverseSPD,3.07195860224e-17,PASS +UJMP,hilbert,8,lu,0.00000000000,PASS +UJMP,hilbert,8,solve,1.62617287389e-17,PASS +UJMP,hilbert,8,cholesky,0.00000000000,PASS +UJMP,hilbert,8,inverseSPD,2.18612692870e-16,PASS +UJMP,rank-deficient,8,qr,0.00000000000,PASS +UJMP,rank-deficient,8,svd,0.00000000000,PASS diff --git a/benchmark/results/2026-10-18-fairness-audit/difficult-correctness.log b/benchmark/results/2026-10-18-fairness-audit/difficult-correctness.log new file mode 100644 index 0000000..3fdba52 --- /dev/null +++ b/benchmark/results/2026-10-18-fairness-audit/difficult-correctness.log @@ -0,0 +1,7 @@ +EJML/rank-deficient/qr: java.lang.IllegalStateException: QR failed +Difficult-input failures: 3 (diagnostic, never silently excluded) +-- org.jblas INFO Deleting /tmp/jblas14766524964538109776/libjblas.so +-- org.jblas INFO Deleting /tmp/jblas14766524964538109776/libgfortran-4.so +-- org.jblas INFO Deleting /tmp/jblas14766524964538109776/libjblas_arch_flavor.so +-- org.jblas INFO Deleting /tmp/jblas14766524964538109776/libquadmath-0.so +-- org.jblas INFO Deleting /tmp/jblas14766524964538109776 diff --git a/benchmark/results/2026-10-18-fairness-audit/inverse-bytecode.txt b/benchmark/results/2026-10-18-fairness-audit/inverse-bytecode.txt new file mode 100644 index 0000000..08a8dba --- /dev/null +++ b/benchmark/results/2026-10-18-fairness-audit/inverse-bytecode.txt @@ -0,0 +1,91 @@ +Dispatch-only javap excerpt. Unrelated implementation methods are omitted. +Exact JAR identity is recorded in the audit/provenance. + + private static dev.nm.algebra.linear.matrix.doubles.Matrix a(dev.nm.algebra.linear.matrix.doubles.Matrix, double); + Code: + 0: new #147 // class dev/nm/algebra/linear/matrix/doubles/factorization/triangle/LU + 3: dup + 4: aload_0 + 5: dload_1 + 6: invokespecial #150 // Method dev/nm/algebra/linear/matrix/doubles/factorization/triangle/LU."NM Dev benchmark charts
Fairness revision 2: timed outputs are materialized; inputs and repeat execution are checked outside timing. Short audit timings are diagnostic. See the report for thread settings and uncertainty.
+Hover, tap or keyboard-focus any point to inspect its measurements.
+Lower time ratios are faster. NM Dev is the 1× reference. Confidence intervals describe timing uncertainty; +the displayed ratio is a ratio of means. Results are from a shared VM and different library API paths.
+Full report and raw measurements
+64 × 64 matrices
256 × 256 matrices
This file is self-contained and works offline. GitHub's inline image previews do not expose point tooltips.
+NM Dev benchmark charts
Fairness revision 2: timed outputs are materialized; inputs and repeat execution are checked outside timing. Short audit timings are diagnostic. See the report for thread settings and uncertainty.
+Hover, tap or keyboard-focus any point to inspect its measurements.
+Lower time ratios are faster. NM Dev is the 1× reference. Confidence intervals describe timing uncertainty; +the displayed ratio is a ratio of means. Results are from a shared VM and different library API paths.
+Full report and raw measurements
+512 × 512 matrices
1024 × 1024 matrices
This file is self-contained and works offline. GitHub's inline image previews do not expose point tooltips.
+NM Dev benchmark charts
Fairness revision 2: timed outputs are materialized; inputs and repeat execution are checked outside timing. Short audit timings are diagnostic. See the report for thread settings and uncertainty.
+Hover, tap or keyboard-focus any point to inspect its measurements.
+Lower time ratios are faster. NM Dev is the 1× reference. Confidence intervals describe timing uncertainty; +the displayed ratio is a ratio of means. Results are from a shared VM and different library API paths.
+Full report and raw measurements
+512 × 512 matrices
1024 × 1024 matrices
This file is self-contained and works offline. GitHub's inline image previews do not expose point tooltips.
+NM Dev benchmark charts
Fairness revision 2: timed outputs are materialized; inputs and repeat execution are checked outside timing. Short audit timings are diagnostic. See the report for thread settings and uncertainty.
+Hover, tap or keyboard-focus any point to inspect its measurements.
+Lower time ratios are faster. NM Dev is the 1× reference. Confidence intervals describe timing uncertainty; +the displayed ratio is a ratio of means. Results are from a shared VM and different library API paths.
+Full report and raw measurements
+64 × 64 matrices
256 × 256 matrices
This file is self-contained and works offline. GitHub's inline image previews do not expose point tooltips.
+Fairness revision 2: timed outputs are materialized and native inputs are checked for mutation. Short audit timings are diagnostic. Audit and accuracy findings.
' + else: + notice='Historical harness issue: ojAlgo scaling returned deferred work. Its scaling points are invalid for completed-work comparisons. Read the audit.
' + chart=chart.replace('NM Dev benchmark explorer
','NM Dev benchmark explorer
'+notice) (output/'benchmarks.html').write_text(chart) (output/'.nojekyll').write_text('') build_tutorials(repo, output) - print(f'Generated six benchmark Pages from {source.name}; raw measurements unchanged.') + print(f'Generated {len(pages)+1} benchmark Pages from {source.name}; raw measurements unchanged.') if __name__ == '__main__': main() diff --git a/benchmark/scripts/record-environment.py b/benchmark/scripts/record-environment.py new file mode 100644 index 0000000..93af5f6 --- /dev/null +++ b/benchmark/scripts/record-environment.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3 +"""Record the exact harness, artifact hashes and execution request for an isolated run.""" +import datetime, hashlib, json, os, platform, subprocess, sys, zipfile +from pathlib import Path +root=Path(sys.argv[1]);threads=int(sys.argv[2]) +def read(path): + p=Path(path);return p.read_text().strip() if p.exists() else 'unavailable' +now=datetime.datetime.now(datetime.timezone.utc).isoformat() +cpu=next((line.split(':',1)[1].strip() for line in read('/proc/cpuinfo').splitlines() if line.startswith('model name')),'unrecorded') +env={'fairness_revision':2,'recorded_utc':now,'os':platform.platform(),'cpu_model':cpu,'visible_cpus':os.cpu_count(),'cpu_quota':read('/sys/fs/cgroup/cpu.max'),'memory_limit_bytes':read('/sys/fs/cgroup/memory.max'),'requested_compute_threads':threads,'profile':sys.argv[3],'benchmark':sys.argv[4],'jmh_parameters_and_overrides':sys.argv[5:],'requested_affinity':os.getenv('BENCH_CPUSET'),'launcher_affinity':sorted(os.sched_getaffinity(0)) if hasattr(os,'sched_getaffinity') else None,'native_thread_counts_independently_audited':False,'environment':'Shared VM/container; CPU affinity is not exclusive CPU reservation','requested_native_thread_environment':{k:os.getenv(k) for k in ('OPENBLAS_NUM_THREADS','OMP_NUM_THREADS','MKL_NUM_THREADS','BLIS_NUM_THREADS','VECLIB_MAXIMUM_THREADS')},'jvm':subprocess.run(['java','-version'],capture_output=True,text=True,check=True).stderr} +(root/'environment.json').write_text(json.dumps(env,indent=2)+'\n') +(root/'started-utc.txt').write_text(now+'\n') +artifacts=[{'file':p.name,'sha256':hashlib.sha256(p.read_bytes()).hexdigest(),'bytes':p.stat().st_size} for p in sorted(Path('benchmark/target/lib').glob('*.jar'))] +(root/'artifacts.json').write_text(json.dumps(artifacts,indent=2)+'\n') +paths=list(Path('benchmark/src/main/java').rglob('*.java'))+list(Path('benchmark/scripts').glob('*'))+[Path('benchmark/dependencies.lock.tsv'),Path('benchmark/versions.json')] +(root/'source-checksums.json').write_text(json.dumps({str(p):hashlib.sha256(p.read_bytes()).hexdigest() for p in sorted(paths) if p.is_file()},indent=2)+'\n') + +with zipfile.ZipFile(root/'harness-source.zip','w',zipfile.ZIP_DEFLATED) as archive: + for p in sorted(paths): + if p.is_file(): archive.write(p,str(p)) + +for name in ['versions.json']: + src=Path('benchmark')/name + if src.exists(): (root/name).write_bytes(src.read_bytes()) +proof=Path('benchmark/dependency-audit/2026-10-18/nmdev-validated.json') +if proof.exists(): (root/'validated-nmdev-artifact.json').write_bytes(proof.read_bytes()) diff --git a/benchmark/scripts/report-rectangular.py b/benchmark/scripts/report-rectangular.py new file mode 100644 index 0000000..8f6d405 --- /dev/null +++ b/benchmark/scripts/report-rectangular.py @@ -0,0 +1,27 @@ +#!/usr/bin/env python3 +"""Summarize the separate rectangular multiplication track, preserving its shape parameters.""" +import csv,json,sys +from pathlib import Path +root=Path(sys.argv[1]);rows=json.loads((root/'jmh.json').read_text()) +libs=['NMDEV','EJML','OJALGO','JBLAS'];shapes=sorted({r['params']['shape'] for r in rows}) +index={(r['params']['library'],r['params']['shape']):r for r in rows} +assert len(index)==len(rows)==len(libs)*len(shapes) +lines=['# Rectangular multiplication — '+root.name,'Shapes are m×k×n. All returned results are materialized, inputs are constructed outside timing, and every fork checks arithmetic twice and exact input preservation. This is a separate four-library track, not an eleven-library decomposition comparison.','Short diagnostic run on a shared VM. See the exact fork settings, affinity and thread requests in [environment.json](environment.json). Native jblas worker counts are not independently audited.','| Shape | NMDEV ms/op | EJML ms/op | OJALGO ms/op | JBLAS ms/op |','|---|---:|---:|---:|---:|'] +versions=json.loads((root/'versions.json').read_text()) +lines.insert(3,'**Versions:** '+', '.join(versions[lib] for lib in libs)+'. [Exact version labels](versions.json).') +with (root/'summary.csv').open('w',newline='') as out: + writer=csv.writer(out);writer.writerow(['shape','library','mean_ms_per_op','jmh_99_9_error_ms','ratio_to_nmdev','allocated_bytes_per_op']) + for shape in shapes: + baseline=index['NMDEV',shape]['primaryMetric']['score'];cells=[shape] + for lib in libs: + row=index[lib,shape];metric=row['primaryMetric'];assert metric['scoreUnit']=='ms/op' and metric['score']>0 + assert len(metric['rawData'])==row['forks'] and all(len(f)==row['measurementIterations'] for f in metric['rawData']) + allocation=row.get('secondaryMetrics',{}).get('gc.alloc.rate.norm',{}).get('score') + writer.writerow([shape,lib,metric['score'],metric['scoreError'],metric['score']/baseline,allocation]);cells.append(f"{metric['score']:.6g} ± {metric['scoreError']:.2g}") + lines.append('| '+' | '.join(cells)+' |') +lines+=['Errors are JMH 99.9% confidence half-widths. Wide or overlapping intervals do not support fine rankings. JVM allocation counts do not include all native allocations.','[Raw JMH](jmh.json) · [Log](jmh.log) · [Summary and allocation](summary.csv) · [Artifacts](artifacts.json) · [Exact source snapshot](harness-source.zip) · [Fairness audit](../2026-10-18-fairness-audit/README.md)'] +text='\n\n'.join(lines)+'\n' +# Keep contiguous table rows so Markdown renders an actual table. +text=text.replace(' |\n\n|',' |\n|').replace('|\n\n|','|\n|') +(root/'README.md').write_text(text) +print('Summarized',len(rows),'rectangular cases') diff --git a/benchmark/scripts/report.py b/benchmark/scripts/report.py index 51d4c62..1045658 100644 --- a/benchmark/scripts/report.py +++ b/benchmark/scripts/report.py @@ -8,6 +8,7 @@ OPERATIONS = ["transpose", "addition", "scale", "multiplication", "determinant", "inverse", "inverseSPD", "cholesky", "lu", "qr", "eigenSymmetric", "svd", "solve"] LIBRARIES = ["NMDEV", "EJML", "COMMONS", "JAMA", "COLT", "PARALLEL_COLT", "JBLAS", "F2J_LAPACK", "MTJ", "OJALGO", "UJMP"] +# Historical fallback labels; every current run supplies its exact versions.json. VERSIONS = {"NMDEV":"NM Dev 1.2.6-SNAPSHOT", "EJML":"EJML 0.46.1", "COMMONS":"Commons Math 3.6.1", "JAMA":"JAMA 1.0.3", "COLT":"Colt 1.2.0", "PARALLEL_COLT":"Parallel Colt 0.10.1", "JBLAS":"jblas 1.2.5 (native)", "F2J_LAPACK":"F2J LAPACK/BLAS 0.1", "MTJ":"MTJ 1.0.4 (F2J)", "OJALGO":"ojAlgo 57.5.0", "UJMP":"UJMP 0.3.0 (core)"} COLORS = dict(zip(LIBRARIES,["#126c78","#4678ba","#d18829","#9d608e","#6b8c31","#b8493a","#6750a4","#4d8794","#b77265","#7a756e","#aa8a25"])) @@ -16,15 +17,17 @@ def table(rows): def main(): parser=argparse.ArgumentParser(description=__doc__);parser.add_argument("directory",type=Path);root=parser.parse_args().directory + versions=json.loads((root/"versions.json").read_text()) if (root/"versions.json").exists() else dict(VERSIONS) rows=json.loads((root/"jmh.json").read_text()) index={(r["params"]["library"],int(r["params"]["size"]),r["params"]["operation"]):r for r in rows} libraries=[lib for lib in LIBRARIES if any(r["params"]["library"]==lib for r in rows)] sizes=sorted({int(r["params"]["size"]) for r in rows}) - expected=len(libraries)*len(sizes)*len(OPERATIONS) + operations=[op for op in OPERATIONS if any(r["params"]["operation"]==op for r in rows)] + expected=len(libraries)*len(sizes)*len(operations) assert len(index)==len(rows)==expected, "Incomplete or duplicate benchmark matrix" assert "NMDEV" in libraries for size in sizes: - for op in OPERATIONS: + for op in operations: for lib in libraries: r=index[lib,size,op];metric=r["primaryMetric"] assert metric["scoreUnit"]=="ms/op" and metric["score"]>0 @@ -34,24 +37,34 @@ def main(): with (root/"summary.csv").open("w",newline="") as output: writer=csv.writer(output);writer.writerow(["size","operation","library","version","mean_ms_per_op","jmh_99_9_error_ms","ratio_to_nmdev","measurement_iterations"]) for size in sizes: - for op in OPERATIONS: + for op in operations: baseline=index["NMDEV",size,op]["primaryMetric"]["score"] for lib in libraries: metric=index[lib,size,op]["primaryMetric"] - writer.writerow([size,op,lib,VERSIONS[lib],metric["score"],metric["scoreError"],metric["score"]/baseline,sum(map(len,metric["rawData"]))]) + writer.writerow([size,op,lib,versions[lib],metric["score"],metric["scoreError"],metric["score"]/baseline,sum(map(len,metric["rawData"]))]) + if any("gc.alloc.rate.norm" in row.get("secondaryMetrics",{}) for row in rows): + with (root/"allocation.csv").open("w",newline="") as output: + writer=csv.writer(output);writer.writerow(["size","operation","library","jvm_allocated_bytes_per_op","jmh_99_9_error_bytes"]) + for row in rows: + metric=row.get("secondaryMetrics",{}).get("gc.alloc.rate.norm",{}) + writer.writerow([row["params"]["size"],row["params"]["operation"],row["params"]["library"],metric.get("score"),metric.get("scoreError")]) environment=json.loads((root/"environment.json").read_text()) if (root/"environment.json").exists() else {} first=rows[0] sections=[f"# Linear algebra benchmark results — {root.name}", "[Suite](../../README.md) · [Methodology](../../METHODOLOGY.md) · [Library coverage](../../LIBRARY-COVERAGE.md)", - f"**{len(OPERATIONS)} operations × {len(libraries)} libraries × {len(sizes)} sizes = {len(rows)} measured cases**, containing {iterations:,} measured iteration means. Lower milliseconds per operation is faster. These are observations on a shared VM, not universal performance rankings.", + f"**{len(operations)} operations × {len(libraries)} libraries × {len(sizes)} sizes = {len(rows)} measured cases**, containing {iterations:,} measured iteration means. Lower milliseconds per operation is faster. These are observations on a shared VM, not universal performance rankings.", "**[Interactive charts with tooltips](charts.html)** — download and open the self-contained HTML in a browser. Hover, tap or keyboard-focus a point for its version/backend, timing, confidence interval and ratio to NM Dev. GitHub inline image previews do not expose point tooltips. Standalone SVGs also contain native tooltip titles.", "## Environment", - f"{environment.get('cpu_model','Unrecorded CPU')}; {environment.get('os','Unrecorded OS')}. Visible CPUs: {environment.get('visible_cpus','unrecorded')}; container CPU quota: `{environment.get('cpu_quota','unrecorded')}`. JVM {first['jdkVersion']}. No exclusive CPU reservation or affinity pinning.", - f"{first['forks']} JVM forks per case; {first['warmupIterations']} warmup iterations of {first['warmupTime']}; {first['measurementIterations']} measurement iterations of {first['measurementTime']}. One JMH worker. The fork options fix a 1 GiB heap and one active processor.", - "Native thread environment variables request one thread. Parallel Colt, ojAlgo and UJMP also receive explicit library thread limits; MTJ uses an explicitly forced pure-Java F2J backend. Native jblas thread counts are not independently audited. See the environment and backend records below.", + f"{environment.get('cpu_model','Unrecorded CPU')}; {environment.get('os','Unrecorded OS')}. Visible CPUs: {environment.get('visible_cpus','unrecorded')}; container CPU quota: `{environment.get('cpu_quota','unrecorded')}`. JVM {first['jdkVersion']}. No exclusive CPU reservation; any requested affinity is recorded below.", + f"{first['forks']} JVM forks per case; {first['warmupIterations']} warmup iterations of {first['warmupTime']}; {first['measurementIterations']} measurement iterations of {first['measurementTime']}. One JMH worker. The fork options fix a 1 GiB heap. Requested computational threads: {environment.get('requested_compute_threads',1)}; requested CPU affinity: {environment.get('requested_affinity','unrecorded')}. Affinity does not reserve a CPU exclusively.", + "Native thread environment variables request the recorded computational thread count. Parallel Colt, ojAlgo and UJMP also receive explicit library thread limits; MTJ uses an explicitly forced pure-Java F2J backend. Native jblas thread counts are not independently audited. See the environment and backend records below.", "## Versions and backends"] backend=json.loads((root/"backends.json").read_text()) if (root/"backends.json").exists() else {} - sections.append(table([["Library","Measured version / backend"],["---","---"]]+[[lib,VERSIONS[lib]+(" — "+backend[lib] if lib in backend else "")] for lib in libraries])) + if "OJALGO" in libraries and not environment.get("fairness_revision"): + sections.insert(2,"> Historical run: ojAlgo scalar multiplication returned a deferred expression. Its scaling timings are invalid for completed-work comparisons. Raw measurements are preserved. See the [fairness audit](../2026-10-18-fairness-audit/README.md).") + if environment.get("fairness_revision"): + sections.insert(2,"> Corrected harness: deferred ojAlgo results are materialized inside timing, native inputs are checked for mutation, and every fork validates repeat execution. These short iterations provide a diagnostic baseline; use longer runs on dedicated hardware for fine rankings. See the [fairness audit](../2026-10-18-fairness-audit/README.md).") + sections.append(table([["Library","Measured version / backend"],["---","---"]]+[[lib,versions[lib]+(" — "+backend[lib] if lib in backend else "")] for lib in libraries])) correctness=list(csv.DictReader((root/"correctness.csv").open())) assert all(r["status"]=="PASS" for r in correctness), "Correctness failure in published evidence" sections += ["## Correctness and interpretation",f"All **{len(correctness)} baseline correctness cases passed twice**. Every measured fork also checks its selected result before timing. Reference arithmetic and residual checks are outside timing."] @@ -59,15 +72,15 @@ def main(): pivot=list(csv.DictReader((root/"pivot-correctness.csv").open()));assert all(r["status"]=="PASS" for r in pivot) sections.append(f"All **{len(pivot)} targeted row-swap/row-cycle cases** also passed. These check determinant signs, LU permutation orientation and solves on pivoting inputs.") sections += ["Tables show **mean ± JMH 99.9% confidence half-width**, in ms/op. These error bars are not standard deviations. Overlapping intervals or high uncertainty make fine rankings unreliable. Chart ratios are ratios of means and do not show ratio uncertainty.", - "Native input conversion is outside timing; necessary copies, allocations and full factor output materialization are timed. This compares public API tasks, not identical algorithms. NM Dev uses a general inverse routine on SPD inputs; other adapters may use Cholesky-based inversion. F2J transpose/addition/scaling use adapter-owned Java loops because LAPACK does not define those matrix convenience APIs."] + "Native input conversion is outside timing; necessary copies, allocations and full factor output materialization are timed. This compares public API tasks, not identical algorithms. NM Dev Inverse detects symmetry and attempts Cholesky on SPD inputs, falling back to LU if that path raises a runtime exception. The exact measured JAR was verified separately by bytecode and runtime constructor-breakpoint tracing; see the fairness audit. F2J transpose/addition/scaling use adapter-owned Java loops because LAPACK does not define those matrix convenience APIs."] for size in sizes: sections.append(f"## {size} × {size} matrices") data=[["Operation"]+libraries,["---"]+["---:"]*len(libraries)] - for op in OPERATIONS: + for op in operations: data.append([op]+[f"{index[lib,size,op]['primaryMetric']['score']:.5g} ± {index[lib,size,op]['primaryMetric']['scoreError']:.2g}" for lib in libraries]) sections += [table(data),f"","NM Dev is 1×. Below 1× means less time than NM Dev; above 1× means more time. Logarithmic scale. Use the interactive HTML for exact point details."] sections += ["## Limits", "No aggregate speedup is reported because averaging operation ratios imposes an arbitrary workload weighting. This run uses one deterministic family of well-conditioned square matrices. It does not measure sparse matrices, conic optimization, parallel scaling, very large inputs or stability on difficult problems. Parameter order is deterministic, and the VM is shared. Reproduce your own workload on dedicated hardware before making broad performance claims.","## Raw evidence"] - evidence=[("jmh.json","JMH JSON and per-fork iteration means"),("jmh.log","Complete JMH log"),("summary.csv","Summary CSV"),("correctness.csv","Baseline correctness residuals"),("pivot-correctness.csv","Targeted pivot/sign checks"),("environment.json","Environment and requested thread settings"),("backends.json","Backend records"),("backend-observation.txt","Post-run backend configuration probe"),("artifacts.json","Artifact versions and SHA-256 checksums"),("source-checksums.json","Measured source checksums"),("dependency-metadata.json","Dependency release metadata")] + evidence=[("jmh.json","JMH JSON and per-fork iteration means"),("jmh.log","Complete JMH log"),("summary.csv","Summary CSV"),("allocation.csv","JVM allocation per operation; native allocations are not fully included"),("correctness.csv","Baseline correctness residuals"),("pivot-correctness.csv","Targeted pivot/sign checks"),("environment.json","Environment and requested thread settings"),("backends.json","Backend records"),("backend-observation.txt","Post-run backend configuration probe"),("artifacts.json","Artifact versions and SHA-256 checksums"),("source-checksums.json","Measured source checksums"),("dependency-metadata.json","Dependency release metadata"),("harness-source.zip","Exact harness source snapshot"),("fairness-check.txt","Deferred-output and mutation regression checks")] sections.append("\n".join(f"- [{label}]({name})" for name,label in evidence if (root/name).exists())) sections.append(f"Regenerate with `python3 benchmark/scripts/report.py benchmark/results/{root.name}`. Python only summarizes measurements and renders charts; all measured calculations run in Java/JMH. Matplotlib is required to regenerate charts and the self-contained HTML.") (root/"README.md").write_text("\n\n".join(sections)+"\n") @@ -78,19 +91,19 @@ def main(): for size in sizes: fig,ax=plt.subplots(figsize=(13,12 if len(libraries)>4 else 8));tooltips={} for j,lib in enumerate(libraries): - for i,op in enumerate(OPERATIONS): + for i,op in enumerate(operations): metric=index[lib,size,op]["primaryMetric"];ratio=metric["score"]/index["NMDEV",size,op]["primaryMetric"]["score"] y=i+(j-(len(libraries)-1)/2)*(.78/len(libraries)) - point=ax.scatter([ratio],[y],s=28,color=COLORS[lib],label=VERSIONS[lib] if i==0 else None,zorder=3) + point=ax.scatter([ratio],[y],s=28,color=COLORS[lib],label=versions[lib] if i==0 else None,zorder=3) key=f"point-{size}-{lib}-{op}";point.set_gid(key) low,high=metric["scoreConfidence"] - tooltips[key]=(f"{VERSIONS[lib]} · {op} · {size} × {size}\n" + tooltips[key]=(f"{versions[lib]} · {op} · {size} × {size}\n" f"Mean: {metric['score']:.6g} ms/op\n" f"99.9% confidence interval: [{low:.6g}, {high:.6g}] ms/op\n" f"JMH error half-width: ±{metric['scoreError']:.3g} ms/op\n" f"Time / NM Dev: {ratio:.4f}× (lower is faster)\n" f"{first['forks']} JVM forks · {first['forks']*first['measurementIterations']} measured iteration means") - ax.set_yticks(range(len(OPERATIONS)),OPERATIONS);ax.invert_yaxis();ax.set_xscale("log") + ax.set_yticks(range(len(operations)),operations);ax.invert_yaxis();ax.set_xscale("log") ax.axvline(1,color="#333333",linewidth=1,linestyle="--");ax.set_xlabel("Time / NM Dev time (log scale; lower is faster)") ax.set_title(f"Dense linear algebra — {size} × {size}\n{len(libraries)} libraries · ratios of measured means on a shared VM",loc="left",fontweight="bold",pad=15) ax.grid(axis="x",alpha=.2);ax.set_axisbelow(True) @@ -99,6 +112,13 @@ def main(): fig.tight_layout(rect=(0,.04,1,1));fig.savefig(root/f"relative-times-{size}.svg",metadata={"Date":None});plt.close(fig) annotate_svg(root/f"relative-times-{size}.svg",tooltips) write_html(root,sizes) + chart_path=root/"charts.html" + chart=chart_path.read_text() + if "OJALGO" in libraries and not environment.get("fairness_revision"): + notice='Historical harness issue: ojAlgo scaling returned deferred work. Its scaling points do not measure completed scaling and must not be used for speed comparisons. See the fairness audit.
' + else: + notice='Fairness revision 2: timed outputs are materialized; inputs and repeat execution are checked outside timing. Short audit timings are diagnostic. See the report for thread settings and uncertainty.
' + chart_path.write_text(chart.replace('NM Dev benchmark charts
','NM Dev benchmark charts
'+notice)) print(f"Summarized {len(rows)} cases and {iterations} iteration means; generated {len(rows)} tooltip points") if __name__=="__main__":main() diff --git a/benchmark/scripts/run.sh b/benchmark/scripts/run.sh index eaba37b..bd0483c 100755 --- a/benchmark/scripts/run.sh +++ b/benchmark/scripts/run.sh @@ -2,41 +2,49 @@ set -euo pipefail repo_root="$(cd "$(dirname "$0")/../.." && pwd)" cd "$repo_root" -output="${1:?Supply a new output directory, for example benchmark/results/my-run}" +output="${1:?Supply a new output directory}" shift if [[ -e "$output/jmh.json" || -e "$output/jmh.log" ]]; then - echo 'Choose a new output directory to preserve earlier results' >&2 - exit 1 -fi -if [[ ! -f nm.lic ]]; then - echo 'Place your valid NM Dev license at the repository root as nm.lic' >&2 - exit 1 + echo 'Choose a new output directory to preserve earlier results' >&2; exit 1 fi +[[ -f nm.lic ]] || { echo 'Place your valid NM Dev license at the repository root as nm.lic' >&2; exit 1; } +threads="${BENCH_THREADS:-1}" +[[ "$threads" =~ ^[1-9][0-9]*$ ]] || { echo 'BENCH_THREADS must be a positive integer' >&2; exit 1; } +profile="${BENCH_PROFILE:-baseline}" +case "$profile" in + baseline) benchmark='dev.nm.benchmark.MatrixBenchmark.compute'; parameters=(-p size=64,256) ;; + large) benchmark='dev.nm.benchmark.MatrixBenchmark.compute'; parameters=(-p size=512,1024 -p library=NMDEV,EJML,OJALGO,JBLAS -p operation=multiplication,lu,cholesky,qr,solve) ;; + rectangular) benchmark='dev.nm.benchmark.RectangularBenchmark.compute'; parameters=() ;; + *) echo 'BENCH_PROFILE must be baseline, large or rectangular' >&2; exit 1 ;; +esac +NMDEV_JAR="$repo_root/benchmark/target/lib/nmdev-1.2.6.jar" bash benchmark/scripts/build-direct.sh mkdir -p "$output" classpath='benchmark/target/classes:benchmark/target/lib/*' -# Request single-thread native execution and explicitly select MTJ's Java backend. -export OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 BLIS_NUM_THREADS=1 VECLIB_MAXIMUM_THREADS=1 +export OPENBLAS_NUM_THREADS="$threads" OMP_NUM_THREADS="$threads" MKL_NUM_THREADS="$threads" BLIS_NUM_THREADS="$threads" VECLIB_MAXIMUM_THREADS="$threads" backend_flags=( -Dcom.github.fommil.netlib.BLAS=com.github.fommil.netlib.F2jBLAS -Dcom.github.fommil.netlib.LAPACK=com.github.fommil.netlib.F2jLAPACK -Dcom.github.fommil.netlib.ARPACK=com.github.fommil.netlib.F2jARPACK ) - +fork_flags="-Xms1g -Xmx1g -XX:ActiveProcessorCount=$threads -Dbenchmark.threads=$threads ${backend_flags[*]}" +runner=() +if [[ -n "${BENCH_CPUSET:-}" ]]; then + command -v taskset >/dev/null || { echo 'taskset is required for BENCH_CPUSET' >&2; exit 1; } + runner=(taskset -c "$BENCH_CPUSET") +fi { - date -u - uname -a - java -version + date -u; uname -a; java -version if command -v lscpu >/dev/null; then lscpu; fi if [[ -r /sys/fs/cgroup/cpu.max ]]; then cat /sys/fs/cgroup/cpu.max; fi if [[ -r /sys/fs/cgroup/memory.max ]]; then cat /sys/fs/cgroup/memory.max; fi } > "$output/environment.txt" 2>&1 sha256sum benchmark/target/lib/*.jar > "$output/artifact-checksums.txt" -java -Xms1g -Xmx1g -XX:ActiveProcessorCount=1 "${backend_flags[@]}" -cp "$classpath" \ - dev.nm.benchmark.CorrectnessMain > "$output/correctness.csv" -java -Xms1g -Xmx1g -XX:ActiveProcessorCount=1 "${backend_flags[@]}" -cp "$classpath" \ - dev.nm.benchmark.PivotValidationMain > "$output/pivot-correctness.csv" +python3 benchmark/scripts/record-environment.py "$output" "$threads" "$profile" "$benchmark" "${parameters[@]}" "$@" +"${runner[@]}" java -cp "$classpath" dev.nm.benchmark.OriginValidationMain > "$output/class-origins.csv" +"${runner[@]}" java -Xms1g -Xmx1g -XX:ActiveProcessorCount="$threads" -Dbenchmark.threads="$threads" "${backend_flags[@]}" -cp "$classpath" dev.nm.benchmark.FairnessValidationMain > "$output/fairness-check.txt" 2> "$output/fairness-check.log" +"${runner[@]}" java -Xms1g -Xmx1g -XX:ActiveProcessorCount="$threads" -Dbenchmark.threads="$threads" "${backend_flags[@]}" -cp "$classpath" dev.nm.benchmark.CorrectnessMain > "$output/correctness.csv" 2> "$output/correctness.log" +"${runner[@]}" java -Xms1g -Xmx1g -XX:ActiveProcessorCount="$threads" -Dbenchmark.threads="$threads" "${backend_flags[@]}" -cp "$classpath" dev.nm.benchmark.PivotValidationMain > "$output/pivot-correctness.csv" 2> "$output/pivot-correctness.log" echo "Running JMH; follow progress in $output/jmh.log" -java -cp "$classpath" org.openjdk.jmh.Main \ - dev.nm.benchmark.MatrixBenchmark.compute -foe true -rf json -rff "$output/jmh.json" \ - "$@" > "$output/jmh.log" 2>&1 +"${runner[@]}" java -cp "$classpath" org.openjdk.jmh.Main "$benchmark" "${parameters[@]}" -jvmArgsAppend "$fork_flags" -prof gc -foe true -rf json -rff "$output/jmh.json" "$@" > "$output/jmh.log" 2>&1 +date -u +%FT%TZ > "$output/completed-utc.txt" echo "Raw results written to $output" diff --git a/benchmark/scripts/validate-archive.py b/benchmark/scripts/validate-archive.py new file mode 100644 index 0000000..84ca4a9 --- /dev/null +++ b/benchmark/scripts/validate-archive.py @@ -0,0 +1,25 @@ +#!/usr/bin/env python3 +"""Check result completeness, raw samples, profiler evidence and artifact consistency.""" +import csv,hashlib,json,sys,zipfile +from pathlib import Path +root=Path(sys.argv[1]);rows=json.loads((root/'jmh.json').read_text());env=json.loads((root/'environment.json').read_text()) +expected=int(sys.argv[2]);assert len(rows)==expected +keys=[] +for r in rows: + keys.append((r['benchmark'],tuple(sorted(r['params'].items())))) + assert r['primaryMetric']['scoreUnit']=='ms/op' and r['primaryMetric']['score']>0 + assert len(r['primaryMetric']['rawData'])==r['forks'] + assert all(len(f)==r['measurementIterations'] for f in r['primaryMetric']['rawData']) + assert 'gc.alloc.rate.norm' in r['secondaryMetrics'] + assert '-XX:ActiveProcessorCount='+str(env['requested_compute_threads']) in r['jvmArgs'] + assert '-Dbenchmark.threads='+str(env['requested_compute_threads']) in r['jvmArgs'] +assert len(set(keys))==expected +for name,count in [('correctness.csv',429),('pivot-correctness.csv',66)]: + checks=list(csv.DictReader((root/name).open()));assert len(checks)==count and all(c['status']=='PASS' for c in checks) +with zipfile.ZipFile(root/'harness-source.zip') as archive: + for name,sha in json.loads((root/'source-checksums.json').read_text()).items():assert hashlib.sha256(archive.read(name)).hexdigest()==sha +jars=json.loads((root/'artifacts.json').read_text());nm=next(j for j in jars if j['file'].startswith('nmdev')) +proof=root/'validated-nmdev-artifact.json' +assert nm['sha256']==(json.loads(proof.read_text())['sha256'] if proof.exists() else 'a3e2b2492e79530b8e12920b48f48bfc295786b138aa26300a16b53bafe1386c') +assert (root/'completed-utc.txt').exists() +print(root.name,':',expected,'cases; input/correctness guards, exact source snapshot, artifact hash, thread flags and allocation metrics verified') diff --git a/benchmark/site/libraries.md b/benchmark/site/libraries.md index e3c58ac..9a16ec0 100644 --- a/benchmark/site/libraries.md +++ b/benchmark/site/libraries.md @@ -4,10 +4,10 @@ All eleven adapters below compile and have passed 429 baseline correctness cases | Library | Version | Integration result | |---|---|---| -| NM Dev | 1.2.6-SNAPSHOT | Passed; combined binary checksum recorded | -| Apache Commons Math | 3.6.1 | Passed | +| NM Dev | 1.2.6 | Passed; combined binary checksum recorded | +| Apache Commons Math | 4.0-SNAPSHOT (20261018.085925-1575) | Passed | | Colt | 1.2.0 | Passed | -| Parallel Colt | 0.10.1 | Passed; added JPLASMA/core-lapack dependencies and one-thread setting | +| Parallel Colt | 0.11.4 (upstream source tag) | Passed; added JPLASMA/core-lapack dependencies and one-thread setting | | EJML | 0.46.1 | Passed | | JAMA | 1.0.3 | Passed | | jblas | 1.2.5 | Passed; bundled native libraries loaded on this Linux x86-64 environment | @@ -25,3 +25,8 @@ All eleven adapters below compile and have passed 429 baseline correctness cases - Parallel Colt, ojAlgo and UJMP receive explicit one-thread settings, in addition to the common one-worker JMH configuration. The original four-library report remains available as a separate historical run. Its omissions were integration scope decisions, not evidence that other libraries could not run. + + +## Current version selection + +The latest comparison upgrades the primary Commons adapter to **Commons Math 4.0-SNAPSHOT**, Parallel Colt to **0.11.4** from its upstream release tag, and NM Dev to the validated **1.2.6** release JAR. See the [complete version audit](https://github.com/nmltd/nmdev-examples/blob/main/benchmark/dependency-audit/2026-10-18/README.md) for all eleven libraries, release-versus-snapshot distinctions and exact checksums. Version labels below are read from the selected run. diff --git a/benchmark/site/methodology.md b/benchmark/site/methodology.md index 5e3ea2a..ae893af 100644 --- a/benchmark/site/methodology.md +++ b/benchmark/site/methodology.md @@ -2,7 +2,7 @@ ## What is measured -JMH 1.37 measures average milliseconds per completed operation. Each library receives the same seeded input values and uses its native matrix representation. The benchmark returns the resulting matrix, scalar or array of factor objects to JMH's generated consumer. No network requests or random generation occur in timed operations. +JMH 1.37 measures average milliseconds per completed operation. Each library receives the same seeded input values and uses its native matrix representation. The benchmark returns the resulting materialized matrix, scalar or array of factor objects to JMH's generated consumer. Returning a lazy expression alone is insufficient. The corrected ojAlgo adapter materializes scalar expressions inside timing; existing physical stores are returned directly to avoid unnecessary extra copies. No network requests or random generation occur in timed operations. Inputs are created and converted in trial setup. Outputs are allocated inside the operation, as required by the selected APIs. EJML decomposition APIs receive a fresh copy when the adapter uses an in-place decomposition, protecting repeated inputs; that copy is timed. Other APIs may make internal copies. This measures the cost of the documented adapter calls, not a lower-level allocation-free kernel. @@ -12,7 +12,7 @@ Each operation includes a string switch and the adapter invocation. This common `java.util.Random` uses seed `20261018 + n` for each matrix size. A has bounded off-diagonal entries `(uniform[0,1)-0.5)/n`, with `1+i/n` added to diagonal entry i. B has similarly bounded entries without the diagonal addition. The SPD matrix is `A^T A`; the right-hand side has n rows and one column. -These deterministic dense inputs are well-conditioned and intentionally avoid near-singularity. Results say nothing about sparse, ill-conditioned, rectangular, low-rank or highly structured inputs. QR and SVD use square matrices in this run, unlike some historical defaults. A single seed is not a workload distribution. +These baseline dense inputs are well-conditioned and intentionally avoid near-singularity. Separate rectangular multiplication and difficult-input accuracy tracks extend coverage; their results are never pooled with the baseline. No sparse performance is measured. QR and SVD use square matrices in this run, unlike some historical defaults. A single seed is not a workload distribution. ## Equivalent mathematical tasks and API differences @@ -28,7 +28,7 @@ These deterministic dense inputs are well-conditioned and intentionally avoid ne | SVD | Full U, diagonal singular values, V; reconstruction and both orthogonality checks | | Solve | New factorization plus solve for one RHS; residual against RHS | -NM Dev uses `Inverse` for both general and SPD inverse, matching the original adapter. EJML, Commons Math and JAMA use Cholesky-based paths for the SPD inverse. General inversion paths can also differ by library. These are task-level API comparisons; a result is not a controlled comparison of identical algorithms. +NM Dev uses `Inverse` for both general and SPD inverse. `Inverse` detects symmetry and attempts Cholesky, with an LU fallback on a runtime exception. Bytecode and debugger constructor-breakpoint checks against the exact measured JAR confirm the SPD fixture takes Cholesky, without LU fallback. See the [fairness audit](https://github.com/nmltd/nmdev-examples/blob/main/benchmark/results/2026-10-18-fairness-audit/README.md). EJML, Commons Math and JAMA use Cholesky-based paths for the SPD inverse. General inversion paths can also differ by library. These are task-level API comparisons; a result is not a controlled comparison of identical algorithms. Factor retrieval and any required materialization are timed. Libraries differ in pivot representation and factor storage. EJML's symmetric-eigen adapter assembles dense Q and D from the returned eigenpairs so it returns the same mathematical outputs. No attempt is made to tune each library beyond the adapters shown in source. Review the adapter source before drawing conclusions about API choices. @@ -60,3 +60,21 @@ The expanded rerun preserves the initial harness timing settings and input seed. - UJMP uses only its core algorithms, with optional external backends disabled. Its results pass independent arithmetic and factorization checks. These choices are visible in source and are part of the measured API workload. They should not be interpreted as a claim that every adapter is optimally tuned for its library. + +## Fairness revision 2 and expanded tracks + +The earlier expanded archive timed an unevaluated ojAlgo scaling expression. Those scaling values are invalid for completed-work comparisons. Historical JSON remains unchanged and the archive report is marked. The corrected run reruns the full 286-case matrix; it does not splice corrected values into an older run. + +The corrected trial setup runs each operation twice, validates outputs and checks bit-for-bit preservation of native A/B/SPD/RHS inputs. It rejects any ojAlgo matrix output that is not a physical store. `FairnessValidationMain` demonstrates rejection of the old deferred scalar result and of a changed input. These checks are outside timed methods. + +The baseline still has 13 operations and eleven adapters. The large profile has sizes 512/1024 and multiplication, LU, Cholesky, QR and solve for NM Dev, EJML, ojAlgo and native jblas. The rectangular profile measures multiplication for those same four libraries at shapes 128×512×64, 512×128×256, 1024×256×64 and 255×257×129 (m×k×n). Each measured rectangular fork independently checks arithmetic twice and exact input preservation. + +`BENCH_THREADS` configures JVM-visible processors, explicit adapter thread limits and requested native thread environment variables. `BENCH_CPUSET` optionally pins the launcher and inherited fork processes. One JMH worker is retained: this measures internal library scaling, not concurrent client throughput. EJML's selected DDRM calls remain sequential; increasing the processor budget does not substitute a different parallel API. Native jblas worker counts remain unaudited. Affinity does not reserve physical cores or eliminate shared-host interference. + +All runs include JMH's GC profiler, raw allocation metrics, exact invocation overrides, dependency hashes and a zipped harness-source snapshot. Short audit runs are diagnostic; longer defaults remain available for stronger timing estimates. Sizes, thread budgets and fork counts must be compared within their own run. + +## Difficult-input accuracy diagnostics + +`DifficultValidationMain` tests row scaling with pivoting, tightly clustered eigenvalues, an 8×8 Hilbert matrix, and exact rank deficiency. It records every PASS/FAIL and exits nonzero if any case fails. Failed cases are retained in the audit and are not silently dropped or represented by a zero timing. + +For solves and inverses, these diagnostics use Frobenius-norm backward error `||AX-B|| / (||A|| ||X|| + ||B||)`, checking both multiplication orientations for an inverse. This assesses the residual relative to the computation's scale; it does not imply a small forward error for an ill-conditioned system. Other operations retain reconstruction/orthogonality checks and the common `1e-8` threshold. A failed rank-deficient QR call is recorded as an API failure, not proof that a mathematically valid QR factorization cannot exist. diff --git a/benchmark/site/reproduce.md b/benchmark/site/reproduce.md index e2f2a44..0fa5e22 100644 --- a/benchmark/site/reproduce.md +++ b/benchmark/site/reproduce.md @@ -11,7 +11,7 @@ bash benchmark/scripts/run.sh benchmark/results/my-run Alternatively use an existing published combined NM Dev JAR and the checked dependency lockfile: ```bash -export NMDEV_JAR=/absolute/path/to/nmdev-dist-1.2.6-SNAPSHOT.jar +export NMDEV_JAR=/absolute/path/to/nmdev-1.2.6.jar bash benchmark/scripts/build-direct.sh bash benchmark/scripts/run.sh benchmark/results/my-run ``` @@ -22,7 +22,7 @@ Allow roughly 35–45 minutes for the complete expanded run on similar hardware. ## Checks and report generation -The run stops on failed correctness checks. See [Validation](validation.html) for the checks and evidence. +The run stops on failed baseline correctness, deferred-output or native-input-preservation checks. Difficult-input diagnostics are a separate command and deliberately retain failures. See [Validation](validation.html) for the checks and evidence. ```bash python3 benchmark/scripts/report.py benchmark/results/my-run @@ -43,3 +43,58 @@ python3 benchmark/scripts/publish-pages.py benchmark/results/my-run ``` Commit the generated `docs/*.html` and `docs/.nojekyll` together with the result archive. The Documentation workflow builds from Markdown, checks internal links, uploads the generated site and publishes it through GitHub Pages on pushes to `main`. Pull requests build and check without deployment. Publishing the site does not rerun the Java benchmarks. Edit shared benchmark guide content in `benchmark/site/*.md`; the HTML pages are generated. + +## Coverage and computational thread profiles + +The defaults remain two forks, three 300 ms warmups and five 500 ms measurements. To reproduce the shorter fairness audit, add `-f 2 -wi 2 -w 200ms -i 3 -r 200ms`. Every invocation records its actual JMH overrides. Do not compare short and long runs as if they had identical uncertainty. + +```bash +# Pin a single computational worker; affinity is optional and must be valid on your host. +BENCH_CPUSET=0 BENCH_THREADS=1 bash benchmark/scripts/run.sh benchmark/results/my-fair-baseline + +# Larger square matrices: four libraries, sizes 512/1024, five operations. +BENCH_CPUSET=0 BENCH_PROFILE=large bash benchmark/scripts/run.sh benchmark/results/my-large + +# Tall, wide and odd-sized multiplication, with four explicitly supported adapters. +BENCH_CPUSET=0 BENCH_PROFILE=rectangular bash benchmark/scripts/run.sh benchmark/results/my-rectangular + +# Four-computational-thread comparison, one JMH client worker. Run serially after the one-thread track. +BENCH_CPUSET=0-3 BENCH_THREADS=4 BENCH_PROFILE=large bash benchmark/scripts/run.sh benchmark/results/my-four-thread + +# Separate difficult-input diagnostics (nonzero exit if any failures are recorded). +java -XX:ActiveProcessorCount=1 -cp 'benchmark/target/classes:benchmark/target/lib/*' \ + dev.nm.benchmark.DifficultValidationMain > difficult-correctness.csv +``` + +The large profile intentionally avoids automatically scheduling every decomposition at historical 5,000×5,000 sizes. Use explicit JMH `-p size=... -p operation=... -p library=...` overrides when resources permit. Historical-sized tests are a separate experiment, not implied by the 512/1024 track. + +`run.sh` rebuilds the harness against the local combined JAR, verifies locked third-party artifacts, runs all baseline guards, and records allocation metrics with `-prof gc`. Use new directories for each profile; previous JSON/logs are never overwritten. If your Maven build uses a differently named NM Dev JAR, run the direct build first with `NMDEV_JAR` pointing at that file. + +## Verify the measured binary's SPD inverse path + +The JDI helper needs a full JDK with `jdk.jdi` and compiler modules. It sets constructor breakpoints only on Chol and LU, without modifying the JAR. The debug run is separate from timing. + +```bash +java com.sun.tools.javac.Main --add-modules jdk.jdi -d benchmark/target/classes benchmark/scripts/InversePathTrace.java +java --add-modules jdk.jdi -cp 'benchmark/target/classes:benchmark/target/lib/*' InversePathTrace spd 64 +java --add-modules jdk.jdi -cp 'benchmark/target/classes:benchmark/target/lib/*' InversePathTrace spd 256 +java --add-modules jdk.jdi -cp 'benchmark/target/classes:benchmark/target/lib/*' InversePathTrace general 64 +java com.sun.tools.javap.Main -p -c -classpath benchmark/target/lib/nmdev-1.2.6.jar \ + dev.nm.algebra.linear.matrix.doubles.operation.Inverse +``` + +Expected: SPD calls Chol and not LU; the nonsymmetric control calls LU and not Chol. Verify the JAR's SHA-256 against the run's `artifacts.json` before interpreting the trace. The helper exits nonzero if the observed constructor counts disagree. + +## Current dependency versions + +See the [version audit](https://github.com/nmltd/nmdev-examples/blob/main/benchmark/dependency-audit/2026-10-18/README.md). The current primary Commons adapter uses the pinned 4.0-SNAPSHOT build; its timestamped module URLs and checksums are in the lockfile. Parallel Colt 0.11.4 is built from the exact upstream release tag by the direct build script because its old Maven Central coordinate stops at 0.10.1. Use the direct build route with JDK 17.0.20 to reproduce this source-built dependency's pinned binary hash. + +If you use the Maven route, first run the direct builder and install the resulting source-built Parallel Colt artifact into your local Maven repository: + +```bash +mvn install:install-file -Dfile=benchmark/target/lib/parallelcolt-0.11.4.jar \ + -DpomFile=benchmark/target/parallelcolt-source/pom.xml +mvn -f benchmark/pom.xml package +``` + +The direct route is the validated build used for these measurements. Per-run `versions.json` and `validated-nmdev-artifact.json` preserve labels and proof of the tested customer JAR. Current version labels do not replace the labels of historical archives. diff --git a/benchmark/src/main/java/dev/nm/benchmark/Adapter.java b/benchmark/src/main/java/dev/nm/benchmark/Adapter.java index ab7f73a..ac888a3 100644 --- a/benchmark/src/main/java/dev/nm/benchmark/Adapter.java +++ b/benchmark/src/main/java/dev/nm/benchmark/Adapter.java @@ -3,6 +3,8 @@ /** Native objects stay native throughout timing; conversion is used only for validation. */ interface Adapter { + /** Native input objects, used only by checks outside timing. */ + Object[] inputs(); Object run(String operation); double[][] array(Object matrix); } diff --git a/benchmark/src/main/java/dev/nm/benchmark/ColtAdapter.java b/benchmark/src/main/java/dev/nm/benchmark/ColtAdapter.java index d4053ba..75dd469 100644 --- a/benchmark/src/main/java/dev/nm/benchmark/ColtAdapter.java +++ b/benchmark/src/main/java/dev/nm/benchmark/ColtAdapter.java @@ -8,7 +8,8 @@ final class ColtAdapter implements Adapter { final DoubleMatrix2D a,b,spd,rhs; final Algebra algebra=new Algebra(); ColtAdapter(Inputs in){a=new DenseDoubleMatrix2D(in.a);b=new DenseDoubleMatrix2D(in.b);spd=new DenseDoubleMatrix2D(in.spd);rhs=new DenseDoubleMatrix2D(in.rhs);} - public Object run(String op){switch(op){ + public Object[] inputs(){return new Object[]{a,b,spd,rhs};} + public Object run(String op){switch(op){ case "transpose":return a.viewDice().copy();case "addition":return a.copy().assign(b,Functions.plus);case "scale":return a.copy().assign(Functions.mult(1.5));case "multiplication":return a.zMult(b,null); case "determinant":return algebra.det(a);case "inverse":return algebra.inverse(a);case "inverseSPD":return new CholeskyDecomposition(spd).solve(DoubleFactory2D.dense.identity(a.rows()));case "cholesky":return new CholeskyDecomposition(spd).getL(); case "lu":{LUDecomposition f=new LUDecomposition(a);return new Object[]{f.getL(),f.getU(),f.getPivot()};} diff --git a/benchmark/src/main/java/dev/nm/benchmark/CommonsAdapter.java b/benchmark/src/main/java/dev/nm/benchmark/CommonsAdapter.java index 120a837..5f543b8 100644 --- a/benchmark/src/main/java/dev/nm/benchmark/CommonsAdapter.java +++ b/benchmark/src/main/java/dev/nm/benchmark/CommonsAdapter.java @@ -1,11 +1,12 @@ /* Copyright (c) NM LTD. See the repository LICENSE. */ package dev.nm.benchmark; -import org.apache.commons.math3.linear.*; +import org.apache.commons.math4.legacy.linear.*; final class CommonsAdapter implements Adapter { final RealMatrix a,b,spd,rhs; CommonsAdapter(Inputs in){a=new Array2DRowRealMatrix(in.a);b=new Array2DRowRealMatrix(in.b);spd=new Array2DRowRealMatrix(in.spd);rhs=new Array2DRowRealMatrix(in.rhs);} + public Object[] inputs(){return new Object[]{a,b,spd,rhs};} public Object run(String op) { switch(op) { case "transpose": return a.transpose(); diff --git a/benchmark/src/main/java/dev/nm/benchmark/CorrectnessMain.java b/benchmark/src/main/java/dev/nm/benchmark/CorrectnessMain.java index 1883c7a..b23d1dd 100644 --- a/benchmark/src/main/java/dev/nm/benchmark/CorrectnessMain.java +++ b/benchmark/src/main/java/dev/nm/benchmark/CorrectnessMain.java @@ -15,9 +15,7 @@ public static void main(String[] args){ for(int size:new int[]{4,64,256})for(String library:libraries)for(String op:operations){ try { Inputs inputs=new Inputs(size);Adapter adapter=MatrixBenchmark.create(library,inputs); - double residual=Validation.check(op,inputs,adapter,adapter.run(op)); - // A second invocation checks that repeat execution remains valid. - Validation.check(op,inputs,adapter,adapter.run(op)); + double residual=Validation.checkRepeated(op, inputs, adapter); csv.printf(Locale.ROOT,"%s,%d,%s,%.12g,PASS%n",library,size,op,residual); } catch(Throwable error) { failures++; csv.printf("%s,%d,%s,,FAIL%n",library,size,op); System.err.println(library+"/"+size+"/"+op);error.printStackTrace(); } } diff --git a/benchmark/src/main/java/dev/nm/benchmark/DifficultValidationMain.java b/benchmark/src/main/java/dev/nm/benchmark/DifficultValidationMain.java new file mode 100644 index 0000000..2d36b8e --- /dev/null +++ b/benchmark/src/main/java/dev/nm/benchmark/DifficultValidationMain.java @@ -0,0 +1,54 @@ +package dev.nm.benchmark; + +import java.util.Locale; + +/** Accuracy diagnostics are kept separate from baseline speed rankings. All failures are recorded. */ +public final class DifficultValidationMain { + public static void main(String[] args) { + String[] libraries=args.length>0?args[0].split(","):new String[]{"NMDEV","EJML","COMMONS","JAMA","COLT","PARALLEL_COLT","JBLAS","F2J_LAPACK","MTJ","OJALGO","UJMP"}; + java.io.PrintStream csv=System.out;System.setOut(System.err); + csv.println("library,fixture,size,operation,residual,status");int failures=0; + for(String library:libraries)for(String fixture:new String[]{"scaled-pivot","clustered-spd","hilbert","rank-deficient"}) { + Inputs in=fixture(fixture); + String[] ops=fixture.equals("rank-deficient")?new String[]{"qr","svd"}: + fixture.equals("clustered-spd")?new String[]{"eigenSymmetric","svd","cholesky","inverseSPD"}: + fixture.equals("hilbert")?new String[]{"lu","solve","cholesky","inverseSPD"}:new String[]{"lu","solve","inverse","qr","svd"}; + for(String op:ops)try { + Adapter adapter=MatrixBenchmark.create(library,in); + Object[] nativeInputs=adapter.inputs();double[][][] before=new double[nativeInputs.length][][]; + for(int i=0;iSource provenance
-Starter copied from nmltd/numerical-methods-java at commit d941c30553488363a5a5258a7c85f49082df70fb, directory samples/new-nmdev-project. HelloWorld, Graph and HelloWorldTest are retained; nm.lic is excluded from publication; root README is replaced, its original guide is adapted under docs/starter-project.md, and Maven identity/exec configuration and ignore rules are updated.
Starter copied from nmltd/numerical-methods-java at commit d941c30553488363a5a5258a7c85f49082df70fb, directory samples/new-nmdev-project. HelloWorld, Graph and HelloWorldTest are retained; the public trial nm.lic was initially excluded and is now copied unchanged from source commit b6d096f8e3dd29d57ec07e10c76188ca82943d09; root README is replaced, its original guide is adapted under docs/starter-project.md, and Maven identity/exec configuration and ignore rules are updated.
Tutorials adapt these tests from nmltd/nmdev at commit fa2904ccec99820798ef7760425daf1d629d83c1:
modules/basic/src/test/java/dev/nm/algebra/linear/matrix/doubles/factorization/triangle/cholesky/CholeskyBanachiewiczTest.java
diff --git a/docs/benchmarks.html b/docs/benchmarks.html
index c11bd58..8d8c5a9 100644
--- a/docs/benchmarks.html
+++ b/docs/benchmarks.html
@@ -1,8 +1,8 @@
-- JMH JSON and per-fork iteration means -
- Complete JMH log -
- Summary CSV -
- Baseline correctness residuals -
- Targeted pivot/sign checks -
- Environment and requested thread settings -
- Backend records -
- Post-run backend configuration probe -
- Artifact versions and SHA-256 checksums -
- Measured source checksums -
- Dependency release metadata +
- JMH JSON and per-fork iteration means +
- Complete JMH log +
- Summary CSV +
- JVM allocation per operation; native allocations are not fully included +
- Baseline correctness residuals +
- Targeted pivot/sign checks +
- Environment and requested thread settings +
- Backend records +
- Artifact versions and SHA-256 checksums +
- Measured source checksums +
- Dependency release metadata +
- Exact harness source snapshot +
- Deferred-output and mutation regression checks
NM Dev benchmark explorer
Compare saved measurements from 2026-10-18-expanded. Hover, tap or keyboard-focus a point for timing, uncertainty and backend details.
Lower values mean less time. Both modes use a logarithmic horizontal axis. Ratios are ratios of measured means, without confidence intervals for the ratios. Results describe the recorded shared VM and backend configurations, not universal performance rankings.
- - +NM Dev benchmark explorer
Fairness revision 2: timed outputs are materialized and native inputs are checked for mutation. Short audit timings are diagnostic. Audit and accuracy findings.
Compare saved measurements from 2026-10-18-latest. Hover, tap or keyboard-focus a point for timing, uncertainty and backend details.
Lower values mean less time. Both modes use a logarithmic horizontal axis. Ratios are ratios of measured means, without confidence intervals for the ratios. Results describe the recorded shared VM and backend configurations, not universal performance rankings.
+ + -Results tables · Methodology · Versions and backends · Reproduce
SVG downloads contain the selected chart, labels, units and run identifier. CSV exports include original millisecond timings, confidence limits and the selected reference ratios. All calculations and exports use the saved measurements; no benchmark is run in your browser.
New NMDEV project
-This sample uses the published obfuscated combined dev.nm:nmdev-dist:1.2.6-SNAPSHOT JAR from GitHub Packages. Java 8 or later and Maven are required.
This sample uses the published obfuscated combined dev.nm:nmdev:1.2.6 JAR from GitHub Packages. Java 8 or later and Maven are required.
Authentication
-The binary package is public at https://maven.pkg.github.com/nmltd/maven-packages. No access to the private NM Dev source repository is required. Create a personal access token (classic) with read:packages using GitHub's instructions.
The binary package is hosted at https://maven.pkg.github.com/nmltd/nmdev. Your GitHub account and token must have access to the package; contact NM Dev for access. Create a personal access token (classic) with read:packages using GitHub's instructions.
Add the following server to ~/.m2/settings.xml (merge it into existing settings):
<settings xmlns="http://maven.apache.org/SETTINGS/1.2.0">
<servers>
@@ -15,7 +15,7 @@ Authentication
Set those environment variables locally. Never commit a token.
License, build and run
-A license is not included. Obtain a trial or commercial license from NM Dev and save it as nm.lic in the repository root. The runtime loads this external file from the current working directory; run the commands below from the repository root. Trial validity and entitlements are enforced by NMDEV. Replace it with your own valid license when the trial expires. The license is not embedded in the application JAR. Run these commands from the repository root:
The original starter's public trial license is included as nm.lic in the repository root. Obtain a replacement trial or commercial license from NM Dev when needed. The runtime loads this external file from the current working directory; run the commands below from the repository root. Trial validity and entitlements are enforced by NMDEV. Replace it with your own valid license when the trial expires. The license is not embedded in the application JAR. Run these commands from the repository root:
mvn -U clean verify
java -jar target/nmdev-examples-1.0.0-SNAPSHOT.jar
java -cp "target/nmdev-examples-1.0.0-SNAPSHOT.jar:target/lib/*" dev.nm.Graph
diff --git a/docs/validation.html b/docs/validation.html
index d1021e6..d105536 100644
--- a/docs/validation.html
+++ b/docs/validation.html
@@ -1,20 +1,22 @@
-Validation — NM Dev benchmarks Validation and interpretation
+Validation — NM Dev benchmarks Validation and interpretation
All 429 baseline correctness cases passed twice. Every measured fork also checks its selected result before timing. Reference arithmetic and residual checks are outside timing.
All 66 targeted row-swap/row-cycle cases also passed. These check determinant signs, LU permutation orientation and solves on pivoting inputs.
Tables show mean ± JMH 99.9% confidence half-width, in ms/op. These error bars are not standard deviations. Overlapping intervals or high uncertainty make fine rankings unreliable. Chart ratios are ratios of means and do not show ratio uncertainty.
-Native input conversion is outside timing; necessary copies, allocations and full factor output materialization are timed. This compares public API tasks, not identical algorithms. NM Dev uses a general inverse routine on SPD inputs; other adapters may use Cholesky-based inversion. F2J transpose/addition/scaling use adapter-owned Java loops because LAPACK does not define those matrix convenience APIs.
+Native input conversion is outside timing; necessary copies, allocations and full factor output materialization are timed. This compares public API tasks, not identical algorithms. NM Dev Inverse detects symmetry and attempts Cholesky on SPD inputs, falling back to LU if that path raises a runtime exception. The exact measured JAR was verified separately by bytecode and runtime constructor-breakpoint tracing; see the fairness audit. F2J transpose/addition/scaling use adapter-owned Java loops because LAPACK does not define those matrix convenience APIs.
Evidence
-
See Methodology for input construction and residual checks.