From b8aa7ad0119c1f4911ac04add5ff645c26c7fc36 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 11 Nov 2026 12:30:18 +0700 Subject: [PATCH 1/2] hexagon: pack Q8_0 files for the NPU and measure them on the IQ-10 scripts/quantize_gguf.py gains three flags: - --pad-m32 zero-pads FFN widths that are not a multiple of 32 in the file. The HMX takes no other width for a packed weight, and the loader pads an FFN at load only when its weights are float. - --dit also packs the DiT's weight matrices, except adaLN and the time embedding. - --keep-float REGEX leaves the matching weights at their source precision. tests/py/test_converters.py covers them. docs/benchmark/qualcomm-iq10.md reports latency, memory and fidelity of each Q8_0 configuration of the 13 models on the IQ-10's Hexagon NPU, against the CPU-only build at F32. The runtime is unchanged. --- CHANGELOG.md | 6 ++ docs/backend/hexagon.md | 20 ++++++ docs/benchmark/qualcomm-iq10.md | 122 ++++++++++++++++++++++++++++++++ scripts/quantize_gguf.py | 60 +++++++++++++++- tests/py/test_converters.py | 22 ++++++ 5 files changed, 228 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5743416a..9c7c0733 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -51,6 +51,12 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke against a CPU-only build of the same commit. GR00T N1.5 and N1.6 are above the bar on a served LIBERO observation, as the CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install and the numbers. +- **Q8_0 files for the Hexagon NPU.** `scripts/quantize_gguf.py` gains + `--pad-m32` (zero-pads FFN widths that are not a multiple of 32, which the HMX + needs for a packed weight), `--dit` (also packs the DiT's weight matrices + except adaLN and the time embedding) and `--keep-float REGEX` (leaves the + matching weights float). `docs/benchmark/qualcomm-iq10.md` reports latency, + memory and fidelity on the IQ-10 for each Q8_0 configuration of the 13 models. - **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200` (Radeon RX 9060 XT) with ROCm 7.14 and llama.cpp `b11223`, every arch except diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index d87a1ee2..c205d672 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -430,6 +430,26 @@ slightly further (1.0e-2 against 9.0e-3). They are sensitive to 16-bit precision rather than miscomputed. Running them with F32 weights on the NPU has not been tried yet. +### Q8_0 weights + +`scripts/quantize_gguf.py --type Q8_0` packs a checkpoint's weights to Q8_0. +Three flags shape a file for the NPU: + +- `--pad-m32` zero-pads in the file the FFN widths that are not a multiple of + 32. The HMX takes no other width for a packed weight, and the loader pads an + FFN at load only when its weights are float. +- `--dit` also packs the DiT's weight matrices, except adaLN and the time + embedding. +- `--keep-float REGEX` leaves the weights whose name matches REGEX float. + +```bash +python scripts/quantize_gguf.py --in gr00tn1d6-libero.gguf --out gr00tn1d6-q8_0.gguf \ + --type Q8_0 --vision --pad-m32 --dit +``` + +Latency, memory and fidelity per configuration: +[benchmark/qualcomm-iq10.md](../benchmark/qualcomm-iq10.md#q8_0-weights). + ## IQ-10, imagined Nothing about a second-generation part is knowable from this repository, so what diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index d583607b..0ce2d555 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -146,6 +146,128 @@ inputs), N1.7 6.2e-3 on fixed inputs (2.7e-3 served). +## Q8_0 weights + +These rows run GGUFs that `scripts/quantize_gguf.py --type Q8_0` made from the +checkpoints above (ggml's block-32 Q8_0 weights). The setup is the one above, +except: + +| | | +|---|---| +| Commit | `71cdb09` | +| Date | 2026-11-11 10:10 to 11:49 (UTC+07) for latency, memory and served fidelity; fixed-input fidelity 2026-11-10 | +| Runtime flags | defaults | + +Each configuration packs the weights of the one before it, plus: + +| Config | `quantize_gguf.py` flags | Packs | +|---|---|---| +| F16 | - | none: the shipped file, loaded as F16 (`--weight-dtype f16`, the default) | +| default | `--type Q8_0` | the 2-D weights outside the vision tower and the action head. Embeddings, the output head, norms, patch embeddings and position tables stay float in every configuration | +| vision, unpadded | `--vision` | the vision tower | +| vision | `--vision --pad-m32` | the vision tower, with FFN widths that are not a multiple of 32 zero-padded (SigLIP's 4304 to 4320 in π0, π0.5, GR00T N1.5 and N1.6, VLA-Adapter and OpenVLA-OFT) | +| vision + DiT | `--vision --pad-m32 --dit` | the DiT's weight matrices, except adaLN and the time embedding | +| ..., keep-float (*kind*, *n*) | the flags of the named configuration plus `--keep-float REGEX` | that configuration with *n* weights left float | + +The `--keep-float` lists come from a greedy search: each group of blocks (a +quarter of a stack, attention or FFN) is packed alone and measured, the groups +that move the action chunk most are left float until the deviation is under the +target, then blocks are packed back one at a time while it stays under. *fixed* +uses the fixed-input deviation and *served* the served one, with a target of +2.9e-3; GR00T N1.5 and N1.6 take 1.1 times their F16 served deviation instead +(1.3e-2, 1.1e-2). *union* leaves both lists float. + +`fixed max|Δ|` is measured as in [Fidelity](../backend/hexagon.md#fidelity) and +`served max|Δ|` as in [Serving](../backend/hexagon.md#serving), against the +CPU-only build at F32. + +| Model | Config | Q8_0 tensors | GGUF GiB | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | mean vs F16 | RAM vs F16 | fixed max\|Δ\| | served max\|Δ\| | +|---|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:| +| Octo-Small | F16 | - | 0.51 | 81.5 | 84.1 | 82.3 | 88.3 | 20.7 | 252 | 725 | - | - | 5.0e-4 | 1.4e-4 | +| Octo-Small | default | 121 / 318 | 0.22 | 79.5 | 81.6 | 80.7 | 85.5 | 20.4 | 258 | 405 | -3.0% | -44.1% | 2.5e-3 | 2.0e-3 | +| Octo-Small | vision | 123 / 318 | 0.22 | 80.2 | 83.5 | 81.1 | 90.3 | 21.6 | 258 | 402 | -0.7% | -44.6% | 1.9e-3 | 1.9e-3 | +| ACT | F16 | - | 0.13 | 147.0 | 152.5 | 152.3 | 156.0 | - | 194 | 469 | - | - | 6.1e-4 | 1.0e-3 | +| ACT | default | 35 / 147 | 0.06 | 145.9 | 150.9 | 150.9 | 152.8 | - | 192 | 405 | -1.0% | -13.6% | 2.1e-3 | 2.4e-3 | +| TurboVLA | F16 | - | 0.80 | 186.6 | 190.2 | 189.1 | 195.3 | - | 248 | 1079 | - | - | 3.7e-4 | 8.5e-4 | +| TurboVLA | default | 133 / 657 | 0.53 | 184.8 | 186.5 | 186.4 | 188.2 | - | 248 | 814 | -1.9% | -24.6% | 1.3e-2 | 4.2e-3 | +| TurboVLA | vision | 208 / 657 | 0.30 | 182.6 | 185.7 | 184.1 | 190.7 | - | 245 | 485 | -2.4% | -55.1% | 1.4e-2 | 8.3e-3 | +| TurboVLA | vision, keep-float (fixed, 32) | 176 / 657 | 0.39 | 181.2 | 185.8 | 183.6 | 191.4 | - | 245 | 611 | -2.3% | -43.4% | 2.6e-3 | 6.9e-3 | +| VLA-JEPA | F16 | - | 4.25 | 489.1 | 496.8 | 494.6 | 506.4 | 197.8 | 184 | 4018 | - | - | 2.1e-3 | 2.2e-3 | +| VLA-JEPA | default | 196 / 873 | 3.02 | 425.6 | 432.1 | 430.3 | 437.5 | 199.1 | 184 | 2718 | -13.0% | -32.4% | 1.4e-2 | 1.0e-1 | +| VLA-JEPA | vision | 300 / 873 | 2.67 | 404.8 | 411.6 | 410.8 | 416.8 | 178.4 | 96 | 2350 | -17.1% | -41.5% | 1.5e-2 | 1.6e-1 | +| VLA-JEPA | vision + DiT | 396 / 873 | 2.56 | 387.0 | 393.5 | 392.9 | 400.2 | 179.2 | 97 | 2288 | -20.8% | -43.1% | 1.3e-2 | 1.3e-1 | +| VLA-JEPA | vision + DiT, keep-float (fixed, 233) | 163 / 873 | 3.88 | 456.6 | 461.4 | 459.9 | 467.7 | 185.0 | 184 | 3614 | -7.1% | -10.1% | 2.9e-3 | 8.3e-3 | +| VLA-JEPA | vision + DiT, keep-float (served, 143) | 253 / 873 | 3.38 | 430.1 | 436.6 | 434.2 | 444.7 | 182.3 | 184 | 3117 | -12.1% | -22.4% | 8.4e-3 | 2.1e-3 | +| VLA-JEPA | vision + DiT, keep-float (union, 244) | 152 / 873 | 3.93 | 452.6 | 461.6 | 460.7 | 469.3 | 185.3 | 184 | 3679 | -7.1% | -8.4% | 3.7e-3 | 5.1e-3 | +| GR00T N1.5 | F16 | - | 6.46 | 527.3 | 529.3 | 528.9 | 531.3 | 138.7 | 161 | 3659 | - | - | 1.9e-3 | 1.2e-2 | +| GR00T N1.5 | default | 85 / 887 | 5.93 | 499.5 | 500.2 | 500.0 | 501.3 | 138.3 | 161 | 3148 | -5.5% | -14.0% | 5.1e-3 | 8.0e-2 | +| GR00T N1.5 | vision | 247 / 887 | 5.57 | 479.1 | 481.1 | 479.7 | 483.8 | 118.8 | 161 | 2754 | -9.1% | -24.7% | 7.2e-3 | 4.6e-2 | +| GR00T N1.5 | vision + DiT | 345 / 887 | 4.28 | 398.7 | 399.2 | 399.1 | 399.5 | 117.9 | 161 | 2339 | -24.6% | -36.1% | 8.4e-3 | 3.6e-2 | +| GR00T N1.5 | vision + DiT, keep-float (fixed, 191) | 154 / 887 | 5.57 | 448.3 | 449.0 | 448.7 | 450.0 | 131.8 | 162 | 3061 | -15.2% | -16.3% | 2.7e-3 | 2.8e-2 | +| GR00T N1.5 | vision + DiT, keep-float (served, 223) | 122 / 887 | 5.86 | 471.7 | 472.9 | 472.6 | 474.0 | 133.1 | 161 | 3172 | -10.7% | -13.3% | 6.8e-3 | 9.1e-3 | +| GR00T N1.5 | vision + DiT, keep-float (union, 272) | 73 / 887 | 6.27 | 489.0 | 491.2 | 489.7 | 494.8 | 135.1 | 161 | 3413 | -7.2% | -6.7% | 3.3e-3 | 1.1e-2 | +| SmolVLA | F16 | - | 1.05 | 583.4 | 584.0 | 583.8 | 584.1 | 229.9 | 146 | 1226 | - | - | 7.4e-4 | 1.9e-3 | +| SmolVLA | default | 225 / 791 | 0.76 | 568.2 | 569.0 | 568.6 | 569.8 | 228.8 | 104 | 919 | -2.6% | -25.0% | 1.2e-2 | 1.3e-2 | +| SmolVLA | vision | 297 / 791 | 0.69 | 560.4 | 561.0 | 560.9 | 561.6 | 221.3 | 104 | 848 | -3.9% | -30.8% | 3.4e-2 | 1.6e-2 | +| SmolVLA | vision, keep-float (fixed, 58) | 239 / 791 | 0.73 | 563.5 | 564.2 | 564.1 | 564.5 | 222.9 | 104 | 896 | -3.4% | -26.9% | 2.0e-3 | 1.6e-2 | +| VLA-Adapter | F16 | - | 2.59 | 664.7 | 669.2 | 666.6 | 677.5 | 257.6 | 872 | 3569 | - | - | 3.9e-4 | 3.9e-4 | +| VLA-Adapter | default | 171 / 1455 | 2.25 | 649.4 | 653.5 | 651.8 | 658.0 | 256.6 | 872 | 3200 | -2.3% | -10.3% | 4.3e-3 | 6.3e-3 | +| VLA-Adapter | vision | 367 / 1455 | 1.65 | 617.9 | 623.4 | 622.1 | 629.0 | 226.5 | 852 | 2555 | -6.8% | -28.4% | 3.1e-3 | 6.0e-3 | +| VLA-Adapter | vision, keep-float (fixed, 8) | 359 / 1455 | 1.66 | 618.2 | 622.7 | 622.6 | 626.9 | 226.5 | 852 | 2516 | -6.9% | -29.5% | 2.8e-3 | 6.3e-3 | +| VLA-Adapter | vision, keep-float (served, 40) | 327 / 1455 | 1.78 | 623.6 | 627.7 | 625.5 | 632.3 | 227.1 | 849 | 2494 | -6.2% | -30.1% | 2.9e-3 | 2.9e-3 | +| GR00T N1.6 | F16 | - | 8.53 | 681.0 | 682.6 | 682.3 | 684.1 | 145.0 | 146 | 4929 | - | - | 1.3e-3 | 1.0e-2 | +| GR00T N1.6 | default | 114 / 1094 | 7.44 | 643.4 | 644.5 | 643.9 | 646.6 | 144.2 | 134 | 3959 | -5.6% | -19.7% | 7.2e-3 | 3.4e-2 | +| GR00T N1.6 | vision, unpadded | 249 / 1094 | 7.20 | 815.6 | 817.3 | 817.3 | 818.5 | 316.7 | 134 | 3738 | +19.7% | -24.2% | 7.1e-3 | 2.5e-2 | +| GR00T N1.6 | vision | 276 / 1094 | 7.08 | 621.6 | 622.5 | 622.1 | 623.2 | 124.2 | 134 | 3588 | -8.8% | -27.2% | 8.0e-3 | 2.9e-2 | +| GR00T N1.6 | vision + DiT | 470 / 1094 | 4.52 | 461.4 | 462.2 | 461.9 | 462.9 | 124.2 | 116 | 2777 | -32.3% | -43.7% | 8.1e-3 | 2.9e-2 | +| GR00T N1.6 | vision + DiT, keep-float (fixed, 57) | 413 / 1094 | 5.26 | 481.7 | 482.3 | 482.2 | 482.7 | 125.5 | 146 | 3076 | -29.3% | -37.6% | 2.8e-3 | 3.6e-2 | +| GR00T N1.6 | vision + DiT, keep-float (served, 3) | 467 / 1094 | 4.55 | 463.2 | 464.0 | 463.6 | 465.0 | 124.3 | 146 | 2800 | -32.0% | -43.2% | 7.6e-3 | 9.2e-3 | +| GR00T N1.6 | vision + DiT, keep-float (union, 60) | 410 / 1094 | 5.29 | 480.1 | 480.7 | 480.5 | 481.2 | 125.2 | 147 | 3114 | -29.6% | -36.8% | 3.0e-3 | 8.6e-3 | +| GR00T N1.7 | F16 | - | 5.86 | 740.1 | 744.9 | 745.0 | 749.0 | 196.9 | 190 | 5108 | - | - | 2.0e-3 | 2.0e-3 | +| GR00T N1.7 | default | 112 / 1030 | 5.16 | 699.0 | 703.5 | 701.5 | 709.3 | 196.1 | 190 | 4356 | -5.6% | -14.7% | 1.1e-2 | 2.0e-2 | +| GR00T N1.7 | vision | 216 / 1030 | 4.81 | 683.3 | 688.4 | 686.0 | 694.1 | 179.5 | 190 | 3996 | -7.6% | -21.8% | 2.0e-2 | 2.2e-2 | +| GR00T N1.7 | vision + DiT | 410 / 1030 | 3.99 | 528.8 | 536.1 | 534.6 | 542.1 | 181.6 | 191 | 3151 | -28.0% | -38.3% | 2.8e-2 | 2.1e-2 | +| GR00T N1.7 | vision + DiT, keep-float (fixed, 185) | 225 / 1030 | 4.90 | 599.2 | 606.0 | 605.4 | 611.7 | 186.9 | 191 | 4051 | -18.6% | -20.7% | 2.6e-3 | 5.3e-3 | +| GR00T N1.7 | vision + DiT, keep-float (served, 157) | 253 / 1030 | 4.68 | 573.7 | 579.8 | 578.0 | 590.8 | 187.3 | 190 | 3870 | -22.2% | -24.2% | 4.2e-2 | 2.8e-3 | +| GR00T N1.7 | vision + DiT, keep-float (union, 233) | 177 / 1030 | 5.06 | 609.4 | 614.4 | 614.1 | 619.1 | 188.2 | 190 | 4249 | -17.5% | -16.8% | 3.9e-3 | 6.5e-3 | +| π0.5 | F16 | - | 11.72 | 1106.0 | 1107.8 | 1107.0 | 1111.5 | 277.0 | 284 | 5980 | - | - | 4.8e-4 | 3.7e-4 | +| π0.5 | default | 127 / 819 | 6.30 | 969.3 | 971.0 | 969.8 | 974.7 | 277.3 | 126 | 3997 | -12.3% | -33.2% | 4.9e-4 | 1.9e-3 | +| π0.5 | vision, unpadded | 262 / 819 | 6.06 | 1318.2 | 1320.2 | 1318.9 | 1324.5 | 621.0 | 126 | 3760 | +19.2% | -37.1% | 8.5e-4 | 2.7e-3 | +| π0.5 | vision | 289 / 819 | 5.94 | 939.6 | 941.3 | 940.8 | 943.1 | 236.7 | 103 | 3676 | -15.0% | -38.5% | 9.2e-4 | 2.8e-3 | +| π0 | F16 | - | 6.04 | 1118.9 | 1123.3 | 1123.7 | 1125.3 | 276.9 | 284 | 5662 | - | - | 7.7e-4 | 5.8e-4 | +| π0 | default | 127 / 780 | 4.31 | 976.4 | 978.8 | 977.8 | 981.2 | 277.0 | 126 | 3723 | -12.9% | -34.2% | 1.8e-2 | 1.7e-3 | +| π0 | vision | 289 / 780 | 3.95 | 935.7 | 937.6 | 937.3 | 938.8 | 235.9 | 103 | 3337 | -16.5% | -41.1% | 1.6e-2 | 1.7e-3 | +| π0 | default, keep-float (fixed, 8) | 119 / 780 | 4.32 | 977.0 | 978.8 | 978.3 | 980.2 | 276.9 | 126 | 3711 | -12.9% | -34.5% | 1.4e-3 | 5.2e-4 | +| OpenVLA-OFT | F16 | - | 14.03 | 1122.2 | 1125.2 | 1125.0 | 1127.6 | 262.7 | 844 | 15266 | - | - | 3.1e-4 | 7.5e-4 | +| OpenVLA-OFT | default | 227 / 958 | 8.32 | 843.5 | 845.7 | 844.8 | 848.9 | 259.5 | 844 | 9403 | -24.8% | -38.4% | 5.9e-4 | 3.5e-3 | +| OpenVLA-OFT | vision | 423 / 958 | 7.72 | 810.1 | 812.5 | 810.6 | 816.4 | 228.7 | 833 | 8821 | -27.8% | -42.2% | 7.9e-4 | 3.2e-3 | +| OpenVLA-OFT | vision, keep-float (served, 12) | 411 / 958 | 7.86 | 818.7 | 821.4 | 820.4 | 824.8 | 230.0 | 839 | 8959 | -27.0% | -41.3% | 7.9e-4 | 2.5e-3 | +| Evo-1 | F16 | - | 1.45 | 1640.1 | 1659.0 | 1653.3 | 1675.4 | 818.6 | 318 | 1582 | - | - | 6.2e-4 | 1.1e-3 | +| Evo-1 | default | 100 / 636 | 1.26 | 1643.5 | 1655.4 | 1653.6 | 1660.8 | 820.1 | 312 | 1459 | -0.2% | -7.8% | 5.4e-3 | 1.2e-2 | +| Evo-1 | vision | 196 / 636 | 1.00 | 1620.6 | 1638.8 | 1639.0 | 1645.7 | 808.1 | 312 | 1121 | -1.2% | -29.1% | 6.3e-3 | 1.4e-2 | +| Evo-1 | vision, keep-float (fixed, 25) | 171 / 636 | 1.05 | 1624.7 | 1634.5 | 1632.9 | 1647.6 | 803.5 | 319 | 1171 | -1.5% | -26.0% | 2.4e-3 | 1.1e-2 | + +A per-op profile of the default, vision and vision + DiT files puts every Q8_0 +matmul on the HMX. The loader zero-pads a 4304-wide FFN to 4320 at load only +when its weights are float, so without `--pad-m32` the vision tower's FFNs of +π0.5 and GR00T N1.6 keep their 4304 width (rows *vision, unpadded*). + +Fixed-input deviation of the default configuration on the NPU and on the +CPU-only build, against the CPU-only build at F32: + +| Model | NPU | CPU-only build | +|---|--:|--:| +| π0 | 1.8e-2 | 1.9e-2 | +| π0.5 | 4.9e-4 | 1.1e-3 | +| SmolVLA | 1.2e-2 | 1.8e-2 | +| Evo-1 | 5.4e-3 | 7.3e-3 | +| GR00T N1.5 | 5.1e-3 | 1.2e-2 | +| GR00T N1.6 | 7.2e-3 | 1.2e-2 | +| GR00T N1.7 | 1.1e-2 | 4.4e-2 | +| VLA-JEPA | 1.4e-2 | 2.7e-2 | +| VLA-Adapter | 4.4e-3 | 6.2e-3 | +| TurboVLA | 1.3e-2 | 2.1e-2 | + ## Not run - **BitVLA**: N/A off CUDA. The only published GGUF is int2-packed, and only diff --git a/scripts/quantize_gguf.py b/scripts/quantize_gguf.py index f364d920..f13fe409 100644 --- a/scripts/quantize_gguf.py +++ b/scripts/quantize_gguf.py @@ -31,6 +31,7 @@ """ import argparse +import re import numpy as np import gguf @@ -60,6 +61,9 @@ "time" ) SKIP_VISION = ("vit", "vision", "vis.d.", "vis.s.", "octo.obs.") +# With --dit, the DiT's weight matrices are packed too; these entries of SKIP +# are what kept them float. +DIT_UNSKIP = ("dit", "aex.", "ah.") # Block size per row (ne0 must divide this). Only the types the gguf writer can # pack are offered; Q8_0 is near-lossless, Q4_0/Q4_1 are 4-bit. @@ -81,21 +85,56 @@ def to_f32(t) -> np.ndarray: return None # already packed / unsupported source -def eligible(name: str, shape, qtype: str, skip) -> bool: +def eligible(name: str, shape, qtype: str, skip, dit: bool = False, keep=None) -> bool: if len(shape) != 2 or not name.endswith(".weight"): return False + if keep is not None and keep.search(name): + return False + if dit and ".dit." in name: + skip = tuple(s for s in skip if s not in DIT_UNSKIP) if any(s in name for s in skip): return False return int(shape[0]) % QK[qtype] == 0 # ne0 is the quantization axis +def ffn_pads(r) -> dict: + """FFN blocks whose width is not a multiple of 32: prefix -> (width, padded).""" + pads = {} + for t in r.tensors: + if t.name.endswith(".fc1.weight") and len(t.shape) == 2 and int(t.shape[1]) % 32: + width = int(t.shape[1]) + pads[t.name[:-len(".fc1.weight")]] = (width, (width + 31) // 32 * 32) + return pads + + +def pad_ffn(t, old: int, new: int) -> np.ndarray: + """fc1 rows and bias, or fc2 columns, zero-padded from old to new.""" + arr = to_f32(t) + if t.name.endswith(".fc1.weight"): + return np.concatenate([arr, np.zeros((new - old, arr.shape[1]), np.float32)], axis=0) + if t.name.endswith(".fc1.bias"): + return np.concatenate([arr, np.zeros((new - old,), np.float32)]) + return np.concatenate([arr, np.zeros((arr.shape[0], new - old), np.float32)], axis=1) + + def main() -> None: ap = argparse.ArgumentParser() ap.add_argument("--in", dest="src", required=True) ap.add_argument("--out", dest="dst", required=True) ap.add_argument("--type", default="Q8_0", choices=sorted(QK)) ap.add_argument("--vision", action="store_true", help="also pack the vision tower") + ap.add_argument("--dit", action="store_true", + help="also pack the DiT's weight matrices (adaLN, the time embedding, " + "norms and the state/action encoders stay float)") + ap.add_argument("--keep-float", metavar="REGEX", + help="keep the weights whose name matches REGEX at their source precision, " + "e.g. '^vlm\\.blk\\.(0|1)\\.' for an LM's first two blocks") + ap.add_argument("--pad-m32", action="store_true", + help="zero-pad FFN widths that are not a multiple of 32 (fc1 rows and bias, " + "fc2 columns): Hexagon's matrix unit takes no other width for a packed " + "weight. The output is unchanged") args = ap.parse_args() + keep = re.compile(args.keep_float) if args.keep_float else None skip = SKIP if args.vision else SKIP + SKIP_VISION r = gguf.GGUFReader(args.src) @@ -108,12 +147,29 @@ def main() -> None: copy_kv(r, w) qtype = getattr(gguf.GGMLQuantizationType, args.type) + pads = ffn_pads(r) if args.pad_m32 else {} + if pads: + old, new = next(iter(pads.values())) + print(f"--pad-m32: {len(pads)} FFN blocks, e.g. {old} -> {new}") + padded = (".fc1.weight", ".fc1.bias", ".fc2.weight") n_q = 0 bytes_in = bytes_out = 0 for t in r.tensors: src_bytes = int(t.data.nbytes) bytes_in += src_bytes - f32 = to_f32(t) if eligible(t.name, t.shape, args.type, skip) else None + blk = t.name.rsplit(".", 2)[0] if t.name.endswith(padded) else None + if blk in pads and to_f32(t) is not None: + arr = pad_ffn(t, *pads[blk]) + if eligible(t.name, arr.shape[::-1], args.type, skip, args.dit, keep): + packed = gguf.quants.quantize(arr, qtype) + w.add_tensor(t.name, packed, raw_dtype=qtype) + bytes_out += int(packed.nbytes) + n_q += 1 + else: + w.add_tensor(t.name, arr, raw_dtype=gguf.GGMLQuantizationType.F32) + bytes_out += int(arr.nbytes) + continue + f32 = to_f32(t) if eligible(t.name, t.shape, args.type, skip, args.dit, keep) else None if f32 is not None: packed = gguf.quants.quantize(f32, qtype) # rows are byte-sized; shape inferred w.add_tensor(t.name, packed, raw_dtype=qtype) diff --git a/tests/py/test_converters.py b/tests/py/test_converters.py index 4c7febd4..0a426a61 100644 --- a/tests/py/test_converters.py +++ b/tests/py/test_converters.py @@ -408,6 +408,28 @@ def test_quantize_skip_names(): assert Q.eligible("vis.d.blk.0.qkv.weight", (64, 64), "Q8_0", Q.SKIP) + +def test_quantize_dit_keep_float_pad(): + import importlib + import re + Q = importlib.import_module("quantize_gguf") + default = Q.SKIP + Q.SKIP_VISION + dit = ["aex.dit.0.attn_q.weight", "aex.dit.0.ff0.weight", "aex.dit.proj_out1.weight", + "ah.dit.3.attn_o.weight"] + for n in dit: + assert not Q.eligible(n, (64, 64), "Q8_0", default), n + assert Q.eligible(n, (64, 64), "Q8_0", default, dit=True), n + for n in ("aex.dit.0.adaln.weight", "aex.dit.time_emb.l1.weight", "ah.act_enc.l2.weight"): + assert not Q.eligible(n, (64, 64), "Q8_0", default, dit=True), n + keep = re.compile(r"^vlm\.blk\.0\.") + assert not Q.eligible("vlm.blk.0.attn_q.weight", (64, 64), "Q8_0", default, keep=keep) + assert Q.eligible("vlm.blk.1.attn_q.weight", (64, 64), "Q8_0", default, keep=keep) + t = lambda n, s: types.SimpleNamespace(name=n, shape=s) + r = types.SimpleNamespace(tensors=[t("vit.blk.0.fc1.weight", [1152, 4304]), + t("vit.blk.0.fc2.weight", [4304, 1152]), + t("lm.blk.0.fc1.weight", [2048, 8192])]) + assert Q.ffn_pads(r) == {"vit.blk.0": (4304, 4320)} + def test_find_sidecar(): import tempfile from gguf_common import find_sidecar From 2c64c737ccbf96d3af7e320261c92cc52638c464 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 11 Nov 2026 12:38:51 +0700 Subject: [PATCH 2/2] docs: drop the file-size column from the IQ-10 Q8_0 table Its F16 rows gave the size of the shipped file, which stores F32 or BF16, so they did not compare with the Q8_0 rows; System RAM is the memory measure. The tensor column now says what it counts. --- docs/benchmark/qualcomm-iq10.md | 130 ++++++++++++++++---------------- 1 file changed, 65 insertions(+), 65 deletions(-) diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 0ce2d555..c7a2e36d 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -181,71 +181,71 @@ uses the fixed-input deviation and *served* the served one, with a target of `served max|Δ|` as in [Serving](../backend/hexagon.md#serving), against the CPU-only build at F32. -| Model | Config | Q8_0 tensors | GGUF GiB | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | mean vs F16 | RAM vs F16 | fixed max\|Δ\| | served max\|Δ\| | -|---|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| Octo-Small | F16 | - | 0.51 | 81.5 | 84.1 | 82.3 | 88.3 | 20.7 | 252 | 725 | - | - | 5.0e-4 | 1.4e-4 | -| Octo-Small | default | 121 / 318 | 0.22 | 79.5 | 81.6 | 80.7 | 85.5 | 20.4 | 258 | 405 | -3.0% | -44.1% | 2.5e-3 | 2.0e-3 | -| Octo-Small | vision | 123 / 318 | 0.22 | 80.2 | 83.5 | 81.1 | 90.3 | 21.6 | 258 | 402 | -0.7% | -44.6% | 1.9e-3 | 1.9e-3 | -| ACT | F16 | - | 0.13 | 147.0 | 152.5 | 152.3 | 156.0 | - | 194 | 469 | - | - | 6.1e-4 | 1.0e-3 | -| ACT | default | 35 / 147 | 0.06 | 145.9 | 150.9 | 150.9 | 152.8 | - | 192 | 405 | -1.0% | -13.6% | 2.1e-3 | 2.4e-3 | -| TurboVLA | F16 | - | 0.80 | 186.6 | 190.2 | 189.1 | 195.3 | - | 248 | 1079 | - | - | 3.7e-4 | 8.5e-4 | -| TurboVLA | default | 133 / 657 | 0.53 | 184.8 | 186.5 | 186.4 | 188.2 | - | 248 | 814 | -1.9% | -24.6% | 1.3e-2 | 4.2e-3 | -| TurboVLA | vision | 208 / 657 | 0.30 | 182.6 | 185.7 | 184.1 | 190.7 | - | 245 | 485 | -2.4% | -55.1% | 1.4e-2 | 8.3e-3 | -| TurboVLA | vision, keep-float (fixed, 32) | 176 / 657 | 0.39 | 181.2 | 185.8 | 183.6 | 191.4 | - | 245 | 611 | -2.3% | -43.4% | 2.6e-3 | 6.9e-3 | -| VLA-JEPA | F16 | - | 4.25 | 489.1 | 496.8 | 494.6 | 506.4 | 197.8 | 184 | 4018 | - | - | 2.1e-3 | 2.2e-3 | -| VLA-JEPA | default | 196 / 873 | 3.02 | 425.6 | 432.1 | 430.3 | 437.5 | 199.1 | 184 | 2718 | -13.0% | -32.4% | 1.4e-2 | 1.0e-1 | -| VLA-JEPA | vision | 300 / 873 | 2.67 | 404.8 | 411.6 | 410.8 | 416.8 | 178.4 | 96 | 2350 | -17.1% | -41.5% | 1.5e-2 | 1.6e-1 | -| VLA-JEPA | vision + DiT | 396 / 873 | 2.56 | 387.0 | 393.5 | 392.9 | 400.2 | 179.2 | 97 | 2288 | -20.8% | -43.1% | 1.3e-2 | 1.3e-1 | -| VLA-JEPA | vision + DiT, keep-float (fixed, 233) | 163 / 873 | 3.88 | 456.6 | 461.4 | 459.9 | 467.7 | 185.0 | 184 | 3614 | -7.1% | -10.1% | 2.9e-3 | 8.3e-3 | -| VLA-JEPA | vision + DiT, keep-float (served, 143) | 253 / 873 | 3.38 | 430.1 | 436.6 | 434.2 | 444.7 | 182.3 | 184 | 3117 | -12.1% | -22.4% | 8.4e-3 | 2.1e-3 | -| VLA-JEPA | vision + DiT, keep-float (union, 244) | 152 / 873 | 3.93 | 452.6 | 461.6 | 460.7 | 469.3 | 185.3 | 184 | 3679 | -7.1% | -8.4% | 3.7e-3 | 5.1e-3 | -| GR00T N1.5 | F16 | - | 6.46 | 527.3 | 529.3 | 528.9 | 531.3 | 138.7 | 161 | 3659 | - | - | 1.9e-3 | 1.2e-2 | -| GR00T N1.5 | default | 85 / 887 | 5.93 | 499.5 | 500.2 | 500.0 | 501.3 | 138.3 | 161 | 3148 | -5.5% | -14.0% | 5.1e-3 | 8.0e-2 | -| GR00T N1.5 | vision | 247 / 887 | 5.57 | 479.1 | 481.1 | 479.7 | 483.8 | 118.8 | 161 | 2754 | -9.1% | -24.7% | 7.2e-3 | 4.6e-2 | -| GR00T N1.5 | vision + DiT | 345 / 887 | 4.28 | 398.7 | 399.2 | 399.1 | 399.5 | 117.9 | 161 | 2339 | -24.6% | -36.1% | 8.4e-3 | 3.6e-2 | -| GR00T N1.5 | vision + DiT, keep-float (fixed, 191) | 154 / 887 | 5.57 | 448.3 | 449.0 | 448.7 | 450.0 | 131.8 | 162 | 3061 | -15.2% | -16.3% | 2.7e-3 | 2.8e-2 | -| GR00T N1.5 | vision + DiT, keep-float (served, 223) | 122 / 887 | 5.86 | 471.7 | 472.9 | 472.6 | 474.0 | 133.1 | 161 | 3172 | -10.7% | -13.3% | 6.8e-3 | 9.1e-3 | -| GR00T N1.5 | vision + DiT, keep-float (union, 272) | 73 / 887 | 6.27 | 489.0 | 491.2 | 489.7 | 494.8 | 135.1 | 161 | 3413 | -7.2% | -6.7% | 3.3e-3 | 1.1e-2 | -| SmolVLA | F16 | - | 1.05 | 583.4 | 584.0 | 583.8 | 584.1 | 229.9 | 146 | 1226 | - | - | 7.4e-4 | 1.9e-3 | -| SmolVLA | default | 225 / 791 | 0.76 | 568.2 | 569.0 | 568.6 | 569.8 | 228.8 | 104 | 919 | -2.6% | -25.0% | 1.2e-2 | 1.3e-2 | -| SmolVLA | vision | 297 / 791 | 0.69 | 560.4 | 561.0 | 560.9 | 561.6 | 221.3 | 104 | 848 | -3.9% | -30.8% | 3.4e-2 | 1.6e-2 | -| SmolVLA | vision, keep-float (fixed, 58) | 239 / 791 | 0.73 | 563.5 | 564.2 | 564.1 | 564.5 | 222.9 | 104 | 896 | -3.4% | -26.9% | 2.0e-3 | 1.6e-2 | -| VLA-Adapter | F16 | - | 2.59 | 664.7 | 669.2 | 666.6 | 677.5 | 257.6 | 872 | 3569 | - | - | 3.9e-4 | 3.9e-4 | -| VLA-Adapter | default | 171 / 1455 | 2.25 | 649.4 | 653.5 | 651.8 | 658.0 | 256.6 | 872 | 3200 | -2.3% | -10.3% | 4.3e-3 | 6.3e-3 | -| VLA-Adapter | vision | 367 / 1455 | 1.65 | 617.9 | 623.4 | 622.1 | 629.0 | 226.5 | 852 | 2555 | -6.8% | -28.4% | 3.1e-3 | 6.0e-3 | -| VLA-Adapter | vision, keep-float (fixed, 8) | 359 / 1455 | 1.66 | 618.2 | 622.7 | 622.6 | 626.9 | 226.5 | 852 | 2516 | -6.9% | -29.5% | 2.8e-3 | 6.3e-3 | -| VLA-Adapter | vision, keep-float (served, 40) | 327 / 1455 | 1.78 | 623.6 | 627.7 | 625.5 | 632.3 | 227.1 | 849 | 2494 | -6.2% | -30.1% | 2.9e-3 | 2.9e-3 | -| GR00T N1.6 | F16 | - | 8.53 | 681.0 | 682.6 | 682.3 | 684.1 | 145.0 | 146 | 4929 | - | - | 1.3e-3 | 1.0e-2 | -| GR00T N1.6 | default | 114 / 1094 | 7.44 | 643.4 | 644.5 | 643.9 | 646.6 | 144.2 | 134 | 3959 | -5.6% | -19.7% | 7.2e-3 | 3.4e-2 | -| GR00T N1.6 | vision, unpadded | 249 / 1094 | 7.20 | 815.6 | 817.3 | 817.3 | 818.5 | 316.7 | 134 | 3738 | +19.7% | -24.2% | 7.1e-3 | 2.5e-2 | -| GR00T N1.6 | vision | 276 / 1094 | 7.08 | 621.6 | 622.5 | 622.1 | 623.2 | 124.2 | 134 | 3588 | -8.8% | -27.2% | 8.0e-3 | 2.9e-2 | -| GR00T N1.6 | vision + DiT | 470 / 1094 | 4.52 | 461.4 | 462.2 | 461.9 | 462.9 | 124.2 | 116 | 2777 | -32.3% | -43.7% | 8.1e-3 | 2.9e-2 | -| GR00T N1.6 | vision + DiT, keep-float (fixed, 57) | 413 / 1094 | 5.26 | 481.7 | 482.3 | 482.2 | 482.7 | 125.5 | 146 | 3076 | -29.3% | -37.6% | 2.8e-3 | 3.6e-2 | -| GR00T N1.6 | vision + DiT, keep-float (served, 3) | 467 / 1094 | 4.55 | 463.2 | 464.0 | 463.6 | 465.0 | 124.3 | 146 | 2800 | -32.0% | -43.2% | 7.6e-3 | 9.2e-3 | -| GR00T N1.6 | vision + DiT, keep-float (union, 60) | 410 / 1094 | 5.29 | 480.1 | 480.7 | 480.5 | 481.2 | 125.2 | 147 | 3114 | -29.6% | -36.8% | 3.0e-3 | 8.6e-3 | -| GR00T N1.7 | F16 | - | 5.86 | 740.1 | 744.9 | 745.0 | 749.0 | 196.9 | 190 | 5108 | - | - | 2.0e-3 | 2.0e-3 | -| GR00T N1.7 | default | 112 / 1030 | 5.16 | 699.0 | 703.5 | 701.5 | 709.3 | 196.1 | 190 | 4356 | -5.6% | -14.7% | 1.1e-2 | 2.0e-2 | -| GR00T N1.7 | vision | 216 / 1030 | 4.81 | 683.3 | 688.4 | 686.0 | 694.1 | 179.5 | 190 | 3996 | -7.6% | -21.8% | 2.0e-2 | 2.2e-2 | -| GR00T N1.7 | vision + DiT | 410 / 1030 | 3.99 | 528.8 | 536.1 | 534.6 | 542.1 | 181.6 | 191 | 3151 | -28.0% | -38.3% | 2.8e-2 | 2.1e-2 | -| GR00T N1.7 | vision + DiT, keep-float (fixed, 185) | 225 / 1030 | 4.90 | 599.2 | 606.0 | 605.4 | 611.7 | 186.9 | 191 | 4051 | -18.6% | -20.7% | 2.6e-3 | 5.3e-3 | -| GR00T N1.7 | vision + DiT, keep-float (served, 157) | 253 / 1030 | 4.68 | 573.7 | 579.8 | 578.0 | 590.8 | 187.3 | 190 | 3870 | -22.2% | -24.2% | 4.2e-2 | 2.8e-3 | -| GR00T N1.7 | vision + DiT, keep-float (union, 233) | 177 / 1030 | 5.06 | 609.4 | 614.4 | 614.1 | 619.1 | 188.2 | 190 | 4249 | -17.5% | -16.8% | 3.9e-3 | 6.5e-3 | -| π0.5 | F16 | - | 11.72 | 1106.0 | 1107.8 | 1107.0 | 1111.5 | 277.0 | 284 | 5980 | - | - | 4.8e-4 | 3.7e-4 | -| π0.5 | default | 127 / 819 | 6.30 | 969.3 | 971.0 | 969.8 | 974.7 | 277.3 | 126 | 3997 | -12.3% | -33.2% | 4.9e-4 | 1.9e-3 | -| π0.5 | vision, unpadded | 262 / 819 | 6.06 | 1318.2 | 1320.2 | 1318.9 | 1324.5 | 621.0 | 126 | 3760 | +19.2% | -37.1% | 8.5e-4 | 2.7e-3 | -| π0.5 | vision | 289 / 819 | 5.94 | 939.6 | 941.3 | 940.8 | 943.1 | 236.7 | 103 | 3676 | -15.0% | -38.5% | 9.2e-4 | 2.8e-3 | -| π0 | F16 | - | 6.04 | 1118.9 | 1123.3 | 1123.7 | 1125.3 | 276.9 | 284 | 5662 | - | - | 7.7e-4 | 5.8e-4 | -| π0 | default | 127 / 780 | 4.31 | 976.4 | 978.8 | 977.8 | 981.2 | 277.0 | 126 | 3723 | -12.9% | -34.2% | 1.8e-2 | 1.7e-3 | -| π0 | vision | 289 / 780 | 3.95 | 935.7 | 937.6 | 937.3 | 938.8 | 235.9 | 103 | 3337 | -16.5% | -41.1% | 1.6e-2 | 1.7e-3 | -| π0 | default, keep-float (fixed, 8) | 119 / 780 | 4.32 | 977.0 | 978.8 | 978.3 | 980.2 | 276.9 | 126 | 3711 | -12.9% | -34.5% | 1.4e-3 | 5.2e-4 | -| OpenVLA-OFT | F16 | - | 14.03 | 1122.2 | 1125.2 | 1125.0 | 1127.6 | 262.7 | 844 | 15266 | - | - | 3.1e-4 | 7.5e-4 | -| OpenVLA-OFT | default | 227 / 958 | 8.32 | 843.5 | 845.7 | 844.8 | 848.9 | 259.5 | 844 | 9403 | -24.8% | -38.4% | 5.9e-4 | 3.5e-3 | -| OpenVLA-OFT | vision | 423 / 958 | 7.72 | 810.1 | 812.5 | 810.6 | 816.4 | 228.7 | 833 | 8821 | -27.8% | -42.2% | 7.9e-4 | 3.2e-3 | -| OpenVLA-OFT | vision, keep-float (served, 12) | 411 / 958 | 7.86 | 818.7 | 821.4 | 820.4 | 824.8 | 230.0 | 839 | 8959 | -27.0% | -41.3% | 7.9e-4 | 2.5e-3 | -| Evo-1 | F16 | - | 1.45 | 1640.1 | 1659.0 | 1653.3 | 1675.4 | 818.6 | 318 | 1582 | - | - | 6.2e-4 | 1.1e-3 | -| Evo-1 | default | 100 / 636 | 1.26 | 1643.5 | 1655.4 | 1653.6 | 1660.8 | 820.1 | 312 | 1459 | -0.2% | -7.8% | 5.4e-3 | 1.2e-2 | -| Evo-1 | vision | 196 / 636 | 1.00 | 1620.6 | 1638.8 | 1639.0 | 1645.7 | 808.1 | 312 | 1121 | -1.2% | -29.1% | 6.3e-3 | 1.4e-2 | -| Evo-1 | vision, keep-float (fixed, 25) | 171 / 636 | 1.05 | 1624.7 | 1634.5 | 1632.9 | 1647.6 | 803.5 | 319 | 1171 | -1.5% | -26.0% | 2.4e-3 | 1.1e-2 | +| Model | Config | Q8_0 / all tensors | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | mean vs F16 | RAM vs F16 | fixed max\|Δ\| | served max\|Δ\| | +|---|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:| +| Octo-Small | F16 | - | 81.5 | 84.1 | 82.3 | 88.3 | 20.7 | 252 | 725 | - | - | 5.0e-4 | 1.4e-4 | +| Octo-Small | default | 121 / 318 | 79.5 | 81.6 | 80.7 | 85.5 | 20.4 | 258 | 405 | -3.0% | -44.1% | 2.5e-3 | 2.0e-3 | +| Octo-Small | vision | 123 / 318 | 80.2 | 83.5 | 81.1 | 90.3 | 21.6 | 258 | 402 | -0.7% | -44.6% | 1.9e-3 | 1.9e-3 | +| ACT | F16 | - | 147.0 | 152.5 | 152.3 | 156.0 | - | 194 | 469 | - | - | 6.1e-4 | 1.0e-3 | +| ACT | default | 35 / 147 | 145.9 | 150.9 | 150.9 | 152.8 | - | 192 | 405 | -1.0% | -13.6% | 2.1e-3 | 2.4e-3 | +| TurboVLA | F16 | - | 186.6 | 190.2 | 189.1 | 195.3 | - | 248 | 1079 | - | - | 3.7e-4 | 8.5e-4 | +| TurboVLA | default | 133 / 657 | 184.8 | 186.5 | 186.4 | 188.2 | - | 248 | 814 | -1.9% | -24.6% | 1.3e-2 | 4.2e-3 | +| TurboVLA | vision | 208 / 657 | 182.6 | 185.7 | 184.1 | 190.7 | - | 245 | 485 | -2.4% | -55.1% | 1.4e-2 | 8.3e-3 | +| TurboVLA | vision, keep-float (fixed, 32) | 176 / 657 | 181.2 | 185.8 | 183.6 | 191.4 | - | 245 | 611 | -2.3% | -43.4% | 2.6e-3 | 6.9e-3 | +| VLA-JEPA | F16 | - | 489.1 | 496.8 | 494.6 | 506.4 | 197.8 | 184 | 4018 | - | - | 2.1e-3 | 2.2e-3 | +| VLA-JEPA | default | 196 / 873 | 425.6 | 432.1 | 430.3 | 437.5 | 199.1 | 184 | 2718 | -13.0% | -32.4% | 1.4e-2 | 1.0e-1 | +| VLA-JEPA | vision | 300 / 873 | 404.8 | 411.6 | 410.8 | 416.8 | 178.4 | 96 | 2350 | -17.1% | -41.5% | 1.5e-2 | 1.6e-1 | +| VLA-JEPA | vision + DiT | 396 / 873 | 387.0 | 393.5 | 392.9 | 400.2 | 179.2 | 97 | 2288 | -20.8% | -43.1% | 1.3e-2 | 1.3e-1 | +| VLA-JEPA | vision + DiT, keep-float (fixed, 233) | 163 / 873 | 456.6 | 461.4 | 459.9 | 467.7 | 185.0 | 184 | 3614 | -7.1% | -10.1% | 2.9e-3 | 8.3e-3 | +| VLA-JEPA | vision + DiT, keep-float (served, 143) | 253 / 873 | 430.1 | 436.6 | 434.2 | 444.7 | 182.3 | 184 | 3117 | -12.1% | -22.4% | 8.4e-3 | 2.1e-3 | +| VLA-JEPA | vision + DiT, keep-float (union, 244) | 152 / 873 | 452.6 | 461.6 | 460.7 | 469.3 | 185.3 | 184 | 3679 | -7.1% | -8.4% | 3.7e-3 | 5.1e-3 | +| GR00T N1.5 | F16 | - | 527.3 | 529.3 | 528.9 | 531.3 | 138.7 | 161 | 3659 | - | - | 1.9e-3 | 1.2e-2 | +| GR00T N1.5 | default | 85 / 887 | 499.5 | 500.2 | 500.0 | 501.3 | 138.3 | 161 | 3148 | -5.5% | -14.0% | 5.1e-3 | 8.0e-2 | +| GR00T N1.5 | vision | 247 / 887 | 479.1 | 481.1 | 479.7 | 483.8 | 118.8 | 161 | 2754 | -9.1% | -24.7% | 7.2e-3 | 4.6e-2 | +| GR00T N1.5 | vision + DiT | 345 / 887 | 398.7 | 399.2 | 399.1 | 399.5 | 117.9 | 161 | 2339 | -24.6% | -36.1% | 8.4e-3 | 3.6e-2 | +| GR00T N1.5 | vision + DiT, keep-float (fixed, 191) | 154 / 887 | 448.3 | 449.0 | 448.7 | 450.0 | 131.8 | 162 | 3061 | -15.2% | -16.3% | 2.7e-3 | 2.8e-2 | +| GR00T N1.5 | vision + DiT, keep-float (served, 223) | 122 / 887 | 471.7 | 472.9 | 472.6 | 474.0 | 133.1 | 161 | 3172 | -10.7% | -13.3% | 6.8e-3 | 9.1e-3 | +| GR00T N1.5 | vision + DiT, keep-float (union, 272) | 73 / 887 | 489.0 | 491.2 | 489.7 | 494.8 | 135.1 | 161 | 3413 | -7.2% | -6.7% | 3.3e-3 | 1.1e-2 | +| SmolVLA | F16 | - | 583.4 | 584.0 | 583.8 | 584.1 | 229.9 | 146 | 1226 | - | - | 7.4e-4 | 1.9e-3 | +| SmolVLA | default | 225 / 791 | 568.2 | 569.0 | 568.6 | 569.8 | 228.8 | 104 | 919 | -2.6% | -25.0% | 1.2e-2 | 1.3e-2 | +| SmolVLA | vision | 297 / 791 | 560.4 | 561.0 | 560.9 | 561.6 | 221.3 | 104 | 848 | -3.9% | -30.8% | 3.4e-2 | 1.6e-2 | +| SmolVLA | vision, keep-float (fixed, 58) | 239 / 791 | 563.5 | 564.2 | 564.1 | 564.5 | 222.9 | 104 | 896 | -3.4% | -26.9% | 2.0e-3 | 1.6e-2 | +| VLA-Adapter | F16 | - | 664.7 | 669.2 | 666.6 | 677.5 | 257.6 | 872 | 3569 | - | - | 3.9e-4 | 3.9e-4 | +| VLA-Adapter | default | 171 / 1455 | 649.4 | 653.5 | 651.8 | 658.0 | 256.6 | 872 | 3200 | -2.3% | -10.3% | 4.3e-3 | 6.3e-3 | +| VLA-Adapter | vision | 367 / 1455 | 617.9 | 623.4 | 622.1 | 629.0 | 226.5 | 852 | 2555 | -6.8% | -28.4% | 3.1e-3 | 6.0e-3 | +| VLA-Adapter | vision, keep-float (fixed, 8) | 359 / 1455 | 618.2 | 622.7 | 622.6 | 626.9 | 226.5 | 852 | 2516 | -6.9% | -29.5% | 2.8e-3 | 6.3e-3 | +| VLA-Adapter | vision, keep-float (served, 40) | 327 / 1455 | 623.6 | 627.7 | 625.5 | 632.3 | 227.1 | 849 | 2494 | -6.2% | -30.1% | 2.9e-3 | 2.9e-3 | +| GR00T N1.6 | F16 | - | 681.0 | 682.6 | 682.3 | 684.1 | 145.0 | 146 | 4929 | - | - | 1.3e-3 | 1.0e-2 | +| GR00T N1.6 | default | 114 / 1094 | 643.4 | 644.5 | 643.9 | 646.6 | 144.2 | 134 | 3959 | -5.6% | -19.7% | 7.2e-3 | 3.4e-2 | +| GR00T N1.6 | vision, unpadded | 249 / 1094 | 815.6 | 817.3 | 817.3 | 818.5 | 316.7 | 134 | 3738 | +19.7% | -24.2% | 7.1e-3 | 2.5e-2 | +| GR00T N1.6 | vision | 276 / 1094 | 621.6 | 622.5 | 622.1 | 623.2 | 124.2 | 134 | 3588 | -8.8% | -27.2% | 8.0e-3 | 2.9e-2 | +| GR00T N1.6 | vision + DiT | 470 / 1094 | 461.4 | 462.2 | 461.9 | 462.9 | 124.2 | 116 | 2777 | -32.3% | -43.7% | 8.1e-3 | 2.9e-2 | +| GR00T N1.6 | vision + DiT, keep-float (fixed, 57) | 413 / 1094 | 481.7 | 482.3 | 482.2 | 482.7 | 125.5 | 146 | 3076 | -29.3% | -37.6% | 2.8e-3 | 3.6e-2 | +| GR00T N1.6 | vision + DiT, keep-float (served, 3) | 467 / 1094 | 463.2 | 464.0 | 463.6 | 465.0 | 124.3 | 146 | 2800 | -32.0% | -43.2% | 7.6e-3 | 9.2e-3 | +| GR00T N1.6 | vision + DiT, keep-float (union, 60) | 410 / 1094 | 480.1 | 480.7 | 480.5 | 481.2 | 125.2 | 147 | 3114 | -29.6% | -36.8% | 3.0e-3 | 8.6e-3 | +| GR00T N1.7 | F16 | - | 740.1 | 744.9 | 745.0 | 749.0 | 196.9 | 190 | 5108 | - | - | 2.0e-3 | 2.0e-3 | +| GR00T N1.7 | default | 112 / 1030 | 699.0 | 703.5 | 701.5 | 709.3 | 196.1 | 190 | 4356 | -5.6% | -14.7% | 1.1e-2 | 2.0e-2 | +| GR00T N1.7 | vision | 216 / 1030 | 683.3 | 688.4 | 686.0 | 694.1 | 179.5 | 190 | 3996 | -7.6% | -21.8% | 2.0e-2 | 2.2e-2 | +| GR00T N1.7 | vision + DiT | 410 / 1030 | 528.8 | 536.1 | 534.6 | 542.1 | 181.6 | 191 | 3151 | -28.0% | -38.3% | 2.8e-2 | 2.1e-2 | +| GR00T N1.7 | vision + DiT, keep-float (fixed, 185) | 225 / 1030 | 599.2 | 606.0 | 605.4 | 611.7 | 186.9 | 191 | 4051 | -18.6% | -20.7% | 2.6e-3 | 5.3e-3 | +| GR00T N1.7 | vision + DiT, keep-float (served, 157) | 253 / 1030 | 573.7 | 579.8 | 578.0 | 590.8 | 187.3 | 190 | 3870 | -22.2% | -24.2% | 4.2e-2 | 2.8e-3 | +| GR00T N1.7 | vision + DiT, keep-float (union, 233) | 177 / 1030 | 609.4 | 614.4 | 614.1 | 619.1 | 188.2 | 190 | 4249 | -17.5% | -16.8% | 3.9e-3 | 6.5e-3 | +| π0.5 | F16 | - | 1106.0 | 1107.8 | 1107.0 | 1111.5 | 277.0 | 284 | 5980 | - | - | 4.8e-4 | 3.7e-4 | +| π0.5 | default | 127 / 819 | 969.3 | 971.0 | 969.8 | 974.7 | 277.3 | 126 | 3997 | -12.3% | -33.2% | 4.9e-4 | 1.9e-3 | +| π0.5 | vision, unpadded | 262 / 819 | 1318.2 | 1320.2 | 1318.9 | 1324.5 | 621.0 | 126 | 3760 | +19.2% | -37.1% | 8.5e-4 | 2.7e-3 | +| π0.5 | vision | 289 / 819 | 939.6 | 941.3 | 940.8 | 943.1 | 236.7 | 103 | 3676 | -15.0% | -38.5% | 9.2e-4 | 2.8e-3 | +| π0 | F16 | - | 1118.9 | 1123.3 | 1123.7 | 1125.3 | 276.9 | 284 | 5662 | - | - | 7.7e-4 | 5.8e-4 | +| π0 | default | 127 / 780 | 976.4 | 978.8 | 977.8 | 981.2 | 277.0 | 126 | 3723 | -12.9% | -34.2% | 1.8e-2 | 1.7e-3 | +| π0 | vision | 289 / 780 | 935.7 | 937.6 | 937.3 | 938.8 | 235.9 | 103 | 3337 | -16.5% | -41.1% | 1.6e-2 | 1.7e-3 | +| π0 | default, keep-float (fixed, 8) | 119 / 780 | 977.0 | 978.8 | 978.3 | 980.2 | 276.9 | 126 | 3711 | -12.9% | -34.5% | 1.4e-3 | 5.2e-4 | +| OpenVLA-OFT | F16 | - | 1122.2 | 1125.2 | 1125.0 | 1127.6 | 262.7 | 844 | 15266 | - | - | 3.1e-4 | 7.5e-4 | +| OpenVLA-OFT | default | 227 / 958 | 843.5 | 845.7 | 844.8 | 848.9 | 259.5 | 844 | 9403 | -24.8% | -38.4% | 5.9e-4 | 3.5e-3 | +| OpenVLA-OFT | vision | 423 / 958 | 810.1 | 812.5 | 810.6 | 816.4 | 228.7 | 833 | 8821 | -27.8% | -42.2% | 7.9e-4 | 3.2e-3 | +| OpenVLA-OFT | vision, keep-float (served, 12) | 411 / 958 | 818.7 | 821.4 | 820.4 | 824.8 | 230.0 | 839 | 8959 | -27.0% | -41.3% | 7.9e-4 | 2.5e-3 | +| Evo-1 | F16 | - | 1640.1 | 1659.0 | 1653.3 | 1675.4 | 818.6 | 318 | 1582 | - | - | 6.2e-4 | 1.1e-3 | +| Evo-1 | default | 100 / 636 | 1643.5 | 1655.4 | 1653.6 | 1660.8 | 820.1 | 312 | 1459 | -0.2% | -7.8% | 5.4e-3 | 1.2e-2 | +| Evo-1 | vision | 196 / 636 | 1620.6 | 1638.8 | 1639.0 | 1645.7 | 808.1 | 312 | 1121 | -1.2% | -29.1% | 6.3e-3 | 1.4e-2 | +| Evo-1 | vision, keep-float (fixed, 25) | 171 / 636 | 1624.7 | 1634.5 | 1632.9 | 1647.6 | 803.5 | 319 | 1171 | -1.5% | -26.0% | 2.4e-3 | 1.1e-2 | A per-op profile of the default, vision and vision + DiT files puts every Q8_0 matmul on the HMX. The loader zero-pads a 4304-wide FFN to 4320 at load only