Skip to content

hexagon: pack Q8_0 files for the NPU and measure them on the IQ-10 #83

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
thanhdq1o2 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

hexagon: pack Q8_0 files for the NPU and measure them on the IQ-10 #83

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b8aa7ad
hexagon: pack Q8_0 files for the NPU and measure them on the IQ-10
thanhdq1o2 Nov 11, 2026
2c64c73
docs: drop the file-size column from the IQ-10 Q8_0 table
thanhdq1o2 Nov 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,12 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke
against a CPU-only build of the same commit. GR00T N1.5 and N1.6 are above
the bar on a served LIBERO observation, as the CPU is at F16. See
`docs/backend/hexagon.md` for the build, the skel install and the numbers.
- **Q8_0 files for the Hexagon NPU.** `scripts/quantize_gguf.py` gains
`--pad-m32` (zero-pads FFN widths that are not a multiple of 32, which the HMX
needs for a packed weight), `--dit` (also packs the DiT's weight matrices
except adaLN and the time embedding) and `--keep-float REGEX` (leaves the
matching weights float). `docs/benchmark/qualcomm-iq10.md` reports latency,
memory and fidelity on the IQ-10 for each Q8_0 configuration of the 13 models.
- **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP
backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200`
(Radeon RX 9060 XT) with ROCm 7.14 and llama.cpp `b11223`, every arch except
Expand Down
20 changes: 20 additions & 0 deletions docs/backend/hexagon.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,26 @@ slightly further (1.0e-2 against 9.0e-3). They are sensitive to 16-bit precision
rather than miscomputed. Running them with F32 weights on the NPU has not been
tried yet.

### Q8_0 weights

`scripts/quantize_gguf.py --type Q8_0` packs a checkpoint's weights to Q8_0.
Three flags shape a file for the NPU:

- `--pad-m32` zero-pads in the file the FFN widths that are not a multiple of
32. The HMX takes no other width for a packed weight, and the loader pads an
FFN at load only when its weights are float.
- `--dit` also packs the DiT's weight matrices, except adaLN and the time
embedding.
- `--keep-float REGEX` leaves the weights whose name matches REGEX float.

```bash
python scripts/quantize_gguf.py --in gr00tn1d6-libero.gguf --out gr00tn1d6-q8_0.gguf \
--type Q8_0 --vision --pad-m32 --dit
```

Latency, memory and fidelity per configuration:
[benchmark/qualcomm-iq10.md](../benchmark/qualcomm-iq10.md#q8_0-weights).

## IQ-10, imagined

Nothing about a second-generation part is knowable from this repository, so what
Expand Down
122 changes: 122 additions & 0 deletions docs/benchmark/qualcomm-iq10.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,128 @@ inputs), N1.7 6.2e-3 on fixed inputs (2.7e-3 served).

</details>

## Q8_0 weights

These rows run GGUFs that `scripts/quantize_gguf.py --type Q8_0` made from the
checkpoints above (ggml's block-32 Q8_0 weights). The setup is the one above,
except:

| | |
|---|---|
| Commit | `71cdb09` |
| Date | 2026-11-11 10:10 to 11:49 (UTC+07) for latency, memory and served fidelity; fixed-input fidelity 2026-11-10 |
| Runtime flags | defaults |

Each configuration packs the weights of the one before it, plus:

| Config | `quantize_gguf.py` flags | Packs |
|---|---|---|
| F16 | - | none: the shipped file, loaded as F16 (`--weight-dtype f16`, the default) |
| default | `--type Q8_0` | the 2-D weights outside the vision tower and the action head. Embeddings, the output head, norms, patch embeddings and position tables stay float in every configuration |
| vision, unpadded | `--vision` | the vision tower |
| vision | `--vision --pad-m32` | the vision tower, with FFN widths that are not a multiple of 32 zero-padded (SigLIP's 4304 to 4320 in π0, π0.5, GR00T N1.5 and N1.6, VLA-Adapter and OpenVLA-OFT) |
| vision + DiT | `--vision --pad-m32 --dit` | the DiT's weight matrices, except adaLN and the time embedding |
| ..., keep-float (*kind*, *n*) | the flags of the named configuration plus `--keep-float REGEX` | that configuration with *n* weights left float |

The `--keep-float` lists come from a greedy search: each group of blocks (a
quarter of a stack, attention or FFN) is packed alone and measured, the groups
that move the action chunk most are left float until the deviation is under the
target, then blocks are packed back one at a time while it stays under. *fixed*
uses the fixed-input deviation and *served* the served one, with a target of
2.9e-3; GR00T N1.5 and N1.6 take 1.1 times their F16 served deviation instead
(1.3e-2, 1.1e-2). *union* leaves both lists float.

`fixed max|Δ|` is measured as in [Fidelity](../backend/hexagon.md#fidelity) and
`served max|Δ|` as in [Serving](../backend/hexagon.md#serving), against the
CPU-only build at F32.

| Model | Config | Q8_0 / all tensors | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | mean vs F16 | RAM vs F16 | fixed max\|Δ\| | served max\|Δ\| |
|---|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|
| Octo-Small | F16 | - | 81.5 | 84.1 | 82.3 | 88.3 | 20.7 | 252 | 725 | - | - | 5.0e-4 | 1.4e-4 |
| Octo-Small | default | 121 / 318 | 79.5 | 81.6 | 80.7 | 85.5 | 20.4 | 258 | 405 | -3.0% | -44.1% | 2.5e-3 | 2.0e-3 |
| Octo-Small | vision | 123 / 318 | 80.2 | 83.5 | 81.1 | 90.3 | 21.6 | 258 | 402 | -0.7% | -44.6% | 1.9e-3 | 1.9e-3 |
| ACT | F16 | - | 147.0 | 152.5 | 152.3 | 156.0 | - | 194 | 469 | - | - | 6.1e-4 | 1.0e-3 |
| ACT | default | 35 / 147 | 145.9 | 150.9 | 150.9 | 152.8 | - | 192 | 405 | -1.0% | -13.6% | 2.1e-3 | 2.4e-3 |
| TurboVLA | F16 | - | 186.6 | 190.2 | 189.1 | 195.3 | - | 248 | 1079 | - | - | 3.7e-4 | 8.5e-4 |
| TurboVLA | default | 133 / 657 | 184.8 | 186.5 | 186.4 | 188.2 | - | 248 | 814 | -1.9% | -24.6% | 1.3e-2 | 4.2e-3 |
| TurboVLA | vision | 208 / 657 | 182.6 | 185.7 | 184.1 | 190.7 | - | 245 | 485 | -2.4% | -55.1% | 1.4e-2 | 8.3e-3 |
| TurboVLA | vision, keep-float (fixed, 32) | 176 / 657 | 181.2 | 185.8 | 183.6 | 191.4 | - | 245 | 611 | -2.3% | -43.4% | 2.6e-3 | 6.9e-3 |
| VLA-JEPA | F16 | - | 489.1 | 496.8 | 494.6 | 506.4 | 197.8 | 184 | 4018 | - | - | 2.1e-3 | 2.2e-3 |
| VLA-JEPA | default | 196 / 873 | 425.6 | 432.1 | 430.3 | 437.5 | 199.1 | 184 | 2718 | -13.0% | -32.4% | 1.4e-2 | 1.0e-1 |
| VLA-JEPA | vision | 300 / 873 | 404.8 | 411.6 | 410.8 | 416.8 | 178.4 | 96 | 2350 | -17.1% | -41.5% | 1.5e-2 | 1.6e-1 |
| VLA-JEPA | vision + DiT | 396 / 873 | 387.0 | 393.5 | 392.9 | 400.2 | 179.2 | 97 | 2288 | -20.8% | -43.1% | 1.3e-2 | 1.3e-1 |
| VLA-JEPA | vision + DiT, keep-float (fixed, 233) | 163 / 873 | 456.6 | 461.4 | 459.9 | 467.7 | 185.0 | 184 | 3614 | -7.1% | -10.1% | 2.9e-3 | 8.3e-3 |
| VLA-JEPA | vision + DiT, keep-float (served, 143) | 253 / 873 | 430.1 | 436.6 | 434.2 | 444.7 | 182.3 | 184 | 3117 | -12.1% | -22.4% | 8.4e-3 | 2.1e-3 |
| VLA-JEPA | vision + DiT, keep-float (union, 244) | 152 / 873 | 452.6 | 461.6 | 460.7 | 469.3 | 185.3 | 184 | 3679 | -7.1% | -8.4% | 3.7e-3 | 5.1e-3 |
| GR00T N1.5 | F16 | - | 527.3 | 529.3 | 528.9 | 531.3 | 138.7 | 161 | 3659 | - | - | 1.9e-3 | 1.2e-2 |
| GR00T N1.5 | default | 85 / 887 | 499.5 | 500.2 | 500.0 | 501.3 | 138.3 | 161 | 3148 | -5.5% | -14.0% | 5.1e-3 | 8.0e-2 |
| GR00T N1.5 | vision | 247 / 887 | 479.1 | 481.1 | 479.7 | 483.8 | 118.8 | 161 | 2754 | -9.1% | -24.7% | 7.2e-3 | 4.6e-2 |
| GR00T N1.5 | vision + DiT | 345 / 887 | 398.7 | 399.2 | 399.1 | 399.5 | 117.9 | 161 | 2339 | -24.6% | -36.1% | 8.4e-3 | 3.6e-2 |
| GR00T N1.5 | vision + DiT, keep-float (fixed, 191) | 154 / 887 | 448.3 | 449.0 | 448.7 | 450.0 | 131.8 | 162 | 3061 | -15.2% | -16.3% | 2.7e-3 | 2.8e-2 |
| GR00T N1.5 | vision + DiT, keep-float (served, 223) | 122 / 887 | 471.7 | 472.9 | 472.6 | 474.0 | 133.1 | 161 | 3172 | -10.7% | -13.3% | 6.8e-3 | 9.1e-3 |
| GR00T N1.5 | vision + DiT, keep-float (union, 272) | 73 / 887 | 489.0 | 491.2 | 489.7 | 494.8 | 135.1 | 161 | 3413 | -7.2% | -6.7% | 3.3e-3 | 1.1e-2 |
| SmolVLA | F16 | - | 583.4 | 584.0 | 583.8 | 584.1 | 229.9 | 146 | 1226 | - | - | 7.4e-4 | 1.9e-3 |
| SmolVLA | default | 225 / 791 | 568.2 | 569.0 | 568.6 | 569.8 | 228.8 | 104 | 919 | -2.6% | -25.0% | 1.2e-2 | 1.3e-2 |
| SmolVLA | vision | 297 / 791 | 560.4 | 561.0 | 560.9 | 561.6 | 221.3 | 104 | 848 | -3.9% | -30.8% | 3.4e-2 | 1.6e-2 |
| SmolVLA | vision, keep-float (fixed, 58) | 239 / 791 | 563.5 | 564.2 | 564.1 | 564.5 | 222.9 | 104 | 896 | -3.4% | -26.9% | 2.0e-3 | 1.6e-2 |
| VLA-Adapter | F16 | - | 664.7 | 669.2 | 666.6 | 677.5 | 257.6 | 872 | 3569 | - | - | 3.9e-4 | 3.9e-4 |
| VLA-Adapter | default | 171 / 1455 | 649.4 | 653.5 | 651.8 | 658.0 | 256.6 | 872 | 3200 | -2.3% | -10.3% | 4.3e-3 | 6.3e-3 |
| VLA-Adapter | vision | 367 / 1455 | 617.9 | 623.4 | 622.1 | 629.0 | 226.5 | 852 | 2555 | -6.8% | -28.4% | 3.1e-3 | 6.0e-3 |
| VLA-Adapter | vision, keep-float (fixed, 8) | 359 / 1455 | 618.2 | 622.7 | 622.6 | 626.9 | 226.5 | 852 | 2516 | -6.9% | -29.5% | 2.8e-3 | 6.3e-3 |
| VLA-Adapter | vision, keep-float (served, 40) | 327 / 1455 | 623.6 | 627.7 | 625.5 | 632.3 | 227.1 | 849 | 2494 | -6.2% | -30.1% | 2.9e-3 | 2.9e-3 |
| GR00T N1.6 | F16 | - | 681.0 | 682.6 | 682.3 | 684.1 | 145.0 | 146 | 4929 | - | - | 1.3e-3 | 1.0e-2 |
| GR00T N1.6 | default | 114 / 1094 | 643.4 | 644.5 | 643.9 | 646.6 | 144.2 | 134 | 3959 | -5.6% | -19.7% | 7.2e-3 | 3.4e-2 |
| GR00T N1.6 | vision, unpadded | 249 / 1094 | 815.6 | 817.3 | 817.3 | 818.5 | 316.7 | 134 | 3738 | +19.7% | -24.2% | 7.1e-3 | 2.5e-2 |
| GR00T N1.6 | vision | 276 / 1094 | 621.6 | 622.5 | 622.1 | 623.2 | 124.2 | 134 | 3588 | -8.8% | -27.2% | 8.0e-3 | 2.9e-2 |
| GR00T N1.6 | vision + DiT | 470 / 1094 | 461.4 | 462.2 | 461.9 | 462.9 | 124.2 | 116 | 2777 | -32.3% | -43.7% | 8.1e-3 | 2.9e-2 |
| GR00T N1.6 | vision + DiT, keep-float (fixed, 57) | 413 / 1094 | 481.7 | 482.3 | 482.2 | 482.7 | 125.5 | 146 | 3076 | -29.3% | -37.6% | 2.8e-3 | 3.6e-2 |
| GR00T N1.6 | vision + DiT, keep-float (served, 3) | 467 / 1094 | 463.2 | 464.0 | 463.6 | 465.0 | 124.3 | 146 | 2800 | -32.0% | -43.2% | 7.6e-3 | 9.2e-3 |
| GR00T N1.6 | vision + DiT, keep-float (union, 60) | 410 / 1094 | 480.1 | 480.7 | 480.5 | 481.2 | 125.2 | 147 | 3114 | -29.6% | -36.8% | 3.0e-3 | 8.6e-3 |
| GR00T N1.7 | F16 | - | 740.1 | 744.9 | 745.0 | 749.0 | 196.9 | 190 | 5108 | - | - | 2.0e-3 | 2.0e-3 |
| GR00T N1.7 | default | 112 / 1030 | 699.0 | 703.5 | 701.5 | 709.3 | 196.1 | 190 | 4356 | -5.6% | -14.7% | 1.1e-2 | 2.0e-2 |
| GR00T N1.7 | vision | 216 / 1030 | 683.3 | 688.4 | 686.0 | 694.1 | 179.5 | 190 | 3996 | -7.6% | -21.8% | 2.0e-2 | 2.2e-2 |
| GR00T N1.7 | vision + DiT | 410 / 1030 | 528.8 | 536.1 | 534.6 | 542.1 | 181.6 | 191 | 3151 | -28.0% | -38.3% | 2.8e-2 | 2.1e-2 |
| GR00T N1.7 | vision + DiT, keep-float (fixed, 185) | 225 / 1030 | 599.2 | 606.0 | 605.4 | 611.7 | 186.9 | 191 | 4051 | -18.6% | -20.7% | 2.6e-3 | 5.3e-3 |
| GR00T N1.7 | vision + DiT, keep-float (served, 157) | 253 / 1030 | 573.7 | 579.8 | 578.0 | 590.8 | 187.3 | 190 | 3870 | -22.2% | -24.2% | 4.2e-2 | 2.8e-3 |
| GR00T N1.7 | vision + DiT, keep-float (union, 233) | 177 / 1030 | 609.4 | 614.4 | 614.1 | 619.1 | 188.2 | 190 | 4249 | -17.5% | -16.8% | 3.9e-3 | 6.5e-3 |
| π0.5 | F16 | - | 1106.0 | 1107.8 | 1107.0 | 1111.5 | 277.0 | 284 | 5980 | - | - | 4.8e-4 | 3.7e-4 |
| π0.5 | default | 127 / 819 | 969.3 | 971.0 | 969.8 | 974.7 | 277.3 | 126 | 3997 | -12.3% | -33.2% | 4.9e-4 | 1.9e-3 |
| π0.5 | vision, unpadded | 262 / 819 | 1318.2 | 1320.2 | 1318.9 | 1324.5 | 621.0 | 126 | 3760 | +19.2% | -37.1% | 8.5e-4 | 2.7e-3 |
| π0.5 | vision | 289 / 819 | 939.6 | 941.3 | 940.8 | 943.1 | 236.7 | 103 | 3676 | -15.0% | -38.5% | 9.2e-4 | 2.8e-3 |
| π0 | F16 | - | 1118.9 | 1123.3 | 1123.7 | 1125.3 | 276.9 | 284 | 5662 | - | - | 7.7e-4 | 5.8e-4 |
| π0 | default | 127 / 780 | 976.4 | 978.8 | 977.8 | 981.2 | 277.0 | 126 | 3723 | -12.9% | -34.2% | 1.8e-2 | 1.7e-3 |
| π0 | vision | 289 / 780 | 935.7 | 937.6 | 937.3 | 938.8 | 235.9 | 103 | 3337 | -16.5% | -41.1% | 1.6e-2 | 1.7e-3 |
| π0 | default, keep-float (fixed, 8) | 119 / 780 | 977.0 | 978.8 | 978.3 | 980.2 | 276.9 | 126 | 3711 | -12.9% | -34.5% | 1.4e-3 | 5.2e-4 |
| OpenVLA-OFT | F16 | - | 1122.2 | 1125.2 | 1125.0 | 1127.6 | 262.7 | 844 | 15266 | - | - | 3.1e-4 | 7.5e-4 |
| OpenVLA-OFT | default | 227 / 958 | 843.5 | 845.7 | 844.8 | 848.9 | 259.5 | 844 | 9403 | -24.8% | -38.4% | 5.9e-4 | 3.5e-3 |
| OpenVLA-OFT | vision | 423 / 958 | 810.1 | 812.5 | 810.6 | 816.4 | 228.7 | 833 | 8821 | -27.8% | -42.2% | 7.9e-4 | 3.2e-3 |
| OpenVLA-OFT | vision, keep-float (served, 12) | 411 / 958 | 818.7 | 821.4 | 820.4 | 824.8 | 230.0 | 839 | 8959 | -27.0% | -41.3% | 7.9e-4 | 2.5e-3 |
| Evo-1 | F16 | - | 1640.1 | 1659.0 | 1653.3 | 1675.4 | 818.6 | 318 | 1582 | - | - | 6.2e-4 | 1.1e-3 |
| Evo-1 | default | 100 / 636 | 1643.5 | 1655.4 | 1653.6 | 1660.8 | 820.1 | 312 | 1459 | -0.2% | -7.8% | 5.4e-3 | 1.2e-2 |
| Evo-1 | vision | 196 / 636 | 1620.6 | 1638.8 | 1639.0 | 1645.7 | 808.1 | 312 | 1121 | -1.2% | -29.1% | 6.3e-3 | 1.4e-2 |
| Evo-1 | vision, keep-float (fixed, 25) | 171 / 636 | 1624.7 | 1634.5 | 1632.9 | 1647.6 | 803.5 | 319 | 1171 | -1.5% | -26.0% | 2.4e-3 | 1.1e-2 |

A per-op profile of the default, vision and vision + DiT files puts every Q8_0
matmul on the HMX. The loader zero-pads a 4304-wide FFN to 4320 at load only
when its weights are float, so without `--pad-m32` the vision tower's FFNs of
π0.5 and GR00T N1.6 keep their 4304 width (rows *vision, unpadded*).

Fixed-input deviation of the default configuration on the NPU and on the
CPU-only build, against the CPU-only build at F32:

| Model | NPU | CPU-only build |
|---|--:|--:|
| π0 | 1.8e-2 | 1.9e-2 |
| π0.5 | 4.9e-4 | 1.1e-3 |
| SmolVLA | 1.2e-2 | 1.8e-2 |
| Evo-1 | 5.4e-3 | 7.3e-3 |
| GR00T N1.5 | 5.1e-3 | 1.2e-2 |
| GR00T N1.6 | 7.2e-3 | 1.2e-2 |
| GR00T N1.7 | 1.1e-2 | 4.4e-2 |
| VLA-JEPA | 1.4e-2 | 2.7e-2 |
| VLA-Adapter | 4.4e-3 | 6.2e-3 |
| TurboVLA | 1.3e-2 | 2.1e-2 |

## Not run

- **BitVLA**: N/A off CUDA. The only published GGUF is int2-packed, and only
Expand Down
60 changes: 58 additions & 2 deletions scripts/quantize_gguf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
"""

import argparse
import re
import numpy as np
import gguf

Expand Down Expand Up @@ -60,6 +61,9 @@
"time"
)
SKIP_VISION = ("vit", "vision", "vis.d.", "vis.s.", "octo.obs.")
# With --dit, the DiT's weight matrices are packed too; these entries of SKIP
# are what kept them float.
DIT_UNSKIP = ("dit", "aex.", "ah.")

# Block size per row (ne0 must divide this). Only the types the gguf writer can
# pack are offered; Q8_0 is near-lossless, Q4_0/Q4_1 are 4-bit.
Expand All @@ -81,21 +85,56 @@ def to_f32(t) -> np.ndarray:
return None # already packed / unsupported source


def eligible(name: str, shape, qtype: str, skip) -> bool:
def eligible(name: str, shape, qtype: str, skip, dit: bool = False, keep=None) -> bool:
if len(shape) != 2 or not name.endswith(".weight"):
return False
if keep is not None and keep.search(name):
return False
if dit and ".dit." in name:
skip = tuple(s for s in skip if s not in DIT_UNSKIP)
if any(s in name for s in skip):
return False
return int(shape[0]) % QK[qtype] == 0 # ne0 is the quantization axis


def ffn_pads(r) -> dict:
"""FFN blocks whose width is not a multiple of 32: prefix -> (width, padded)."""
pads = {}
for t in r.tensors:
if t.name.endswith(".fc1.weight") and len(t.shape) == 2 and int(t.shape[1]) % 32:
width = int(t.shape[1])
pads[t.name[:-len(".fc1.weight")]] = (width, (width + 31) // 32 * 32)
return pads


def pad_ffn(t, old: int, new: int) -> np.ndarray:
"""fc1 rows and bias, or fc2 columns, zero-padded from old to new."""
arr = to_f32(t)
if t.name.endswith(".fc1.weight"):
return np.concatenate([arr, np.zeros((new - old, arr.shape[1]), np.float32)], axis=0)
if t.name.endswith(".fc1.bias"):
return np.concatenate([arr, np.zeros((new - old,), np.float32)])
return np.concatenate([arr, np.zeros((arr.shape[0], new - old), np.float32)], axis=1)


def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--in", dest="src", required=True)
ap.add_argument("--out", dest="dst", required=True)
ap.add_argument("--type", default="Q8_0", choices=sorted(QK))
ap.add_argument("--vision", action="store_true", help="also pack the vision tower")
ap.add_argument("--dit", action="store_true",
help="also pack the DiT's weight matrices (adaLN, the time embedding, "
"norms and the state/action encoders stay float)")
ap.add_argument("--keep-float", metavar="REGEX",
help="keep the weights whose name matches REGEX at their source precision, "
"e.g. '^vlm\\.blk\\.(0|1)\\.' for an LM's first two blocks")
ap.add_argument("--pad-m32", action="store_true",
help="zero-pad FFN widths that are not a multiple of 32 (fc1 rows and bias, "
"fc2 columns): Hexagon's matrix unit takes no other width for a packed "
"weight. The output is unchanged")
args = ap.parse_args()
keep = re.compile(args.keep_float) if args.keep_float else None
skip = SKIP if args.vision else SKIP + SKIP_VISION

r = gguf.GGUFReader(args.src)
Expand All @@ -108,12 +147,29 @@ def main() -> None:
copy_kv(r, w)

qtype = getattr(gguf.GGMLQuantizationType, args.type)
pads = ffn_pads(r) if args.pad_m32 else {}
if pads:
old, new = next(iter(pads.values()))
print(f"--pad-m32: {len(pads)} FFN blocks, e.g. {old} -> {new}")
padded = (".fc1.weight", ".fc1.bias", ".fc2.weight")
n_q = 0
bytes_in = bytes_out = 0
for t in r.tensors:
src_bytes = int(t.data.nbytes)
bytes_in += src_bytes
f32 = to_f32(t) if eligible(t.name, t.shape, args.type, skip) else None
blk = t.name.rsplit(".", 2)[0] if t.name.endswith(padded) else None
if blk in pads and to_f32(t) is not None:
arr = pad_ffn(t, *pads[blk])
if eligible(t.name, arr.shape[::-1], args.type, skip, args.dit, keep):
packed = gguf.quants.quantize(arr, qtype)
w.add_tensor(t.name, packed, raw_dtype=qtype)
bytes_out += int(packed.nbytes)
n_q += 1
else:
w.add_tensor(t.name, arr, raw_dtype=gguf.GGMLQuantizationType.F32)
bytes_out += int(arr.nbytes)
continue
f32 = to_f32(t) if eligible(t.name, t.shape, args.type, skip, args.dit, keep) else None
if f32 is not None:
packed = gguf.quants.quantize(f32, qtype) # rows are byte-sized; shape inferred
w.add_tensor(t.name, packed, raw_dtype=qtype)
Expand Down
22 changes: 22 additions & 0 deletions tests/py/test_converters.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,6 +408,28 @@ def test_quantize_skip_names():
assert Q.eligible("vis.d.blk.0.qkv.weight", (64, 64), "Q8_0", Q.SKIP)



def test_quantize_dit_keep_float_pad():
import importlib
import re
Q = importlib.import_module("quantize_gguf")
default = Q.SKIP + Q.SKIP_VISION
dit = ["aex.dit.0.attn_q.weight", "aex.dit.0.ff0.weight", "aex.dit.proj_out1.weight",
"ah.dit.3.attn_o.weight"]
for n in dit:
assert not Q.eligible(n, (64, 64), "Q8_0", default), n
assert Q.eligible(n, (64, 64), "Q8_0", default, dit=True), n
for n in ("aex.dit.0.adaln.weight", "aex.dit.time_emb.l1.weight", "ah.act_enc.l2.weight"):
assert not Q.eligible(n, (64, 64), "Q8_0", default, dit=True), n
keep = re.compile(r"^vlm\.blk\.0\.")
assert not Q.eligible("vlm.blk.0.attn_q.weight", (64, 64), "Q8_0", default, keep=keep)
assert Q.eligible("vlm.blk.1.attn_q.weight", (64, 64), "Q8_0", default, keep=keep)
t = lambda n, s: types.SimpleNamespace(name=n, shape=s)
r = types.SimpleNamespace(tensors=[t("vit.blk.0.fc1.weight", [1152, 4304]),
t("vit.blk.0.fc2.weight", [4304, 1152]),
t("lm.blk.0.fc1.weight", [2048, 8192])])
assert Q.ffn_pads(r) == {"vit.blk.0": (4304, 4320)}

def test_find_sidecar():
import tempfile
from gguf_common import find_sidecar
Expand Down
Loading