diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d2436c7..5a91d22f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,15 +9,23 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke - **Hexagon NPU on the Qualcomm Dragonwing IQ-10 (Linux).** `scripts/build_linux_hexagon.sh` cross-builds vla.cpp with `GGML_HEXAGON=ON` in Qualcomm's toolchain container, `vla-server` included. On the IQ-10 (v81, - llama.cpp `b11223`) SmolVLA, π0.5, Evo-1 and GR00T N1.5 / N1.6 / N1.7 run - within 2.9e-3 of the CPU build on fixed inputs and serve through - `eval/client`. Concats the NPU refuses on buffers above 4 GB of DSP address - space and matmuls of a 2D weight against a batch, which it computes wrong - (GR00T N1.6 with two views), run on the CPU, and GR00T N1.7's LM leaves - flash attention (F16 scores on the HMX) unless `--flash-attn` is given. - GR00T N1.5 and N1.6 are above the bar on a served LIBERO observation, as the - CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install - and the numbers. + llama.cpp `b11223`) SmolVLA, π0, π0.5, Evo-1, GR00T N1.5 / N1.6 / N1.7, + VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small and TurboVLA run within 2.9e-3 + of the CPU build on fixed inputs and serve through `eval/client`. Concats the + NPU refuses on buffers above 4 GB of DSP address space, copies that read a + weight (also answered `NO_SUPPORT`; VLA-Adapter), matmuls of a 2D weight + against a batch, which it computes wrong (GR00T N1.6 with two views), and + repeats of a computed tensor, which can read stale cache lines (TurboVLA), run + on the CPU, and GR00T N1.7's LM leaves flash attention (F16 scores on the + HMX) unless `--flash-attn` is given. Three matmuls that the HMX does not take + move onto it: SigLIP's 4304-wide MLP is zero-padded to 4320 at load, π0.5's + SigLIP takes flash attention, and Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, + π0.5 and Evo-1 run 27-40% faster, within the bar on fixed inputs. π0.5's VLM + and action expert take flash attention too, and the experts' KV concats run as + row copies that stay on the NPU (π0, π0.5, SmolVLA). GR00T N1.5, N1.6 and + VLA-JEPA are above the bar on a served LIBERO observation, as the CPU is at + F16. See `docs/backend/hexagon.md` for the build, the skel install and the + numbers. - **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200` (Radeon RX 9060 XT) with ROCm 7.14 and llama.cpp `b11223`, every arch except diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 6df93b30..d9196bb4 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -5,9 +5,9 @@ to take once it ran. > **Status: running on Windows on Arm and on a Dragonwing IQ-10 under Linux.** > vla.cpp builds with `GGML_HEXAGON=ON` and runs eleven checkpoints on a -> Snapdragon X laptop's v73 NPU and six on an IQ-10's v81 NPU under Linux, each -> checked against a CPU reference. The laptop build, its measurements and the -> upstream bugs found along the way are in +> Snapdragon X laptop's v73 NPU and twelve on an IQ-10's v81 NPU under Linux, +> each checked against a CPU reference. The laptop build, its measurements and +> the upstream bugs found along the way are in > [hexagon-windows.md](hexagon-windows.md); the Linux board is in [Linux > (IQ-10)](#linux-iq-10). What follows is the backend in general: the parts that > hold on any Snapdragon part, and the IQ-9 and multi-NPU notes from before @@ -94,7 +94,10 @@ changed all three conclusions: it. - **`GELU_ERF` is still missing.** It now runs on the CPU (next point) in every DINOv2 and SigLIP-so400m tower, and costs what that implies: Evo-1 spends more - on copies than the NPU saves. + on copies than the NPU saves. An HTP kernel was written and checked on the + IQ-10 (b11223): more accurate than the CPU's (relative error 1.5e-6 against + 5.1e-5), and Evo-1 7% faster. It was left out: GR00T N1.7 sits close to the + bar, and the kernel moved its fixed-input chunk from 2.0e-3 to 3.0e-3. - **Absent is no longer fatal.** `src/backend_fallback.cpp` wraps the NPU in a backend that runs whatever it rejects on the CPU, without touching an arch. That also covers ops refused for shape rather than type: softmax rows that are @@ -109,6 +112,7 @@ these, and they compute something else: | `ADD`/`MUL`/... | `src1` broadcast over one dim but not a higher one | Refused, runs on the CPU | | `IM2COL` | Wrong unless kernel = stride > 1 and no padding | Refused, runs on the CPU | | `CPY` F32→F16 | Reads back wrong on the host | Copies feeding a CPU op run on the CPU | +| `REPEAT` | Reads stale cache lines of a source written by DMA earlier in the same op batch | Refused unless the source is a weight, runs on the CPU | | fused `MUL_MAT+ADD`, `RMS_NORM+MUL` | ~10x the error of the unfused ops | `GGML_HEXAGON_OPFUSION=0` by default | Each one flips the gripper channel, or comes close, on some checkpoint. The @@ -117,11 +121,21 @@ was right: a missing op fails at load, and these only show in the actions. [hexagon-windows.md](hexagon-windows.md#hexagon-issues-found-and-worked-around) has the numbers and how each was bisected. They belong upstream. -One more op is claimed and then refused at run time. On buffers mapped above +Two more ops are claimed and then refused at run time. On buffers mapped above 4 GB of DSP address space, which the IQ-10 under Linux hands out, the NPU runs `CONCAT` only as the dim-1 contiguous DMA copy and answers any other concat with `NO_SUPPORT`, and the host aborts on the first predict. The wrapper sends those -concats to the CPU. +concats to the CPU. A `CPY` that reads a weight gets the same answer +(VLA-Adapter casts its F16 action queries to F32 in the graph), and also runs on +the CPU. + +The experts' KV concats (π0, π0.5, SmolVLA) were such concats: along the token +dim of contiguous [head dim, heads, tokens] tensors, once per layer per denoise +step. Appending along that dim appends whole rows, so on Hexagon +`concat_dim2()` in `src/backend.h` does them as a dim-1 concat of the +[head dim x heads, tokens] views, which writes the same bytes and stays on the +NPU: π0.5 1379 to 1133 ms, SmolVLA 810 to 638 ms, outputs unchanged. π0 takes +the same path. Flash attention on the HMX (v81) keeps scores and probabilities in F16. That is a precision limit rather than a wrong op, but in GR00T N1.7's LM it moves the @@ -133,6 +147,28 @@ of 1, activations above 1) is claimed and comes out wrong. GR00T N1.6's SigLIP2 tower takes two camera views that way: its action chunk lands 0.16 from the CPU reference. The wrapper runs those matmuls on the CPU: 1.9e-3, and 25% faster. +The matrix unit (HMX) takes a matmul only when its inner dim K and its output +dim N are multiples of 32, and with a batch only an F16 src0. Anything else runs +on the vector units, at about 1/20 of the speed. `GGML_HEXAGON_PROFILE=1` on the +IQ-10 showed three such matmuls dominating, and vla.cpp now avoids them in +Hexagon builds: + +- **SigLIP-so400m's MLP is 4304 wide**: 440 ms of GR00T N1.5's 565 ms tower. + The loader zero-pads fc1, its bias and fc2 to 4320 (`WeightLoader::pad`, + `gemm_width()` in `src/backend.h`), which leaves the output unchanged. +- **π0.5's attention ran explicit scores** in SigLIP (head dim 72, F32 x F32), + the VLM and the action expert, because its flash attention was tied to SYCL's + view-taking kernel. The VLM's and the expert's single KV head also sent their + matmuls to the CPU (the batched `MUL_MAT` rule above). All three now take the + plain flash attention on Hexagon (`use_flash_attention()` in `pi05.cpp`). +- **Evo-1's LM attention multiplied F32 K and V against 14 query heads.** K and + V now go to F16 (`fa_kv()`); the scores still accumulate in F32. + +What is left on the vector units: GR00T's DiT attention (head dim 48) and Evo-1's +DiT cross-attention (head dim 112, 1025 keys), whose shapes the HMX does not +take, and Evo-1's InternViT attention, which stays on the CPU because F16 K/V +there measured 3.9e-2. + ## What changed in `vla.cpp` The shape of the diff was predicted correctly, apart from the scheduler: @@ -280,43 +316,57 @@ export LD_LIBRARY_PATH=$PWD/pkg/lib `vla_predict_check` at the Hexagon defaults, against the same binary with `VLA_DEVICE=cpu --weight-dtype f32 --flash-attn 0`. A Hexagon build turns flash attention on by default even on the CPU, so the reference asks for the explicit -path that other builds run: - -| Model | max\|Δ\| | -|---|--:| -| SmolVLA | 7.2e-4 | -| π0.5 | 2.7e-4 | -| Evo-1 | 4.1e-4 | -| GR00T N1.5 | 1.7e-3 | -| GR00T N1.6 | 1.4e-3 | -| GR00T N1.7 | 2.0e-3 | +path that other builds run. Views are the ones the LIBERO client sends, so they +differ from the benchmark's for VLA-JEPA, VLA-Adapter, OpenVLA-OFT and Octo: -GR00T N1.6 here takes one view; with two it is 1.9e-3. The other archs have -not been run on the board yet. +| Model | Views | max\|Δ\| | +|---|--:|--:| +| SmolVLA | 2 | 7.2e-4 | +| π0 | 2 | 1.2e-3 | +| π0.5 | 2 | 4.8e-4 | +| Evo-1 | 1 | 5.5e-4 | +| GR00T N1.5 | 1 | 2.4e-3 | +| GR00T N1.6 | 1 | 1.4e-3 | +| GR00T N1.7 | 1 | 2.0e-3 | +| VLA-JEPA | 2 | 2.8e-3 | +| VLA-Adapter | 2 | 4.2e-4 | +| OpenVLA-OFT | 2 | 3.5e-4 | +| Octo-Small | 1 | 4.1e-4 | +| TurboVLA | 2 | 3.1e-4 | + +GR00T N1.6 with two views is 1.9e-3. At the benchmark's single view VLA-JEPA is +1.9e-3, VLA-Adapter 4.8e-4 and OpenVLA-OFT 2.2e-4. ### Serving `vla-server` on the board, driven from the host by `eval/client/vla_cpp_client.py` and the adapters `run_sim_client_direct.py` -uses, on one fixed LIBERO observation (two views, the task prompt, a fixed -robot state) with fixed noise. The same request goes to a Hexagon server and to -a CPU server with `--weight-dtype f32 --flash-attn 0`. Every model returns a -finite chunk, the same chunk for a repeated request, an error reply to a -malformed or an invalid request, and keeps serving afterwards. +uses, on one fixed LIBERO observation (two views, the task prompt, a fixed robot +state), with fixed noise for the archs that draw it. The same request goes to a +Hexagon server and to a CPU server with `--weight-dtype f32 --flash-attn 0`. +Every model returns a finite chunk, the same chunk for a repeated request, an +error reply to a malformed or an invalid request, and keeps serving afterwards. | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 | |---|--:|--:| | SmolVLA | 1.3e-3 | | -| π0.5 | 4.3e-4 | | -| Evo-1 | 4.3e-4 | | -| GR00T N1.5 | 8.3e-3 | 4.3e-2 | +| π0.5 | 3.5e-4 | | +| Evo-1 | 1.1e-3 | | +| GR00T N1.5 | 1.1e-2 | 4.3e-2 | | GR00T N1.6 | 1.3e-2 | 1.1e-2 | | GR00T N1.7 | 2.5e-3 | | - -GR00T N1.5 and N1.6 are above the bar on this input, by about what the CPU -itself gives with F16 weights: they are sensitive to 16-bit precision rather -than miscomputed. Running them with F32 weights on the NPU has not been tried -yet. +| π0 | 5.8e-4 | | +| VLA-JEPA | 6.7e-3 | 2.1e-2 | +| VLA-Adapter | 4.5e-4 | | +| OpenVLA-OFT | 5.0e-4 | | +| Octo-Small | 9.3e-5 | | +| TurboVLA | 2.2e-4 | | + +GR00T N1.5, N1.6 and VLA-JEPA are above the bar on this input, and so is the CPU +itself with F16 weights: N1.5 and VLA-JEPA land closer to the reference than the +CPU at F16, N1.6 slightly further (1.3e-2 against 1.1e-2). They are sensitive to +16-bit precision rather than miscomputed. Running them with F32 weights on the +NPU has not been tried yet. ## IQ-10, imagined diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index 7c381c8f..9a127c17 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`e8dfa34` (PR #69, same llama.cpp pin). Each report +`1cab1c3` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 28.5 | 116.3 | 170.2 | 106.0 | 122.8 | 131.4 | N/A | 213.1 | 620.5 | 232.5 | 1262.5 | 566.0 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 1020.9 | 1110.3 | 747.5 | N/A | 810.3 | n/r | 2914.6 | n/r | 3519.4 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 121.6 | 428.1 | 440.2 | 1733.6 | 613.7 | 705.8 | 747.0 | N/A | 631.0 | 1124.4 | 1117.4 | 4097.7 | 2579.1 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 23.5 | 90.7 | 126.7 | 134.0 | 161.1 | 144.9 | N/A | 161.3 | 438.3 | 430.2 | 405.8 | 256.3 | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index b511afd0..6dc1d5c8 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `e8dfa34` (PR #69, on top of #68) | +| Commit | `1cab1c3` | | llama.cpp | `b11223` | -| Date | 2026-11-02 11:02 to 11:48 (UTC+07) | +| Date | 2026-11-04 11:52 to 12:30 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -65,12 +65,18 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.7 | 1 | 256 | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | -| SmolVLA | 2 | 512 | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | -| GR00T N1.5 | 1 | 224 | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | -| GR00T N1.6 | 1 | 224 | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | -| π0.5 | 2 | 224 | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | -| Evo-1 | 1 | 448 | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | +| Octo-Small | 2 | 256 + 128 | 121.6 | 123.0 | 122.3 | 125.8 | 20.5 | 253 | 804 | +| TurboVLA | 2 | 256 | 428.1 | 433.0 | 431.6 | 436.8 | - | 438 | 1327 | +| VLA-JEPA | 1 | 256 | 440.2 | 446.3 | 444.7 | 450.2 | 195.2 | 185 | 3971 | +| GR00T N1.5 | 1 | 224 | 613.7 | 615.7 | 615.7 | 618.0 | 148.3 | 162 | 3636 | +| SmolVLA | 2 | 512 | 631.0 | 632.4 | 631.7 | 634.6 | 281.8 | 146 | 1190 | +| GR00T N1.6 | 1 | 224 | 705.8 | 708.5 | 708.0 | 710.7 | 145.0 | 146 | 4869 | +| GR00T N1.7 | 1 | 256 | 747.0 | 756.0 | 755.0 | 764.5 | 197.2 | 191 | 5090 | +| π0.5 | 2 | 224 | 1117.4 | 1121.0 | 1120.8 | 1123.3 | 292.5 | 285 | 5790 | +| π0 | 2 | 224 | 1124.4 | 1128.7 | 1128.6 | 1130.5 | 293.0 | 285 | 5534 | +| VLA-Adapter | 1 | 224 | 1733.6 | 1752.3 | 1749.5 | 1765.6 | 1187.2 | 872 | 3566 | +| Evo-1 | 1 | 448 | 2579.1 | 2595.5 | 2592.1 | 2611.9 | 854.0 | 319 | 1551 | +| OpenVLA-OFT | 1 | 224 | 4097.7 | 4107.8 | 4107.8 | 4114.2 | 1187.3 | 844 | 15327 | ### Fastest configuration @@ -94,23 +100,35 @@ rates measured at the defaults therefore do not carry over to these rows. | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.7 | *(defaults)* | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | - | -| SmolVLA | *(defaults)* | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | - | -| GR00T N1.5 | *(defaults)* | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | - | -| GR00T N1.6 | *(defaults)* | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | - | -| π0.5 | *(defaults)* | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | - | -| Evo-1 | *(defaults)* | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | - | +| Octo-Small | *(defaults)* | 121.6 | 123.0 | 122.3 | 125.8 | 20.5 | 253 | 804 | - | +| TurboVLA | *(defaults)* | 428.1 | 433.0 | 431.6 | 436.8 | - | 438 | 1327 | - | +| VLA-JEPA | *(defaults)* | 440.2 | 446.3 | 444.7 | 450.2 | 195.2 | 185 | 3971 | - | +| GR00T N1.5 | *(defaults)* | 613.7 | 615.7 | 615.7 | 618.0 | 148.3 | 162 | 3636 | - | +| SmolVLA | *(defaults)* | 631.0 | 632.4 | 631.7 | 634.6 | 281.8 | 146 | 1190 | - | +| GR00T N1.6 | *(defaults)* | 705.8 | 708.5 | 708.0 | 710.7 | 145.0 | 146 | 4869 | - | +| GR00T N1.7 | *(defaults)* | 747.0 | 756.0 | 755.0 | 764.5 | 197.2 | 191 | 5090 | - | +| π0.5 | *(defaults)* | 1117.4 | 1121.0 | 1120.8 | 1123.3 | 292.5 | 285 | 5790 | - | +| π0 | *(defaults)* | 1124.4 | 1128.7 | 1128.6 | 1130.5 | 293.0 | 285 | 5534 | - | +| VLA-Adapter | *(defaults)* | 1733.6 | 1752.3 | 1749.5 | 1765.6 | 1187.2 | 872 | 3566 | - | +| Evo-1 | *(defaults)* | 2579.1 | 2595.5 | 2592.1 | 2611.9 | 854.0 | 319 | 1551 | - | +| OpenVLA-OFT | *(defaults)* | 4097.7 | 4107.8 | 4107.8 | 4114.2 | 1187.3 | 844 | 15327 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| GR00T N1.7 | 768.7 | 748.5 | 761.3 | 744.5 | 1086.6 | -| SmolVLA | 823.1 | 818.9 | 819.5 | 818.5 | 3018.8 | -| GR00T N1.5 | 1026.1 | 1023.0 | 1025.3 | 1026.1 | 1501.2 | -| GR00T N1.6 | 1114.4 | 1115.1 | 1114.4 | 1113.3 | 1290.3 | -| π0.5 | 2939.0 | 2956.9 | 2959.0 | 2962.7 | 2959.6 | -| Evo-1 | 3583.7 | 3593.9 | 3591.8 | 3582.6 | 5828.5 | +| Octo-Small | 125.3 | 128.8 | 124.4 | 123.5 | 124.6 | +| TurboVLA | 438.8 | 436.5 | 439.3 | 438.2 | 535.1 | +| VLA-JEPA | 446.7 | 445.8 | 444.8 | 445.2 | 817.8 | +| GR00T N1.5 | 615.8 | 619.3 | 615.3 | 618.4 | 1096.5 | +| SmolVLA | 632.1 | 633.0 | 632.5 | 632.6 | 2834.8 | +| GR00T N1.6 | 714.2 | 712.1 | 718.7 | 714.9 | 885.7 | +| GR00T N1.7 | 765.2 | 742.0 | 761.5 | 743.6 | 1081.0 | +| π0.5 | 1121.1 | 1121.9 | 1120.5 | 1121.4 | 2062.9 | +| π0 | 1129.8 | 1128.8 | 1131.8 | 1128.5 | 2090.5 | +| VLA-Adapter | 1766.1 | 1760.1 | 1756.1 | 1757.2 | 1767.1 | +| Evo-1 | 2613.0 | 2613.7 | 2617.3 | 2617.9 | 4840.1 | +| OpenVLA-OFT | 4110.3 | 4101.9 | 4101.7 | 4108.2 | 4107.9 |
@@ -118,8 +136,6 @@ rates measured at the defaults therefore do not carry over to these rows. - **BitVLA**: N/A off CUDA. The only published GGUF is int2-packed, and only CUDA builds load it. -- **π0, VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small, TurboVLA**: not run in - this round; their GGUFs were not on the board. ## Notes @@ -132,9 +148,9 @@ its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). These rows time the defaults; they do not vouch for accuracy. On a served LIBERO -observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar, about as -far off as the CPU at F16; see -[backend/hexagon.md](../backend/hexagon.md#serving). +observation, GR00T N1.5, N1.6 and VLA-JEPA land above the 2.9e-3 fidelity bar: +N1.5 and VLA-JEPA closer to the reference than the CPU at F16, N1.6 slightly +further; see [backend/hexagon.md](../backend/hexagon.md#serving). ## Reproducing diff --git a/src/backend.h b/src/backend.h index bd574c0d..3b50682a 100644 --- a/src/backend.h +++ b/src/backend.h @@ -191,6 +191,44 @@ inline ggml_tensor * fa_kv([[maybe_unused]] ggml_context * ctx, ggml_tensor * t) #endif } +/** + * @brief `ggml_concat(a, b, 2)` of [d0, d1, n] tensors, as the compiled-in + * backend runs it. + * + * Hexagon runs a concat on the NPU only as a dim-1 copy of whole rows; any other + * goes to the CPU fallback (src/backend_fallback.cpp). For contiguous inputs a + * dim-2 concat appends whole d0*d1 rows, so it is done as a dim-1 concat of the + * [d0*d1, n] views, which writes the same bytes. Everywhere else, and for + * non-contiguous inputs, this is ggml_concat(a, b, 2). + */ +inline ggml_tensor * concat_dim2(ggml_context * ctx, ggml_tensor * a, ggml_tensor * b) { +#ifdef GGML_USE_HEXAGON + if (ggml_is_contiguous(a) && ggml_is_contiguous(b) && a->ne[3] == 1 && b->ne[3] == 1 && + a->ne[0] == b->ne[0] && a->ne[1] == b->ne[1]) { + ggml_tensor * rows = ggml_concat(ctx, ggml_reshape_2d(ctx, a, a->ne[0] * a->ne[1], a->ne[2]), + ggml_reshape_2d(ctx, b, b->ne[0] * b->ne[1], b->ne[2]), 1); + return ggml_reshape_3d(ctx, rows, a->ne[0], a->ne[1], a->ne[2] + b->ne[2]); + } +#endif + return ggml_concat(ctx, a, b, 2); +} + +/** + * @brief A GEMM's inner width as the compiled-in backend's matmul takes it. + * + * Hexagon's matrix unit (HMX) takes an F16 weight only when both of its dims + * are multiples of 32; anything else runs on the vector units. SigLIP's + * 4304-wide MLP spends 440 ms there per tower on the IQ-10, against about + * 0.5 ms a matmul on the HMX. Everywhere else this returns @p n. + */ +inline int64_t gemm_width(int64_t n) { +#ifdef GGML_USE_HEXAGON + return (n + 31) / 32 * 32; +#else + return n; +#endif +} + /** * @brief `ggml_gelu` (the tanh form), in ops the compiled-in backend gets right. * diff --git a/src/backend_fallback.cpp b/src/backend_fallback.cpp index 7e0d4035..151692b9 100644 --- a/src/backend_fallback.cpp +++ b/src/backend_fallback.cpp @@ -351,6 +351,23 @@ bool accel_runs(FallbackCtx * fc, const ggml_tensor * t, const char ** why) { *why = "miscomputes a 2D weight against a batch in"; return false; } + if (fc->hexagon && t->op == GGML_OP_CPY && is_weight(t->src[0])) { + // Claimed, but a copy that reads a weight is answered with NO_SUPPORT + // and the host aborts. VLA-Adapter casts its F16 action queries + // [896,64] to F32 in the graph; with the cast on the CPU it is within + // 4.2e-4 of the CPU F32 reference. + *why = "cannot run (NO_SUPPORT) a weight-reading"; + return false; + } + if (fc->hexagon && t->op == GGML_OP_REPEAT && !is_weight(t->src[0])) { + // The NPU's REPEAT reads its source through the data cache, which is + // invalidated only between op batches, while the op that wrote the + // source used DMA, so within one batch it can read stale lines. + // TurboVLA repeats its ViT prefix [768,5] over two views: max|delta| + // 0.19 and different on every run, 3.1e-4 with the repeat on the CPU. + *why = "can read stale data in a"; + return false; + } return ggml_backend_supports_op(fc->accel, t); } diff --git a/src/loader.cpp b/src/loader.cpp index 54826859..fc6d95f7 100644 --- a/src/loader.cpp +++ b/src/loader.cpp @@ -20,6 +20,7 @@ #include #include #include +#include namespace vla { @@ -39,11 +40,25 @@ void WeightLoader::as_float(const std::string & name, int64_t K, int64_t N, Floa as_float_[name] = AsFloat{K, N, std::move(make)}; } +void WeightLoader::pad(const std::string & name, int64_t ne0, int64_t ne1) { + pad_[name] = {ne0, ne1}; +} + bool WeightLoader::read_resident(const std::string & name, ggml_type type, bool gemma_norm, std::vector & out) { const auto it = as_float_.find(name); if (it == as_float_.end()) { out = g_.read_convert(name.c_str(), type, gemma_norm); - return !out.empty(); + const auto p = pad_.find(name); + if (p == pad_.end() || out.empty()) + return !out.empty(); + const ggml_tensor * src = g_.meta(name.c_str()); + const size_t rs = ggml_row_size(type, src->ne[0]); + const size_t rp = ggml_row_size(type, p->second.first); + std::vector padded(rp * (size_t) p->second.second, 0); + for (int64_t r = 0; r < src->ne[1]; ++r) + std::memcpy(padded.data() + r * rp, out.data() + r * rs, rs); + out.swap(padded); + return true; } const AsFloat & a = it->second; std::vector f; @@ -97,7 +112,23 @@ ggml_tensor * WeightLoader::declare(ggml_type want, bool required, bool gemma_no return nullptr; } - ggml_tensor * t = ggml_new_tensor(ctx_, g_.resident_type(src, want), ggml_n_dims(src), src->ne); + const ggml_type rt = g_.resident_type(src, want); + int64_t ne[GGML_MAX_DIMS]; + std::memcpy(ne, src->ne, sizeof(ne)); + int n_dims = ggml_n_dims(src); + if (const auto p = pad_.find(name); p != pad_.end()) { + const bool float_type = rt == GGML_TYPE_F32 || rt == GGML_TYPE_F16 || rt == GGML_TYPE_BF16; + if (!float_type || n_dims > 2 || p->second.first < ne[0] || p->second.second < ne[1]) { + std::fprintf(stderr, "vla(%s): cannot pad %s to [%lld, %lld]\n", arch_, name, + (long long) p->second.first, (long long) p->second.second); + ok_ = false; + return nullptr; + } + ne[0] = p->second.first; + ne[1] = p->second.second; + n_dims = 2; + } + ggml_tensor * t = ggml_new_tensor(ctx_, rt, n_dims, ne); if (!t) { std::fprintf(stderr, "vla(%s): ggml_new_tensor failed for %s (weight context too small?)\n", arch_, name); ok_ = false; diff --git a/src/loader.h b/src/loader.h index 2f71d099..e44d01e6 100644 --- a/src/loader.h +++ b/src/loader.h @@ -82,6 +82,10 @@ class WeightLoader { using FloatMaker = std::function &)>; void as_float(const std::string & name, int64_t K, int64_t N, FloatMaker make); + // Declare 2D (or 1D) float tensor `name` as [ne0, ne1], at least its shape in + // the file, with the extra elements zero. Call before declaring it. + void pad(const std::string & name, int64_t ne0, int64_t ne1); + // For helpers that inspect the file before declaring (foldquant.cpp). gguf_reader & reader() { return g_; @@ -111,6 +115,7 @@ class WeightLoader { FloatMaker make; }; std::map as_float_; + std::map> pad_; const char * arch_; gguf_reader & g_; diff --git a/src/models/evo1.cpp b/src/models/evo1.cpp index 547e0024..ce31ffab 100644 --- a/src/models/evo1.cpp +++ b/src/models/evo1.cpp @@ -188,8 +188,12 @@ ggml_tensor * build_qwen2_layer(ggml_context * C, const Evo1ModelArch & m, const att = ggml_reshape_2d(C, fa, hq, seq); } else { ggml_tensor * Q = ggml_cont(C, ggml_permute(C, q_rope, 0, 2, 1, 3)); - ggml_tensor * K = ggml_cont(C, ggml_permute(C, k_rope, 0, 2, 1, 3)); - ggml_tensor * V = ggml_cont(C, ggml_permute(C, ggml_reshape_3d(C, vp, hd, n_kv, seq), 1, 2, 0, 3)); + // fa_kv: on Hexagon K and V go to F16, so both matmuls run on the HMX, which + // takes no batched F32 src0 (1 s of vector-unit time per chunk on the IQ-10). + // The scores still accumulate in F32 and the softmax stays F32: 4.3e-4 from + // the CPU F32 reference there. Other backends keep F32 K/V. + ggml_tensor * K = fa_kv(C, ggml_cont(C, ggml_permute(C, k_rope, 0, 2, 1, 3))); + ggml_tensor * V = fa_kv(C, ggml_cont(C, ggml_permute(C, ggml_reshape_3d(C, vp, hd, n_kv, seq), 1, 2, 0, 3))); ggml_tensor * kq = ggml_mul_mat(C, K, Q); ggml_prec_set_acc(kq, GGML_PREC_F32); ggml_tensor * aw = ggml_soft_max_ext(C, kq, mask, scale, 0.0f); ggml_tensor * kqv = ggml_mul_mat(C, V, aw); diff --git a/src/models/pi05.cpp b/src/models/pi05.cpp index 1f4faa31..6eeece50 100644 --- a/src/models/pi05.cpp +++ b/src/models/pi05.cpp @@ -146,6 +146,14 @@ namespace { // straight off the [hd, H, S] views. Otherwise the unfused graph is used. bool use_fused_attention() { return vla::flash_attn_enabled() && vla::fa_takes_views(); } +// Where flash attention is on by default (Hexagon), the three sites also take the +// plain flash attention when the kernel does not take views. Their explicit +// scores are matmuls the NPU does not run: SigLIP's at head dim 72, and the VLM's +// and the expert's, whose single KV head the fallback wrapper sends to the CPU. +bool use_flash_attention() { + return use_fused_attention() || (vla::default_flash_attn() && vla::flash_attn_enabled()); +} + ggml_tensor * build_adarms( ggml_context * ctx, ggml_tensor * x, ggml_tensor * mod, int64_t h, float eps, ggml_tensor ** gate_out) { @@ -185,7 +193,7 @@ ggml_tensor * build_expert_layer( ggml_tensor * gate_ffn = ggml_view_1d(ctx, mod_ffn, h, (size_t) 2*h * sizeof(float)); ggml_tensor * xq_attn = fq_act_mod(ctx, &w.act_mod_qkv, w.fq_qkv, x_in, fqx->scale1_attn, fqx->shift_attn); ggml_tensor * h1 = gemma_attn(ctx, w, nullptr, positions, cfg, seq, cached_K, cached_V, mask, nullptr, nullptr, - GGML_TYPE_F32, use_fused_attention(), nullptr, x_in, nullptr, fqx->K16, fqx->V16, + GGML_TYPE_F32, use_flash_attention(), nullptr, x_in, nullptr, fqx->K16, fqx->V16, xq_attn, gate_attn); ggml_tensor * xq_ffn = fq_act_mod(ctx, &w.act_mod_gu, w.fq_gu, h1, fqx->scale1_ffn, fqx->shift_ffn); return gemma_mlp(ctx, w, nullptr, GGML_TYPE_F32, nullptr, h1, xq_ffn, gate_ffn); @@ -207,13 +215,13 @@ ggml_tensor * build_expert_layer( // out * gate + x reading the GEMM output in whatever type it has. ggml_tensor * xn = fused_adarms(ctx, x_in, mod_attn, at, *fa); ggml_tensor * o = gemma_attn(ctx, w, xn, positions, cfg, seq, cached_K, cached_V, mask, - nullptr, nullptr, at, use_fused_attention(), nullptr, nullptr, kv); + nullptr, nullptr, at, use_flash_attention(), nullptr, nullptr, kv); ggml_tensor * h1 = fused_gateres(ctx, x_in, o, mod_attn, *fg); ggml_tensor * xm = fused_adarms(ctx, h1, mod_ffn, at, *fa); return fused_gateres(ctx, h1, gemma_mlp(ctx, w, xm, at), mod_ffn, *fg); } - if (!mask && !kv && gemma_fused(w, at, use_fused_attention(), seq, seq + (cached_K ? cached_K->ne[2] : 0))) { + if (!mask && !kv && gemma_fused(w, at, use_flash_attention(), seq, seq + (cached_K ? cached_K->ne[2] : 0))) { auto part = [&](ggml_tensor * mod, int j) { return ggml_view_1d(ctx, mod, h, (size_t) j*h*sizeof(float)); }; @@ -237,7 +245,7 @@ ggml_tensor * build_expert_layer( // mm_act does not convert its input: a BF16 GEMM must be handed BF16 rows, // but adaRMS produces F32, whose bits would be misread as BF16. ggml_tensor * o_out = gemma_attn(ctx, w, as_type(ctx, x_norm, at), positions, cfg, seq, cached_K, cached_V, mask, - nullptr, nullptr, at, use_fused_attention(), x_norm, nullptr, kv, + nullptr, nullptr, at, use_flash_attention(), x_norm, nullptr, kv, fqx ? fqx->K16 : nullptr, fqx ? fqx->V16 : nullptr); o_out = as_type(ctx, o_out, GGML_TYPE_F32); @@ -544,7 +552,7 @@ std::unique_ptr pi05_create(const std::string& mmproj_path, std::printf("vla(pi05): expert weights INT4 (W4A8), %s\n", aex_w4_group > 0 ? ("group " + std::to_string(aex_w4_group)).c_str() : "per channel"); } - m->vis.vit.enc.cfg.flash_attn = use_fused_attention(); + m->vis.vit.enc.cfg.flash_attn = use_flash_attention(); // Fused SigLIP blocks (modules/encoder.h, EncCfg::fused): SYCL, BF16, fused // attention. A FoldQuant GGUF takes them too: its integer sites are the LM's // and the expert's, and its vision tower is BF16 like any other checkpoint's. @@ -837,7 +845,7 @@ std::vector Pi05ModelArch::predict(const Inputs& in) { pl.rms_spec); else h = gemma_layer(C, pl.blk[i], h, t_prefix_pos, cfg, n_prefix_g, - nullptr, nullptr, t_prefix_mask, &cK[i], &cV[i], act_type, use_fused_attention(), + nullptr, nullptr, t_prefix_mask, &cK[i], &cV[i], act_type, use_flash_attention(), kv_inplace ? &kv[i] : nullptr); } (void) h; diff --git a/src/models/smolvla.cpp b/src/models/smolvla.cpp index d298ec11..02c11414 100644 --- a/src/models/smolvla.cpp +++ b/src/models/smolvla.cpp @@ -732,8 +732,8 @@ ggml_tensor * build_expert_self_attn_layer( ggml_tensor * k_h = ggml_reshape_3d(ctx, mm_w(ctx, w.Wk, x_norm), cfg.head_dim, cfg.n_kv_heads, cfg.n_suffix); ggml_tensor * v_h = ggml_reshape_3d(ctx, mm_w(ctx, w.Wv, x_norm), cfg.head_dim, cfg.n_kv_heads, cfg.n_suffix); - ggml_tensor * K_full = ggml_concat(ctx, cached_K, rope(ctx, rope_spec(cfg), k_h, positions_full), 2); - ggml_tensor * V_full = ggml_concat(ctx, cached_V, v_h, 2); + ggml_tensor * K_full = concat_dim2(ctx, cached_K, rope(ctx, rope_spec(cfg), k_h, positions_full)); + ggml_tensor * V_full = concat_dim2(ctx, cached_V, v_h); return attn_mlp(ctx, w, x_in, rope(ctx, rope_spec(cfg), q_h, positions_full), K_full, V_full, mask_full, cfg, cfg.n_suffix); } diff --git a/src/modules/encoder.cpp b/src/modules/encoder.cpp index d7083ce6..f450bb66 100644 --- a/src/modules/encoder.cpp +++ b/src/modules/encoder.cpp @@ -14,6 +14,7 @@ #include "modules/encoder.h" +#include "backend.h" #include "layers/attn.h" #include "layers/ffn.h" #include "layers/fused_ops.h" @@ -27,10 +28,32 @@ #include "ggml-backend.h" #include +#include #include namespace vla { +namespace { + +// Widens the MLP to gemm_width() with zeros. The extra fc1 rows and bias are 0, +// GELU(0) is 0 and the extra fc2 columns are 0, so the block's output is the same. +void pad_mlp(WeightLoader & L, const std::string & fc1, const std::string & fc2) { + const ggml_tensor * w1 = L.reader().meta((fc1 + ".weight").c_str()); + const ggml_tensor * w2 = L.reader().meta((fc2 + ".weight").c_str()); + if (!w1 || !w2 || ggml_is_quantized(L.gemm_type()) || ggml_is_quantized(w1->type) || + ggml_is_quantized(w2->type)) + return; + const int64_t ff = w1->ne[1]; + const int64_t fp = gemm_width(ff); + if (fp == ff || w2->ne[0] != ff) + return; + L.pad(fc1 + ".weight", w1->ne[0], fp); + L.pad(fc1 + ".bias", fp, 1); + L.pad(fc2 + ".weight", fp, w2->ne[1]); +} + +} + void EncStack::declare(WeightLoader & L, const char * prefix, int64_t layers, const EncNames & n) { blk.resize(layers); ln_spec.eps = cfg.ln_eps; @@ -64,6 +87,8 @@ void EncStack::declare(WeightLoader & L, const char * prefix, int64_t layers, co w.q_fc1 = L.quant_i8(b + n.fc1 + ".q8", {b + n.fc1 + ".weight"}); w.q_fc2 = L.quant_i8(b + n.fc2 + ".q8", {b + n.fc2 + ".weight"}); } else { + const std::string b = std::string(prefix) + "." + std::to_string(i) + "."; + pad_mlp(L, b + n.fc1, b + n.fc2); w.Wo = L.gemm("%s.%lld.attn_o.weight", prefix, (long long)i); w.Wfc1 = L.gemm("%s.%lld.%s.weight", prefix, (long long)i, n.fc1); w.Wfc2 = L.gemm("%s.%lld.%s.weight", prefix, (long long)i, n.fc2); diff --git a/src/modules/gemma_expert.h b/src/modules/gemma_expert.h index 1c66fe3f..e7641166 100644 --- a/src/modules/gemma_expert.h +++ b/src/modules/gemma_expert.h @@ -525,8 +525,8 @@ inline ggml_tensor * gemma_attn( ggml_tensor * K_full = k_rope; ggml_tensor * V_full = v_h; if (cached_K && cached_V) { - K_full = ggml_concat(ctx, cached_K, k_rope, 2); - V_full = ggml_concat(ctx, cached_V, v_h, 2); + K_full = concat_dim2(ctx, cached_K, k_rope); + V_full = concat_dim2(ctx, cached_V, v_h); } const float scale = 1.f/std::sqrt((float) hd);