hexagon: explicit attention on the HMX on the IQ-10

CHANGELOG.md

@@ -35,8 +35,13 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke
   reference (1.0e-3 served): pads that read a weight, also answered
   `NO_SUPPORT`, run on the CPU, and its ResNet's im2col matrices are
   zero-padded to a multiple of 32 so the convs' matmuls take the HMX (152.9 ms,
-  170.8 ms on the CPU). GR00T N1.5 and N1.6 are above the bar on a served LIBERO
-  observation, as the CPU is at F16. See
+  170.8 ms on the CPU). Explicit attention takes F16 K/V with its head dim and
+  keys zero-padded to a multiple of 32, the padding masked out, so both matmuls
+  run on the HMX: OpenVLA-OFT's LM, Octo's block transformer, Evo-1's DiT
+  cross-attention and GR00T's DiT (OpenVLA-OFT 35% faster, Evo-1 29%, Octo
+  29%, GR00T N1.5 11%); VLA-Adapter's LM only for V. Fidelity is measured
+  against a CPU-only build of the same commit. GR00T N1.5 and N1.6 are above
+  the bar on a served LIBERO observation, as the CPU is at F16. See
   `docs/backend/hexagon.md` for the build, the skel install and the numbers.
 - **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP
   backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200`

README.md

@@ -105,12 +105,6 @@ Use [ggml-compiler](https://github.com/khanhnd61-vr/ggml-compiler) and follow [t
 
 Legend: `Y` = supported (released and benchmarked), `~` = in progress, `-` = planned.
 
-The Hexagon column is the Dragonwing IQ-10 under Linux
-([docs/backend/hexagon.md](docs/backend/hexagon.md)). On a Snapdragon X laptop
-under Windows, π0 and VLA-JEPA land above the 2.9e-3 fidelity bar, OpenVLA-OFT
-was not attempted (about 14 GB at F16, over its 8 GB budget) and ACT was not run
-([docs/backend/hexagon-windows.md](docs/backend/hexagon-windows.md)).
-
 ---
 
 ## Rollout on a real robot

docs/backend/hexagon-windows.md

@@ -130,7 +130,7 @@ PyTorch and LIBERO; the smoke test used a 20-line pyzmq client instead.
 
 | Variable / flag | Effect in the Hexagon and OpenCL builds |
 |---|---|
-| `VLA_DEVICE=cpu` | Skip the accelerator. The same binary then produces the CPU reference. |
+| `VLA_DEVICE=cpu` | Skip the accelerator. Not a fidelity reference: the Hexagon build's NPU changes (F16 K/V, padding) run on the CPU too, so compare against a CPU-only build with `--weight-dtype f32 --flash-attn 0`. |
 | `VLA_DEVICE=<n>` | Registry device index: an HTP session, or an OpenCL GPU |
 | `--weight-dtype f16\|f32\|bf16` | Resident dtype for GEMM weights. New: `f16`, which is also the default on both accelerators. |
 | `--flash-attn on\|off` | On by default in Hexagon builds, off elsewhere |

docs/backend/hexagon.md

@@ -143,11 +143,11 @@ the same path.
 
 Flash attention on the HMX (v81) keeps scores and probabilities in F16. That is
 a precision limit rather than a wrong op, but in GR00T N1.7's LM it moves the
-action chunk 5.4e-3 from the CPU reference. N1.7's LM therefore takes the
+action chunk 6.2e-3 from the CPU reference. N1.7's LM therefore takes the
 explicit path unless `--flash-attn` is given: 2.0e-3, at the same latency.
 VLA-JEPA's LM does the same: with flash attention a served LIBERO observation
-lands 6.7e-3 from the reference (the CPU with flash attention 5.5e-3), without
-it 2.6e-3, at a cost of 683 to 810 ms with two views.
+lands 6.2e-3 from the reference, without it 2.2e-3, at a cost of 683 to 810 ms
+with two views.
 
 The softmax takes rows of up to 32 elements or a multiple of 32. Where a
 sequence can grow without changing its result, vla.cpp pads it to
@@ -165,7 +165,7 @@ tower takes two camera views that way: its action chunk lands 0.16 from the CPU
 reference. The wrapper runs those matmuls on the CPU. On Hexagon `linear()` in
 `src/layers/linear.h` multiplies a contiguous batch as one 2D matrix of rows
 instead, the same product, so most of them never reach the rule: GR00T N1.6
-with two views lands 8.8e-4 from the reference, and the matmuls of TurboVLA's
+with two views lands 7.5e-4 from the reference, and the matmuls of TurboVLA's
 DINOv3, which takes its two views as a batch, run on the NPU.
 
 A copy that transposes the innermost dim, such as the `CONT` of a permuted V
@@ -196,6 +196,16 @@ Hexagon builds:
   311 ms.
 - **Evo-1's LM attention multiplied F32 K and V against 14 query heads.** K and
   V now go to F16 (`fa_kv()`); the scores still accumulate in F32.
+- **Explicit attention elsewhere multiplied F32 K and V too**, and its key
+  counts and head dims are often not multiples of 32. OpenVLA-OFT's LM (330
+  keys), Octo's block transformer, Evo-1's DiT cross-attention (head dim 112,
+  1025 keys) and GR00T's DiT (head dim 48) now take F16 K/V with the head dim
+  and the keys zero-padded to `gemm_width()` and the padded keys masked out
+  (`hmx_kv()` in `dit_head.cpp`): OpenVLA-OFT 1725 to 1120 ms, Evo-1 2348 to
+  1661 ms, Octo-Small 119 to 85 ms, GR00T N1.5 595 to 532 ms. Two LMs keep part
+  of it in F32: in VLA-Adapter's LM only V goes to F16, since F16 scores put the
+  chunk 3.9e-3 off, and VLA-JEPA's cross-attention K/V overflow F16 (NaN
+  actions), so only its DiT self-attention takes the HMX.
 - **ACT's ResNet convs are im2col and one matmul** with the patches as its
   first operand. The stem's K of 147 and layer3's and layer4's 1200 and 300
   output positions (640x480) are not multiples of 32: 80 ms of the 101 ms spent
@@ -206,10 +216,10 @@ Hexagon builds:
   benchmark's protocol ACT takes 152.9 ms on the NPU against 170.8 ms on the
   CPU.
 
-What is left on the vector units: GR00T's DiT attention (head dim 48) and Evo-1's
-DiT cross-attention (head dim 112, 1025 keys), whose shapes the HMX does not
-take, and Evo-1's InternViT attention, which stays on the CPU because F16 K/V
-there measured 3.9e-2.
+What is left on the vector units: VLA-Adapter's QK scores and VLA-JEPA's LM and
+DiT cross-attention scores, which stay F32 for the reasons above, and Evo-1's
+InternViT attention, which stays on the CPU because F16 K/V there measured
+3.9e-2.
 
 ## What changed in `vla.cpp`
 
@@ -220,7 +230,8 @@ The shape of the diff was predicted correctly, apart from the scheduler:
 2. **A rung in [src/backend.h](../../src/backend.h).**
    - It goes through the registry, because `ggml_backend_hexagon_init` is
      declared but not defined upstream.
-   - `VLA_DEVICE` indexes HTP sessions, and `VLA_DEVICE=cpu` gives the reference.
+   - `VLA_DEVICE` indexes HTP sessions, and `VLA_DEVICE=cpu` skips them. That
+     run is not a fidelity reference (see [Fidelity](#fidelity)).
    - It sets the defaults above and points `ADSP_LIBRARY_PATH` at the
      executable's folder.
 3. **Weights.**
@@ -254,9 +265,11 @@ found.
    to the CPU and how much was copied (`VLA_FALLBACK_STATS=1` for per-graph
    detail).
 4. **Fidelity before latency.**
-   - Compare `vla_predict_check` output against the same binary with
-     `VLA_DEVICE=cpu`, and against `--weight-dtype f32`. Add `--flash-attn 0`
-     to the reference: a Hexagon build defaults to flash attention on the CPU too.
+   - Compare `vla_predict_check` output against a CPU-only build
+     (`-DGGML_HEXAGON=OFF`) of the same commit, run with `--weight-dtype f32
+     --flash-attn 0`. The Hexagon binary with `VLA_DEVICE=cpu` is not a
+     reference: its NPU changes (flash attention, F16 K/V, padding to
+     `gemm_width()`) run on the CPU too.
    - The bar is 2.9e-3.
    - A run with a wrong kernel still prints a latency, and it is usually a good
      one.
@@ -355,30 +368,31 @@ export LD_LIBRARY_PATH=$PWD/pkg/lib
 
 ### Fidelity
 
-`vla_predict_check` at the Hexagon defaults, against the same binary with
-`VLA_DEVICE=cpu --weight-dtype f32 --flash-attn 0`. A Hexagon build turns flash
-attention on by default even on the CPU, so the reference asks for the explicit
-path that other builds run. Views are the ones the LIBERO client sends, so they
+`vla_predict_check` at the Hexagon defaults, against a CPU-only build
+(`-DGGML_HEXAGON=OFF`) of the same commit with `--weight-dtype f32
+--flash-attn 0`. The Hexagon build's own CPU run is not a reference: what it
+changes for the NPU (F16 K/V, padding to `gemm_width()`) is compiled in and runs
+on the CPU too. Views are the ones the LIBERO client sends, so they
 differ from the benchmark's for VLA-JEPA, VLA-Adapter, OpenVLA-OFT and Octo:
 
 | Model | Views | max\|Δ\| |
 |---|--:|--:|
-| SmolVLA | 2 | 7.2e-4 |
-| π0 | 2 | 1.2e-3 |
+| SmolVLA | 2 | 7.4e-4 |
+| π0 | 2 | 7.7e-4 |
 | π0.5 | 2 | 4.8e-4 |
-| Evo-1 | 1 | 5.5e-4 |
-| GR00T N1.5 | 1 | 2.4e-3 |
-| GR00T N1.6 | 1 | 1.4e-3 |
+| Evo-1 | 1 | 6.2e-4 |
+| GR00T N1.5 | 1 | 1.9e-3 |
+| GR00T N1.6 | 1 | 1.3e-3 |
 | GR00T N1.7 | 1 | 2.0e-3 |
-| VLA-JEPA | 2 | 2.2e-3 |
-| VLA-Adapter | 2 | 5.4e-4 |
-| OpenVLA-OFT | 2 | 3.2e-4 |
-| Octo-Small | 1 | 5.3e-4 |
+| VLA-JEPA | 2 | 2.1e-3 |
+| VLA-Adapter | 2 | 3.9e-4 |
+| OpenVLA-OFT | 2 | 3.1e-4 |
+| Octo-Small | 1 | 5.0e-4 |
 | TurboVLA | 2 | 3.7e-4 |
 | ACT | 1 | 6.1e-4 |
 
-GR00T N1.6 with two views is 8.8e-4. At the benchmark's single view VLA-JEPA is
-1.9e-3, VLA-Adapter 5.7e-4 and OpenVLA-OFT 2.1e-4. ACT takes its one 640x480
+GR00T N1.6 with two views is 7.5e-4. At the benchmark's single view VLA-JEPA is
+1.8e-3, VLA-Adapter 4.8e-4 and OpenVLA-OFT 1.8e-4. ACT takes its one 640x480
 view, the benchmark's input.
 
 ### Serving
@@ -387,31 +401,32 @@ view, the benchmark's input.
 `eval/client/vla_cpp_client.py` and the adapters `run_sim_client_direct.py`
 uses, on one fixed LIBERO observation (two views, the task prompt, a fixed robot
 state), with fixed noise for the archs that draw it. The same request goes to a
-Hexagon server and to a CPU server with `--weight-dtype f32 --flash-attn 0`.
+Hexagon server and to a server from the CPU-only build with `--weight-dtype f32
+--flash-attn 0`.
 Every model returns a finite chunk, the same chunk for a repeated request, an
 error reply to a malformed or an invalid request, and keeps serving afterwards.
 ACT has no `eval/client` preset; it gets the same checks from raw `vla.proto`
 requests carrying one fixed 640x480 view and a fixed 14-dim state.
 
 | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 |
 |---|--:|--:|
-| SmolVLA | 1.3e-3 | |
-| π0.5 | 3.5e-4 | |
+| SmolVLA | 1.9e-3 | |
+| π0.5 | 3.7e-4 | |
 | Evo-1 | 1.1e-3 | |
-| GR00T N1.5 | 1.1e-2 | 4.3e-2 |
-| GR00T N1.6 | 1.3e-2 | 1.1e-2 |
-| GR00T N1.7 | 2.5e-3 | |
+| GR00T N1.5 | 1.2e-2 | 2.8e-2 |
+| GR00T N1.6 | 1.0e-2 | 9.0e-3 |
+| GR00T N1.7 | 2.0e-3 | |
 | π0 | 5.8e-4 | |
-| VLA-JEPA | 2.6e-3 | 2.1e-2 |
-| VLA-Adapter | 5.4e-4 | |
-| OpenVLA-OFT | 5.9e-4 | |
-| Octo-Small | 2.4e-4 | |
+| VLA-JEPA | 2.2e-3 | |
+| VLA-Adapter | 3.9e-4 | |
+| OpenVLA-OFT | 7.5e-4 | |
+| Octo-Small | 1.4e-4 | |
 | TurboVLA | 8.5e-4 | |
 | ACT | 1.0e-3 | |
 
 GR00T N1.5 and N1.6 are above the bar on this input, and so is the CPU itself
 with F16 weights: N1.5 lands closer to the reference than the CPU at F16, N1.6
-slightly further (1.3e-2 against 1.1e-2). They are sensitive to 16-bit precision
+slightly further (1.0e-2 against 9.0e-3). They are sensitive to 16-bit precision
 rather than miscomputed. Running them with F32 weights on the NPU has not been
 tried yet.
 

docs/benchmark/README.md

@@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path
 ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with
 fixed-input checks against the CPU build in
 [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at
-`7c76ea9`, its ACT cell at `0f3e010` (same llama.cpp pin). Each report
+`bd166b8` (same llama.cpp pin). Each report
 lists the device, the build, each model's configuration, latency and peak memory
 at the defaults, and the fastest runtime flags for that device.
 
@@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why.
 | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | n/r | 28.5 | 116.3 | 170.2 | 106.0 | 122.8 | 131.4 | N/A | 213.1 | 620.5 | 232.5 | 1262.5 | 566.0 |
 | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | n/r | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 |
 | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | n/r | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 |
-| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 117.6 | 148.5 | 188.1 | 505.4 | 703.2 | 591.6 | 696.5 | 759.5 | N/A | 583.6 | 1114.0 | 1113.0 | 1720.3 | 2333.6 |
+| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 81.7 | 147.8 | 184.6 | 489.7 | 664.0 | 530.2 | 680.9 | 739.6 | N/A | 583.3 | 1117.6 | 1114.0 | 1117.7 | 1653.5 |
 | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | n/r | 23.5 | 90.7 | 126.7 | 134.0 | 161.1 | 144.9 | N/A | 161.3 | 438.3 | 430.2 | 405.8 | 256.3 |
 | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | n/r | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 |
 | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | n/r | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 |