Repository navigation
hexagon: explicit attention on the HMX on the IQ-10 - #77
Merged
Merged
Conversation
vla-bench under TEMPLATE.md's protocol at e8dfa34, for the six models whose GGUFs are on the board. The flag screen keeps F16 weights and varies only --flash-attn and --mm-prec: the NPU has no BF16 kernels, and F16 is the default. Memory adds the drop in system MemAvailable, since the weights sit in FastRPC buffers that peak RSS leaves out. The CPU section and the archs without GGUFs on the board are left for a later round.
Minimum latency at the defaults from qualcomm-iq10.md, measured at e8dfa34. Models whose GGUFs were not on the board are marked n/r; BitVLA is N/A off CUDA.
ggml-hexagon takes an F16 weight on the matrix unit only when both dims are multiples of 32. SigLIP-so400m's MLP is 4304 wide, so fc1 and fc2 ran on the vector units: 440 ms of GR00T N1.5's 565 ms vision tower on the IQ-10. WeightLoader::pad() declares a float weight larger than in the file, with the extra elements zero, and EncStack pads fc1, its bias and fc2 to gemm_width(), which rounds up to 32 in Hexagon builds and is the identity elsewhere. The padded units are zero through GELU and fc2, so the block's output is the same. On the IQ-10 against the unpadded CPU F32 reference: pi0.5 2926 -> 2122 ms (3.1e-4), GR00T N1.5 1024 -> 610 ms (2.4e-3), N1.6 1117 -> 709 ms (1.4e-3). SmolVLA, Evo-1 and N1.7 do not use this MLP and are unchanged.
pi0.5 ties SigLIP's flash attention to a view-taking kernel, which only SYCL has, so on Hexagon its tower ran the explicit scores: an F32 x F32 matmul at head dim 72 that the HMX does not take, 266 ms on the vector units. Where flash attention is on by default (Hexagon), SigLIP now takes the plain flash attention that GR00T's identical tower already uses. Other backends are unchanged. IQ-10: 2118 -> 1833 ms, 2.8e-4 from the CPU F32 reference.
The LM's explicit attention multiplies F32 K and V against a batch of 14 query heads. ggml-hexagon's HMX takes no batched F32 src0, so both matmuls ran on the vector units, about 1 s per chunk on the IQ-10. On Hexagon K and V now go to F16 through fa_kv(); the scores still accumulate in F32 and the softmax stays F32. Other backends keep F32 K/V. IQ-10: 3261 -> 2336 ms, 4.3e-4 from the CPU F32 reference (5.2e-4 before). The InternViT flash attention stays on the CPU: F16 K/V there measured 3.9e-2.
hexagon.md: which matmuls the HMX takes, the three that now move onto it, what is left on the vector units, and why the GELU_ERF kernel was left out. The fidelity and serving tables are re-measured at this commit. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 74e145d: GR00T N1.5 613.6 ms, N1.6 706.9, pi0.5 1789.4, Evo-1 2589.2; SmolVLA and N1.7 unchanged. The defaults stay the fastest configuration.
VinRobotics#70 was squash-merged into main; its two commits are already on this branch, and the IQ-10 benchmark page and README row keep this branch's re-measured numbers.
The VLM's and the expert's attention took flash attention only on a view-taking kernel (SYCL). On Hexagon they ran explicit scores against a single KV head, which the fallback wrapper sends to the CPU. Where flash attention is on by default (Hexagon), all three sites now take the plain flash attention; the view-only paths keep use_fused_attention(). IQ-10: 1773 -> 1320 ms, 4.8e-4 from the CPU F32 reference, 3.5e-4 on a served LIBERO observation. Other models unchanged.
The NPU runs a concat only as a dim-1 copy of whole rows, so the expert's dim-2 KV concats (π0, π0.5, SmolVLA) went to the CPU fallback every step. concat_dim2() in backend.h does them on Hexagon as a dim-1 concat of the [d0*d1, n] views, which writes the same bytes; other backends keep ggml_concat(a, b, 2). IQ-10: π0.5 1379 -> 1133 ms, SmolVLA 810 -> 638 ms, both with outputs identical to before on fixed inputs and on a served observation.
…rk at 0b78eb9 hexagon.md: π0.5's VLM and action expert take flash attention on Hexagon, and the experts' KV concats run as row copies that stay on the NPU (π0 shares the code but has not been run on the board). π0.5's fidelity is re-measured: 4.8e-4 on fixed inputs, 3.5e-4 on a served observation. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 0b78eb9: π0.5 1117.2 ms, SmolVLA 633.0 ms; the other models are unchanged. The defaults stay the fastest configuration.
ggml-hexagon claims a CPY whose source is a weight, then answers the batch with NO_SUPPORT and the host aborts. VLA-Adapter casts its F16 action queries to F32 in the graph and aborted on every predict on the Dragonwing IQ-10. With the cast on the CPU it is within 4.2e-4 of the CPU F32 reference, 4.5e-4 when served.
The NPU's REPEAT reads its source through the data cache, which is invalidated only between op batches, while ops write their output by DMA. When the source is written earlier in the same batch, REPEAT can read stale lines. TurboVLA's ViT prefix, repeated over two views, came out 0.19 off the CPU F32 reference and different on every run; with the repeat on the CPU it is 3.1e-4 and repeatable, 2.2e-4 when served.
…file pad_mlp() checked only the loader's GEMM type. scripts/quantize_gguf.py --vision packs SigLIP's fc1 [1152, 4304] to Q8_0 while fc2, 4304 wide, stays float; the loader keeps fc1 packed and declare() refused to pad it, so such a GGUF failed to load in Hexagon builds. An MLP with either weight quantized in the file now loads unpadded, as it did before the padding. Float checkpoints are unchanged: π0, π0.5, GR00T N1.5/N1.6/N1.7, VLA-Adapter and OpenVLA-OFT give bit-identical actions on the Dragonwing IQ-10.
… summary The previous docs commit added six archs to the IQ-10 pages but left two summaries behind: the status note in docs/backend/hexagon.md still said six checkpoints run on the IQ-10, and the IQ-10 row of docs/benchmark/README.md still carried the 0b78eb9 numbers with the six new archs marked not run. Both now match docs/benchmark/qualcomm-iq10.md at 1cab1c3.
OpenVLA-OFT and VLA-Adapter declare their DINOv2 and SigLIP towers in modules/dual_tower.h rather than through EncStack, so the zero padding of SigLIP's 4304-wide MLP did not reach them, and its fc1 and fc2 ran on the vector units: 425 ms per predict on the Dragonwing IQ-10. pad_mlp() moves into WeightLoader so both declare paths use it. VLA-Adapter goes from 1752 to 1352 ms and OpenVLA-OFT from 4108 to 3711 ms (mean, vla-bench), within 4.8e-4 and 3.4e-4 of the CPU F32 reference and 6.7e-4 / 5.6e-4 when served. The other ten archs give bit-identical actions.
The towers' explicit attention multiplies F32 scores on the NPU's vector units, sends the softmax to the CPU and transposes V with a strided copy. Where flash attention is on by default (Hexagon) they now take it, as π0.5's SigLIP does; other backends keep the explicit path. On the Dragonwing IQ-10 the vision stage goes from 790 to 311 ms: VLA-Adapter from 1352 to 872 ms and OpenVLA-OFT from 3711 to 3235 ms (mean, vla-bench), within 5.6e-4 and 2.9e-4 of the CPU F32 reference and 6.1e-4 / 5.2e-4 when served. The other ten archs give bit-identical actions.
The NPU computes one 2D weight against a batch of activations wrong, so the fallback wrapper runs those matmuls on the CPU. On Hexagon, linear() now multiplies a contiguous batch as one 2D matrix of rows and reshapes the result back, which is the same product and stays on the NPU. Other backends are unchanged. On the Dragonwing IQ-10 TurboVLA's DINOv3 over two views leaves the CPU: 433 to about 296 ms (mean, vla-bench, 5 reps), within 3.7e-4 of the CPU F32 reference and 8.5e-4 when served. GR00T N1.6 with two views moves from 1.9e-3 to 8.8e-4. The other archs give bit-identical actions.
A CONT whose source is strided in its innermost dim is a gather on the NPU and takes tens of milliseconds for a large tensor: 48 ms for OpenVLA-OFT's [330,128,32] V transpose, 32 ms for Evo-1's [1025,112,8], 18 ms for TurboVLA's [512,256,4]. Those of 256K elements or more now run on the CPU. Smaller ones take tens of microseconds on the NPU, less than the trip to the CPU, and stay there. On the Dragonwing IQ-10 (mean, vla-bench, 5 reps): OpenVLA-OFT 3247 to 1763 ms, TurboVLA 300 to 196 ms, Evo-1 2605 to 2357 ms, SmolVLA 639 to 585 ms, GR00T N1.5, π0 and π0.5 2-3% faster, the rest within 1.5%. A copy computes the same bytes on either side: all twelve archs give bit-identical actions.
Where flash attention is the default (Hexagon), the LM's F16 K/V put a served LIBERO request 6.7e-3 from the CPU F32 reference on the Dragonwing IQ-10, above the 2.9e-3 bar; the CPU with flash attention on is 5.5e-3. With the explicit path it is 2.6e-3 served and 2.2e-3 on the fixed input (was 2.8e-3), at a cost of 683 to 810 ms with two views (mean, vla-bench). --flash-attn 1 turns it back on. Other backends are unchanged and the other eleven archs give bit-identical actions.
The LM's softmax rows were 337 wide, which the NPU refuses, so every layer's softmax ran on the CPU. On Hexagon the LM now runs on gemm_width(SEQ) tokens with zeros appended; the mask is causal, so no real token sees them, and the conditioning keeps the first SEQ rows. Other backends run the same sequence as before. On the Dragonwing IQ-10, VLA-Adapter goes from 859 to 830 ms (mean, vla-bench, one view), 5.7e-4 from the CPU F32 reference on the fixed input and 5.0e-4 served.
Each head block scored its own tokens, the action queries and the vision tokens separately and concatenated the scores and the values along their columns, which the NPU refuses, then took a 329-wide softmax it also refuses. Where gemm_width() pads (Hexagon), the keys and values are now stacked as rows and padded to a multiple of 32, the padding masked out, with rg on the vision keys instead of their scores (RoPE is linear). Other backends keep the previous graph. On the Dragonwing IQ-10, VLA-Adapter goes from 830 to 747 ms (mean, vla-bench, one view), 5.1e-4 from the CPU F32 reference on the fixed input and 5.3e-4 served.
ggml-hexagon has no GELU_ERF, so DINOv2's MLP and the projector sent their activations through the CPU in every layer. backend.h gains gelu_erf(): on Hexagon x*sigmoid(p(x)), p an odd cubic in x clamped to [-5, 5] fitted to logit(Phi(x)), within 2.5e-5 of the exact GELU for every x; elsewhere ggml_gelu_erf, so other backends build the same graph. On the Dragonwing IQ-10, VLA-Adapter goes from 740 to 705 ms and OpenVLA-OFT from 1771 to 1740 ms (mean, vla-bench, one view), within 6.2e-4 / 3.2e-4 of the CPU F32 reference on the fixed input and 5.4e-4 / 5.9e-4 served.
…agon Its softmax rows were 353 wide, which the NPU refuses, so every block's softmax ran on the CPU. Where gemm_width() pads (Hexagon), zero tokens are appended and masked out as keys; the readouts are gathered by index, so the extra rows never reach them. Other backends run the same sequence as before. On the Dragonwing IQ-10, Octo-Small goes from 126.5 to 122.6 ms (mean, vla-bench, two views), 5.3e-4 from the CPU F32 reference on the fixed input and 2.4e-4 served.
…rk at 7c76ea9 hexagon.md: VLA-JEPA's LM leaves flash attention as N1.7's does (6.7e-3 to 2.6e-3 served); VLA-Adapter's LM and action head and Octo's block transformer pad their softmax rows to a multiple of 32; VLA-Adapter's and OpenVLA-OFT's DINOv2 build GELU (erf) from ops the NPU runs. Fidelity and serving tables re-measured: only GR00T N1.5 and N1.6 stay above the bar served. qualcomm-iq10.md and the summary row re-measured at 7c76ea9: VLA-Adapter 853.7 to 708.4 ms, Octo-Small 124.0 to 119.4 ms, OpenVLA-OFT 1768.2 to 1725.5 ms, VLA-JEPA 452.9 to 512.6 ms.
The NPU claims a PAD whose source is a weight, answers NO_SUPPORT, and the host aborts, as it does for a weight-reading CPY. The wrapper now runs those pads on the CPU. On the Dragonwing IQ-10 the twelve archs already on the NPU give the same actions as before (13 fixed-input configurations against the CPU F32 reference).
The HMX takes a matmul only when K and the rows of its first operand are multiples of 32. ACT's stem (K = 147) and layer3 and layer4 (1200 and 300 output positions at 640x480) ran their convs' matmuls on the vector units: 80 ms of the 101 ms spent on the NPU. Where gemm_width() pads (Hexagon), for one view, the im2col matrix and the stem's kernel are zero-padded and the extra output rows dropped, which leaves the product unchanged. Other backends and multi-view inputs keep conv_2d_f32. On the Dragonwing IQ-10, ACT goes from 228.5 to 155.6 ms (mean, vla-bench, one 640x480 view; the same build on the CPU 170.8 ms), 6.1e-4 from the CPU F32 reference on the fixed input and 1.0e-3 served.
hexagon.md: pads that read a weight run on the CPU; ACT's ResNet convs pad their patch matrices for the HMX; ACT in the fidelity (6.1e-4) and serving (1.0e-3) tables. qualcomm-iq10.md: ACT's configuration, latency (152.9 ms mean, 170.8 ms on the CPU) and flag screen at 0f3e010, and the 7c76ea9 run's date given in UTC+07 (the board's clock is UTC). CHANGELOG: ACT.
The LM's explicit attention multiplied F32 K and V against 32 heads on the NPU's vector units, 626 ms per predict on the Dragonwing IQ-10, and 330 keys are not a width the HMX takes. On Hexagon K and V now go to F16 (fa_kv()) and the keys are padded with zeros to gemm_width(SEQ), masked out in the softmax; the queries keep SEQ rows. Scores accumulate in F32 and the softmax stays F32. Other backends keep F32 K/V and SEQ keys. OpenVLA-OFT goes from 1729 to 1127 ms (mean, vla-bench, one view), 3.1e-4 from the CPU F32 reference on the fixed input and 7.5e-4 served.
README: the Hexagon column follows the Dragonwing IQ-10 under Linux, where π0, VLA-JEPA, OpenVLA-OFT and ACT run within the 2.9e-3 bar and are benchmarked; a note gives the Snapdragon X laptop's state, and the column and the backend list link docs/backend/hexagon.md, which covers both. docs/benchmark/README.md: an ACT column, 148.5 ms on the IQ-10 at 0f3e010 and n/r elsewhere.
The LM's explicit attention multiplied F32 K and V on the NPU's vector units. On Hexagon V now goes to F16 (fa_kv()), so the PV matmul runs on the HMX. K stays F32: F16 scores put the action chunk 3.9e-3 from the CPU F32 reference on the Dragonwing IQ-10 (1.2e-2 served). Other backends keep F32 V. VLA-Adapter goes from 710 to 672 ms (mean, vla-bench, one view), 4.5e-4 from the CPU F32 reference on the fixed input and 3.9e-4 served.
The block transformer's explicit attention multiplied F32 K and V on the NPU's vector units. On Hexagon K and V now go to F16 (fa_kv()), so both matmuls run on the HMX; the sequence is already padded to a multiple of 32. Scores accumulate in F32 and the softmax stays F32. Other backends keep F32 K/V. On the Dragonwing IQ-10, Octo-Small goes from 120.1 to 86.9 ms (mean, vla-bench, two views), 5.0e-4 from the CPU F32 reference on the fixed input and 1.4e-4 served.
Each of the 32 denoising steps scored 50 action queries against the 1025-row context as F32 x F32 matmuls on the NPU's vector units: head dim 112 and 1025 keys are not widths the HMX takes, 574 ms per predict on the Dragonwing IQ-10. On Hexagon the head dim and the keys are now padded with zeros to gemm_width() (128, 1056), the padded keys masked out, and K and V go to F16 once per predict. Scores accumulate in F32 and the softmax stays F32. Other backends build the same graph. Evo-1 goes from 2346 to 1660 ms (mean, vla-bench), 5.9e-4 from the CPU F32 reference on the fixed input and 2.1e-3 served (2.1e-3 before).
GR00T's DiT attention multiplied F32 K and V on the NPU's vector units: head dim 48 and 41-272 keys are not widths the HMX takes. On Hexagon hmx_kv() pads the head dim and the keys with zeros to gemm_width() and casts K and V to F16; the queries take the head-dim padding, the padded keys are masked out (one mask per shape per graph) and the output drops the padded columns. DitCfg::cross_kv_f16 keeps cross-attention K/V F32 where they overflow F16: VLA-JEPA's give NaN actions otherwise. Other backends build the same graph. On the Dragonwing IQ-10 (mean, vla-bench): GR00T N1.5 598 to 525 ms, N1.6 700 to 678 ms, N1.7 756 to 746 ms, VLA-JEPA 515 to 499 ms; against a CPU-only F32 build at the same commit, 1.9e-3, 1.3e-3, 2.0e-3 and 2.1e-3 on fixed inputs, and N1.7 2.0e-3, VLA-JEPA 2.2e-3 served.
…ark at bd166b8 hexagon.md: OpenVLA-OFT's LM, Octo's block transformer, Evo-1's DiT cross-attention and GR00T's DiT take F16 K/V with the head dim and keys padded to gemm_width(); VLA-Adapter's LM only for V, VLA-JEPA's cross-attention stays F32. Fidelity and serving are re-measured against a CPU-only build of the same commit, since a Hexagon build's CPU run carries the NPU-only changes. qualcomm-iq10.md and the summary row at bd166b8: OpenVLA-OFT 1725.5 to 1120.2 ms, Evo-1 2348.3 to 1661.4 ms, Octo-Small 119.4 to 84.6 ms, GR00T N1.5 594.9 to 532.0 ms.
TEMPLATE.md sorts the latency and fastest-flag tables by min; they were sorted by mean (π0 and OpenVLA-OFT swapped). The fidelity quoted for the two non-default fastest sets is re-measured against the CPU-only build: VLA-JEPA with --flash-attn 6.2e-3 served, GR00T N1.7 with --flash-attn 6.2e-3 on fixed inputs, both outside the bar. The BF16 timing (SmolVLA 18.6 s against 0.58 s), N1.6's two-view fidelity (7.5e-4) and its CPU fallback note are updated to bd166b8.
… stale VLA_DEVICE=cpu reference
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Profiling the IQ-10 at #76 showed explicit attention as the largest cost left on the NPU: QK and PV matmuls with F32 K and V run on the vector units (HVX), and many key counts and head dims are not the multiples of 32 the matrix unit (HMX) takes. On Hexagon this PR casts K and V to F16 and pads the head dim and the keys with zeros to
gemm_width(), masking the padded keys out, so both matmuls run on the HMX. Scores accumulate in F32 and the softmax stays F32. Other backends build the same graphs.Changes
07b4ca9b23efd90f684e2a51c789f0a670dDitHead(GR00T, VLA-JEPA):hmx_kv()pads head dim 48 to 64 and the keys, F16 K/V, one in-graph mask per shape.DitCfg::cross_kv_f16keeps VLA-JEPA's cross-attention K/V F32: they overflow F16 (NaN actions)bc10b41hexagon.md, CHANGELOG, IQ-10 benchmark atbd166b853de2fcReference. A Hexagon build's own CPU run (
VLA_DEVICE=cpu) is not a reference: what the build changes for the NPU (F16 K/V,gemm_width()padding) is compiled in and runs on the CPU too. Aftera51c789it put Evo-1 3.8e-3 from the NPU, against 6.2e-4 from a CPU-only build. The numbers below and in the docs are against a CPU-only build (-DGGML_HEXAGON=OFF) of the same commit with--weight-dtype f32 --flash-attn 0, for fixed inputs and served alike.IQ-10,
vla-benchunder TEMPLATE.md, mean msbd166b8)The bar is 2.9e-3. The fastest-flag screen keeps the defaults for 11 archs. For VLA-JEPA (
--flash-attn, −12%) and GR00T N1.7 (--flash-attn --mm-prec default, −4%) the faster set turns the LM's flash attention back on, which is outside the bar (VLA-JEPA 6.2e-3 served, N1.7 6.2e-3 on fixed inputs); the defaults are the fastest settings within it.Verified
vla_predict_check, GR00T N1.6 with two views too), and a repeated run gives the same output.vla-server,eval/client, rawvla.protofor ACT), each arch returns a finite chunk, the same chunk for a repeated request, an error reply to a malformed and to an invalid request, and keeps serving. GR00T N1.5 (1.2e-2) and N1.6 (1.0e-2) stay above the bar on it, as the CPU-only build at F16 does (2.8e-2, 9.0e-3).ctestpasses (10/10).Tried and left out
Qwen3LM, shared with the GR00T LMs): would need key padding in the shared module, with GR00T N1.7 at 2.0e-3, for about 40 ms.