Repository navigation
hexagon: speed up the new archs on the IQ-10 and run ACT - #76
Merged
Merged
Conversation
vla-bench under TEMPLATE.md's protocol at e8dfa34, for the six models whose GGUFs are on the board. The flag screen keeps F16 weights and varies only --flash-attn and --mm-prec: the NPU has no BF16 kernels, and F16 is the default. Memory adds the drop in system MemAvailable, since the weights sit in FastRPC buffers that peak RSS leaves out. The CPU section and the archs without GGUFs on the board are left for a later round.
Minimum latency at the defaults from qualcomm-iq10.md, measured at e8dfa34. Models whose GGUFs were not on the board are marked n/r; BitVLA is N/A off CUDA.
ggml-hexagon takes an F16 weight on the matrix unit only when both dims are multiples of 32. SigLIP-so400m's MLP is 4304 wide, so fc1 and fc2 ran on the vector units: 440 ms of GR00T N1.5's 565 ms vision tower on the IQ-10. WeightLoader::pad() declares a float weight larger than in the file, with the extra elements zero, and EncStack pads fc1, its bias and fc2 to gemm_width(), which rounds up to 32 in Hexagon builds and is the identity elsewhere. The padded units are zero through GELU and fc2, so the block's output is the same. On the IQ-10 against the unpadded CPU F32 reference: pi0.5 2926 -> 2122 ms (3.1e-4), GR00T N1.5 1024 -> 610 ms (2.4e-3), N1.6 1117 -> 709 ms (1.4e-3). SmolVLA, Evo-1 and N1.7 do not use this MLP and are unchanged.
pi0.5 ties SigLIP's flash attention to a view-taking kernel, which only SYCL has, so on Hexagon its tower ran the explicit scores: an F32 x F32 matmul at head dim 72 that the HMX does not take, 266 ms on the vector units. Where flash attention is on by default (Hexagon), SigLIP now takes the plain flash attention that GR00T's identical tower already uses. Other backends are unchanged. IQ-10: 2118 -> 1833 ms, 2.8e-4 from the CPU F32 reference.
The LM's explicit attention multiplies F32 K and V against a batch of 14 query heads. ggml-hexagon's HMX takes no batched F32 src0, so both matmuls ran on the vector units, about 1 s per chunk on the IQ-10. On Hexagon K and V now go to F16 through fa_kv(); the scores still accumulate in F32 and the softmax stays F32. Other backends keep F32 K/V. IQ-10: 3261 -> 2336 ms, 4.3e-4 from the CPU F32 reference (5.2e-4 before). The InternViT flash attention stays on the CPU: F16 K/V there measured 3.9e-2.
hexagon.md: which matmuls the HMX takes, the three that now move onto it, what is left on the vector units, and why the GELU_ERF kernel was left out. The fidelity and serving tables are re-measured at this commit. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 74e145d: GR00T N1.5 613.6 ms, N1.6 706.9, pi0.5 1789.4, Evo-1 2589.2; SmolVLA and N1.7 unchanged. The defaults stay the fastest configuration.
VinRobotics#70 was squash-merged into main; its two commits are already on this branch, and the IQ-10 benchmark page and README row keep this branch's re-measured numbers.
The VLM's and the expert's attention took flash attention only on a view-taking kernel (SYCL). On Hexagon they ran explicit scores against a single KV head, which the fallback wrapper sends to the CPU. Where flash attention is on by default (Hexagon), all three sites now take the plain flash attention; the view-only paths keep use_fused_attention(). IQ-10: 1773 -> 1320 ms, 4.8e-4 from the CPU F32 reference, 3.5e-4 on a served LIBERO observation. Other models unchanged.
The NPU runs a concat only as a dim-1 copy of whole rows, so the expert's dim-2 KV concats (π0, π0.5, SmolVLA) went to the CPU fallback every step. concat_dim2() in backend.h does them on Hexagon as a dim-1 concat of the [d0*d1, n] views, which writes the same bytes; other backends keep ggml_concat(a, b, 2). IQ-10: π0.5 1379 -> 1133 ms, SmolVLA 810 -> 638 ms, both with outputs identical to before on fixed inputs and on a served observation.
…rk at 0b78eb9 hexagon.md: π0.5's VLM and action expert take flash attention on Hexagon, and the experts' KV concats run as row copies that stay on the NPU (π0 shares the code but has not been run on the board). π0.5's fidelity is re-measured: 4.8e-4 on fixed inputs, 3.5e-4 on a served observation. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 0b78eb9: π0.5 1117.2 ms, SmolVLA 633.0 ms; the other models are unchanged. The defaults stay the fastest configuration.
ggml-hexagon claims a CPY whose source is a weight, then answers the batch with NO_SUPPORT and the host aborts. VLA-Adapter casts its F16 action queries to F32 in the graph and aborted on every predict on the Dragonwing IQ-10. With the cast on the CPU it is within 4.2e-4 of the CPU F32 reference, 4.5e-4 when served.
The NPU's REPEAT reads its source through the data cache, which is invalidated only between op batches, while ops write their output by DMA. When the source is written earlier in the same batch, REPEAT can read stale lines. TurboVLA's ViT prefix, repeated over two views, came out 0.19 off the CPU F32 reference and different on every run; with the repeat on the CPU it is 3.1e-4 and repeatable, 2.2e-4 when served.
…file pad_mlp() checked only the loader's GEMM type. scripts/quantize_gguf.py --vision packs SigLIP's fc1 [1152, 4304] to Q8_0 while fc2, 4304 wide, stays float; the loader keeps fc1 packed and declare() refused to pad it, so such a GGUF failed to load in Hexagon builds. An MLP with either weight quantized in the file now loads unpadded, as it did before the padding. Float checkpoints are unchanged: π0, π0.5, GR00T N1.5/N1.6/N1.7, VLA-Adapter and OpenVLA-OFT give bit-identical actions on the Dragonwing IQ-10.
… summary The previous docs commit added six archs to the IQ-10 pages but left two summaries behind: the status note in docs/backend/hexagon.md still said six checkpoints run on the IQ-10, and the IQ-10 row of docs/benchmark/README.md still carried the 0b78eb9 numbers with the six new archs marked not run. Both now match docs/benchmark/qualcomm-iq10.md at 1cab1c3.
OpenVLA-OFT and VLA-Adapter declare their DINOv2 and SigLIP towers in modules/dual_tower.h rather than through EncStack, so the zero padding of SigLIP's 4304-wide MLP did not reach them, and its fc1 and fc2 ran on the vector units: 425 ms per predict on the Dragonwing IQ-10. pad_mlp() moves into WeightLoader so both declare paths use it. VLA-Adapter goes from 1752 to 1352 ms and OpenVLA-OFT from 4108 to 3711 ms (mean, vla-bench), within 4.8e-4 and 3.4e-4 of the CPU F32 reference and 6.7e-4 / 5.6e-4 when served. The other ten archs give bit-identical actions.
The towers' explicit attention multiplies F32 scores on the NPU's vector units, sends the softmax to the CPU and transposes V with a strided copy. Where flash attention is on by default (Hexagon) they now take it, as π0.5's SigLIP does; other backends keep the explicit path. On the Dragonwing IQ-10 the vision stage goes from 790 to 311 ms: VLA-Adapter from 1352 to 872 ms and OpenVLA-OFT from 3711 to 3235 ms (mean, vla-bench), within 5.6e-4 and 2.9e-4 of the CPU F32 reference and 6.1e-4 / 5.2e-4 when served. The other ten archs give bit-identical actions.
The NPU computes one 2D weight against a batch of activations wrong, so the fallback wrapper runs those matmuls on the CPU. On Hexagon, linear() now multiplies a contiguous batch as one 2D matrix of rows and reshapes the result back, which is the same product and stays on the NPU. Other backends are unchanged. On the Dragonwing IQ-10 TurboVLA's DINOv3 over two views leaves the CPU: 433 to about 296 ms (mean, vla-bench, 5 reps), within 3.7e-4 of the CPU F32 reference and 8.5e-4 when served. GR00T N1.6 with two views moves from 1.9e-3 to 8.8e-4. The other archs give bit-identical actions.
A CONT whose source is strided in its innermost dim is a gather on the NPU and takes tens of milliseconds for a large tensor: 48 ms for OpenVLA-OFT's [330,128,32] V transpose, 32 ms for Evo-1's [1025,112,8], 18 ms for TurboVLA's [512,256,4]. Those of 256K elements or more now run on the CPU. Smaller ones take tens of microseconds on the NPU, less than the trip to the CPU, and stay there. On the Dragonwing IQ-10 (mean, vla-bench, 5 reps): OpenVLA-OFT 3247 to 1763 ms, TurboVLA 300 to 196 ms, Evo-1 2605 to 2357 ms, SmolVLA 639 to 585 ms, GR00T N1.5, π0 and π0.5 2-3% faster, the rest within 1.5%. A copy computes the same bytes on either side: all twelve archs give bit-identical actions.
Where flash attention is the default (Hexagon), the LM's F16 K/V put a served LIBERO request 6.7e-3 from the CPU F32 reference on the Dragonwing IQ-10, above the 2.9e-3 bar; the CPU with flash attention on is 5.5e-3. With the explicit path it is 2.6e-3 served and 2.2e-3 on the fixed input (was 2.8e-3), at a cost of 683 to 810 ms with two views (mean, vla-bench). --flash-attn 1 turns it back on. Other backends are unchanged and the other eleven archs give bit-identical actions.
The LM's softmax rows were 337 wide, which the NPU refuses, so every layer's softmax ran on the CPU. On Hexagon the LM now runs on gemm_width(SEQ) tokens with zeros appended; the mask is causal, so no real token sees them, and the conditioning keeps the first SEQ rows. Other backends run the same sequence as before. On the Dragonwing IQ-10, VLA-Adapter goes from 859 to 830 ms (mean, vla-bench, one view), 5.7e-4 from the CPU F32 reference on the fixed input and 5.0e-4 served.
Each head block scored its own tokens, the action queries and the vision tokens separately and concatenated the scores and the values along their columns, which the NPU refuses, then took a 329-wide softmax it also refuses. Where gemm_width() pads (Hexagon), the keys and values are now stacked as rows and padded to a multiple of 32, the padding masked out, with rg on the vision keys instead of their scores (RoPE is linear). Other backends keep the previous graph. On the Dragonwing IQ-10, VLA-Adapter goes from 830 to 747 ms (mean, vla-bench, one view), 5.1e-4 from the CPU F32 reference on the fixed input and 5.3e-4 served.
ggml-hexagon has no GELU_ERF, so DINOv2's MLP and the projector sent their activations through the CPU in every layer. backend.h gains gelu_erf(): on Hexagon x*sigmoid(p(x)), p an odd cubic in x clamped to [-5, 5] fitted to logit(Phi(x)), within 2.5e-5 of the exact GELU for every x; elsewhere ggml_gelu_erf, so other backends build the same graph. On the Dragonwing IQ-10, VLA-Adapter goes from 740 to 705 ms and OpenVLA-OFT from 1771 to 1740 ms (mean, vla-bench, one view), within 6.2e-4 / 3.2e-4 of the CPU F32 reference on the fixed input and 5.4e-4 / 5.9e-4 served.
…agon Its softmax rows were 353 wide, which the NPU refuses, so every block's softmax ran on the CPU. Where gemm_width() pads (Hexagon), zero tokens are appended and masked out as keys; the readouts are gathered by index, so the extra rows never reach them. Other backends run the same sequence as before. On the Dragonwing IQ-10, Octo-Small goes from 126.5 to 122.6 ms (mean, vla-bench, two views), 5.3e-4 from the CPU F32 reference on the fixed input and 2.4e-4 served.
…rk at 7c76ea9 hexagon.md: VLA-JEPA's LM leaves flash attention as N1.7's does (6.7e-3 to 2.6e-3 served); VLA-Adapter's LM and action head and Octo's block transformer pad their softmax rows to a multiple of 32; VLA-Adapter's and OpenVLA-OFT's DINOv2 build GELU (erf) from ops the NPU runs. Fidelity and serving tables re-measured: only GR00T N1.5 and N1.6 stay above the bar served. qualcomm-iq10.md and the summary row re-measured at 7c76ea9: VLA-Adapter 853.7 to 708.4 ms, Octo-Small 124.0 to 119.4 ms, OpenVLA-OFT 1768.2 to 1725.5 ms, VLA-JEPA 452.9 to 512.6 ms.
The NPU claims a PAD whose source is a weight, answers NO_SUPPORT, and the host aborts, as it does for a weight-reading CPY. The wrapper now runs those pads on the CPU. On the Dragonwing IQ-10 the twelve archs already on the NPU give the same actions as before (13 fixed-input configurations against the CPU F32 reference).
The HMX takes a matmul only when K and the rows of its first operand are multiples of 32. ACT's stem (K = 147) and layer3 and layer4 (1200 and 300 output positions at 640x480) ran their convs' matmuls on the vector units: 80 ms of the 101 ms spent on the NPU. Where gemm_width() pads (Hexagon), for one view, the im2col matrix and the stem's kernel are zero-padded and the extra output rows dropped, which leaves the product unchanged. Other backends and multi-view inputs keep conv_2d_f32. On the Dragonwing IQ-10, ACT goes from 228.5 to 155.6 ms (mean, vla-bench, one 640x480 view; the same build on the CPU 170.8 ms), 6.1e-4 from the CPU F32 reference on the fixed input and 1.0e-3 served.
hexagon.md: pads that read a weight run on the CPU; ACT's ResNet convs pad their patch matrices for the HMX; ACT in the fidelity (6.1e-4) and serving (1.0e-3) tables. qualcomm-iq10.md: ACT's configuration, latency (152.9 ms mean, 170.8 ms on the CPU) and flag screen at 0f3e010, and the 7c76ea9 run's date given in UTC+07 (the board's clock is UTC). CHANGELOG: ACT.
thanhdq1o2
changed the title
README: the Hexagon column follows the Dragonwing IQ-10 under Linux, where π0, VLA-JEPA, OpenVLA-OFT and ACT run within the 2.9e-3 bar and are benchmarked; a note gives the Snapdragon X laptop's state, and the column and the backend list link docs/backend/hexagon.md, which covers both. docs/benchmark/README.md: an ACT column, 148.5 ms on the IQ-10 at 0f3e010 and n/r elsewhere.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
At
1cab1c3, π0, VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small and TurboVLA run on the IQ-10 NPU with much of their graphs on the CPU fallback, and VLA-JEPA is above the fidelity bar on a served observation. This PR moves most of that work onto the NPU, brings VLA-JEPA within the bar, and runs ACT on the NPU. Other backends build the same graphs as before.Changes
9f44354pad_mlp()moves intoWeightLoader, so the towers indual_tower.hget SigLIP's 4304 to 4320 padding tooa4d443bdual_tower.htake flash attention on Hexagon, as π0.5's SigLIP does1105408linear()multiplies a contiguous batch of activations as one 2D matrix of rows (same product)MUL_MATrule)f7b0e68CONTthat transposes its innermost dim runs on the CPU from 256K elements up (same bytes)2b3424dhexagon.md, CHANGELOG, IQ-10 benchmark atf7b0e682affa26--flash-attnis given, as N1.7's doese89190cgemm_width(SEQ)tokens on Hexagon, zeros appended; the mask is causal, so no real token sees themee69ea2rgon the vision keys instead of their scorese84913dvla::gelu_erf(): on Hexagon x·sigmoid(p(x)), p an odd cubic in x clamped to [−5, 5], within 2.5e-5 of the exact GELU; used indual_tower.hGELU_ERFon the CPU in every DINOv2 layer and the projector7c76ea95af7b7ehexagon.md, CHANGELOG, IQ-10 benchmark at7c76ea98abda27PADthat reads a weight runs on the CPU. The NPU claims it, answersNO_SUPPORT, and the host aborts, as with a weight-readingCPY.0f3e010gemm_width()on Hexagon, one view, and drop the extra rows11861b3hexagon.md, CHANGELOG, ACT on the IQ-10 benchmark page at0f3e010; the page's date for the7c76ea9run corrected to UTC+07f7ad698docs/benchmark/README.mdIQ-10,
vla-benchunder TEMPLATE.md, min ms1cab1c3)7c76ea9)0f3e010The reference is the same build on the CPU with
--weight-dtype f32 --flash-attn 0; the bar is 2.9e-3. VLA-JEPA is slower because its LM leaves flash attention, which is what brings it within the bar served (6.7e-3 before). ACT runs one 640x480 view of its ALOHA checkpoint (ravediamond/act-aloha-sim-transfer-cube-gguf); before0f3e010it took 228.5 ms (mean, 5 reps).Verified
vla_predict_check), and a repeated run gives the same output. After8abda27and0f3e010the other twelve give the same values as at7c76ea9.vla-server), each arch returns a finite chunk, the same chunk for a repeated request, an error reply to a malformed and to an invalid request, and keeps serving. The twelve LIBERO archs take one fixed LIBERO observation througheval/client; GR00T N1.5 (1.1e-2) and N1.6 (1.3e-2) stay above the bar on it, as the CPU at F16 does (4.3e-2, 1.1e-2). ACT has noeval/clientpreset and takes rawvla.protorequests (one fixed 640x480 view, a fixed 14-dim state).ctestpasses (10/10).Tried and left out
im2col+ one matmul (no transposing copy): within noise.NO_SUPPORT.