Repository navigation
hexagon: keep more work on the IQ-10 NPU and run six more archs - #71
Closed
thanhdq1o2 wants to merge 17 commits into
Closed
thanhdq1o2 wants to merge 17 commits into
thanhdq1o2 wants to merge 17 commits into
Conversation
vla-bench under TEMPLATE.md's protocol at e8dfa34, for the six models whose GGUFs are on the board. The flag screen keeps F16 weights and varies only --flash-attn and --mm-prec: the NPU has no BF16 kernels, and F16 is the default. Memory adds the drop in system MemAvailable, since the weights sit in FastRPC buffers that peak RSS leaves out. The CPU section and the archs without GGUFs on the board are left for a later round.
Minimum latency at the defaults from qualcomm-iq10.md, measured at e8dfa34. Models whose GGUFs were not on the board are marked n/r; BitVLA is N/A off CUDA.
ggml-hexagon takes an F16 weight on the matrix unit only when both dims are multiples of 32. SigLIP-so400m's MLP is 4304 wide, so fc1 and fc2 ran on the vector units: 440 ms of GR00T N1.5's 565 ms vision tower on the IQ-10. WeightLoader::pad() declares a float weight larger than in the file, with the extra elements zero, and EncStack pads fc1, its bias and fc2 to gemm_width(), which rounds up to 32 in Hexagon builds and is the identity elsewhere. The padded units are zero through GELU and fc2, so the block's output is the same. On the IQ-10 against the unpadded CPU F32 reference: pi0.5 2926 -> 2122 ms (3.1e-4), GR00T N1.5 1024 -> 610 ms (2.4e-3), N1.6 1117 -> 709 ms (1.4e-3). SmolVLA, Evo-1 and N1.7 do not use this MLP and are unchanged.
pi0.5 ties SigLIP's flash attention to a view-taking kernel, which only SYCL has, so on Hexagon its tower ran the explicit scores: an F32 x F32 matmul at head dim 72 that the HMX does not take, 266 ms on the vector units. Where flash attention is on by default (Hexagon), SigLIP now takes the plain flash attention that GR00T's identical tower already uses. Other backends are unchanged. IQ-10: 2118 -> 1833 ms, 2.8e-4 from the CPU F32 reference.
The LM's explicit attention multiplies F32 K and V against a batch of 14 query heads. ggml-hexagon's HMX takes no batched F32 src0, so both matmuls ran on the vector units, about 1 s per chunk on the IQ-10. On Hexagon K and V now go to F16 through fa_kv(); the scores still accumulate in F32 and the softmax stays F32. Other backends keep F32 K/V. IQ-10: 3261 -> 2336 ms, 4.3e-4 from the CPU F32 reference (5.2e-4 before). The InternViT flash attention stays on the CPU: F16 K/V there measured 3.9e-2.
hexagon.md: which matmuls the HMX takes, the three that now move onto it, what is left on the vector units, and why the GELU_ERF kernel was left out. The fidelity and serving tables are re-measured at this commit. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 74e145d: GR00T N1.5 613.6 ms, N1.6 706.9, pi0.5 1789.4, Evo-1 2589.2; SmolVLA and N1.7 unchanged. The defaults stay the fastest configuration.
VinRobotics#70 was squash-merged into main; its two commits are already on this branch, and the IQ-10 benchmark page and README row keep this branch's re-measured numbers.
The VLM's and the expert's attention took flash attention only on a view-taking kernel (SYCL). On Hexagon they ran explicit scores against a single KV head, which the fallback wrapper sends to the CPU. Where flash attention is on by default (Hexagon), all three sites now take the plain flash attention; the view-only paths keep use_fused_attention(). IQ-10: 1773 -> 1320 ms, 4.8e-4 from the CPU F32 reference, 3.5e-4 on a served LIBERO observation. Other models unchanged.
The NPU runs a concat only as a dim-1 copy of whole rows, so the expert's dim-2 KV concats (π0, π0.5, SmolVLA) went to the CPU fallback every step. concat_dim2() in backend.h does them on Hexagon as a dim-1 concat of the [d0*d1, n] views, which writes the same bytes; other backends keep ggml_concat(a, b, 2). IQ-10: π0.5 1379 -> 1133 ms, SmolVLA 810 -> 638 ms, both with outputs identical to before on fixed inputs and on a served observation.
…rk at 0b78eb9 hexagon.md: π0.5's VLM and action expert take flash attention on Hexagon, and the experts' KV concats run as row copies that stay on the NPU (π0 shares the code but has not been run on the board). π0.5's fidelity is re-measured: 4.8e-4 on fixed inputs, 3.5e-4 on a served observation. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 0b78eb9: π0.5 1117.2 ms, SmolVLA 633.0 ms; the other models are unchanged. The defaults stay the fastest configuration.
thanhdq1o2
changed the title
ggml-hexagon claims a CPY whose source is a weight, then answers the batch with NO_SUPPORT and the host aborts. VLA-Adapter casts its F16 action queries to F32 in the graph and aborted on every predict on the Dragonwing IQ-10. With the cast on the CPU it is within 4.2e-4 of the CPU F32 reference, 4.5e-4 when served.
The NPU's REPEAT reads its source through the data cache, which is invalidated only between op batches, while ops write their output by DMA. When the source is written earlier in the same batch, REPEAT can read stale lines. TurboVLA's ViT prefix, repeated over two views, came out 0.19 off the CPU F32 reference and different on every run; with the repeat on the CPU it is 3.1e-4 and repeatable, 2.2e-4 when served.
thanhdq1o2
changed the title
…file pad_mlp() checked only the loader's GEMM type. scripts/quantize_gguf.py --vision packs SigLIP's fc1 [1152, 4304] to Q8_0 while fc2, 4304 wide, stays float; the loader keeps fc1 packed and declare() refused to pad it, so such a GGUF failed to load in Hexagon builds. An MLP with either weight quantized in the file now loads unpadded, as it did before the padding. Float checkpoints are unchanged: π0, π0.5, GR00T N1.5/N1.6/N1.7, VLA-Adapter and OpenVLA-OFT give bit-identical actions on the Dragonwing IQ-10.
… summary The previous docs commit added six archs to the IQ-10 pages but left two summaries behind: the status note in docs/backend/hexagon.md still said six checkpoints run on the IQ-10, and the IQ-10 row of docs/benchmark/README.md still carried the 0b78eb9 numbers with the six new archs marked not run. Both now match docs/benchmark/qualcomm-iq10.md at 1cab1c3.
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Profiling the IQ-10 (
GGML_HEXAGON_PROFILE=1,VLA_FALLBACK_STATS=1) showed matmuls the matrix unit (HMX) does not take and ops the NPU sends to the CPU. The HMX takes a matmul only when K and N are multiples of 32, and with a batch only an F16 src0. The NPU runs a concat only as a dim-1 row copy. This PR keeps that work on the NPU in Hexagon builds, and sends two ops the NPU claims but cannot run correctly to the CPU, which brings π0, VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small and TurboVLA onto the board. Other backends are unchanged.Changes
b33ead8WeightLoader::pad()zero-pads SigLIP-so400m's MLP from 4304 to 4320 (fc1, its bias, fc2). The output is unchanged.b85e17474e145dfa_kv()). Scores accumulate in F32, softmax stays F32.bdacd9ec10cb80use_flash_attention())MUL_MATrule0b78eb9concat_dim2(): the experts' dim-2 KV concats run as a dim-1 concat of the[d0*d1, n]views (same bytes)26ad84bhexagon.md, CHANGELOG, IQ-10 benchmark re-measured at0b78eb91ca8633CPYthat reads a weight runs on the CPU. The NPU claims it, answersNO_SUPPORT, and the host aborts.1cab1c3REPEATof a computed tensor runs on the CPU. The NPU reads the source through the data cache, invalidated only between op batches, so a source written earlier in the same batch can be read stale.6aec070hexagon.md, CHANGELOG, IQ-10 benchmark for 12 archs at1cab1c39d49a36scripts/quantize_gguf.py --visionloads as it does on main. Float checkpoints give bit-identical actions.declare()refused to pad the packed fc1 and the load failed7dc3571hexagon.mdand the IQ-10 row ofdocs/benchmark/README.mdlist the twelve archs at1cab1c30b78eb9numbersThe two new rules check for a Hexagon session. On the board they fire only for VLA-Adapter (the CPY), TurboVLA (its ViT prefix repeated over two views) and Octo-Small (its task-token repeat); the other nine archs' graphs are unchanged by them.
IQ-10,
vla-benchunder TEMPLATE.md, min msThe reference is the same build on the CPU with
--weight-dtype f32 --flash-attn 0; the bar is 2.9e-3. Fixed inputs use the views the LIBERO client sends, and the benchmark uses its own inputs (both listed in the docs). BitVLA is not run: its only published GGUF is int2-packed and loads only in CUDA builds.Verified
vla_predict_check), and a repeated run gives the same output.vla-serverandeval/client), each arch returns a finite chunk, the same chunk for a repeated request, an error reply to a malformed and to an invalid request, and keeps serving. GR00T N1.5 (1.1e-2), N1.6 (1.3e-2) and VLA-JEPA (6.7e-3) are above the bar on it; so is the CPU at F16 (4.3e-2, 1.1e-2, 2.1e-2).ctestpasses (10/10).Tried and left out