From 2a66f9faf30d45a5b8df74294bb249b59c4f1d9c Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Mon, 2 Nov 2026 15:04:25 +0700 Subject: [PATCH 01/38] docs: benchmark the Qualcomm Dragonwing IQ-10 Hexagon NPU vla-bench under TEMPLATE.md's protocol at e8dfa34, for the six models whose GGUFs are on the board. The flag screen keeps F16 weights and varies only --flash-attn and --mm-prec: the NPU has no BF16 kernels, and F16 is the default. Memory adds the drop in system MemAvailable, since the weights sit in FastRPC buffers that peak RSS leaves out. The CPU section and the archs without GGUFs on the board are left for a later round. --- docs/benchmark/qualcomm-iq10.md | 148 ++++++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 docs/benchmark/qualcomm-iq10.md diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md new file mode 100644 index 00000000..b511afd0 --- /dev/null +++ b/docs/benchmark/qualcomm-iq10.md @@ -0,0 +1,148 @@ +# Qualcomm Dragonwing IQ-10: Hexagon NPU + +`vla-bench` times `predict()` in-process on synthetic inputs. It measures the +engine only: no transport, no simulator, and no claim about task success. + +The graph runs on the Hexagon NPU, and ops the NPU rejects run on the CPU +through vla.cpp's fallback wrapper. + +## Test setup + +| | | +|---|---| +| Device | Qualcomm Dragonwing IQ-10 (IQX EVK): QCS8797, 18 Oryon cores, Hexagon v81 NPU (one of its four CDSP sessions in use), 60.8 GiB RAM visible | +| Threads | 16 (the default: all hardware threads, capped at 16), for the NPU's CPU fallback | +| Power | CPU governor `performance` | +| OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | +| Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | +| Commit | `e8dfa34` (PR #69, on top of #68) | +| llama.cpp | `b11223` | +| Date | 2026-11-02 11:02 to 11:48 (UTC+07) | +| Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | +| Runtime flags | defaults; the fastest flags per model are in the second table | +| Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | + +**Peak RSS** is the process's peak resident set (`getrusage`, via +`/usr/bin/time`). It leaves out the weights, which live in FastRPC buffers +mapped for the NPU. **System RAM** is the largest drop in the system's +`MemAvailable` while the process ran, sampled every 0.5 s; it counts those +buffers. Nothing else ran on the board. + +## Model configuration + +Each model runs at its native view count and input size, with weights as +shipped. Checkpoints are the `vrfai/*` GGUFs; where a repo ships several, we use +the `libero_object` variant. Settings not listed are the defaults. + +| Model | Checkpoint | Views | Input | Lang tokens | Action chunk | Action dim | Action head | Steps | +|---|---|--:|--:|--:|--:|--:|---|--:| +| SmolVLA | `smolvla-libero-gguf` | 2 | 512 | 16 | 50 | 7 | flow matching | 10 | +| π0 | `pi0-libero-finetuned-v044-gguf` | 2 | 224 | 16 | 50 | 7 | flow matching | 10 | +| π0.5 | `pi05-libero-gguf` | 2 | 224 | 16 | 50 | 7 | flow matching | 10 | +| GR00T N1.5 | `gr00tn1d5-libero-object-gguf` | 1 | 224 | 16 | 16 | 32 | flow matching (DiT) | 4 | +| GR00T N1.6 | `gr00tn1d6-libero-gguf` | 1 | 224 | 16 | 50 | 128 | flow matching (DiT) | 4 | +| GR00T N1.7 | `gr00tn1d7-libero-gguf` | 1 | 256 | 16 | 40 | 132 | flow matching (DiT) | 4 | +| VLA-JEPA | `vla-jepa-libero` | 1 | 256 | 16 + 32 | 7 | 7 | flow matching (DiT) | 4 | +| Evo-1 | `evo1-libero-gguf` | 1 | 448 | 16 | 50 | 7 | flow matching | 32 | +| BitVLA | `bitvla-libero-gguf` (int2) | 1 | 224 | 16 | 8 | 7 | parallel decoding | 1 | +| VLA-Adapter | `vla-adapter-libero-gguf` | 1 | 224 | 16 | 8 | 7 | bridge-attention policy | 1 | +| OpenVLA-OFT | `openvla-oft-libero-gguf` | 1 | 224 | 16 | 8 | 7 | L1 regression head | 1 | +| Octo-Small | `octo-small-libero-gguf` | 2 | 256 + 128 | 16 | 4 | 7 | DDPM diffusion | 20 | +| TurboVLA | `turbovla-libero-gguf` | 2 | 256 | 16 | 12 | 7 | single-pass decoder | 1 | + +- **Lang tokens** is the synthetic prompt length. VLA-JEPA adds its 32 + `` tokens with `--extra-token 151697 --extra-count 32`. +- **Action dim** is the checkpoint's padded width. LIBERO uses the first 7 dims. +- **GR00T** needs its embodiment set in the environment: + `VLA_GR00T_EMBODIMENT=new_embodiment` for N1.5, `libero_panda` for N1.6 and + `libero_sim` for N1.7. **Octo** needs `VLA_OCTO_UNNORM_DATASET=libero_object`. +- **Octo** takes a 256 px primary view and a 128 px wrist view. + +## Latency and memory + +Rows are sorted by latency. `vision ms` is `-` for archs that do not time their +vision stage separately. + +| Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | +|---|--:|--:|--:|--:|--:|--:|--:|--:|--:| +| GR00T N1.7 | 1 | 256 | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | +| SmolVLA | 2 | 512 | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | +| GR00T N1.5 | 1 | 224 | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | +| GR00T N1.6 | 1 | 224 | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | +| π0.5 | 2 | 224 | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | +| Evo-1 | 1 | 448 | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | + +### Fastest configuration + +We screen each model over these runtime flag sets, then re-run the fastest set +under the full protocol: + +- the defaults (F16 weights, the Hexagon build's default); +- `--flash-attn` and `--mm-prec default`, alone and together; +- `--flash-attn 0`, because flash attention is on by default in Hexagon builds. + +The weight dtype stays F16 in every set. `--weight-dtype f16` is the default, +and `--weight-dtype bf16` is left out: the NPU has no BF16 kernels, so every +matmul would run on the CPU (SmolVLA: 18.7 s against 0.82 s). + +The screen runs 2 warmups and 5 reps in one process per flag set. A flag set +replaces the defaults only if its mean is more than 3% lower, both in the screen +and in the full re-run. + +Flash attention and the matmul precision can move the action chunk. Success +rates measured at the defaults therefore do not carry over to these rows. + +| Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | +|---|---|--:|--:|--:|--:|--:|--:|--:|--:| +| GR00T N1.7 | *(defaults)* | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | - | +| SmolVLA | *(defaults)* | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | - | +| GR00T N1.5 | *(defaults)* | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | - | +| GR00T N1.6 | *(defaults)* | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | - | +| π0.5 | *(defaults)* | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | - | +| Evo-1 | *(defaults)* | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | - | + +
Screen means (ms) per flag set + +| Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | +|---|--:|--:|--:|--:|--:| +| GR00T N1.7 | 768.7 | 748.5 | 761.3 | 744.5 | 1086.6 | +| SmolVLA | 823.1 | 818.9 | 819.5 | 818.5 | 3018.8 | +| GR00T N1.5 | 1026.1 | 1023.0 | 1025.3 | 1026.1 | 1501.2 | +| GR00T N1.6 | 1114.4 | 1115.1 | 1114.4 | 1113.3 | 1290.3 | +| π0.5 | 2939.0 | 2956.9 | 2959.0 | 2962.7 | 2959.6 | +| Evo-1 | 3583.7 | 3593.9 | 3591.8 | 3582.6 | 5828.5 | + +
+ +## Not run + +- **BitVLA**: N/A off CUDA. The only published GGUF is int2-packed, and only + CUDA builds load it. +- **π0, VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small, TurboVLA**: not run in + this round; their GGUFs were not on the board. + +## Notes + +The HTP skel has to be processed for this board and installed in +`/vendor/dsp/cdsp0..3`: its FastRPC ignores `ADSP_LIBRARY_PATH`. See +[backend/hexagon.md](../backend/hexagon.md#linux-iq-10). + +GR00T N1.6 runs one view here, as the configuration above fixes. With two views +its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see +[backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). + +These rows time the defaults; they do not vouch for accuracy. On a served LIBERO +observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar, about as +far off as the CPU at F16; see +[backend/hexagon.md](../backend/hexagon.md#serving). + +## Reproducing + +```bash +scripts/build_linux_hexagon.sh +# copy build-lnx-htp/pkg to the board, install its HTP skel (see backend/hexagon.md), then on the board: +export LD_LIBRARY_PATH=$PWD/pkg/lib + +# one row, for example SmolVLA +./pkg/bin/vla-bench --ckpt smolvla-libero.gguf --images 2 --size 512 --warmup 3 --reps 20 +``` From cc3c2bb1ae1064516e10dba4baec8cca0145e230 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Mon, 2 Nov 2026 15:46:39 +0700 Subject: [PATCH 02/38] docs: add the IQ-10 Hexagon row to the benchmark table Minimum latency at the defaults from qualcomm-iq10.md, measured at e8dfa34. Models whose GGUFs were not on the board are marked n/r; BitVLA is N/A off CUDA. --- docs/benchmark/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index 6dfc44ae..c75ebc14 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -11,7 +11,8 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path `1012c05`, before that path, for comparison. The Radeon RX 9060 XT ROCm row ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in -[backend/rocm.md](../backend/rocm.md). Each report +[backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at +`e8dfa34` (PR #69, same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -35,6 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 1020.9 | 1110.3 | 747.5 | N/A | 810.3 | n/r | 2914.6 | n/r | 3519.4 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | From b33ead8ac6adf79789e882e2f4793503bea68c47 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 09:42:06 +0700 Subject: [PATCH 03/38] hexagon: pad SigLIP's MLP to a multiple of 32 so it runs on the HMX ggml-hexagon takes an F16 weight on the matrix unit only when both dims are multiples of 32. SigLIP-so400m's MLP is 4304 wide, so fc1 and fc2 ran on the vector units: 440 ms of GR00T N1.5's 565 ms vision tower on the IQ-10. WeightLoader::pad() declares a float weight larger than in the file, with the extra elements zero, and EncStack pads fc1, its bias and fc2 to gemm_width(), which rounds up to 32 in Hexagon builds and is the identity elsewhere. The padded units are zero through GELU and fc2, so the block's output is the same. On the IQ-10 against the unpadded CPU F32 reference: pi0.5 2926 -> 2122 ms (3.1e-4), GR00T N1.5 1024 -> 610 ms (2.4e-3), N1.6 1117 -> 709 ms (1.4e-3). SmolVLA, Evo-1 and N1.7 do not use this MLP and are unchanged. --- src/backend.h | 16 ++++++++++++++++ src/loader.cpp | 35 +++++++++++++++++++++++++++++++++-- src/loader.h | 5 +++++ src/modules/encoder.cpp | 24 ++++++++++++++++++++++++ 4 files changed, 78 insertions(+), 2 deletions(-) diff --git a/src/backend.h b/src/backend.h index bd574c0d..894f4004 100644 --- a/src/backend.h +++ b/src/backend.h @@ -191,6 +191,22 @@ inline ggml_tensor * fa_kv([[maybe_unused]] ggml_context * ctx, ggml_tensor * t) #endif } +/** + * @brief A GEMM's inner width as the compiled-in backend's matmul takes it. + * + * Hexagon's matrix unit (HMX) takes an F16 weight only when both of its dims + * are multiples of 32; anything else runs on the vector units. SigLIP's + * 4304-wide MLP spends 440 ms there per tower on the IQ-10, against about + * 0.5 ms a matmul on the HMX. Everywhere else this returns @p n. + */ +inline int64_t gemm_width(int64_t n) { +#ifdef GGML_USE_HEXAGON + return (n + 31) / 32 * 32; +#else + return n; +#endif +} + /** * @brief `ggml_gelu` (the tanh form), in ops the compiled-in backend gets right. * diff --git a/src/loader.cpp b/src/loader.cpp index 54826859..fc6d95f7 100644 --- a/src/loader.cpp +++ b/src/loader.cpp @@ -20,6 +20,7 @@ #include #include #include +#include namespace vla { @@ -39,11 +40,25 @@ void WeightLoader::as_float(const std::string & name, int64_t K, int64_t N, Floa as_float_[name] = AsFloat{K, N, std::move(make)}; } +void WeightLoader::pad(const std::string & name, int64_t ne0, int64_t ne1) { + pad_[name] = {ne0, ne1}; +} + bool WeightLoader::read_resident(const std::string & name, ggml_type type, bool gemma_norm, std::vector & out) { const auto it = as_float_.find(name); if (it == as_float_.end()) { out = g_.read_convert(name.c_str(), type, gemma_norm); - return !out.empty(); + const auto p = pad_.find(name); + if (p == pad_.end() || out.empty()) + return !out.empty(); + const ggml_tensor * src = g_.meta(name.c_str()); + const size_t rs = ggml_row_size(type, src->ne[0]); + const size_t rp = ggml_row_size(type, p->second.first); + std::vector padded(rp * (size_t) p->second.second, 0); + for (int64_t r = 0; r < src->ne[1]; ++r) + std::memcpy(padded.data() + r * rp, out.data() + r * rs, rs); + out.swap(padded); + return true; } const AsFloat & a = it->second; std::vector f; @@ -97,7 +112,23 @@ ggml_tensor * WeightLoader::declare(ggml_type want, bool required, bool gemma_no return nullptr; } - ggml_tensor * t = ggml_new_tensor(ctx_, g_.resident_type(src, want), ggml_n_dims(src), src->ne); + const ggml_type rt = g_.resident_type(src, want); + int64_t ne[GGML_MAX_DIMS]; + std::memcpy(ne, src->ne, sizeof(ne)); + int n_dims = ggml_n_dims(src); + if (const auto p = pad_.find(name); p != pad_.end()) { + const bool float_type = rt == GGML_TYPE_F32 || rt == GGML_TYPE_F16 || rt == GGML_TYPE_BF16; + if (!float_type || n_dims > 2 || p->second.first < ne[0] || p->second.second < ne[1]) { + std::fprintf(stderr, "vla(%s): cannot pad %s to [%lld, %lld]\n", arch_, name, + (long long) p->second.first, (long long) p->second.second); + ok_ = false; + return nullptr; + } + ne[0] = p->second.first; + ne[1] = p->second.second; + n_dims = 2; + } + ggml_tensor * t = ggml_new_tensor(ctx_, rt, n_dims, ne); if (!t) { std::fprintf(stderr, "vla(%s): ggml_new_tensor failed for %s (weight context too small?)\n", arch_, name); ok_ = false; diff --git a/src/loader.h b/src/loader.h index 2f71d099..e44d01e6 100644 --- a/src/loader.h +++ b/src/loader.h @@ -82,6 +82,10 @@ class WeightLoader { using FloatMaker = std::function &)>; void as_float(const std::string & name, int64_t K, int64_t N, FloatMaker make); + // Declare 2D (or 1D) float tensor `name` as [ne0, ne1], at least its shape in + // the file, with the extra elements zero. Call before declaring it. + void pad(const std::string & name, int64_t ne0, int64_t ne1); + // For helpers that inspect the file before declaring (foldquant.cpp). gguf_reader & reader() { return g_; @@ -111,6 +115,7 @@ class WeightLoader { FloatMaker make; }; std::map as_float_; + std::map> pad_; const char * arch_; gguf_reader & g_; diff --git a/src/modules/encoder.cpp b/src/modules/encoder.cpp index 8dbb8b89..5c6fa428 100644 --- a/src/modules/encoder.cpp +++ b/src/modules/encoder.cpp @@ -14,6 +14,7 @@ #include "modules/encoder.h" +#include "backend.h" #include "layers/attn.h" #include "layers/ffn.h" #include "layers/fused_ops.h" @@ -25,9 +26,30 @@ #include "layers/qlinear.h" #include +#include namespace vla { +namespace { + +// Widens the MLP to gemm_width() with zeros. The extra fc1 rows and bias are 0, +// GELU(0) is 0 and the extra fc2 columns are 0, so the block's output is the same. +void pad_mlp(WeightLoader & L, const std::string & fc1, const std::string & fc2) { + const ggml_tensor * w1 = L.reader().meta((fc1 + ".weight").c_str()); + const ggml_tensor * w2 = L.reader().meta((fc2 + ".weight").c_str()); + if (!w1 || !w2 || ggml_is_quantized(L.gemm_type())) + return; + const int64_t ff = w1->ne[1]; + const int64_t fp = gemm_width(ff); + if (fp == ff || w2->ne[0] != ff) + return; + L.pad(fc1 + ".weight", w1->ne[0], fp); + L.pad(fc1 + ".bias", fp, 1); + L.pad(fc2 + ".weight", fp, w2->ne[1]); +} + +} + void EncStack::declare(WeightLoader & L, const char * prefix, int64_t layers, const EncNames & n) { blk.resize(layers); ln_spec.eps = cfg.ln_eps; @@ -61,6 +83,8 @@ void EncStack::declare(WeightLoader & L, const char * prefix, int64_t layers, co w.q_fc1 = L.quant_i8(b + n.fc1 + ".q8", {b + n.fc1 + ".weight"}); w.q_fc2 = L.quant_i8(b + n.fc2 + ".q8", {b + n.fc2 + ".weight"}); } else { + const std::string b = std::string(prefix) + "." + std::to_string(i) + "."; + pad_mlp(L, b + n.fc1, b + n.fc2); w.Wo = L.gemm("%s.%lld.attn_o.weight", prefix, (long long)i); w.Wfc1 = L.gemm("%s.%lld.%s.weight", prefix, (long long)i, n.fc1); w.Wfc2 = L.gemm("%s.%lld.%s.weight", prefix, (long long)i, n.fc2); From b85e1746a860ea5ef4d7b43379604aaeded24b37 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 10:00:45 +0700 Subject: [PATCH 04/38] pi05: flash attention in SigLIP on Hexagon pi0.5 ties SigLIP's flash attention to a view-taking kernel, which only SYCL has, so on Hexagon its tower ran the explicit scores: an F32 x F32 matmul at head dim 72 that the HMX does not take, 266 ms on the vector units. Where flash attention is on by default (Hexagon), SigLIP now takes the plain flash attention that GR00T's identical tower already uses. Other backends are unchanged. IQ-10: 2118 -> 1833 ms, 2.8e-4 from the CPU F32 reference. --- src/models/pi05.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/models/pi05.cpp b/src/models/pi05.cpp index 1f4faa31..a73e916a 100644 --- a/src/models/pi05.cpp +++ b/src/models/pi05.cpp @@ -544,7 +544,10 @@ std::unique_ptr pi05_create(const std::string& mmproj_path, std::printf("vla(pi05): expert weights INT4 (W4A8), %s\n", aex_w4_group > 0 ? ("group " + std::to_string(aex_w4_group)).c_str() : "per channel"); } - m->vis.vit.enc.cfg.flash_attn = use_fused_attention(); + // On Hexagon (flash attention on by default) SigLIP takes the plain flash + // attention too: its explicit scores, head dim 72 against 256 keys, are an + // F32 x F32 matmul the HMX does not take, 266 ms of vector-unit time. + m->vis.vit.enc.cfg.flash_attn = use_fused_attention() || (vla::default_flash_attn() && vla::flash_attn_enabled()); // Fused SigLIP blocks (modules/encoder.h, EncCfg::fused): SYCL, BF16, fused // attention. A FoldQuant GGUF takes them too: its integer sites are the LM's // and the expert's, and its vision tower is BF16 like any other checkpoint's. From 74e145de94673e6628b20c12dbc7b217ba99a387 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 10:04:51 +0700 Subject: [PATCH 05/38] evo1: F16 K/V in the LM's attention on Hexagon The LM's explicit attention multiplies F32 K and V against a batch of 14 query heads. ggml-hexagon's HMX takes no batched F32 src0, so both matmuls ran on the vector units, about 1 s per chunk on the IQ-10. On Hexagon K and V now go to F16 through fa_kv(); the scores still accumulate in F32 and the softmax stays F32. Other backends keep F32 K/V. IQ-10: 3261 -> 2336 ms, 4.3e-4 from the CPU F32 reference (5.2e-4 before). The InternViT flash attention stays on the CPU: F16 K/V there measured 3.9e-2. --- src/models/evo1.cpp | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/models/evo1.cpp b/src/models/evo1.cpp index f6e56563..c61adc1d 100644 --- a/src/models/evo1.cpp +++ b/src/models/evo1.cpp @@ -162,8 +162,12 @@ ggml_tensor * build_qwen2_layer(ggml_context * C, const Evo1ModelArch & m, const ggml_tensor * q_rope = ggml_rope_ext(C, ggml_reshape_3d(C, qp, hd, n_q, seq), positions, nullptr, (int) hd, GGML_ROPE_TYPE_NEOX, 0, m.lm_rope_base, 1.0f, 0.0f, 1.0f, 32.0f, 1.0f); ggml_tensor * k_rope = ggml_rope_ext(C, ggml_reshape_3d(C, kp, hd, n_kv, seq), positions, nullptr, (int) hd, GGML_ROPE_TYPE_NEOX, 0, m.lm_rope_base, 1.0f, 0.0f, 1.0f, 32.0f, 1.0f); ggml_tensor * Q = ggml_cont(C, ggml_permute(C, q_rope, 0, 2, 1, 3)); - ggml_tensor * K = ggml_cont(C, ggml_permute(C, k_rope, 0, 2, 1, 3)); - ggml_tensor * V = ggml_cont(C, ggml_permute(C, ggml_reshape_3d(C, vp, hd, n_kv, seq), 1, 2, 0, 3)); + // fa_kv: on Hexagon K and V go to F16, so both matmuls run on the HMX, which + // takes no batched F32 src0 (1 s of vector-unit time per chunk on the IQ-10). + // The scores still accumulate in F32 and the softmax stays F32: 4.3e-4 from + // the CPU F32 reference there. Other backends keep F32 K/V. + ggml_tensor * K = fa_kv(C, ggml_cont(C, ggml_permute(C, k_rope, 0, 2, 1, 3))); + ggml_tensor * V = fa_kv(C, ggml_cont(C, ggml_permute(C, ggml_reshape_3d(C, vp, hd, n_kv, seq), 1, 2, 0, 3))); ggml_tensor * kq = ggml_mul_mat(C, K, Q); ggml_prec_set_acc(kq, GGML_PREC_F32); ggml_tensor * aw = ggml_soft_max_ext(C, kq, mask, scale, 0.0f); ggml_tensor * kqv = ggml_mul_mat(C, V, aw); From bdacd9e54f5e1d43c22cc9eb7a0f059aef991831 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 12:05:15 +0700 Subject: [PATCH 06/38] docs: Hexagon matmuls on the HMX, IQ-10 benchmark at 74e145d hexagon.md: which matmuls the HMX takes, the three that now move onto it, what is left on the vector units, and why the GELU_ERF kernel was left out. The fidelity and serving tables are re-measured at this commit. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 74e145d: GR00T N1.5 613.6 ms, N1.6 706.9, pi0.5 1789.4, Evo-1 2589.2; SmolVLA and N1.7 unchanged. The defaults stay the fastest configuration. --- CHANGELOG.md | 4 +++ docs/backend/hexagon.md | 44 +++++++++++++++++++++++------- docs/benchmark/README.md | 4 +-- docs/benchmark/qualcomm-iq10.md | 47 ++++++++++++++++++--------------- 4 files changed, 65 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d2436c7..3be3ac7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,10 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke space and matmuls of a 2D weight against a batch, which it computes wrong (GR00T N1.6 with two views), run on the CPU, and GR00T N1.7's LM leaves flash attention (F16 scores on the HMX) unless `--flash-attn` is given. + Three matmuls that the HMX does not take move onto it: SigLIP's 4304-wide + MLP is zero-padded to 4320 at load, π0.5's SigLIP takes flash attention, and + Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, π0.5 and Evo-1 run + 27-40% faster, within the bar on fixed inputs. GR00T N1.5 and N1.6 are above the bar on a served LIBERO observation, as the CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install and the numbers. diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 6df93b30..c664eb90 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -94,7 +94,10 @@ changed all three conclusions: it. - **`GELU_ERF` is still missing.** It now runs on the CPU (next point) in every DINOv2 and SigLIP-so400m tower, and costs what that implies: Evo-1 spends more - on copies than the NPU saves. + on copies than the NPU saves. An HTP kernel was written and checked on the + IQ-10 (b11223): more accurate than the CPU's (relative error 1.5e-6 against + 5.1e-5), and Evo-1 7% faster. It was left out: GR00T N1.7 sits close to the + bar, and the kernel moved its fixed-input chunk from 2.0e-3 to 3.0e-3. - **Absent is no longer fatal.** `src/backend_fallback.cpp` wraps the NPU in a backend that runs whatever it rejects on the CPU, without touching an arch. That also covers ops refused for shape rather than type: softmax rows that are @@ -133,6 +136,26 @@ of 1, activations above 1) is claimed and comes out wrong. GR00T N1.6's SigLIP2 tower takes two camera views that way: its action chunk lands 0.16 from the CPU reference. The wrapper runs those matmuls on the CPU: 1.9e-3, and 25% faster. +The matrix unit (HMX) takes a matmul only when its inner dim K and its output +dim N are multiples of 32, and with a batch only an F16 src0. Anything else runs +on the vector units, at about 1/20 of the speed. `GGML_HEXAGON_PROFILE=1` on the +IQ-10 showed three such matmuls dominating, and vla.cpp now avoids them in +Hexagon builds: + +- **SigLIP-so400m's MLP is 4304 wide**: 440 ms of GR00T N1.5's 565 ms tower. + The loader zero-pads fc1, its bias and fc2 to 4320 (`WeightLoader::pad`, + `gemm_width()` in `src/backend.h`), which leaves the output unchanged. +- **π0.5's SigLIP ran explicit attention** (head dim 72, F32 x F32), because + its flash attention was tied to SYCL's view-taking kernel. It now takes the + plain flash attention, as GR00T's tower does. +- **Evo-1's LM attention multiplied F32 K and V against 14 query heads.** K and + V now go to F16 (`fa_kv()`); the scores still accumulate in F32. + +What is left on the vector units: GR00T's DiT attention (head dim 48) and Evo-1's +DiT cross-attention (head dim 112, 1025 keys), whose shapes the HMX does not +take, and Evo-1's InternViT attention, which stays on the CPU because F16 K/V +there measured 3.9e-2. + ## What changed in `vla.cpp` The shape of the diff was predicted correctly, apart from the scheduler: @@ -285,9 +308,9 @@ path that other builds run: | Model | max\|Δ\| | |---|--:| | SmolVLA | 7.2e-4 | -| π0.5 | 2.7e-4 | -| Evo-1 | 4.1e-4 | -| GR00T N1.5 | 1.7e-3 | +| π0.5 | 2.8e-4 | +| Evo-1 | 5.0e-4 | +| GR00T N1.5 | 2.4e-3 | | GR00T N1.6 | 1.4e-3 | | GR00T N1.7 | 2.0e-3 | @@ -307,15 +330,16 @@ malformed or an invalid request, and keeps serving afterwards. | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 | |---|--:|--:| | SmolVLA | 1.3e-3 | | -| π0.5 | 4.3e-4 | | -| Evo-1 | 4.3e-4 | | -| GR00T N1.5 | 8.3e-3 | 4.3e-2 | +| π0.5 | 4.2e-4 | | +| Evo-1 | 1.1e-3 | | +| GR00T N1.5 | 1.1e-2 | 4.3e-2 | | GR00T N1.6 | 1.3e-2 | 1.1e-2 | | GR00T N1.7 | 2.5e-3 | | -GR00T N1.5 and N1.6 are above the bar on this input, by about what the CPU -itself gives with F16 weights: they are sensitive to 16-bit precision rather -than miscomputed. Running them with F32 weights on the NPU has not been tried +GR00T N1.5 and N1.6 are above the bar on this input, and so is the CPU itself +with F16 weights: N1.5 lands closer to the reference than the CPU at F16, N1.6 +slightly further (1.3e-2 against 1.1e-2). They are sensitive to 16-bit +precision rather than miscomputed. Running them with F32 weights on the NPU has not been tried yet. ## IQ-10, imagined diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index c75ebc14..a2befb19 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`e8dfa34` (PR #69, same llama.cpp pin). Each report +`74e145d` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 1020.9 | 1110.3 | 747.5 | N/A | 810.3 | n/r | 2914.6 | n/r | 3519.4 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 613.6 | 706.9 | 746.5 | N/A | 811.5 | n/r | 1789.4 | n/r | 2589.2 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index b511afd0..9415b204 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `e8dfa34` (PR #69, on top of #68) | +| Commit | `74e145d` (branch `hexagon-b11223-ops`, on top of #68 to #70) | | llama.cpp | `b11223` | -| Date | 2026-11-02 11:02 to 11:48 (UTC+07) | +| Date | 2026-11-03 10:43 to 11:02 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -65,12 +65,12 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.7 | 1 | 256 | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | -| SmolVLA | 2 | 512 | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | -| GR00T N1.5 | 1 | 224 | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | -| GR00T N1.6 | 1 | 224 | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | -| π0.5 | 2 | 224 | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | -| Evo-1 | 1 | 448 | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | +| GR00T N1.5 | 1 | 224 | 613.6 | 615.7 | 614.8 | 618.5 | 148.2 | 162 | 3644 | +| GR00T N1.6 | 1 | 224 | 706.9 | 710.9 | 710.3 | 715.0 | 145.3 | 147 | 4688 | +| GR00T N1.7 | 1 | 256 | 746.5 | 754.0 | 753.4 | 760.6 | 195.7 | 191 | 5047 | +| SmolVLA | 2 | 512 | 811.5 | 816.3 | 815.9 | 820.2 | 283.1 | 146 | 1234 | +| π0.5 | 2 | 224 | 1789.4 | 1820.3 | 1818.5 | 1850.2 | 296.7 | 285 | 5783 | +| Evo-1 | 1 | 448 | 2589.2 | 2602.0 | 2598.8 | 2613.3 | 853.9 | 319 | 1588 | ### Fastest configuration @@ -94,23 +94,23 @@ rates measured at the defaults therefore do not carry over to these rows. | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.7 | *(defaults)* | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | - | -| SmolVLA | *(defaults)* | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | - | -| GR00T N1.5 | *(defaults)* | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | - | -| GR00T N1.6 | *(defaults)* | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | - | -| π0.5 | *(defaults)* | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | - | -| Evo-1 | *(defaults)* | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | - | +| GR00T N1.5 | *(defaults)* | 613.6 | 615.7 | 614.8 | 618.5 | 148.2 | 162 | 3644 | - | +| GR00T N1.6 | *(defaults)* | 706.9 | 710.9 | 710.3 | 715.0 | 145.3 | 147 | 4688 | - | +| GR00T N1.7 | *(defaults)* | 746.5 | 754.0 | 753.4 | 760.6 | 195.7 | 191 | 5047 | - | +| SmolVLA | *(defaults)* | 811.5 | 816.3 | 815.9 | 820.2 | 283.1 | 146 | 1234 | - | +| π0.5 | *(defaults)* | 1789.4 | 1820.3 | 1818.5 | 1850.2 | 296.7 | 285 | 5783 | - | +| Evo-1 | *(defaults)* | 2589.2 | 2602.0 | 2598.8 | 2613.3 | 853.9 | 319 | 1588 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| GR00T N1.7 | 768.7 | 748.5 | 761.3 | 744.5 | 1086.6 | -| SmolVLA | 823.1 | 818.9 | 819.5 | 818.5 | 3018.8 | -| GR00T N1.5 | 1026.1 | 1023.0 | 1025.3 | 1026.1 | 1501.2 | -| GR00T N1.6 | 1114.4 | 1115.1 | 1114.4 | 1113.3 | 1290.3 | -| π0.5 | 2939.0 | 2956.9 | 2959.0 | 2962.7 | 2959.6 | -| Evo-1 | 3583.7 | 3593.9 | 3591.8 | 3582.6 | 5828.5 | +| GR00T N1.5 | 616.1 | 615.0 | 619.5 | 615.9 | 1096.1 | +| GR00T N1.6 | 716.2 | 698.0 | 700.7 | 696.3 | 873.4 | +| GR00T N1.7 | 753.6 | 733.7 | 764.7 | 731.8 | 1070.6 | +| SmolVLA | 828.8 | 818.5 | 812.5 | 822.8 | 3012.6 | +| π0.5 | 1815.6 | 1816.9 | 11581.7 | 1822.6 | 2144.2 | +| Evo-1 | 2614.6 | 2599.1 | 2606.5 | 2605.6 | 4870.0 |
@@ -131,9 +131,12 @@ GR00T N1.6 runs one view here, as the configuration above fixes. With two views its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). +In the screen, π0.5 with `--mm-prec default` took 11.6 s against 1.8 s at the +defaults. That was not investigated. + These rows time the defaults; they do not vouch for accuracy. On a served LIBERO -observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar, about as -far off as the CPU at F16; see +observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar: N1.5 +closer to the reference than the CPU at F16, N1.6 slightly further; see [backend/hexagon.md](../backend/hexagon.md#serving). ## Reproducing From c10cb805342c95f1215acc6f1de2a5609d809884 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 17:00:49 +0700 Subject: [PATCH 07/38] pi05: flash attention in the VLM prefix and action expert on Hexagon The VLM's and the expert's attention took flash attention only on a view-taking kernel (SYCL). On Hexagon they ran explicit scores against a single KV head, which the fallback wrapper sends to the CPU. Where flash attention is on by default (Hexagon), all three sites now take the plain flash attention; the view-only paths keep use_fused_attention(). IQ-10: 1773 -> 1320 ms, 4.8e-4 from the CPU F32 reference, 3.5e-4 on a served LIBERO observation. Other models unchanged. --- src/models/pi05.cpp | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/src/models/pi05.cpp b/src/models/pi05.cpp index a73e916a..6eeece50 100644 --- a/src/models/pi05.cpp +++ b/src/models/pi05.cpp @@ -146,6 +146,14 @@ namespace { // straight off the [hd, H, S] views. Otherwise the unfused graph is used. bool use_fused_attention() { return vla::flash_attn_enabled() && vla::fa_takes_views(); } +// Where flash attention is on by default (Hexagon), the three sites also take the +// plain flash attention when the kernel does not take views. Their explicit +// scores are matmuls the NPU does not run: SigLIP's at head dim 72, and the VLM's +// and the expert's, whose single KV head the fallback wrapper sends to the CPU. +bool use_flash_attention() { + return use_fused_attention() || (vla::default_flash_attn() && vla::flash_attn_enabled()); +} + ggml_tensor * build_adarms( ggml_context * ctx, ggml_tensor * x, ggml_tensor * mod, int64_t h, float eps, ggml_tensor ** gate_out) { @@ -185,7 +193,7 @@ ggml_tensor * build_expert_layer( ggml_tensor * gate_ffn = ggml_view_1d(ctx, mod_ffn, h, (size_t) 2*h * sizeof(float)); ggml_tensor * xq_attn = fq_act_mod(ctx, &w.act_mod_qkv, w.fq_qkv, x_in, fqx->scale1_attn, fqx->shift_attn); ggml_tensor * h1 = gemma_attn(ctx, w, nullptr, positions, cfg, seq, cached_K, cached_V, mask, nullptr, nullptr, - GGML_TYPE_F32, use_fused_attention(), nullptr, x_in, nullptr, fqx->K16, fqx->V16, + GGML_TYPE_F32, use_flash_attention(), nullptr, x_in, nullptr, fqx->K16, fqx->V16, xq_attn, gate_attn); ggml_tensor * xq_ffn = fq_act_mod(ctx, &w.act_mod_gu, w.fq_gu, h1, fqx->scale1_ffn, fqx->shift_ffn); return gemma_mlp(ctx, w, nullptr, GGML_TYPE_F32, nullptr, h1, xq_ffn, gate_ffn); @@ -207,13 +215,13 @@ ggml_tensor * build_expert_layer( // out * gate + x reading the GEMM output in whatever type it has. ggml_tensor * xn = fused_adarms(ctx, x_in, mod_attn, at, *fa); ggml_tensor * o = gemma_attn(ctx, w, xn, positions, cfg, seq, cached_K, cached_V, mask, - nullptr, nullptr, at, use_fused_attention(), nullptr, nullptr, kv); + nullptr, nullptr, at, use_flash_attention(), nullptr, nullptr, kv); ggml_tensor * h1 = fused_gateres(ctx, x_in, o, mod_attn, *fg); ggml_tensor * xm = fused_adarms(ctx, h1, mod_ffn, at, *fa); return fused_gateres(ctx, h1, gemma_mlp(ctx, w, xm, at), mod_ffn, *fg); } - if (!mask && !kv && gemma_fused(w, at, use_fused_attention(), seq, seq + (cached_K ? cached_K->ne[2] : 0))) { + if (!mask && !kv && gemma_fused(w, at, use_flash_attention(), seq, seq + (cached_K ? cached_K->ne[2] : 0))) { auto part = [&](ggml_tensor * mod, int j) { return ggml_view_1d(ctx, mod, h, (size_t) j*h*sizeof(float)); }; @@ -237,7 +245,7 @@ ggml_tensor * build_expert_layer( // mm_act does not convert its input: a BF16 GEMM must be handed BF16 rows, // but adaRMS produces F32, whose bits would be misread as BF16. ggml_tensor * o_out = gemma_attn(ctx, w, as_type(ctx, x_norm, at), positions, cfg, seq, cached_K, cached_V, mask, - nullptr, nullptr, at, use_fused_attention(), x_norm, nullptr, kv, + nullptr, nullptr, at, use_flash_attention(), x_norm, nullptr, kv, fqx ? fqx->K16 : nullptr, fqx ? fqx->V16 : nullptr); o_out = as_type(ctx, o_out, GGML_TYPE_F32); @@ -544,10 +552,7 @@ std::unique_ptr pi05_create(const std::string& mmproj_path, std::printf("vla(pi05): expert weights INT4 (W4A8), %s\n", aex_w4_group > 0 ? ("group " + std::to_string(aex_w4_group)).c_str() : "per channel"); } - // On Hexagon (flash attention on by default) SigLIP takes the plain flash - // attention too: its explicit scores, head dim 72 against 256 keys, are an - // F32 x F32 matmul the HMX does not take, 266 ms of vector-unit time. - m->vis.vit.enc.cfg.flash_attn = use_fused_attention() || (vla::default_flash_attn() && vla::flash_attn_enabled()); + m->vis.vit.enc.cfg.flash_attn = use_flash_attention(); // Fused SigLIP blocks (modules/encoder.h, EncCfg::fused): SYCL, BF16, fused // attention. A FoldQuant GGUF takes them too: its integer sites are the LM's // and the expert's, and its vision tower is BF16 like any other checkpoint's. @@ -840,7 +845,7 @@ std::vector Pi05ModelArch::predict(const Inputs& in) { pl.rms_spec); else h = gemma_layer(C, pl.blk[i], h, t_prefix_pos, cfg, n_prefix_g, - nullptr, nullptr, t_prefix_mask, &cK[i], &cV[i], act_type, use_fused_attention(), + nullptr, nullptr, t_prefix_mask, &cK[i], &cV[i], act_type, use_flash_attention(), kv_inplace ? &kv[i] : nullptr); } (void) h; From 0b78eb9726dc1254832d81a67020b592d13f875f Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 17:20:22 +0700 Subject: [PATCH 08/38] hexagon: KV concats as row copies so they stay on the NPU MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The NPU runs a concat only as a dim-1 copy of whole rows, so the expert's dim-2 KV concats (π0, π0.5, SmolVLA) went to the CPU fallback every step. concat_dim2() in backend.h does them on Hexagon as a dim-1 concat of the [d0*d1, n] views, which writes the same bytes; other backends keep ggml_concat(a, b, 2). IQ-10: π0.5 1379 -> 1133 ms, SmolVLA 810 -> 638 ms, both with outputs identical to before on fixed inputs and on a served observation. --- src/backend.h | 22 ++++++++++++++++++++++ src/models/smolvla.cpp | 4 ++-- src/modules/gemma_expert.h | 4 ++-- 3 files changed, 26 insertions(+), 4 deletions(-) diff --git a/src/backend.h b/src/backend.h index 894f4004..3b50682a 100644 --- a/src/backend.h +++ b/src/backend.h @@ -191,6 +191,28 @@ inline ggml_tensor * fa_kv([[maybe_unused]] ggml_context * ctx, ggml_tensor * t) #endif } +/** + * @brief `ggml_concat(a, b, 2)` of [d0, d1, n] tensors, as the compiled-in + * backend runs it. + * + * Hexagon runs a concat on the NPU only as a dim-1 copy of whole rows; any other + * goes to the CPU fallback (src/backend_fallback.cpp). For contiguous inputs a + * dim-2 concat appends whole d0*d1 rows, so it is done as a dim-1 concat of the + * [d0*d1, n] views, which writes the same bytes. Everywhere else, and for + * non-contiguous inputs, this is ggml_concat(a, b, 2). + */ +inline ggml_tensor * concat_dim2(ggml_context * ctx, ggml_tensor * a, ggml_tensor * b) { +#ifdef GGML_USE_HEXAGON + if (ggml_is_contiguous(a) && ggml_is_contiguous(b) && a->ne[3] == 1 && b->ne[3] == 1 && + a->ne[0] == b->ne[0] && a->ne[1] == b->ne[1]) { + ggml_tensor * rows = ggml_concat(ctx, ggml_reshape_2d(ctx, a, a->ne[0] * a->ne[1], a->ne[2]), + ggml_reshape_2d(ctx, b, b->ne[0] * b->ne[1], b->ne[2]), 1); + return ggml_reshape_3d(ctx, rows, a->ne[0], a->ne[1], a->ne[2] + b->ne[2]); + } +#endif + return ggml_concat(ctx, a, b, 2); +} + /** * @brief A GEMM's inner width as the compiled-in backend's matmul takes it. * diff --git a/src/models/smolvla.cpp b/src/models/smolvla.cpp index 91059d62..95764fd8 100644 --- a/src/models/smolvla.cpp +++ b/src/models/smolvla.cpp @@ -708,8 +708,8 @@ ggml_tensor * build_expert_self_attn_layer( ggml_tensor * k_h = ggml_reshape_3d(ctx, mm_w(ctx, w.Wk, x_norm), cfg.head_dim, cfg.n_kv_heads, cfg.n_suffix); ggml_tensor * v_h = ggml_reshape_3d(ctx, mm_w(ctx, w.Wv, x_norm), cfg.head_dim, cfg.n_kv_heads, cfg.n_suffix); - ggml_tensor * K_full = ggml_concat(ctx, cached_K, rope(ctx, rope_spec(cfg), k_h, positions_full), 2); - ggml_tensor * V_full = ggml_concat(ctx, cached_V, v_h, 2); + ggml_tensor * K_full = concat_dim2(ctx, cached_K, rope(ctx, rope_spec(cfg), k_h, positions_full)); + ggml_tensor * V_full = concat_dim2(ctx, cached_V, v_h); return attn_mlp(ctx, w, x_in, rope(ctx, rope_spec(cfg), q_h, positions_full), K_full, V_full, mask_full, cfg, cfg.n_suffix); } diff --git a/src/modules/gemma_expert.h b/src/modules/gemma_expert.h index 1c66fe3f..e7641166 100644 --- a/src/modules/gemma_expert.h +++ b/src/modules/gemma_expert.h @@ -525,8 +525,8 @@ inline ggml_tensor * gemma_attn( ggml_tensor * K_full = k_rope; ggml_tensor * V_full = v_h; if (cached_K && cached_V) { - K_full = ggml_concat(ctx, cached_K, k_rope, 2); - V_full = ggml_concat(ctx, cached_V, v_h, 2); + K_full = concat_dim2(ctx, cached_K, k_rope); + V_full = concat_dim2(ctx, cached_V, v_h); } const float scale = 1.f/std::sqrt((float) hd); From 26ad84b88bb388c605b3f2e38e893f2a0b7ac332 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 17:42:54 +0700 Subject: [PATCH 09/38] =?UTF-8?q?docs:=20=CF=800.5=20attention=20and=20KV?= =?UTF-8?q?=20concats=20on=20the=20Hexagon=20NPU,=20IQ-10=20benchmark=20at?= =?UTF-8?q?=200b78eb9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hexagon.md: π0.5's VLM and action expert take flash attention on Hexagon, and the experts' KV concats run as row copies that stay on the NPU (π0 shares the code but has not been run on the board). π0.5's fidelity is re-measured: 4.8e-4 on fixed inputs, 3.5e-4 on a served observation. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 0b78eb9: π0.5 1117.2 ms, SmolVLA 633.0 ms; the other models are unchanged. The defaults stay the fastest configuration. --- CHANGELOG.md | 3 +++ docs/backend/hexagon.md | 20 +++++++++++---- docs/benchmark/README.md | 4 +-- docs/benchmark/qualcomm-iq10.md | 43 +++++++++++++++------------------ 4 files changed, 40 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3be3ac7b..7a0a0d1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,9 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke MLP is zero-padded to 4320 at load, π0.5's SigLIP takes flash attention, and Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, π0.5 and Evo-1 run 27-40% faster, within the bar on fixed inputs. + π0.5's VLM and action expert take flash attention too, and the experts' KV + concats run as row copies that stay on the NPU (π0.5, SmolVLA; π0 shares the + code, not yet run on the board). GR00T N1.5 and N1.6 are above the bar on a served LIBERO observation, as the CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install and the numbers. diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index c664eb90..2112f061 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -126,6 +126,14 @@ One more op is claimed and then refused at run time. On buffers mapped above `NO_SUPPORT`, and the host aborts on the first predict. The wrapper sends those concats to the CPU. +The experts' KV concats (π0, π0.5, SmolVLA) were such concats: along the token +dim of contiguous [head dim, heads, tokens] tensors, once per layer per denoise +step. Appending along that dim appends whole rows, so on Hexagon +`concat_dim2()` in `src/backend.h` does them as a dim-1 concat of the +[head dim x heads, tokens] views, which writes the same bytes and stays on the +NPU: π0.5 1379 to 1133 ms, SmolVLA 810 to 638 ms, outputs unchanged. π0 shares +the code but has not been run on the board yet. + Flash attention on the HMX (v81) keeps scores and probabilities in F16. That is a precision limit rather than a wrong op, but in GR00T N1.7's LM it moves the action chunk 5.4e-3 from the CPU reference. N1.7's LM therefore takes the @@ -145,9 +153,11 @@ Hexagon builds: - **SigLIP-so400m's MLP is 4304 wide**: 440 ms of GR00T N1.5's 565 ms tower. The loader zero-pads fc1, its bias and fc2 to 4320 (`WeightLoader::pad`, `gemm_width()` in `src/backend.h`), which leaves the output unchanged. -- **π0.5's SigLIP ran explicit attention** (head dim 72, F32 x F32), because - its flash attention was tied to SYCL's view-taking kernel. It now takes the - plain flash attention, as GR00T's tower does. +- **π0.5's attention ran explicit scores** in SigLIP (head dim 72, F32 x F32), + the VLM and the action expert, because its flash attention was tied to SYCL's + view-taking kernel. The VLM's and the expert's single KV head also sent their + matmuls to the CPU (the batched `MUL_MAT` rule above). All three now take the + plain flash attention on Hexagon (`use_flash_attention()` in `pi05.cpp`). - **Evo-1's LM attention multiplied F32 K and V against 14 query heads.** K and V now go to F16 (`fa_kv()`); the scores still accumulate in F32. @@ -308,7 +318,7 @@ path that other builds run: | Model | max\|Δ\| | |---|--:| | SmolVLA | 7.2e-4 | -| π0.5 | 2.8e-4 | +| π0.5 | 4.8e-4 | | Evo-1 | 5.0e-4 | | GR00T N1.5 | 2.4e-3 | | GR00T N1.6 | 1.4e-3 | @@ -330,7 +340,7 @@ malformed or an invalid request, and keeps serving afterwards. | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 | |---|--:|--:| | SmolVLA | 1.3e-3 | | -| π0.5 | 4.2e-4 | | +| π0.5 | 3.5e-4 | | | Evo-1 | 1.1e-3 | | | GR00T N1.5 | 1.1e-2 | 4.3e-2 | | GR00T N1.6 | 1.3e-2 | 1.1e-2 | diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index a2befb19..b3825840 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`74e145d` (same llama.cpp pin). Each report +`0b78eb9` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 613.6 | 706.9 | 746.5 | N/A | 811.5 | n/r | 1789.4 | n/r | 2589.2 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 614.3 | 696.4 | 750.3 | N/A | 633.0 | n/r | 1117.2 | n/r | 2588.6 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 9415b204..4d843293 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `74e145d` (branch `hexagon-b11223-ops`, on top of #68 to #70) | +| Commit | `0b78eb9` (branch `hexagon-pi05-attn`, on top of #71) | | llama.cpp | `b11223` | -| Date | 2026-11-03 10:43 to 11:02 (UTC+07) | +| Date | 2026-11-03 17:20 to 17:37 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -65,12 +65,12 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.5 | 1 | 224 | 613.6 | 615.7 | 614.8 | 618.5 | 148.2 | 162 | 3644 | -| GR00T N1.6 | 1 | 224 | 706.9 | 710.9 | 710.3 | 715.0 | 145.3 | 147 | 4688 | -| GR00T N1.7 | 1 | 256 | 746.5 | 754.0 | 753.4 | 760.6 | 195.7 | 191 | 5047 | -| SmolVLA | 2 | 512 | 811.5 | 816.3 | 815.9 | 820.2 | 283.1 | 146 | 1234 | -| π0.5 | 2 | 224 | 1789.4 | 1820.3 | 1818.5 | 1850.2 | 296.7 | 285 | 5783 | -| Evo-1 | 1 | 448 | 2589.2 | 2602.0 | 2598.8 | 2613.3 | 853.9 | 319 | 1588 | +| GR00T N1.5 | 1 | 224 | 614.3 | 616.7 | 616.1 | 619.6 | 148.5 | 162 | 3661 | +| SmolVLA | 2 | 512 | 633.0 | 633.5 | 633.4 | 634.0 | 282.1 | 146 | 1193 | +| GR00T N1.6 | 1 | 224 | 696.4 | 701.7 | 700.9 | 706.0 | 144.5 | 147 | 4700 | +| GR00T N1.7 | 1 | 256 | 750.3 | 756.2 | 755.2 | 763.3 | 196.9 | 191 | 5093 | +| π0.5 | 2 | 224 | 1117.2 | 1124.3 | 1123.8 | 1127.1 | 295.3 | 285 | 5786 | +| Evo-1 | 1 | 448 | 2588.6 | 2602.2 | 2600.9 | 2616.2 | 857.1 | 319 | 1573 | ### Fastest configuration @@ -94,23 +94,23 @@ rates measured at the defaults therefore do not carry over to these rows. | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.5 | *(defaults)* | 613.6 | 615.7 | 614.8 | 618.5 | 148.2 | 162 | 3644 | - | -| GR00T N1.6 | *(defaults)* | 706.9 | 710.9 | 710.3 | 715.0 | 145.3 | 147 | 4688 | - | -| GR00T N1.7 | *(defaults)* | 746.5 | 754.0 | 753.4 | 760.6 | 195.7 | 191 | 5047 | - | -| SmolVLA | *(defaults)* | 811.5 | 816.3 | 815.9 | 820.2 | 283.1 | 146 | 1234 | - | -| π0.5 | *(defaults)* | 1789.4 | 1820.3 | 1818.5 | 1850.2 | 296.7 | 285 | 5783 | - | -| Evo-1 | *(defaults)* | 2589.2 | 2602.0 | 2598.8 | 2613.3 | 853.9 | 319 | 1588 | - | +| GR00T N1.5 | *(defaults)* | 614.3 | 616.7 | 616.1 | 619.6 | 148.5 | 162 | 3661 | - | +| SmolVLA | *(defaults)* | 633.0 | 633.5 | 633.4 | 634.0 | 282.1 | 146 | 1193 | - | +| GR00T N1.6 | *(defaults)* | 696.4 | 701.7 | 700.9 | 706.0 | 144.5 | 147 | 4700 | - | +| GR00T N1.7 | *(defaults)* | 750.3 | 756.2 | 755.2 | 763.3 | 196.9 | 191 | 5093 | - | +| π0.5 | *(defaults)* | 1117.2 | 1124.3 | 1123.8 | 1127.1 | 295.3 | 285 | 5786 | - | +| Evo-1 | *(defaults)* | 2588.6 | 2602.2 | 2600.9 | 2616.2 | 857.1 | 319 | 1573 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| GR00T N1.5 | 616.1 | 615.0 | 619.5 | 615.9 | 1096.1 | -| GR00T N1.6 | 716.2 | 698.0 | 700.7 | 696.3 | 873.4 | -| GR00T N1.7 | 753.6 | 733.7 | 764.7 | 731.8 | 1070.6 | -| SmolVLA | 828.8 | 818.5 | 812.5 | 822.8 | 3012.6 | -| π0.5 | 1815.6 | 1816.9 | 11581.7 | 1822.6 | 2144.2 | -| Evo-1 | 2614.6 | 2599.1 | 2606.5 | 2605.6 | 4870.0 | +| GR00T N1.5 | 616.7 | 620.3 | 616.1 | 616.4 | 1095.2 | +| SmolVLA | 634.6 | 634.2 | 634.9 | 634.6 | 2830.8 | +| GR00T N1.6 | 699.9 | 701.5 | 702.9 | 698.8 | 872.8 | +| GR00T N1.7 | 758.4 | 743.5 | 761.2 | 732.3 | 1073.6 | +| π0.5 | 1152.2 | 1151.7 | 1153.3 | 1152.4 | 2127.4 | +| Evo-1 | 2612.8 | 2611.6 | 2600.5 | 2600.4 | 4832.9 |
@@ -131,9 +131,6 @@ GR00T N1.6 runs one view here, as the configuration above fixes. With two views its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). -In the screen, π0.5 with `--mm-prec default` took 11.6 s against 1.8 s at the -defaults. That was not investigated. - These rows time the defaults; they do not vouch for accuracy. On a served LIBERO observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar: N1.5 closer to the reference than the CPU at F16, N1.6 slightly further; see From 1ca86332a973a80f8463701b7593017f2186aa14 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 11:42:34 +0700 Subject: [PATCH 10/38] hexagon: run weight-reading copies on the CPU ggml-hexagon claims a CPY whose source is a weight, then answers the batch with NO_SUPPORT and the host aborts. VLA-Adapter casts its F16 action queries to F32 in the graph and aborted on every predict on the Dragonwing IQ-10. With the cast on the CPU it is within 4.2e-4 of the CPU F32 reference, 4.5e-4 when served. --- src/backend_fallback.cpp | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/backend_fallback.cpp b/src/backend_fallback.cpp index 7e0d4035..a7629eaf 100644 --- a/src/backend_fallback.cpp +++ b/src/backend_fallback.cpp @@ -351,6 +351,14 @@ bool accel_runs(FallbackCtx * fc, const ggml_tensor * t, const char ** why) { *why = "miscomputes a 2D weight against a batch in"; return false; } + if (fc->hexagon && t->op == GGML_OP_CPY && is_weight(t->src[0])) { + // Claimed, but a copy that reads a weight is answered with NO_SUPPORT + // and the host aborts. VLA-Adapter casts its F16 action queries + // [896,64] to F32 in the graph; with the cast on the CPU it is within + // 4.2e-4 of the CPU F32 reference. + *why = "cannot run (NO_SUPPORT) a weight-reading"; + return false; + } return ggml_backend_supports_op(fc->accel, t); } From 1cab1c3c733099b63c0b150dcc7743fd39137da6 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 11:42:34 +0700 Subject: [PATCH 11/38] hexagon: run REPEAT of a computed tensor on the CPU The NPU's REPEAT reads its source through the data cache, which is invalidated only between op batches, while ops write their output by DMA. When the source is written earlier in the same batch, REPEAT can read stale lines. TurboVLA's ViT prefix, repeated over two views, came out 0.19 off the CPU F32 reference and different on every run; with the repeat on the CPU it is 3.1e-4 and repeatable, 2.2e-4 when served. --- src/backend_fallback.cpp | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/backend_fallback.cpp b/src/backend_fallback.cpp index a7629eaf..151692b9 100644 --- a/src/backend_fallback.cpp +++ b/src/backend_fallback.cpp @@ -359,6 +359,15 @@ bool accel_runs(FallbackCtx * fc, const ggml_tensor * t, const char ** why) { *why = "cannot run (NO_SUPPORT) a weight-reading"; return false; } + if (fc->hexagon && t->op == GGML_OP_REPEAT && !is_weight(t->src[0])) { + // The NPU's REPEAT reads its source through the data cache, which is + // invalidated only between op batches, while the op that wrote the + // source used DMA, so within one batch it can read stale lines. + // TurboVLA repeats its ViT prefix [768,5] over two views: max|delta| + // 0.19 and different on every run, 3.1e-4 with the repeat on the CPU. + *why = "can read stale data in a"; + return false; + } return ggml_backend_supports_op(fc->accel, t); } From 6aec07081b6bdcf079b3562a04f6207ab428a925 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 13:50:56 +0700 Subject: [PATCH 12/38] docs: six more archs on the Hexagon NPU, IQ-10 benchmark at 1cab1c3 --- CHANGELOG.md | 33 ++++++++-------- docs/backend/hexagon.md | 70 ++++++++++++++++++++------------- docs/benchmark/qualcomm-iq10.md | 66 +++++++++++++++++++------------ 3 files changed, 101 insertions(+), 68 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a0a0d1d..5a91d22f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,22 +9,23 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke - **Hexagon NPU on the Qualcomm Dragonwing IQ-10 (Linux).** `scripts/build_linux_hexagon.sh` cross-builds vla.cpp with `GGML_HEXAGON=ON` in Qualcomm's toolchain container, `vla-server` included. On the IQ-10 (v81, - llama.cpp `b11223`) SmolVLA, π0.5, Evo-1 and GR00T N1.5 / N1.6 / N1.7 run - within 2.9e-3 of the CPU build on fixed inputs and serve through - `eval/client`. Concats the NPU refuses on buffers above 4 GB of DSP address - space and matmuls of a 2D weight against a batch, which it computes wrong - (GR00T N1.6 with two views), run on the CPU, and GR00T N1.7's LM leaves - flash attention (F16 scores on the HMX) unless `--flash-attn` is given. - Three matmuls that the HMX does not take move onto it: SigLIP's 4304-wide - MLP is zero-padded to 4320 at load, π0.5's SigLIP takes flash attention, and - Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, π0.5 and Evo-1 run - 27-40% faster, within the bar on fixed inputs. - π0.5's VLM and action expert take flash attention too, and the experts' KV - concats run as row copies that stay on the NPU (π0.5, SmolVLA; π0 shares the - code, not yet run on the board). - GR00T N1.5 and N1.6 are above the bar on a served LIBERO observation, as the - CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install - and the numbers. + llama.cpp `b11223`) SmolVLA, π0, π0.5, Evo-1, GR00T N1.5 / N1.6 / N1.7, + VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small and TurboVLA run within 2.9e-3 + of the CPU build on fixed inputs and serve through `eval/client`. Concats the + NPU refuses on buffers above 4 GB of DSP address space, copies that read a + weight (also answered `NO_SUPPORT`; VLA-Adapter), matmuls of a 2D weight + against a batch, which it computes wrong (GR00T N1.6 with two views), and + repeats of a computed tensor, which can read stale cache lines (TurboVLA), run + on the CPU, and GR00T N1.7's LM leaves flash attention (F16 scores on the + HMX) unless `--flash-attn` is given. Three matmuls that the HMX does not take + move onto it: SigLIP's 4304-wide MLP is zero-padded to 4320 at load, π0.5's + SigLIP takes flash attention, and Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, + π0.5 and Evo-1 run 27-40% faster, within the bar on fixed inputs. π0.5's VLM + and action expert take flash attention too, and the experts' KV concats run as + row copies that stay on the NPU (π0, π0.5, SmolVLA). GR00T N1.5, N1.6 and + VLA-JEPA are above the bar on a served LIBERO observation, as the CPU is at + F16. See `docs/backend/hexagon.md` for the build, the skel install and the + numbers. - **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200` (Radeon RX 9060 XT) with ROCm 7.14 and llama.cpp `b11223`, every arch except diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 2112f061..213e22e4 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -112,6 +112,7 @@ these, and they compute something else: | `ADD`/`MUL`/... | `src1` broadcast over one dim but not a higher one | Refused, runs on the CPU | | `IM2COL` | Wrong unless kernel = stride > 1 and no padding | Refused, runs on the CPU | | `CPY` F32→F16 | Reads back wrong on the host | Copies feeding a CPU op run on the CPU | +| `REPEAT` | Reads stale cache lines of a source written by DMA earlier in the same op batch | Refused unless the source is a weight, runs on the CPU | | fused `MUL_MAT+ADD`, `RMS_NORM+MUL` | ~10x the error of the unfused ops | `GGML_HEXAGON_OPFUSION=0` by default | Each one flips the gripper channel, or comes close, on some checkpoint. The @@ -120,19 +121,21 @@ was right: a missing op fails at load, and these only show in the actions. [hexagon-windows.md](hexagon-windows.md#hexagon-issues-found-and-worked-around) has the numbers and how each was bisected. They belong upstream. -One more op is claimed and then refused at run time. On buffers mapped above +Two more ops are claimed and then refused at run time. On buffers mapped above 4 GB of DSP address space, which the IQ-10 under Linux hands out, the NPU runs `CONCAT` only as the dim-1 contiguous DMA copy and answers any other concat with `NO_SUPPORT`, and the host aborts on the first predict. The wrapper sends those -concats to the CPU. +concats to the CPU. A `CPY` that reads a weight gets the same answer +(VLA-Adapter casts its F16 action queries to F32 in the graph), and also runs on +the CPU. The experts' KV concats (π0, π0.5, SmolVLA) were such concats: along the token dim of contiguous [head dim, heads, tokens] tensors, once per layer per denoise step. Appending along that dim appends whole rows, so on Hexagon `concat_dim2()` in `src/backend.h` does them as a dim-1 concat of the [head dim x heads, tokens] views, which writes the same bytes and stays on the -NPU: π0.5 1379 to 1133 ms, SmolVLA 810 to 638 ms, outputs unchanged. π0 shares -the code but has not been run on the board yet. +NPU: π0.5 1379 to 1133 ms, SmolVLA 810 to 638 ms, outputs unchanged. π0 takes +the same path. Flash attention on the HMX (v81) keeps scores and probabilities in F16. That is a precision limit rather than a wrong op, but in GR00T N1.7's LM it moves the @@ -313,29 +316,36 @@ export LD_LIBRARY_PATH=$PWD/pkg/lib `vla_predict_check` at the Hexagon defaults, against the same binary with `VLA_DEVICE=cpu --weight-dtype f32 --flash-attn 0`. A Hexagon build turns flash attention on by default even on the CPU, so the reference asks for the explicit -path that other builds run: +path that other builds run. Views are the ones the LIBERO client sends, so they +differ from the benchmark's for VLA-JEPA, VLA-Adapter, OpenVLA-OFT and Octo: -| Model | max\|Δ\| | -|---|--:| -| SmolVLA | 7.2e-4 | -| π0.5 | 4.8e-4 | -| Evo-1 | 5.0e-4 | -| GR00T N1.5 | 2.4e-3 | -| GR00T N1.6 | 1.4e-3 | -| GR00T N1.7 | 2.0e-3 | - -GR00T N1.6 here takes one view; with two it is 1.9e-3. The other archs have -not been run on the board yet. +| Model | Views | max\|Δ\| | +|---|--:|--:| +| SmolVLA | 2 | 7.2e-4 | +| π0 | 2 | 1.2e-3 | +| π0.5 | 2 | 4.8e-4 | +| Evo-1 | 1 | 5.5e-4 | +| GR00T N1.5 | 1 | 2.4e-3 | +| GR00T N1.6 | 1 | 1.4e-3 | +| GR00T N1.7 | 1 | 2.0e-3 | +| VLA-JEPA | 2 | 2.8e-3 | +| VLA-Adapter | 2 | 4.2e-4 | +| OpenVLA-OFT | 2 | 3.5e-4 | +| Octo-Small | 1 | 4.1e-4 | +| TurboVLA | 2 | 3.1e-4 | + +GR00T N1.6 with two views is 1.9e-3. At the benchmark's single view VLA-JEPA is +1.9e-3, VLA-Adapter 4.8e-4 and OpenVLA-OFT 2.2e-4. ### Serving `vla-server` on the board, driven from the host by `eval/client/vla_cpp_client.py` and the adapters `run_sim_client_direct.py` -uses, on one fixed LIBERO observation (two views, the task prompt, a fixed -robot state) with fixed noise. The same request goes to a Hexagon server and to -a CPU server with `--weight-dtype f32 --flash-attn 0`. Every model returns a -finite chunk, the same chunk for a repeated request, an error reply to a -malformed or an invalid request, and keeps serving afterwards. +uses, on one fixed LIBERO observation (two views, the task prompt, a fixed robot +state), with fixed noise for the archs that draw it. The same request goes to a +Hexagon server and to a CPU server with `--weight-dtype f32 --flash-attn 0`. +Every model returns a finite chunk, the same chunk for a repeated request, an +error reply to a malformed or an invalid request, and keeps serving afterwards. | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 | |---|--:|--:| @@ -345,12 +355,18 @@ malformed or an invalid request, and keeps serving afterwards. | GR00T N1.5 | 1.1e-2 | 4.3e-2 | | GR00T N1.6 | 1.3e-2 | 1.1e-2 | | GR00T N1.7 | 2.5e-3 | | - -GR00T N1.5 and N1.6 are above the bar on this input, and so is the CPU itself -with F16 weights: N1.5 lands closer to the reference than the CPU at F16, N1.6 -slightly further (1.3e-2 against 1.1e-2). They are sensitive to 16-bit -precision rather than miscomputed. Running them with F32 weights on the NPU has not been tried -yet. +| π0 | 5.8e-4 | | +| VLA-JEPA | 6.7e-3 | 2.1e-2 | +| VLA-Adapter | 4.5e-4 | | +| OpenVLA-OFT | 5.0e-4 | | +| Octo-Small | 9.3e-5 | | +| TurboVLA | 2.2e-4 | | + +GR00T N1.5, N1.6 and VLA-JEPA are above the bar on this input, and so is the CPU +itself with F16 weights: N1.5 and VLA-JEPA land closer to the reference than the +CPU at F16, N1.6 slightly further (1.3e-2 against 1.1e-2). They are sensitive to +16-bit precision rather than miscomputed. Running them with F32 weights on the +NPU has not been tried yet. ## IQ-10, imagined diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 4d843293..6dc1d5c8 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `0b78eb9` (branch `hexagon-pi05-attn`, on top of #71) | +| Commit | `1cab1c3` | | llama.cpp | `b11223` | -| Date | 2026-11-03 17:20 to 17:37 (UTC+07) | +| Date | 2026-11-04 11:52 to 12:30 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -65,12 +65,18 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.5 | 1 | 224 | 614.3 | 616.7 | 616.1 | 619.6 | 148.5 | 162 | 3661 | -| SmolVLA | 2 | 512 | 633.0 | 633.5 | 633.4 | 634.0 | 282.1 | 146 | 1193 | -| GR00T N1.6 | 1 | 224 | 696.4 | 701.7 | 700.9 | 706.0 | 144.5 | 147 | 4700 | -| GR00T N1.7 | 1 | 256 | 750.3 | 756.2 | 755.2 | 763.3 | 196.9 | 191 | 5093 | -| π0.5 | 2 | 224 | 1117.2 | 1124.3 | 1123.8 | 1127.1 | 295.3 | 285 | 5786 | -| Evo-1 | 1 | 448 | 2588.6 | 2602.2 | 2600.9 | 2616.2 | 857.1 | 319 | 1573 | +| Octo-Small | 2 | 256 + 128 | 121.6 | 123.0 | 122.3 | 125.8 | 20.5 | 253 | 804 | +| TurboVLA | 2 | 256 | 428.1 | 433.0 | 431.6 | 436.8 | - | 438 | 1327 | +| VLA-JEPA | 1 | 256 | 440.2 | 446.3 | 444.7 | 450.2 | 195.2 | 185 | 3971 | +| GR00T N1.5 | 1 | 224 | 613.7 | 615.7 | 615.7 | 618.0 | 148.3 | 162 | 3636 | +| SmolVLA | 2 | 512 | 631.0 | 632.4 | 631.7 | 634.6 | 281.8 | 146 | 1190 | +| GR00T N1.6 | 1 | 224 | 705.8 | 708.5 | 708.0 | 710.7 | 145.0 | 146 | 4869 | +| GR00T N1.7 | 1 | 256 | 747.0 | 756.0 | 755.0 | 764.5 | 197.2 | 191 | 5090 | +| π0.5 | 2 | 224 | 1117.4 | 1121.0 | 1120.8 | 1123.3 | 292.5 | 285 | 5790 | +| π0 | 2 | 224 | 1124.4 | 1128.7 | 1128.6 | 1130.5 | 293.0 | 285 | 5534 | +| VLA-Adapter | 1 | 224 | 1733.6 | 1752.3 | 1749.5 | 1765.6 | 1187.2 | 872 | 3566 | +| Evo-1 | 1 | 448 | 2579.1 | 2595.5 | 2592.1 | 2611.9 | 854.0 | 319 | 1551 | +| OpenVLA-OFT | 1 | 224 | 4097.7 | 4107.8 | 4107.8 | 4114.2 | 1187.3 | 844 | 15327 | ### Fastest configuration @@ -94,23 +100,35 @@ rates measured at the defaults therefore do not carry over to these rows. | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.5 | *(defaults)* | 614.3 | 616.7 | 616.1 | 619.6 | 148.5 | 162 | 3661 | - | -| SmolVLA | *(defaults)* | 633.0 | 633.5 | 633.4 | 634.0 | 282.1 | 146 | 1193 | - | -| GR00T N1.6 | *(defaults)* | 696.4 | 701.7 | 700.9 | 706.0 | 144.5 | 147 | 4700 | - | -| GR00T N1.7 | *(defaults)* | 750.3 | 756.2 | 755.2 | 763.3 | 196.9 | 191 | 5093 | - | -| π0.5 | *(defaults)* | 1117.2 | 1124.3 | 1123.8 | 1127.1 | 295.3 | 285 | 5786 | - | -| Evo-1 | *(defaults)* | 2588.6 | 2602.2 | 2600.9 | 2616.2 | 857.1 | 319 | 1573 | - | +| Octo-Small | *(defaults)* | 121.6 | 123.0 | 122.3 | 125.8 | 20.5 | 253 | 804 | - | +| TurboVLA | *(defaults)* | 428.1 | 433.0 | 431.6 | 436.8 | - | 438 | 1327 | - | +| VLA-JEPA | *(defaults)* | 440.2 | 446.3 | 444.7 | 450.2 | 195.2 | 185 | 3971 | - | +| GR00T N1.5 | *(defaults)* | 613.7 | 615.7 | 615.7 | 618.0 | 148.3 | 162 | 3636 | - | +| SmolVLA | *(defaults)* | 631.0 | 632.4 | 631.7 | 634.6 | 281.8 | 146 | 1190 | - | +| GR00T N1.6 | *(defaults)* | 705.8 | 708.5 | 708.0 | 710.7 | 145.0 | 146 | 4869 | - | +| GR00T N1.7 | *(defaults)* | 747.0 | 756.0 | 755.0 | 764.5 | 197.2 | 191 | 5090 | - | +| π0.5 | *(defaults)* | 1117.4 | 1121.0 | 1120.8 | 1123.3 | 292.5 | 285 | 5790 | - | +| π0 | *(defaults)* | 1124.4 | 1128.7 | 1128.6 | 1130.5 | 293.0 | 285 | 5534 | - | +| VLA-Adapter | *(defaults)* | 1733.6 | 1752.3 | 1749.5 | 1765.6 | 1187.2 | 872 | 3566 | - | +| Evo-1 | *(defaults)* | 2579.1 | 2595.5 | 2592.1 | 2611.9 | 854.0 | 319 | 1551 | - | +| OpenVLA-OFT | *(defaults)* | 4097.7 | 4107.8 | 4107.8 | 4114.2 | 1187.3 | 844 | 15327 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| GR00T N1.5 | 616.7 | 620.3 | 616.1 | 616.4 | 1095.2 | -| SmolVLA | 634.6 | 634.2 | 634.9 | 634.6 | 2830.8 | -| GR00T N1.6 | 699.9 | 701.5 | 702.9 | 698.8 | 872.8 | -| GR00T N1.7 | 758.4 | 743.5 | 761.2 | 732.3 | 1073.6 | -| π0.5 | 1152.2 | 1151.7 | 1153.3 | 1152.4 | 2127.4 | -| Evo-1 | 2612.8 | 2611.6 | 2600.5 | 2600.4 | 4832.9 | +| Octo-Small | 125.3 | 128.8 | 124.4 | 123.5 | 124.6 | +| TurboVLA | 438.8 | 436.5 | 439.3 | 438.2 | 535.1 | +| VLA-JEPA | 446.7 | 445.8 | 444.8 | 445.2 | 817.8 | +| GR00T N1.5 | 615.8 | 619.3 | 615.3 | 618.4 | 1096.5 | +| SmolVLA | 632.1 | 633.0 | 632.5 | 632.6 | 2834.8 | +| GR00T N1.6 | 714.2 | 712.1 | 718.7 | 714.9 | 885.7 | +| GR00T N1.7 | 765.2 | 742.0 | 761.5 | 743.6 | 1081.0 | +| π0.5 | 1121.1 | 1121.9 | 1120.5 | 1121.4 | 2062.9 | +| π0 | 1129.8 | 1128.8 | 1131.8 | 1128.5 | 2090.5 | +| VLA-Adapter | 1766.1 | 1760.1 | 1756.1 | 1757.2 | 1767.1 | +| Evo-1 | 2613.0 | 2613.7 | 2617.3 | 2617.9 | 4840.1 | +| OpenVLA-OFT | 4110.3 | 4101.9 | 4101.7 | 4108.2 | 4107.9 |
@@ -118,8 +136,6 @@ rates measured at the defaults therefore do not carry over to these rows. - **BitVLA**: N/A off CUDA. The only published GGUF is int2-packed, and only CUDA builds load it. -- **π0, VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small, TurboVLA**: not run in - this round; their GGUFs were not on the board. ## Notes @@ -132,9 +148,9 @@ its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). These rows time the defaults; they do not vouch for accuracy. On a served LIBERO -observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar: N1.5 -closer to the reference than the CPU at F16, N1.6 slightly further; see -[backend/hexagon.md](../backend/hexagon.md#serving). +observation, GR00T N1.5, N1.6 and VLA-JEPA land above the 2.9e-3 fidelity bar: +N1.5 and VLA-JEPA closer to the reference than the CPU at F16, N1.6 slightly +further; see [backend/hexagon.md](../backend/hexagon.md#serving). ## Reproducing From 9d49a3690972030ee6adce283f8a903cef4ef523 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 14:09:52 +0700 Subject: [PATCH 13/38] encoder: leave an MLP unpadded when its weights are quantized in the file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pad_mlp() checked only the loader's GEMM type. scripts/quantize_gguf.py --vision packs SigLIP's fc1 [1152, 4304] to Q8_0 while fc2, 4304 wide, stays float; the loader keeps fc1 packed and declare() refused to pad it, so such a GGUF failed to load in Hexagon builds. An MLP with either weight quantized in the file now loads unpadded, as it did before the padding. Float checkpoints are unchanged: π0, π0.5, GR00T N1.5/N1.6/N1.7, VLA-Adapter and OpenVLA-OFT give bit-identical actions on the Dragonwing IQ-10. --- src/modules/encoder.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/modules/encoder.cpp b/src/modules/encoder.cpp index 5c6fa428..458436ed 100644 --- a/src/modules/encoder.cpp +++ b/src/modules/encoder.cpp @@ -37,7 +37,8 @@ namespace { void pad_mlp(WeightLoader & L, const std::string & fc1, const std::string & fc2) { const ggml_tensor * w1 = L.reader().meta((fc1 + ".weight").c_str()); const ggml_tensor * w2 = L.reader().meta((fc2 + ".weight").c_str()); - if (!w1 || !w2 || ggml_is_quantized(L.gemm_type())) + if (!w1 || !w2 || ggml_is_quantized(L.gemm_type()) || ggml_is_quantized(w1->type) || + ggml_is_quantized(w2->type)) return; const int64_t ff = w1->ne[1]; const int64_t fp = gemm_width(ff); From 7dc357119ca684b9c094be1e5d80106ea7cd6abc Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 16:32:41 +0700 Subject: [PATCH 14/38] docs: list the twelve IQ-10 archs in the Hexagon status and benchmark summary The previous docs commit added six archs to the IQ-10 pages but left two summaries behind: the status note in docs/backend/hexagon.md still said six checkpoints run on the IQ-10, and the IQ-10 row of docs/benchmark/README.md still carried the 0b78eb9 numbers with the six new archs marked not run. Both now match docs/benchmark/qualcomm-iq10.md at 1cab1c3. --- docs/backend/hexagon.md | 6 +++--- docs/benchmark/README.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 213e22e4..d9196bb4 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -5,9 +5,9 @@ to take once it ran. > **Status: running on Windows on Arm and on a Dragonwing IQ-10 under Linux.** > vla.cpp builds with `GGML_HEXAGON=ON` and runs eleven checkpoints on a -> Snapdragon X laptop's v73 NPU and six on an IQ-10's v81 NPU under Linux, each -> checked against a CPU reference. The laptop build, its measurements and the -> upstream bugs found along the way are in +> Snapdragon X laptop's v73 NPU and twelve on an IQ-10's v81 NPU under Linux, +> each checked against a CPU reference. The laptop build, its measurements and +> the upstream bugs found along the way are in > [hexagon-windows.md](hexagon-windows.md); the Linux board is in [Linux > (IQ-10)](#linux-iq-10). What follows is the backend in general: the parts that > hold on any Snapdragon part, and the IQ-9 and multi-NPU notes from before diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index b3825840..54f0f3c5 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`0b78eb9` (same llama.cpp pin). Each report +`1cab1c3` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 614.3 | 696.4 | 750.3 | N/A | 633.0 | n/r | 1117.2 | n/r | 2588.6 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 121.6 | 428.1 | 440.2 | 1733.6 | 613.7 | 705.8 | 747.0 | N/A | 631.0 | 1124.4 | 1117.4 | 4097.7 | 2579.1 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | From 9f443548492354dfb9b2823359d5f790a9ddc565 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 15:49:07 +0700 Subject: [PATCH 15/38] dual_tower: pad SigLIP's MLP to a multiple of 32 on Hexagon OpenVLA-OFT and VLA-Adapter declare their DINOv2 and SigLIP towers in modules/dual_tower.h rather than through EncStack, so the zero padding of SigLIP's 4304-wide MLP did not reach them, and its fc1 and fc2 ran on the vector units: 425 ms per predict on the Dragonwing IQ-10. pad_mlp() moves into WeightLoader so both declare paths use it. VLA-Adapter goes from 1752 to 1352 ms and OpenVLA-OFT from 4108 to 3711 ms (mean, vla-bench), within 4.8e-4 and 3.4e-4 of the CPU F32 reference and 6.7e-4 / 5.6e-4 when served. The other ten archs give bit-identical actions. --- src/loader.cpp | 14 ++++++++++++++ src/loader.h | 6 ++++++ src/modules/dual_tower.h | 2 ++ src/modules/encoder.cpp | 24 +----------------------- 4 files changed, 23 insertions(+), 23 deletions(-) diff --git a/src/loader.cpp b/src/loader.cpp index fc6d95f7..7730d6b5 100644 --- a/src/loader.cpp +++ b/src/loader.cpp @@ -44,6 +44,20 @@ void WeightLoader::pad(const std::string & name, int64_t ne0, int64_t ne1) { pad_[name] = {ne0, ne1}; } +void WeightLoader::pad_mlp(const std::string & fc1, const std::string & fc2) { + const ggml_tensor * w1 = g_.meta((fc1 + ".weight").c_str()); + const ggml_tensor * w2 = g_.meta((fc2 + ".weight").c_str()); + if (!w1 || !w2 || ggml_is_quantized(gemm_) || ggml_is_quantized(w1->type) || ggml_is_quantized(w2->type)) + return; + const int64_t ff = w1->ne[1]; + const int64_t fp = gemm_width(ff); + if (fp == ff || w2->ne[0] != ff) + return; + pad(fc1 + ".weight", w1->ne[0], fp); + pad(fc1 + ".bias", fp, 1); + pad(fc2 + ".weight", fp, w2->ne[1]); +} + bool WeightLoader::read_resident(const std::string & name, ggml_type type, bool gemma_norm, std::vector & out) { const auto it = as_float_.find(name); if (it == as_float_.end()) { diff --git a/src/loader.h b/src/loader.h index e44d01e6..cec75760 100644 --- a/src/loader.h +++ b/src/loader.h @@ -86,6 +86,12 @@ class WeightLoader { // the file, with the extra elements zero. Call before declaring it. void pad(const std::string & name, int64_t ne0, int64_t ne1); + // Widens the MLP fc1 -> GELU -> fc2 (`fc1`/`fc2` without ".weight") to + // gemm_width() with zeros: the extra fc1 rows and bias are 0, GELU(0) is 0 and + // the extra fc2 columns are 0, so the output is the same. Weights quantized in + // the file or by the loader stay as they are. Call before declaring them. + void pad_mlp(const std::string & fc1, const std::string & fc2); + // For helpers that inspect the file before declaring (foldquant.cpp). gguf_reader & reader() { return g_; diff --git a/src/modules/dual_tower.h b/src/modules/dual_tower.h index 330b4838..b08c7299 100644 --- a/src/modules/dual_tower.h +++ b/src/modules/dual_tower.h @@ -73,6 +73,8 @@ struct DualTower { v.resize(n); for (int64_t i=0; itype) || - ggml_is_quantized(w2->type)) - return; - const int64_t ff = w1->ne[1]; - const int64_t fp = gemm_width(ff); - if (fp == ff || w2->ne[0] != ff) - return; - L.pad(fc1 + ".weight", w1->ne[0], fp); - L.pad(fc1 + ".bias", fp, 1); - L.pad(fc2 + ".weight", fp, w2->ne[1]); -} - -} - void EncStack::declare(WeightLoader & L, const char * prefix, int64_t layers, const EncNames & n) { blk.resize(layers); ln_spec.eps = cfg.ln_eps; @@ -85,7 +63,7 @@ void EncStack::declare(WeightLoader & L, const char * prefix, int64_t layers, co w.q_fc2 = L.quant_i8(b + n.fc2 + ".q8", {b + n.fc2 + ".weight"}); } else { const std::string b = std::string(prefix) + "." + std::to_string(i) + "."; - pad_mlp(L, b + n.fc1, b + n.fc2); + L.pad_mlp(b + n.fc1, b + n.fc2); w.Wo = L.gemm("%s.%lld.attn_o.weight", prefix, (long long)i); w.Wfc1 = L.gemm("%s.%lld.%s.weight", prefix, (long long)i, n.fc1); w.Wfc2 = L.gemm("%s.%lld.%s.weight", prefix, (long long)i, n.fc2); From a4d443b94e2329dd546e3123158dfddb00c58f5f Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 15:49:22 +0700 Subject: [PATCH 16/38] dual_tower: flash attention in DINOv2 and SigLIP on Hexagon MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The towers' explicit attention multiplies F32 scores on the NPU's vector units, sends the softmax to the CPU and transposes V with a strided copy. Where flash attention is on by default (Hexagon) they now take it, as π0.5's SigLIP does; other backends keep the explicit path. On the Dragonwing IQ-10 the vision stage goes from 790 to 311 ms: VLA-Adapter from 1352 to 872 ms and OpenVLA-OFT from 3711 to 3235 ms (mean, vla-bench), within 5.6e-4 and 2.9e-4 of the CPU F32 reference and 6.1e-4 / 5.2e-4 when served. The other ten archs give bit-identical actions. --- src/modules/dual_tower.h | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/src/modules/dual_tower.h b/src/modules/dual_tower.h index b08c7299..7ed1ea03 100644 --- a/src/modules/dual_tower.h +++ b/src/modules/dual_tower.h @@ -126,12 +126,14 @@ inline ggml_tensor* vit_block_fused(ggml_context*C, const ViTLayerW&w, ggml_tens return ls?fused_bias_gate_res(C,h,w.bfc2,w.ls2,x):fused_bias_res(C,h,w.bfc2,x); } -inline ggml_tensor* vit_block(ggml_context*C, const ViTLayerW&w, ggml_tensor*x, int64_t N, int64_t hidden, int64_t heads, int64_t hd, float eps, bool ls){ +inline ggml_tensor* vit_block(ggml_context*C, const ViTLayerW&w, ggml_tensor*x, int64_t N, int64_t hidden, int64_t heads, int64_t hd, float eps, bool ls, bool fa){ ggml_tensor*qkv=linear(C,w.Wqkv,w.bqkv,layer_norm(C,x,w.n1w,w.n1b,eps)); ggml_tensor*Q=ggml_cont(C,ggml_permute(C,head_view(C,qkv,hd,heads,N,hidden,3,0),0,2,1,3)); ggml_tensor*K=ggml_cont(C,ggml_permute(C,head_view(C,qkv,hd,heads,N,hidden,3,1),0,2,1,3)); - ggml_tensor*V=ggml_cont(C,ggml_permute(C,head_view(C,qkv,hd,heads,N,hidden,3,2),1,2,0,3)); - ggml_tensor*ao=linear(C,w.Wproj,w.bproj,attention(C,Q,K,V,nullptr,1.0f/std::sqrt((float)hd),hidden,N)); + const float scale=1.0f/std::sqrt((float)hd); + ggml_tensor*att=fa?flash_attention(C,Q,K,ggml_cont(C,ggml_permute(C,head_view(C,qkv,hd,heads,N,hidden,3,2),0,2,1,3)),nullptr,scale) + :attention(C,Q,K,ggml_cont(C,ggml_permute(C,head_view(C,qkv,hd,heads,N,hidden,3,2),1,2,0,3)),nullptr,scale,hidden,N); + ggml_tensor*ao=linear(C,w.Wproj,w.bproj,att); x=ggml_add(C,x,ls?ggml_mul(C,ao,w.ls1):ao); ggml_tensor*h=ffn_gelu_erf(C,w.Wfc1,w.bfc1,w.Wfc2,w.bfc2,layer_norm(C,x,w.n2w,w.n2b,eps)); return ggml_add(C,x,ls?ggml_mul(C,h,w.ls2):h); @@ -139,7 +141,7 @@ inline ggml_tensor* vit_block(ggml_context*C, const ViTLayerW&w, ggml_tensor*x, inline ggml_tensor* tower(ggml_context*C, ggml_tensor*pix, ggml_tensor*pw, ggml_tensor*pb, ggml_tensor*pos, ggml_tensor*cls, ggml_tensor*reg, const std::vector&blk, - int64_t hidden, int64_t heads, int64_t hd, int64_t patch, float eps, bool prefix, bool fused){ + int64_t hidden, int64_t heads, int64_t hd, int64_t patch, float eps, bool prefix, bool fused, bool fa){ ggml_tensor*conv=ggml_conv_2d(C,pw,pix,patch,patch,0,0,1,1); // Patch count from the conv, not a constant: both callers run 224/14 today, // and a different input size would otherwise reshape into the wrong grid. @@ -152,7 +154,7 @@ inline ggml_tensor* tower(ggml_context*C, ggml_tensor*pix, ggml_tensor*pw, ggml_ x=ggml_concat(C,tok,pt,1); } for(size_t i=0;inb[1],nprefix*x->nb[1])); return x; @@ -173,6 +175,10 @@ inline ggml_tensor* DualTower::encode(ggml_backend_t backend, graph_cacheggml_cgraph*{ @@ -181,8 +187,8 @@ inline ggml_tensor* DualTower::encode(ggml_backend_t backend, graph_cache Date: Wed, 4 Nov 2026 15:49:29 +0700 Subject: [PATCH 17/38] linear: fold a batch of activations into rows on Hexagon The NPU computes one 2D weight against a batch of activations wrong, so the fallback wrapper runs those matmuls on the CPU. On Hexagon, linear() now multiplies a contiguous batch as one 2D matrix of rows and reshapes the result back, which is the same product and stays on the NPU. Other backends are unchanged. On the Dragonwing IQ-10 TurboVLA's DINOv3 over two views leaves the CPU: 433 to about 296 ms (mean, vla-bench, 5 reps), within 3.7e-4 of the CPU F32 reference and 8.5e-4 when served. GR00T N1.6 with two views moves from 1.9e-3 to 8.8e-4. The other archs give bit-identical actions. --- src/layers/linear.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/layers/linear.h b/src/layers/linear.h index 0de035a3..bd2fe191 100644 --- a/src/layers/linear.h +++ b/src/layers/linear.h @@ -22,6 +22,17 @@ namespace vla { inline ggml_tensor * linear(ggml_context * C, ggml_tensor * W, ggml_tensor * b, ggml_tensor * x) { +#ifdef GGML_USE_HEXAGON + // Hexagon computes a 2D weight against a batch of activations wrong, and the + // fallback wrapper runs that on the CPU; the same rows as one 2D matmul stay + // on the NPU. + if (W->ne[2] * W->ne[3] == 1 && x->ne[2] * x->ne[3] > 1 && ggml_is_contiguous(x)) { + ggml_tensor * y = ggml_mul_mat(C, W, ggml_reshape_2d(C, x, x->ne[0], x->ne[1] * x->ne[2] * x->ne[3])); + if (b) + y = ggml_add(C, y, b); + return ggml_reshape_4d(C, y, W->ne[1], x->ne[1], x->ne[2], x->ne[3]); + } +#endif ggml_tensor * y = ggml_mul_mat(C, W, x); return b ? ggml_add(C, y, b) : y; } From f7b0e68232c4ea46e3ff67dc8561be2bb6f2214d Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 15:49:37 +0700 Subject: [PATCH 18/38] hexagon: run large transposing copies on the CPU MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A CONT whose source is strided in its innermost dim is a gather on the NPU and takes tens of milliseconds for a large tensor: 48 ms for OpenVLA-OFT's [330,128,32] V transpose, 32 ms for Evo-1's [1025,112,8], 18 ms for TurboVLA's [512,256,4]. Those of 256K elements or more now run on the CPU. Smaller ones take tens of microseconds on the NPU, less than the trip to the CPU, and stay there. On the Dragonwing IQ-10 (mean, vla-bench, 5 reps): OpenVLA-OFT 3247 to 1763 ms, TurboVLA 300 to 196 ms, Evo-1 2605 to 2357 ms, SmolVLA 639 to 585 ms, GR00T N1.5, π0 and π0.5 2-3% faster, the rest within 1.5%. A copy computes the same bytes on either side: all twelve archs give bit-identical actions. --- src/backend_fallback.cpp | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/backend_fallback.cpp b/src/backend_fallback.cpp index 151692b9..47b3181e 100644 --- a/src/backend_fallback.cpp +++ b/src/backend_fallback.cpp @@ -368,6 +368,15 @@ bool accel_runs(FallbackCtx * fc, const ggml_tensor * t, const char ** why) { *why = "can read stale data in a"; return false; } + if (fc->hexagon && t->op == GGML_OP_CONT && t->src[0]->nb[0] != ggml_type_size(t->src[0]->type) && + ggml_nelements(t) >= 256 * 1024) { + // Runs, but a large copy that transposes the innermost dim is a strided + // gather on the NPU: 18 ms for TurboVLA's fusion values [512,256,4], 32 ms + // for Evo-1's [1025,112,8], 48 ms for OpenVLA-OFT's [330,128,32]. Small + // ones take tens of microseconds there, less than a trip to the CPU. + *why = "is slow on a large transposing"; + return false; + } return ggml_backend_supports_op(fc->accel, t); } From 2b3424d2aa7a1fdb19b6902638ec168c9b1fc0fa Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 16:31:10 +0700 Subject: [PATCH 19/38] docs: optimized six archs on the Hexagon NPU, IQ-10 benchmark at f7b0e68 --- CHANGELOG.md | 27 +++++++----- docs/backend/hexagon.md | 38 +++++++++++----- docs/benchmark/README.md | 4 +- docs/benchmark/qualcomm-iq10.md | 78 ++++++++++++++++----------------- 4 files changed, 83 insertions(+), 64 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a91d22f..87340608 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,18 +14,21 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke of the CPU build on fixed inputs and serve through `eval/client`. Concats the NPU refuses on buffers above 4 GB of DSP address space, copies that read a weight (also answered `NO_SUPPORT`; VLA-Adapter), matmuls of a 2D weight - against a batch, which it computes wrong (GR00T N1.6 with two views), and - repeats of a computed tensor, which can read stale cache lines (TurboVLA), run - on the CPU, and GR00T N1.7's LM leaves flash attention (F16 scores on the - HMX) unless `--flash-attn` is given. Three matmuls that the HMX does not take - move onto it: SigLIP's 4304-wide MLP is zero-padded to 4320 at load, π0.5's - SigLIP takes flash attention, and Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, - π0.5 and Evo-1 run 27-40% faster, within the bar on fixed inputs. π0.5's VLM - and action expert take flash attention too, and the experts' KV concats run as - row copies that stay on the NPU (π0, π0.5, SmolVLA). GR00T N1.5, N1.6 and - VLA-JEPA are above the bar on a served LIBERO observation, as the CPU is at - F16. See `docs/backend/hexagon.md` for the build, the skel install and the - numbers. + against a batch, which it computes wrong, and repeats of a computed tensor, + which can read stale cache lines (TurboVLA), run on the CPU, and GR00T N1.7's + LM leaves flash attention (F16 scores on the HMX) unless `--flash-attn` is + given. Three matmuls that the HMX does not take move onto it: SigLIP's + 4304-wide MLP is zero-padded to 4320 at load, π0.5's SigLIP takes flash + attention, and Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, π0.5 and + Evo-1 run 27-40% faster, within the bar on fixed inputs. π0.5's VLM and action + expert take flash attention too, and the experts' KV concats run as row copies + that stay on the NPU (π0, π0.5, SmolVLA). VLA-Adapter's and OpenVLA-OFT's + DINOv2 + SigLIP towers take the same padding and flash attention, `linear()` + multiplies a batch of activations as one matrix of rows, and large transposing + copies run on the CPU: OpenVLA-OFT, VLA-Adapter and TurboVLA run 51-57% + faster, Evo-1 and SmolVLA 8-10%. GR00T N1.5, N1.6 and VLA-JEPA are above the + bar on a served LIBERO observation, as the CPU is at F16. See + `docs/backend/hexagon.md` for the build, the skel install and the numbers. - **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200` (Radeon RX 9060 XT) with ROCm 7.14 and llama.cpp `b11223`, every arch except diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index d9196bb4..90fd01e3 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -145,7 +145,18 @@ explicit path unless `--flash-attn` is given: 2.0e-3, at the same latency. `MUL_MAT` with a 2D weight against a batch of activations (weight `ne[2..3]` of 1, activations above 1) is claimed and comes out wrong. GR00T N1.6's SigLIP2 tower takes two camera views that way: its action chunk lands 0.16 from the CPU -reference. The wrapper runs those matmuls on the CPU: 1.9e-3, and 25% faster. +reference. The wrapper runs those matmuls on the CPU. On Hexagon `linear()` in +`src/layers/linear.h` multiplies a contiguous batch as one 2D matrix of rows +instead, the same product, so most of them never reach the rule: GR00T N1.6 +with two views lands 8.8e-4 from the reference, and the matmuls of TurboVLA's +DINOv3, which takes its two views as a batch, run on the NPU. + +A copy that transposes the innermost dim, such as the `CONT` of a permuted V +for explicit attention, is a strided gather on the NPU: 48 ms for OpenVLA-OFT's +[330,128,32], 32 ms for Evo-1's [1025,112,8], 18 ms for TurboVLA's +[512,256,4]. The wrapper runs those of 256K elements or more on the CPU, which +writes the same bytes; the smaller ones take tens of microseconds on the NPU, +less than the trip to the CPU. The matrix unit (HMX) takes a matmul only when its inner dim K and its output dim N are multiples of 32, and with a batch only an F16 src0. Anything else runs @@ -154,13 +165,18 @@ IQ-10 showed three such matmuls dominating, and vla.cpp now avoids them in Hexagon builds: - **SigLIP-so400m's MLP is 4304 wide**: 440 ms of GR00T N1.5's 565 ms tower. - The loader zero-pads fc1, its bias and fc2 to 4320 (`WeightLoader::pad`, - `gemm_width()` in `src/backend.h`), which leaves the output unchanged. + The loader zero-pads fc1, its bias and fc2 to 4320 + (`WeightLoader::pad_mlp`, `gemm_width()` in `src/backend.h`), which leaves + the output unchanged. VLA-Adapter's and OpenVLA-OFT's SigLIP + (`src/modules/dual_tower.h`) take the same padding. - **π0.5's attention ran explicit scores** in SigLIP (head dim 72, F32 x F32), the VLM and the action expert, because its flash attention was tied to SYCL's view-taking kernel. The VLM's and the expert's single KV head also sent their matmuls to the CPU (the batched `MUL_MAT` rule above). All three now take the plain flash attention on Hexagon (`use_flash_attention()` in `pi05.cpp`). + VLA-Adapter's and OpenVLA-OFT's DINOv2 and SigLIP towers did the same and now + take flash attention on Hexagon too: their vision stage goes from 790 to + 311 ms. - **Evo-1's LM attention multiplied F32 K and V against 14 query heads.** K and V now go to F16 (`fa_kv()`); the scores still accumulate in F32. @@ -329,13 +345,13 @@ differ from the benchmark's for VLA-JEPA, VLA-Adapter, OpenVLA-OFT and Octo: | GR00T N1.6 | 1 | 1.4e-3 | | GR00T N1.7 | 1 | 2.0e-3 | | VLA-JEPA | 2 | 2.8e-3 | -| VLA-Adapter | 2 | 4.2e-4 | -| OpenVLA-OFT | 2 | 3.5e-4 | +| VLA-Adapter | 2 | 5.6e-4 | +| OpenVLA-OFT | 2 | 2.9e-4 | | Octo-Small | 1 | 4.1e-4 | -| TurboVLA | 2 | 3.1e-4 | +| TurboVLA | 2 | 3.7e-4 | -GR00T N1.6 with two views is 1.9e-3. At the benchmark's single view VLA-JEPA is -1.9e-3, VLA-Adapter 4.8e-4 and OpenVLA-OFT 2.2e-4. +GR00T N1.6 with two views is 8.8e-4. At the benchmark's single view VLA-JEPA is +1.9e-3, VLA-Adapter 4.8e-4 and OpenVLA-OFT 1.9e-4. ### Serving @@ -357,10 +373,10 @@ error reply to a malformed or an invalid request, and keeps serving afterwards. | GR00T N1.7 | 2.5e-3 | | | π0 | 5.8e-4 | | | VLA-JEPA | 6.7e-3 | 2.1e-2 | -| VLA-Adapter | 4.5e-4 | | -| OpenVLA-OFT | 5.0e-4 | | +| VLA-Adapter | 6.1e-4 | | +| OpenVLA-OFT | 5.2e-4 | | | Octo-Small | 9.3e-5 | | -| TurboVLA | 2.2e-4 | | +| TurboVLA | 8.5e-4 | | GR00T N1.5, N1.6 and VLA-JEPA are above the bar on this input, and so is the CPU itself with F16 weights: N1.5 and VLA-JEPA land closer to the reference than the diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index 54f0f3c5..59dda10b 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`1cab1c3` (same llama.cpp pin). Each report +`f7b0e68` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 121.6 | 428.1 | 440.2 | 1733.6 | 613.7 | 705.8 | 747.0 | N/A | 631.0 | 1124.4 | 1117.4 | 4097.7 | 2579.1 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 121.3 | 186.8 | 448.1 | 837.8 | 597.1 | 704.6 | 756.6 | N/A | 583.6 | 1116.8 | 1109.9 | 1761.5 | 2329.8 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 6dc1d5c8..55dc1851 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `1cab1c3` | +| Commit | `f7b0e68` | | llama.cpp | `b11223` | -| Date | 2026-11-04 11:52 to 12:30 (UTC+07) | +| Date | 2026-11-04 15:48 to 16:23 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -65,18 +65,18 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| Octo-Small | 2 | 256 + 128 | 121.6 | 123.0 | 122.3 | 125.8 | 20.5 | 253 | 804 | -| TurboVLA | 2 | 256 | 428.1 | 433.0 | 431.6 | 436.8 | - | 438 | 1327 | -| VLA-JEPA | 1 | 256 | 440.2 | 446.3 | 444.7 | 450.2 | 195.2 | 185 | 3971 | -| GR00T N1.5 | 1 | 224 | 613.7 | 615.7 | 615.7 | 618.0 | 148.3 | 162 | 3636 | -| SmolVLA | 2 | 512 | 631.0 | 632.4 | 631.7 | 634.6 | 281.8 | 146 | 1190 | -| GR00T N1.6 | 1 | 224 | 705.8 | 708.5 | 708.0 | 710.7 | 145.0 | 146 | 4869 | -| GR00T N1.7 | 1 | 256 | 747.0 | 756.0 | 755.0 | 764.5 | 197.2 | 191 | 5090 | -| π0.5 | 2 | 224 | 1117.4 | 1121.0 | 1120.8 | 1123.3 | 292.5 | 285 | 5790 | -| π0 | 2 | 224 | 1124.4 | 1128.7 | 1128.6 | 1130.5 | 293.0 | 285 | 5534 | -| VLA-Adapter | 1 | 224 | 1733.6 | 1752.3 | 1749.5 | 1765.6 | 1187.2 | 872 | 3566 | -| Evo-1 | 1 | 448 | 2579.1 | 2595.5 | 2592.1 | 2611.9 | 854.0 | 319 | 1551 | -| OpenVLA-OFT | 1 | 224 | 4097.7 | 4107.8 | 4107.8 | 4114.2 | 1187.3 | 844 | 15327 | +| Octo-Small | 2 | 256 + 128 | 121.3 | 124.0 | 122.8 | 128.5 | 20.4 | 253 | 731 | +| TurboVLA | 2 | 256 | 186.8 | 190.6 | 189.6 | 194.9 | - | 249 | 1009 | +| VLA-JEPA | 1 | 256 | 448.1 | 452.9 | 451.0 | 458.7 | 197.0 | 185 | 3971 | +| SmolVLA | 2 | 512 | 583.6 | 584.3 | 584.2 | 584.9 | 230.0 | 146 | 1197 | +| GR00T N1.5 | 1 | 224 | 597.1 | 599.1 | 598.6 | 600.0 | 138.9 | 162 | 3645 | +| GR00T N1.6 | 1 | 224 | 704.6 | 707.6 | 706.3 | 711.0 | 145.9 | 147 | 4765 | +| GR00T N1.7 | 1 | 256 | 756.6 | 762.6 | 761.2 | 769.1 | 195.5 | 191 | 5084 | +| VLA-Adapter | 1 | 224 | 837.8 | 853.7 | 852.5 | 863.3 | 296.0 | 872 | 3581 | +| π0.5 | 2 | 224 | 1109.9 | 1111.9 | 1112.0 | 1112.7 | 277.1 | 285 | 5839 | +| π0 | 2 | 224 | 1116.8 | 1120.5 | 1120.1 | 1123.1 | 277.6 | 285 | 5535 | +| OpenVLA-OFT | 1 | 224 | 1761.5 | 1768.2 | 1768.8 | 1772.7 | 298.5 | 844 | 15461 | +| Evo-1 | 1 | 448 | 2329.8 | 2346.0 | 2341.9 | 2360.7 | 819.2 | 319 | 1594 | ### Fastest configuration @@ -100,35 +100,35 @@ rates measured at the defaults therefore do not carry over to these rows. | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| Octo-Small | *(defaults)* | 121.6 | 123.0 | 122.3 | 125.8 | 20.5 | 253 | 804 | - | -| TurboVLA | *(defaults)* | 428.1 | 433.0 | 431.6 | 436.8 | - | 438 | 1327 | - | -| VLA-JEPA | *(defaults)* | 440.2 | 446.3 | 444.7 | 450.2 | 195.2 | 185 | 3971 | - | -| GR00T N1.5 | *(defaults)* | 613.7 | 615.7 | 615.7 | 618.0 | 148.3 | 162 | 3636 | - | -| SmolVLA | *(defaults)* | 631.0 | 632.4 | 631.7 | 634.6 | 281.8 | 146 | 1190 | - | -| GR00T N1.6 | *(defaults)* | 705.8 | 708.5 | 708.0 | 710.7 | 145.0 | 146 | 4869 | - | -| GR00T N1.7 | *(defaults)* | 747.0 | 756.0 | 755.0 | 764.5 | 197.2 | 191 | 5090 | - | -| π0.5 | *(defaults)* | 1117.4 | 1121.0 | 1120.8 | 1123.3 | 292.5 | 285 | 5790 | - | -| π0 | *(defaults)* | 1124.4 | 1128.7 | 1128.6 | 1130.5 | 293.0 | 285 | 5534 | - | -| VLA-Adapter | *(defaults)* | 1733.6 | 1752.3 | 1749.5 | 1765.6 | 1187.2 | 872 | 3566 | - | -| Evo-1 | *(defaults)* | 2579.1 | 2595.5 | 2592.1 | 2611.9 | 854.0 | 319 | 1551 | - | -| OpenVLA-OFT | *(defaults)* | 4097.7 | 4107.8 | 4107.8 | 4114.2 | 1187.3 | 844 | 15327 | - | +| Octo-Small | *(defaults)* | 121.3 | 124.0 | 122.8 | 128.5 | 20.4 | 253 | 731 | - | +| TurboVLA | *(defaults)* | 186.8 | 190.6 | 189.6 | 194.9 | - | 249 | 1009 | - | +| VLA-JEPA | *(defaults)* | 448.1 | 452.9 | 451.0 | 458.7 | 197.0 | 185 | 3971 | - | +| SmolVLA | *(defaults)* | 583.6 | 584.3 | 584.2 | 584.9 | 230.0 | 146 | 1197 | - | +| GR00T N1.5 | *(defaults)* | 597.1 | 599.1 | 598.6 | 600.0 | 138.9 | 162 | 3645 | - | +| GR00T N1.6 | *(defaults)* | 704.6 | 707.6 | 706.3 | 711.0 | 145.9 | 147 | 4765 | - | +| GR00T N1.7 | `--flash-attn` | 733.3 | 739.1 | 736.8 | 747.5 | 195.3 | 191 | 5059 | -3.1% | +| VLA-Adapter | *(defaults)* | 837.8 | 853.7 | 852.5 | 863.3 | 296.0 | 872 | 3581 | - | +| π0.5 | *(defaults)* | 1109.9 | 1111.9 | 1112.0 | 1112.7 | 277.1 | 285 | 5839 | - | +| π0 | *(defaults)* | 1116.8 | 1120.5 | 1120.1 | 1123.1 | 277.6 | 285 | 5535 | - | +| OpenVLA-OFT | *(defaults)* | 1761.5 | 1768.2 | 1768.8 | 1772.7 | 298.5 | 844 | 15461 | - | +| Evo-1 | *(defaults)* | 2329.8 | 2346.0 | 2341.9 | 2360.7 | 819.2 | 319 | 1594 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| Octo-Small | 125.3 | 128.8 | 124.4 | 123.5 | 124.6 | -| TurboVLA | 438.8 | 436.5 | 439.3 | 438.2 | 535.1 | -| VLA-JEPA | 446.7 | 445.8 | 444.8 | 445.2 | 817.8 | -| GR00T N1.5 | 615.8 | 619.3 | 615.3 | 618.4 | 1096.5 | -| SmolVLA | 632.1 | 633.0 | 632.5 | 632.6 | 2834.8 | -| GR00T N1.6 | 714.2 | 712.1 | 718.7 | 714.9 | 885.7 | -| GR00T N1.7 | 765.2 | 742.0 | 761.5 | 743.6 | 1081.0 | -| π0.5 | 1121.1 | 1121.9 | 1120.5 | 1121.4 | 2062.9 | -| π0 | 1129.8 | 1128.8 | 1131.8 | 1128.5 | 2090.5 | -| VLA-Adapter | 1766.1 | 1760.1 | 1756.1 | 1757.2 | 1767.1 | -| Evo-1 | 2613.0 | 2613.7 | 2617.3 | 2617.9 | 4840.1 | -| OpenVLA-OFT | 4110.3 | 4101.9 | 4101.7 | 4108.2 | 4107.9 | +| Octo-Small | 124.3 | 128.9 | 129.6 | 127.7 | 124.7 | +| TurboVLA | 194.6 | 192.5 | 188.6 | 194.3 | 251.9 | +| VLA-JEPA | 453.3 | 454.5 | 453.3 | 451.9 | 605.5 | +| SmolVLA | 586.3 | 586.9 | 584.4 | 586.0 | 2147.5 | +| GR00T N1.5 | 601.3 | 600.7 | 601.7 | 600.4 | 877.5 | +| GR00T N1.6 | 708.6 | 708.4 | 706.5 | 707.5 | 876.8 | +| GR00T N1.7 | 763.3 | 737.3 | 765.2 | 741.6 | 869.0 | +| VLA-Adapter | 854.5 | 859.7 | 870.9 | 878.5 | 1151.2 | +| π0.5 | 1113.6 | 1112.9 | 1113.4 | 1113.5 | 2047.2 | +| π0 | 1141.2 | 1140.3 | 1141.9 | 1140.9 | 2044.7 | +| OpenVLA-OFT | 1776.1 | 1775.6 | 1770.0 | 1765.2 | 2045.9 | +| Evo-1 | 2339.7 | 2350.8 | 2340.9 | 2352.5 | 3868.0 |
@@ -144,7 +144,7 @@ The HTP skel has to be processed for this board and installed in [backend/hexagon.md](../backend/hexagon.md#linux-iq-10). GR00T N1.6 runs one view here, as the configuration above fixes. With two views -its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see +one batched `MUL_MAT` of its SigLIP2 tower still runs on the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). These rows time the defaults; they do not vouch for accuracy. On a served LIBERO From 2affa266f354c3baa86e7706c73890bf3d8f1a25 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 11:06:51 +0700 Subject: [PATCH 20/38] vla_jepa: keep the LM off flash attention unless asked Where flash attention is the default (Hexagon), the LM's F16 K/V put a served LIBERO request 6.7e-3 from the CPU F32 reference on the Dragonwing IQ-10, above the 2.9e-3 bar; the CPU with flash attention on is 5.5e-3. With the explicit path it is 2.6e-3 served and 2.2e-3 on the fixed input (was 2.8e-3), at a cost of 683 to 810 ms with two views (mean, vla-bench). --flash-attn 1 turns it back on. Other backends are unchanged and the other eleven archs give bit-identical actions. --- src/models/vla_jepa.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/models/vla_jepa.cpp b/src/models/vla_jepa.cpp index c2c3137f..ff42d56d 100644 --- a/src/models/vla_jepa.cpp +++ b/src/models/vla_jepa.cpp @@ -130,7 +130,10 @@ bool load_config(const gguf_reader & g, const Options & opts, VlaJepaModelArch & m.lm.cfg.head_dim = m.lm_head_dim; m.lm.cfg.inter = m.lm_inter; m.lm.cfg.rms_eps = m.lm_rms_eps; - m.lm.cfg.flash_attn = flash_attn_enabled(); + // The LM takes flash attention only when --flash-attn asks for it. Where it + // is the default (Hexagon), its F16 K/V put a served LIBERO request 6.7e-3 + // from the CPU F32 reference on an IQ-10 (v81), 2.6e-3 with the explicit path. + m.lm.cfg.flash_attn = opts.flash_attn.value_or(false); m.lm.cfg.rope.type = GGML_ROPE_TYPE_IMROPE; m.lm.cfg.rope.n_dims = (int) m.lm_head_dim; m.lm.cfg.rope.freq_base = m.lm_rope_base; From e89190cee5e1835c530b3b8862ae247ac561ad83 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 12:03:24 +0700 Subject: [PATCH 21/38] vla_adapter: pad the LM's sequence to a multiple of 32 on Hexagon The LM's softmax rows were 337 wide, which the NPU refuses, so every layer's softmax ran on the CPU. On Hexagon the LM now runs on gemm_width(SEQ) tokens with zeros appended; the mask is causal, so no real token sees them, and the conditioning keeps the first SEQ rows. Other backends run the same sequence as before. On the Dragonwing IQ-10, VLA-Adapter goes from 859 to 830 ms (mean, vla-bench, one view), 5.7e-4 from the CPU F32 reference on the fixed input and 5.0e-4 served. --- src/models/vla_adapter.cpp | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/src/models/vla_adapter.cpp b/src/models/vla_adapter.cpp index fc13a7bd..4776327b 100644 --- a/src/models/vla_adapter.cpp +++ b/src/models/vla_adapter.cpp @@ -237,6 +237,10 @@ std::vector VlaAdapterModelArch::predict(const Inputs& in) { const int64_t NUM_PROMPT_TOKENS = NPROMPT-1; const int64_t NPATCH = NP * n_views; const int64_t SEQ = 1+NPATCH+(NPROMPT-1)+num_tokens+1; + // The LM runs on LSEQ tokens, zeros appended. The mask is causal, so no real + // token sees them, and on Hexagon the softmax rows become a multiple of 32, + // which is what the NPU takes. + const int64_t LSEQ = gemm_width(SEQ); const auto ti=clock::now(); // LM + action head graph depends only on the sequence layout. const MainKey mkey{ SEQ, n_views, NPROMPT }; @@ -254,12 +258,14 @@ std::vector VlaAdapterModelArch::predict(const Inputs& in) { ggml_tensor*erest=ggml_cont(C,ggml_view_2d(C,emb2,HC,NPROMPT+num_tokens,emb2->nb[1],emb2->nb[1])); ggml_tensor*t_proj=ggml_new_tensor_2d(C,GGML_TYPE_F32,HC,NPATCH); ggml_set_input(t_proj); ggml_tensor*mm_seq=ggml_concat(C,ggml_concat(C,e0,t_proj,1),erest,1); + if(LSEQ>SEQ) + mm_seq=ggml_pad(C,mm_seq,0,(int)(LSEQ-SEQ),0,0); - ggml_tensor*t_pos=ggml_new_tensor_1d(C,GGML_TYPE_I32,SEQ); ggml_set_input(t_pos); ggml_set_output(t_pos); - ggml_tensor*t_mask=ggml_new_tensor_2d(C,GGML_TYPE_F32,SEQ,SEQ); ggml_set_input(t_mask); ggml_set_output(t_mask); + ggml_tensor*t_pos=ggml_new_tensor_1d(C,GGML_TYPE_I32,LSEQ); ggml_set_input(t_pos); ggml_set_output(t_pos); + ggml_tensor*t_mask=ggml_new_tensor_2d(C,GGML_TYPE_F32,LSEQ,LSEQ); ggml_set_input(t_mask); ggml_set_output(t_mask); const float lsc=1.0f/std::sqrt((float)lm_head_dim); std::vector lout(lm_layers); ggml_tensor*x=mm_seq; - const bool fused_lm=fused_ops && cublas_rows(SEQ) && SEQ<=4096; + const bool fused_lm=fused_ops && LSEQ==SEQ && cublas_rows(SEQ) && SEQ<=4096; for(int i=0;i VlaAdapterModelArch::predict(const Inputs& in) { } ggml_tensor*hn=ggml_mul(C,ggml_rms_norm(C,x,lm_rms_eps),l.attn_norm); ggml_tensor*qp=ggml_add(C,ggml_mul_mat(C,l.Wq,hn),l.bq),*kp=ggml_add(C,ggml_mul_mat(C,l.Wk,hn),l.bk),*vp=ggml_add(C,ggml_mul_mat(C,l.Wv,hn),l.bv); - ggml_tensor*qh=ggml_reshape_3d(C,qp,lm_head_dim,n_q,SEQ),*kh=ggml_reshape_3d(C,kp,lm_head_dim,n_kv,SEQ),*vh=ggml_reshape_3d(C,vp,lm_head_dim,n_kv,SEQ); + ggml_tensor*qh=ggml_reshape_3d(C,qp,lm_head_dim,n_q,LSEQ),*kh=ggml_reshape_3d(C,kp,lm_head_dim,n_kv,LSEQ),*vh=ggml_reshape_3d(C,vp,lm_head_dim,n_kv,LSEQ); ggml_tensor*qr=ggml_rope_ext(C,qh,t_pos,nullptr,(int)lm_head_dim,GGML_ROPE_TYPE_NEOX,0,lm_rope_base,1.0f,0.0f,1.0f,32.0f,1.0f); ggml_tensor*kr=ggml_rope_ext(C,kh,t_pos,nullptr,(int)lm_head_dim,GGML_ROPE_TYPE_NEOX,0,lm_rope_base,1.0f,0.0f,1.0f,32.0f,1.0f); ggml_tensor*Q=ggml_cont(C,ggml_permute(C,qr,0,2,1,3)),*K=ggml_cont(C,ggml_permute(C,kr,0,2,1,3)),*V=ggml_cont(C,ggml_permute(C,vh,1,2,0,3)); ggml_tensor*kq=ggml_mul_mat(C,K,Q); ggml_prec_set_acc(kq,GGML_PREC_F32); ggml_tensor*aw=ggml_soft_max_ext(C,kq,t_mask,lsc,0.0f); ggml_tensor*kqv=ggml_mul_mat(C,V,aw); - ggml_tensor*att=ggml_reshape_2d(C,ggml_cont(C,ggml_permute(C,kqv,0,2,1,3)),HC,SEQ); + ggml_tensor*att=ggml_reshape_2d(C,ggml_cont(C,ggml_permute(C,kqv,0,2,1,3)),HC,LSEQ); x=ggml_add(C,x,ggml_mul_mat(C,l.Wo,att)); ggml_tensor*hn2=ggml_mul(C,ggml_rms_norm(C,x,lm_rms_eps),l.ffn_norm); ggml_tensor*gt=ggml_silu(C,ggml_mul_mat(C,l.Wg,hn2)),*ut=ggml_mul_mat(C,l.Wu,hn2); x=ggml_add(C,x,ggml_mul_mat(C,l.Wd,ggml_mul(C,gt,ut))); lout[i]=x; } + if(LSEQ>SEQ) + for(auto&o:lout) + o=ggml_view_2d(C,o,HC,SEQ,o->nb[1],0); ggml_tensor*final_norm=ggml_mul(C,ggml_rms_norm(C,lout[lm_layers-1],lm_rms_eps),lm_out_norm); std::vector cond(head_blocks); @@ -363,11 +372,11 @@ std::vector VlaAdapterModelArch::predict(const Inputs& in) { if(!ggml_are_same_shape(proj,t_proj)){ std::fprintf(stderr,"vla(vla_adapter): projector output does not match the LM width\n"); return {}; } ggml_backend_tensor_copy(proj,t_proj); if(!gio.consts){ - std::vector pp(SEQ); - for(int64_t i=0;i pp(LSEQ); + for(int64_t i=0;i mk; build_causal_mask(SEQ, mk); + std::vector mk; build_causal_mask(LSEQ, mk); ggml_backend_tensor_set(t_mask,mk.data(),0,ggml_nbytes(t_mask)); std::vector zx((size_t)action_dim*HC*chunk,0.0f); ggml_backend_tensor_set(t_x0,zx.data(),0,ggml_nbytes(t_x0)); From ee69ea2694ab47e79bc3fffab23c179e9299f1b5 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 12:38:39 +0700 Subject: [PATCH 22/38] vla_adapter: one padded softmax per action-head block on Hexagon Each head block scored its own tokens, the action queries and the vision tokens separately and concatenated the scores and the values along their columns, which the NPU refuses, then took a 329-wide softmax it also refuses. Where gemm_width() pads (Hexagon), the keys and values are now stacked as rows and padded to a multiple of 32, the padding masked out, with rg on the vision keys instead of their scores (RoPE is linear). Other backends keep the previous graph. On the Dragonwing IQ-10, VLA-Adapter goes from 830 to 747 ms (mean, vla-bench, one view), 5.1e-4 from the CPU F32 reference on the fixed input and 5.3e-4 served. --- src/models/vla_adapter.cpp | 75 ++++++++++++++++++++++++++++++-------- 1 file changed, 60 insertions(+), 15 deletions(-) diff --git a/src/models/vla_adapter.cpp b/src/models/vla_adapter.cpp index 4776327b..dca6f22a 100644 --- a/src/models/vla_adapter.cpp +++ b/src/models/vla_adapter.cpp @@ -31,12 +31,14 @@ #include "layers/norm.h" #include "layers/rope.h" +#include #include #include #include #include #include #include +#include #include namespace vla { @@ -73,6 +75,7 @@ struct VlaAdapterModelArch : public ModelArchBase { ggml_tensor *t_ids=nullptr,*t_proj=nullptr,*t_pos=nullptr,*t_mask=nullptr; ggml_tensor *t_state=nullptr,*t_x0=nullptr,*norm_actions=nullptr; ggml_tensor *cT=nullptr,*sT=nullptr,*cA=nullptr,*sA=nullptr,*cK=nullptr,*sK=nullptr; + ggml_tensor *cJ=nullptr,*sJ=nullptr,*hmask=nullptr; bool consts=false; }; graph_cache main_graph; @@ -314,9 +317,21 @@ std::vector VlaAdapterModelArch::predict(const Inputs& in) { ggml_tensor*ss=ggml_new_tensor_2d(C,GGML_TYPE_F32,HD,Lh); ggml_set_input(ss); ggml_set_output(ss); return {cc,ss}; }; + // Each head block attends its own tokens, the action queries and the vision + // tokens. Where gemm_width() pads (Hexagon), their keys and values are + // stacked as rows and padded to a multiple of 32, masked out: one softmax + // the NPU takes, and no column concats. + const int64_t LJ=chunk+num_tokens+1+NPATCH, LH=gemm_width(LJ); + const bool joint=LH>LJ; auto [cT,sT]=cs_tensor(chunk); - auto [cA,sA]=cs_tensor(num_tokens+1); - auto [cK,sK]=cs_tensor(NPATCH); + ggml_tensor *cA=nullptr,*sA=nullptr,*cK=nullptr,*sK=nullptr,*cJ=nullptr,*sJ=nullptr,*hmask=nullptr; + if(joint){ + std::tie(cJ,sJ)=cs_tensor(LH); + hmask=ggml_new_tensor_2d(C,GGML_TYPE_F32,LH,chunk); ggml_set_input(hmask); ggml_set_output(hmask); + }else{ + std::tie(cA,sA)=cs_tensor(num_tokens+1); + std::tie(cK,sK)=cs_tensor(NPATCH); + } ggml_tensor*t_x0=ggml_new_tensor_2d(C,GGML_TYPE_F32,action_dim*HC,chunk); ggml_set_input(t_x0); ggml_set_output(t_x0); ggml_tensor*hx=ggml_relu(C,linear(C,h_fc1w,h_fc1b,layer_norm(C,t_x0,h_ln1w,h_ln1b,head_ln_eps))); @@ -329,18 +344,31 @@ std::vector VlaAdapterModelArch::predict(const Inputs& in) { auto qk=[&](ggml_tensor*p,ggml_tensor*cs,ggml_tensor*sn){ return rope_pairwise(C,to_heads(C,p,HD,NH,p->ne[1]),cs,sn,HD); }; auto tov=[&](ggml_tensor*p){ return to_heads_v(C,p,HD,NH,p->ne[1]); }; ggml_tensor*q=qk(linear(C,w.Wq,w.bq,hx),cT,sT); - ggml_tensor*kse=qk(linear(C,w.Wks,w.bks,hx),cT,sT); - ggml_tensor*vse=linear(C,w.Wvs,w.bvs,hx); - ggml_tensor*kad=qk(linear(C,w.Wka,w.bka,had),cA,sA); ggml_tensor*vad=linear(C,w.Wva,w.bva,had); - ggml_tensor*kta=qk(linear(C,w.Wkt,w.bkt,ht),cK,sK); ggml_tensor*vta=linear(C,w.Wvt,w.bvt,ht); - ggml_tensor*Vs=tov(vse),*Va=tov(vad),*VT=tov(vta); - ggml_tensor*ss2=ggml_mul_mat(C,kse,q),*sa=ggml_mul_mat(C,kad,q),*sr=ggml_mul_mat(C,kta,q); - ggml_prec_set_acc(ss2,GGML_PREC_F32); ggml_prec_set_acc(sa,GGML_PREC_F32); ggml_prec_set_acc(sr,GGML_PREC_F32); - ggml_tensor*st2=ggml_scale(C,sr,w.rg); - ggml_tensor*scr=ggml_concat(C,ggml_concat(C,ss2,sa,0),st2,0); - ggml_tensor*attn=ggml_soft_max_ext(C,scr,nullptr,hsc,0.0f); - ggml_tensor*Vc=ggml_concat(C,ggml_concat(C,Vs,Va,0),VT,0); - ggml_tensor*kqv=ggml_mul_mat(C,Vc,attn); + ggml_tensor*kqv; + if(joint){ + // rg scales the vision scores; RoPE is linear, so it goes on their keys. + ggml_tensor*kj=ggml_concat(C,ggml_concat(C,linear(C,w.Wks,w.bks,hx),linear(C,w.Wka,w.bka,had),1), + ggml_scale(C,linear(C,w.Wkt,w.bkt,ht),w.rg),1); + ggml_tensor*vj=ggml_concat(C,ggml_concat(C,linear(C,w.Wvs,w.bvs,hx),linear(C,w.Wva,w.bva,had),1), + linear(C,w.Wvt,w.bvt,ht),1); + ggml_tensor*K=qk(ggml_pad(C,kj,0,(int)(LH-LJ),0,0),cJ,sJ); + ggml_tensor*V=tov(ggml_pad(C,vj,0,(int)(LH-LJ),0,0)); + ggml_tensor*sc=ggml_mul_mat(C,K,q); ggml_prec_set_acc(sc,GGML_PREC_F32); + kqv=ggml_mul_mat(C,V,ggml_soft_max_ext(C,sc,hmask,hsc,0.0f)); + }else{ + ggml_tensor*kse=qk(linear(C,w.Wks,w.bks,hx),cT,sT); + ggml_tensor*vse=linear(C,w.Wvs,w.bvs,hx); + ggml_tensor*kad=qk(linear(C,w.Wka,w.bka,had),cA,sA); ggml_tensor*vad=linear(C,w.Wva,w.bva,had); + ggml_tensor*kta=qk(linear(C,w.Wkt,w.bkt,ht),cK,sK); ggml_tensor*vta=linear(C,w.Wvt,w.bvt,ht); + ggml_tensor*Vs=tov(vse),*Va=tov(vad),*VT=tov(vta); + ggml_tensor*ss2=ggml_mul_mat(C,kse,q),*sa=ggml_mul_mat(C,kad,q),*sr=ggml_mul_mat(C,kta,q); + ggml_prec_set_acc(ss2,GGML_PREC_F32); ggml_prec_set_acc(sa,GGML_PREC_F32); ggml_prec_set_acc(sr,GGML_PREC_F32); + ggml_tensor*st2=ggml_scale(C,sr,w.rg); + ggml_tensor*scr=ggml_concat(C,ggml_concat(C,ss2,sa,0),st2,0); + ggml_tensor*attn=ggml_soft_max_ext(C,scr,nullptr,hsc,0.0f); + ggml_tensor*Vc=ggml_concat(C,ggml_concat(C,Vs,Va,0),VT,0); + kqv=ggml_mul_mat(C,Vc,attn); + } ggml_tensor*mg=ggml_reshape_2d(C,ggml_cont(C,ggml_permute(C,kqv,0,2,1,3)),HC,chunk); ggml_tensor*res=ggml_add(C,linear(C,w.Wo,w.bo,mg),hx); hx=ggml_relu(C,linear(C,w.flw,w.flb,layer_norm(C,res,w.flnw,w.flnb,head_ln_eps))); @@ -350,6 +378,7 @@ std::vector VlaAdapterModelArch::predict(const Inputs& in) { gio.t_ids=t_ids; gio.t_proj=t_proj; gio.t_pos=t_pos; gio.t_mask=t_mask; gio.t_state=t_state; gio.t_x0=t_x0; gio.norm_actions=norm_actions; gio.cT=cT; gio.sT=sT; gio.cA=cA; gio.sA=sA; gio.cK=cK; gio.sK=sK; + gio.cJ=cJ; gio.sJ=sJ; gio.hmask=hmask; ggml_cgraph*gf=ggml_new_graph_custom(C,65536,false); ggml_build_forward_expand(gf,norm_actions); return gf; @@ -382,7 +411,23 @@ std::vector VlaAdapterModelArch::predict(const Inputs& in) { ggml_backend_tensor_set(t_x0,zx.data(),0,ggml_nbytes(t_x0)); auto fill_cs=[&](ggml_tensor*cc,ggml_tensor*ss,int64_t Lh){ std::vector cb,sb; rope_pairwise_table(HD,Lh,head_rope_base,cb,sb); ggml_backend_tensor_set(cc,cb.data(),0,ggml_nbytes(cc)); ggml_backend_tensor_set(ss,sb.data(),0,ggml_nbytes(ss)); }; - fill_cs(cT,sT,chunk); fill_cs(cA,sA,num_tokens+1); fill_cs(cK,sK,NPATCH); + fill_cs(cT,sT,chunk); + if(gio.cJ){ + std::vector cb,sb,cj,sj; + for(int64_t Lh:{chunk,num_tokens+1,NPATCH}){ + rope_pairwise_table(HD,Lh,head_rope_base,cb,sb); + cj.insert(cj.end(),cb.begin(),cb.end()); sj.insert(sj.end(),sb.begin(),sb.end()); + } + const int64_t LH=gio.cJ->ne[1], LJ=(int64_t)cj.size()/HD; + cj.resize((size_t)(HD*LH),1.0f); sj.resize((size_t)(HD*LH),0.0f); + ggml_backend_tensor_set(gio.cJ,cj.data(),0,ggml_nbytes(gio.cJ)); ggml_backend_tensor_set(gio.sJ,sj.data(),0,ggml_nbytes(gio.sJ)); + std::vector hm((size_t)(LH*chunk),0.0f); + for(int64_t r=0;r sv(proprio_dim,0.0f); for(int64_t i=0;i Date: Thu, 5 Nov 2026 13:31:11 +0700 Subject: [PATCH 23/38] dual_tower: GELU (erf) in ops the Hexagon NPU runs ggml-hexagon has no GELU_ERF, so DINOv2's MLP and the projector sent their activations through the CPU in every layer. backend.h gains gelu_erf(): on Hexagon x*sigmoid(p(x)), p an odd cubic in x clamped to [-5, 5] fitted to logit(Phi(x)), within 2.5e-5 of the exact GELU for every x; elsewhere ggml_gelu_erf, so other backends build the same graph. On the Dragonwing IQ-10, VLA-Adapter goes from 740 to 705 ms and OpenVLA-OFT from 1771 to 1740 ms (mean, vla-bench, one view), within 6.2e-4 / 3.2e-4 of the CPU F32 reference on the fixed input and 5.4e-4 / 5.9e-4 served. --- src/backend.h | 20 ++++++++++++++++++++ src/modules/dual_tower.h | 6 +++--- 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/src/backend.h b/src/backend.h index 3b50682a..60a51755 100644 --- a/src/backend.h +++ b/src/backend.h @@ -249,6 +249,26 @@ inline ggml_tensor * gelu(ggml_context * ctx, ggml_tensor * x) { #endif } +/** + * @brief `ggml_gelu_erf`, in ops the compiled-in backend runs. + * + * ggml-hexagon has no GELU_ERF, so every MLP that uses it round-trips its + * activations through the CPU. There it is x*sigmoid(p(x)), p an odd cubic in + * x clamped to [-5, 5] fitted to logit(Phi(x)): within 2.5e-5 of the exact + * GELU for every x. Elsewhere this is plain ggml_gelu_erf. + */ +inline ggml_tensor * gelu_erf(ggml_context * ctx, ggml_tensor * x) { +#ifdef GGML_USE_HEXAGON + ggml_tensor * xc = ggml_clamp(ctx, x, -5.0f, 5.0f); + ggml_tensor * x2 = ggml_mul(ctx, xc, xc); + ggml_tensor * p = ggml_mul(ctx, ggml_scale_bias(ctx, x2, -7.0303601711e-4f, 7.4011308986e-2f), x2); + p = ggml_mul(ctx, ggml_scale_bias(ctx, p, 1.0f, 1.5950157475f), xc); + return ggml_mul(ctx, x, ggml_sigmoid(ctx, p)); +#else + return ggml_gelu_erf(ctx, x); +#endif +} + inline ggml_tensor * geglu(ggml_context * ctx, ggml_tensor * gate, ggml_tensor * up) { #if !defined(GGML_USE_METAL) && !defined(GGML_USE_OPENVINO) && \ !defined(GGML_USE_HEXAGON) && !defined(GGML_USE_OPENCL) diff --git a/src/modules/dual_tower.h b/src/modules/dual_tower.h index 7ed1ea03..57aa9a88 100644 --- a/src/modules/dual_tower.h +++ b/src/modules/dual_tower.h @@ -135,7 +135,7 @@ inline ggml_tensor* vit_block(ggml_context*C, const ViTLayerW&w, ggml_tensor*x, :attention(C,Q,K,ggml_cont(C,ggml_permute(C,head_view(C,qkv,hd,heads,N,hidden,3,2),1,2,0,3)),nullptr,scale,hidden,N); ggml_tensor*ao=linear(C,w.Wproj,w.bproj,att); x=ggml_add(C,x,ls?ggml_mul(C,ao,w.ls1):ao); - ggml_tensor*h=ffn_gelu_erf(C,w.Wfc1,w.bfc1,w.Wfc2,w.bfc2,layer_norm(C,x,w.n2w,w.n2b,eps)); + ggml_tensor*h=linear(C,w.Wfc2,w.bfc2,gelu_erf(C,linear(C,w.Wfc1,w.bfc1,layer_norm(C,x,w.n2w,w.n2b,eps)))); return ggml_add(C,x,ls?ggml_mul(C,h,w.ls2):h); } @@ -198,8 +198,8 @@ inline ggml_tensor* DualTower::encode(ggml_backend_t backend, graph_cache Date: Thu, 5 Nov 2026 13:44:58 +0700 Subject: [PATCH 24/38] octo: pad the block transformer's sequence to a multiple of 32 on Hexagon Its softmax rows were 353 wide, which the NPU refuses, so every block's softmax ran on the CPU. Where gemm_width() pads (Hexagon), zero tokens are appended and masked out as keys; the readouts are gathered by index, so the extra rows never reach them. Other backends run the same sequence as before. On the Dragonwing IQ-10, Octo-Small goes from 126.5 to 122.6 ms (mean, vla-bench, two views), 5.3e-4 from the CPU F32 reference on the fixed input and 2.4e-4 served. --- src/models/octo.cpp | 31 +++++++++++++++++++++++-------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/src/models/octo.cpp b/src/models/octo.cpp index da5c0205..4774c722 100644 --- a/src/models/octo.cpp +++ b/src/models/octo.cpp @@ -1083,6 +1083,10 @@ bool run_transformer_graph(OctoRuntime& rt, const int n_readout = (int) layout.readout_seq_idx.size(); if (input.size() != (size_t) kHidden*seq) return false; + // Zero tokens appended to a multiple of 32 where gemm_width() pads (Hexagon), + // masked out as keys: the softmax rows are then a width the NPU takes. The + // readouts are gathered by index, so the extra rows never reach them. + const int pseq = (int) gemm_width(seq); OctoRuntime::BtKey key; for (const OctoSeqRun& r : layout.runs) @@ -1109,12 +1113,12 @@ bool run_transformer_graph(OctoRuntime& rt, if (!out_w || !out_b) return nullptr; - ggml_tensor * x = ggml_new_tensor_2d(C, GGML_TYPE_F32, kHidden, seq); + ggml_tensor * x = ggml_new_tensor_2d(C, GGML_TYPE_F32, kHidden, pseq); ggml_set_name(x, "octo.block_transformer.input"); ggml_set_input(x); io.input = x; - ggml_tensor * mask = ggml_new_tensor_2d(C, GGML_TYPE_F32, seq, seq); + ggml_tensor * mask = ggml_new_tensor_2d(C, GGML_TYPE_F32, pseq, pseq); ggml_set_name(mask, "octo.block_transformer.additive_mask"); ggml_set_input(mask); ggml_set_output(mask); @@ -1127,7 +1131,7 @@ bool run_transformer_graph(OctoRuntime& rt, io.readout_idx = readout_idx; for (int i=0; i<12; ++i) { - if (rt.fused_ops && seq <= 4096) { + if (rt.fused_ops && pseq == seq && seq <= 4096) { // The same block on the fused (bit-identical) kernels: LayerNorm, // bias + head split + V^T in one pass, exact attention with the // mask, and the bias + residual and bias + GELU epilogues. @@ -1144,11 +1148,11 @@ bool run_transformer_graph(OctoRuntime& rt, } ggml_tensor * n1 = layer_norm(C, x, blk_w[i][0], blk_w[i][1], ln_eps); ggml_tensor * qkv = linear(C, blk_w[i][2], blk_w[i][3], n1); - ggml_tensor * Q = ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, seq, kHidden, 3, 0), 0, 2, 1, 3)); - ggml_tensor * K = ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, seq, kHidden, 3, 1), 0, 2, 1, 3)); - ggml_tensor * V = ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, seq, kHidden, 3, 2), 1, 2, 0, 3)); + ggml_tensor * Q = ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, pseq, kHidden, 3, 0), 0, 2, 1, 3)); + ggml_tensor * K = ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, pseq, kHidden, 3, 1), 0, 2, 1, 3)); + ggml_tensor * V = ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, pseq, kHidden, 3, 2), 1, 2, 0, 3)); - ggml_tensor * merged = attention(C, Q, K, V, mask, attn_scale, kHidden, seq); + ggml_tensor * merged = attention(C, Q, K, V, mask, attn_scale, kHidden, pseq); ggml_tensor * residual = ggml_add(C, x, linear(C, blk_w[i][4], blk_w[i][5], merged)); ggml_tensor * n2 = layer_norm(C, residual, blk_w[i][6], blk_w[i][7], ln_eps); @@ -1177,10 +1181,21 @@ bool run_transformer_graph(OctoRuntime& rt, if (fresh) { std::vector mask; build_transformer_mask(layout, mask); + if (pseq > seq) { + std::vector padded((size_t) pseq*pseq, 0.0f); + for (int q=0; q= seq ? -FLT_MAX : q < seq ? mask[(size_t) q*seq+k] : 0.0f; + mask.swap(padded); + } ggml_backend_tensor_set(io.mask, mask.data(), 0, ggml_nbytes(io.mask)); ggml_backend_tensor_set(io.readout_idx, layout.readout_seq_idx.data(), 0, ggml_nbytes(io.readout_idx)); } - ggml_backend_tensor_set(io.input, input.data(), 0, ggml_nbytes(io.input)); + ggml_backend_tensor_set(io.input, input.data(), 0, input.size()*sizeof(float)); + if (pseq > seq) { + const std::vector zeros((size_t) kHidden*(pseq-seq), 0.0f); + ggml_backend_tensor_set(io.input, zeros.data(), input.size()*sizeof(float), zeros.size()*sizeof(float)); + } if (!octo_compute(rt, rt.transformer.graph(), "block transformer")) return false; readout_action.resize((size_t) kHidden*n_readout); From 5af7b7e9edd14a4fd45a5ad4e2c0dc0ea44af3ac Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 15:11:08 +0700 Subject: [PATCH 25/38] docs: VLA-JEPA within the bar served, padded softmaxes, IQ-10 benchmark at 7c76ea9 hexagon.md: VLA-JEPA's LM leaves flash attention as N1.7's does (6.7e-3 to 2.6e-3 served); VLA-Adapter's LM and action head and Octo's block transformer pad their softmax rows to a multiple of 32; VLA-Adapter's and OpenVLA-OFT's DINOv2 build GELU (erf) from ops the NPU runs. Fidelity and serving tables re-measured: only GR00T N1.5 and N1.6 stay above the bar served. qualcomm-iq10.md and the summary row re-measured at 7c76ea9: VLA-Adapter 853.7 to 708.4 ms, Octo-Small 124.0 to 119.4 ms, OpenVLA-OFT 1768.2 to 1725.5 ms, VLA-JEPA 452.9 to 512.6 ms. --- CHANGELOG.md | 9 +++- docs/backend/hexagon.md | 45 +++++++++++------ docs/benchmark/README.md | 4 +- docs/benchmark/qualcomm-iq10.md | 87 +++++++++++++++++---------------- 4 files changed, 85 insertions(+), 60 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 87340608..9c2b7eaa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,8 +26,13 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke DINOv2 + SigLIP towers take the same padding and flash attention, `linear()` multiplies a batch of activations as one matrix of rows, and large transposing copies run on the CPU: OpenVLA-OFT, VLA-Adapter and TurboVLA run 51-57% - faster, Evo-1 and SmolVLA 8-10%. GR00T N1.5, N1.6 and VLA-JEPA are above the - bar on a served LIBERO observation, as the CPU is at F16. See + faster, Evo-1 and SmolVLA 8-10%. VLA-JEPA's LM leaves flash attention unless + `--flash-attn` is given, which brings it within the bar on a served LIBERO + observation at 13% more latency. VLA-Adapter's LM and action head and Octo's + block transformer pad their softmax rows to a multiple of 32, and + VLA-Adapter's and OpenVLA-OFT's DINOv2 build GELU (erf) from ops the NPU runs: + VLA-Adapter 17% faster, Octo 4%. GR00T N1.5 and N1.6 are above the bar on a + served LIBERO observation, as the CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install and the numbers. - **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200` diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 90fd01e3..7ebb82fd 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -98,6 +98,10 @@ changed all three conclusions: IQ-10 (b11223): more accurate than the CPU's (relative error 1.5e-6 against 5.1e-5), and Evo-1 7% faster. It was left out: GR00T N1.7 sits close to the bar, and the kernel moved its fixed-input chunk from 2.0e-3 to 3.0e-3. + VLA-Adapter's and OpenVLA-OFT's DINOv2 MLP and projector build it from ops the + NPU runs instead (`vla::gelu_erf()`: x*sigmoid(p(x)), p an odd cubic in x + clamped to [-5, 5], within 2.5e-5 of the exact GELU): VLA-Adapter 740 to + 705 ms, OpenVLA-OFT 1771 to 1740 ms. - **Absent is no longer fatal.** `src/backend_fallback.cpp` wraps the NPU in a backend that runs whatever it rejects on the CPU, without touching an arch. That also covers ops refused for shape rather than type: softmax rows that are @@ -141,6 +145,19 @@ Flash attention on the HMX (v81) keeps scores and probabilities in F16. That is a precision limit rather than a wrong op, but in GR00T N1.7's LM it moves the action chunk 5.4e-3 from the CPU reference. N1.7's LM therefore takes the explicit path unless `--flash-attn` is given: 2.0e-3, at the same latency. +VLA-JEPA's LM does the same: with flash attention a served LIBERO observation +lands 6.7e-3 from the reference (the CPU with flash attention 5.5e-3), without +it 2.6e-3, at a cost of 683 to 810 ms with two views. + +The softmax takes rows of up to 32 elements or a multiple of 32. Where a +sequence can grow without changing its result, vla.cpp pads it to +`gemm_width()` on Hexagon rather than send every layer's softmax to the CPU: +VLA-Adapter's causal LM and Octo's block transformer get zero tokens appended, +masked out as keys, and each VLA-Adapter action-head block stacks its three +key/value sets as rows, padded and masked, instead of concatenating scores +along their columns. VLA-Adapter goes from 859 to 747 ms, Octo-Small from 126.5 +to 122.6 ms. OpenVLA-OFT's LM gains nothing that way: its 22 padding tokens add +as much 7B matmul as the softmax saves. `MUL_MAT` with a 2D weight against a batch of activations (weight `ne[2..3]` of 1, activations above 1) is claimed and comes out wrong. GR00T N1.6's SigLIP2 @@ -344,14 +361,14 @@ differ from the benchmark's for VLA-JEPA, VLA-Adapter, OpenVLA-OFT and Octo: | GR00T N1.5 | 1 | 2.4e-3 | | GR00T N1.6 | 1 | 1.4e-3 | | GR00T N1.7 | 1 | 2.0e-3 | -| VLA-JEPA | 2 | 2.8e-3 | -| VLA-Adapter | 2 | 5.6e-4 | -| OpenVLA-OFT | 2 | 2.9e-4 | -| Octo-Small | 1 | 4.1e-4 | +| VLA-JEPA | 2 | 2.2e-3 | +| VLA-Adapter | 2 | 5.4e-4 | +| OpenVLA-OFT | 2 | 3.2e-4 | +| Octo-Small | 1 | 5.3e-4 | | TurboVLA | 2 | 3.7e-4 | GR00T N1.6 with two views is 8.8e-4. At the benchmark's single view VLA-JEPA is -1.9e-3, VLA-Adapter 4.8e-4 and OpenVLA-OFT 1.9e-4. +1.9e-3, VLA-Adapter 5.7e-4 and OpenVLA-OFT 2.1e-4. ### Serving @@ -372,17 +389,17 @@ error reply to a malformed or an invalid request, and keeps serving afterwards. | GR00T N1.6 | 1.3e-2 | 1.1e-2 | | GR00T N1.7 | 2.5e-3 | | | π0 | 5.8e-4 | | -| VLA-JEPA | 6.7e-3 | 2.1e-2 | -| VLA-Adapter | 6.1e-4 | | -| OpenVLA-OFT | 5.2e-4 | | -| Octo-Small | 9.3e-5 | | +| VLA-JEPA | 2.6e-3 | 2.1e-2 | +| VLA-Adapter | 5.4e-4 | | +| OpenVLA-OFT | 5.9e-4 | | +| Octo-Small | 2.4e-4 | | | TurboVLA | 8.5e-4 | | -GR00T N1.5, N1.6 and VLA-JEPA are above the bar on this input, and so is the CPU -itself with F16 weights: N1.5 and VLA-JEPA land closer to the reference than the -CPU at F16, N1.6 slightly further (1.3e-2 against 1.1e-2). They are sensitive to -16-bit precision rather than miscomputed. Running them with F32 weights on the -NPU has not been tried yet. +GR00T N1.5 and N1.6 are above the bar on this input, and so is the CPU itself +with F16 weights: N1.5 lands closer to the reference than the CPU at F16, N1.6 +slightly further (1.3e-2 against 1.1e-2). They are sensitive to 16-bit precision +rather than miscomputed. Running them with F32 weights on the NPU has not been +tried yet. ## IQ-10, imagined diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index 59dda10b..007276f3 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`f7b0e68` (same llama.cpp pin). Each report +`7c76ea9` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 121.3 | 186.8 | 448.1 | 837.8 | 597.1 | 704.6 | 756.6 | N/A | 583.6 | 1116.8 | 1109.9 | 1761.5 | 2329.8 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 117.6 | 188.1 | 505.4 | 703.2 | 591.6 | 696.5 | 759.5 | N/A | 583.6 | 1114.0 | 1113.0 | 1720.3 | 2333.6 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 55dc1851..1e153881 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `f7b0e68` | +| Commit | `7c76ea9` | | llama.cpp | `b11223` | -| Date | 2026-11-04 15:48 to 16:23 (UTC+07) | +| Date | 2026-11-05 07:26 to 07:57 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -65,18 +65,18 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| Octo-Small | 2 | 256 + 128 | 121.3 | 124.0 | 122.8 | 128.5 | 20.4 | 253 | 731 | -| TurboVLA | 2 | 256 | 186.8 | 190.6 | 189.6 | 194.9 | - | 249 | 1009 | -| VLA-JEPA | 1 | 256 | 448.1 | 452.9 | 451.0 | 458.7 | 197.0 | 185 | 3971 | -| SmolVLA | 2 | 512 | 583.6 | 584.3 | 584.2 | 584.9 | 230.0 | 146 | 1197 | -| GR00T N1.5 | 1 | 224 | 597.1 | 599.1 | 598.6 | 600.0 | 138.9 | 162 | 3645 | -| GR00T N1.6 | 1 | 224 | 704.6 | 707.6 | 706.3 | 711.0 | 145.9 | 147 | 4765 | -| GR00T N1.7 | 1 | 256 | 756.6 | 762.6 | 761.2 | 769.1 | 195.5 | 191 | 5084 | -| VLA-Adapter | 1 | 224 | 837.8 | 853.7 | 852.5 | 863.3 | 296.0 | 872 | 3581 | -| π0.5 | 2 | 224 | 1109.9 | 1111.9 | 1112.0 | 1112.7 | 277.1 | 285 | 5839 | -| π0 | 2 | 224 | 1116.8 | 1120.5 | 1120.1 | 1123.1 | 277.6 | 285 | 5535 | -| OpenVLA-OFT | 1 | 224 | 1761.5 | 1768.2 | 1768.8 | 1772.7 | 298.5 | 844 | 15461 | -| Evo-1 | 1 | 448 | 2329.8 | 2346.0 | 2341.9 | 2360.7 | 819.2 | 319 | 1594 | +| Octo-Small | 2 | 256 + 128 | 117.6 | 119.4 | 118.4 | 120.7 | 21.2 | 253 | 723 | +| TurboVLA | 2 | 256 | 188.1 | 190.8 | 188.6 | 194.9 | - | 249 | 1075 | +| VLA-JEPA | 1 | 256 | 505.4 | 512.6 | 510.1 | 522.2 | 196.4 | 184 | 3969 | +| SmolVLA | 2 | 512 | 583.6 | 584.4 | 584.4 | 585.1 | 229.9 | 145 | 1196 | +| GR00T N1.5 | 1 | 224 | 591.6 | 594.9 | 593.4 | 597.1 | 138.9 | 162 | 3639 | +| GR00T N1.6 | 1 | 224 | 696.5 | 699.3 | 699.2 | 702.2 | 145.1 | 147 | 4692 | +| VLA-Adapter | 1 | 224 | 703.2 | 708.4 | 708.2 | 715.3 | 258.1 | 872 | 3566 | +| GR00T N1.7 | 1 | 256 | 759.5 | 766.3 | 766.0 | 773.4 | 197.6 | 191 | 5113 | +| π0 | 2 | 224 | 1114.0 | 1116.6 | 1116.4 | 1118.2 | 277.1 | 284 | 5537 | +| π0.5 | 2 | 224 | 1113.0 | 1117.3 | 1116.6 | 1120.7 | 277.9 | 285 | 5826 | +| OpenVLA-OFT | 1 | 224 | 1720.3 | 1725.5 | 1724.7 | 1731.0 | 263.4 | 844 | 15451 | +| Evo-1 | 1 | 448 | 2333.6 | 2348.3 | 2349.9 | 2360.1 | 820.6 | 319 | 1757 | ### Fastest configuration @@ -96,39 +96,42 @@ replaces the defaults only if its mean is more than 3% lower, both in the screen and in the full re-run. Flash attention and the matmul precision can move the action chunk. Success -rates measured at the defaults therefore do not carry over to these rows. +rates measured at the defaults therefore do not carry over to these rows. For +VLA-JEPA and GR00T N1.7, `--flash-attn` puts their LM back on flash attention, +which the defaults leave off because it is outside the fidelity bar (VLA-JEPA +6.7e-3 on a served LIBERO observation, N1.7 5.4e-3 on fixed inputs). | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| Octo-Small | *(defaults)* | 121.3 | 124.0 | 122.8 | 128.5 | 20.4 | 253 | 731 | - | -| TurboVLA | *(defaults)* | 186.8 | 190.6 | 189.6 | 194.9 | - | 249 | 1009 | - | -| VLA-JEPA | *(defaults)* | 448.1 | 452.9 | 451.0 | 458.7 | 197.0 | 185 | 3971 | - | -| SmolVLA | *(defaults)* | 583.6 | 584.3 | 584.2 | 584.9 | 230.0 | 146 | 1197 | - | -| GR00T N1.5 | *(defaults)* | 597.1 | 599.1 | 598.6 | 600.0 | 138.9 | 162 | 3645 | - | -| GR00T N1.6 | *(defaults)* | 704.6 | 707.6 | 706.3 | 711.0 | 145.9 | 147 | 4765 | - | -| GR00T N1.7 | `--flash-attn` | 733.3 | 739.1 | 736.8 | 747.5 | 195.3 | 191 | 5059 | -3.1% | -| VLA-Adapter | *(defaults)* | 837.8 | 853.7 | 852.5 | 863.3 | 296.0 | 872 | 3581 | - | -| π0.5 | *(defaults)* | 1109.9 | 1111.9 | 1112.0 | 1112.7 | 277.1 | 285 | 5839 | - | -| π0 | *(defaults)* | 1116.8 | 1120.5 | 1120.1 | 1123.1 | 277.6 | 285 | 5535 | - | -| OpenVLA-OFT | *(defaults)* | 1761.5 | 1768.2 | 1768.8 | 1772.7 | 298.5 | 844 | 15461 | - | -| Evo-1 | *(defaults)* | 2329.8 | 2346.0 | 2341.9 | 2360.7 | 819.2 | 319 | 1594 | - | +| Octo-Small | *(defaults)* | 117.6 | 119.4 | 118.4 | 120.7 | 21.2 | 253 | 723 | - | +| TurboVLA | *(defaults)* | 188.1 | 190.8 | 188.6 | 194.9 | - | 249 | 1075 | - | +| VLA-JEPA | `--flash-attn --mm-prec default` | 442.1 | 452.4 | 450.2 | 460.6 | 198.3 | 184 | 3985 | -11.7% | +| SmolVLA | *(defaults)* | 583.6 | 584.4 | 584.4 | 585.1 | 229.9 | 145 | 1196 | - | +| GR00T N1.5 | *(defaults)* | 591.6 | 594.9 | 593.4 | 597.1 | 138.9 | 162 | 3639 | - | +| GR00T N1.6 | *(defaults)* | 696.5 | 699.3 | 699.2 | 702.2 | 145.1 | 147 | 4692 | - | +| VLA-Adapter | *(defaults)* | 703.2 | 708.4 | 708.2 | 715.3 | 258.1 | 872 | 3566 | - | +| GR00T N1.7 | `--flash-attn --mm-prec default` | 727.8 | 734.1 | 733.2 | 742.4 | 196.2 | 191 | 5081 | -4.2% | +| π0 | *(defaults)* | 1114.0 | 1116.6 | 1116.4 | 1118.2 | 277.1 | 284 | 5537 | - | +| π0.5 | *(defaults)* | 1113.0 | 1117.3 | 1116.6 | 1120.7 | 277.9 | 285 | 5826 | - | +| OpenVLA-OFT | *(defaults)* | 1720.3 | 1725.5 | 1724.7 | 1731.0 | 263.4 | 844 | 15451 | - | +| Evo-1 | *(defaults)* | 2333.6 | 2348.3 | 2349.9 | 2360.1 | 820.6 | 319 | 1757 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| Octo-Small | 124.3 | 128.9 | 129.6 | 127.7 | 124.7 | -| TurboVLA | 194.6 | 192.5 | 188.6 | 194.3 | 251.9 | -| VLA-JEPA | 453.3 | 454.5 | 453.3 | 451.9 | 605.5 | -| SmolVLA | 586.3 | 586.9 | 584.4 | 586.0 | 2147.5 | -| GR00T N1.5 | 601.3 | 600.7 | 601.7 | 600.4 | 877.5 | -| GR00T N1.6 | 708.6 | 708.4 | 706.5 | 707.5 | 876.8 | -| GR00T N1.7 | 763.3 | 737.3 | 765.2 | 741.6 | 869.0 | -| VLA-Adapter | 854.5 | 859.7 | 870.9 | 878.5 | 1151.2 | -| π0.5 | 1113.6 | 1112.9 | 1113.4 | 1113.5 | 2047.2 | -| π0 | 1141.2 | 1140.3 | 1141.9 | 1140.9 | 2044.7 | -| OpenVLA-OFT | 1776.1 | 1775.6 | 1770.0 | 1765.2 | 2045.9 | -| Evo-1 | 2339.7 | 2350.8 | 2340.9 | 2352.5 | 3868.0 | +| Octo-Small | 121.8 | 120.4 | 121.8 | 123.2 | 123.8 | +| TurboVLA | 196.5 | 189.3 | 192.3 | 190.9 | 257.4 | +| VLA-JEPA | 514.8 | 454.3 | 512.0 | 452.9 | 1662.4 | +| SmolVLA | 584.6 | 585.0 | 585.2 | 584.6 | 2147.2 | +| GR00T N1.5 | 594.7 | 592.6 | 600.1 | 601.0 | 877.2 | +| GR00T N1.6 | 709.6 | 714.5 | 711.1 | 713.1 | 875.3 | +| VLA-Adapter | 712.4 | 711.1 | 713.7 | 712.8 | 989.2 | +| GR00T N1.7 | 765.5 | 739.0 | 758.3 | 733.1 | 865.3 | +| π0 | 1131.6 | 1138.1 | 1118.0 | 1128.8 | 2051.6 | +| π0.5 | 1125.8 | 1109.1 | 1134.2 | 1134.6 | 2036.0 | +| OpenVLA-OFT | 1737.9 | 1722.4 | 1715.7 | 1718.5 | 2005.8 | +| Evo-1 | 2345.2 | 2352.4 | 2355.4 | 2361.4 | 3877.1 |
@@ -148,9 +151,9 @@ one batched `MUL_MAT` of its SigLIP2 tower still runs on the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). These rows time the defaults; they do not vouch for accuracy. On a served LIBERO -observation, GR00T N1.5, N1.6 and VLA-JEPA land above the 2.9e-3 fidelity bar: -N1.5 and VLA-JEPA closer to the reference than the CPU at F16, N1.6 slightly -further; see [backend/hexagon.md](../backend/hexagon.md#serving). +observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar: N1.5 +closer to the reference than the CPU at F16, N1.6 slightly further; see +[backend/hexagon.md](../backend/hexagon.md#serving). ## Reproducing From 8abda27ccd0a8c39bd7198b8ea28a5ca294168c1 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 17:05:05 +0700 Subject: [PATCH 26/38] hexagon: run weight-reading pads on the CPU The NPU claims a PAD whose source is a weight, answers NO_SUPPORT, and the host aborts, as it does for a weight-reading CPY. The wrapper now runs those pads on the CPU. On the Dragonwing IQ-10 the twelve archs already on the NPU give the same actions as before (13 fixed-input configurations against the CPU F32 reference). --- src/backend_fallback.cpp | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/backend_fallback.cpp b/src/backend_fallback.cpp index 47b3181e..a4eaed17 100644 --- a/src/backend_fallback.cpp +++ b/src/backend_fallback.cpp @@ -359,6 +359,13 @@ bool accel_runs(FallbackCtx * fc, const ggml_tensor * t, const char ** why) { *why = "cannot run (NO_SUPPORT) a weight-reading"; return false; } + if (fc->hexagon && t->op == GGML_OP_PAD && is_weight(t->src[0])) { + // The same for a pad that reads a weight: ACT zero-pads its stem kernel + // [147,64] to K = 160 so the conv's matmul takes the HMX, and the host + // aborts with the pad on the NPU. + *why = "cannot run (NO_SUPPORT) a weight-reading"; + return false; + } if (fc->hexagon && t->op == GGML_OP_REPEAT && !is_weight(t->src[0])) { // The NPU's REPEAT reads its source through the data cache, which is // invalidated only between op batches, while the op that wrote the From 0f3e010841424ff9eb4c26666533902a632c10bc Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 17:05:05 +0700 Subject: [PATCH 27/38] act: pad the ResNet's patch matrices to a multiple of 32 on Hexagon The HMX takes a matmul only when K and the rows of its first operand are multiples of 32. ACT's stem (K = 147) and layer3 and layer4 (1200 and 300 output positions at 640x480) ran their convs' matmuls on the vector units: 80 ms of the 101 ms spent on the NPU. Where gemm_width() pads (Hexagon), for one view, the im2col matrix and the stem's kernel are zero-padded and the extra output rows dropped, which leaves the product unchanged. Other backends and multi-view inputs keep conv_2d_f32. On the Dragonwing IQ-10, ACT goes from 228.5 to 155.6 ms (mean, vla-bench, one 640x480 view; the same build on the CPU 170.8 ms), 6.1e-4 from the CPU F32 reference on the fixed input and 1.0e-3 served. --- src/models/act.cpp | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/src/models/act.cpp b/src/models/act.cpp index d05ec7d4..56586def 100644 --- a/src/models/act.cpp +++ b/src/models/act.cpp @@ -176,12 +176,34 @@ struct ActModelArch : public ModelArchBase { namespace { +// conv_2d_f32 with the patch matrix padded with zeros to gemm_width() in both +// dims, for one view. The HMX takes a matmul only when those are multiples of +// 32; the stem's K of 147 and layer3's and layer4's 1200 and 300 rows are not. +// Returns nullptr where nothing would be padded. +ggml_tensor * conv_2d_padded(ggml_context * C, ggml_tensor * w, ggml_tensor * x, int stride, int pad) { + const int64_t ow = (x->ne[0] + 2*pad - w->ne[0]) / stride + 1, oh = (x->ne[1] + 2*pad - w->ne[1]) / stride + 1; + const int64_t K = w->ne[0]*w->ne[1]*w->ne[2], R = ow*oh, Kp = gemm_width(K), Rp = gemm_width(R); + if (x->ne[3] != 1 || (Kp == K && Rp == R)) + return nullptr; + ggml_tensor * col = ggml_im2col(C, w, x, stride, stride, pad, pad, 1, 1, true, GGML_TYPE_F32); + ggml_tensor * c2 = ggml_pad(C, ggml_reshape_2d(C, col, K, R), (int) (Kp - K), (int) (Rp - R), 0, 0); + ggml_tensor * w2 = ggml_reshape_2d(C, w, K, w->ne[3]); + if (Kp > K) + w2 = ggml_pad(C, w2, (int) (Kp - K), 0, 0, 0); + ggml_tensor * y = ggml_mul_mat(C, c2, w2); + if (Rp > R) + y = ggml_cont(C, ggml_view_2d(C, y, R, y->ne[1], y->nb[1], 0)); + return ggml_reshape_4d(C, y, col->ne[1], col->ne[2], w->ne[3], 1); +} + // [w, h, c, view] to [ow, oh, oc, view]. The kernel's type picks the path; see // ActModelArch::conv_direct. ggml_tensor * conv2d(ggml_context * C, const ConvW & cw, ggml_tensor * x, int stride, int pad) { - ggml_tensor * y = cw.w->type == GGML_TYPE_F16 - ? ggml_conv_2d_direct(C, cw.w, x, stride, stride, pad, pad, 1, 1) - : conv_2d_f32(C, cw.w, x, stride, pad); + ggml_tensor * y = nullptr; + if (cw.w->type == GGML_TYPE_F16) + y = ggml_conv_2d_direct(C, cw.w, x, stride, stride, pad, pad, 1, 1); + else if (!(y = conv_2d_padded(C, cw.w, x, stride, pad))) + y = conv_2d_f32(C, cw.w, x, stride, pad); return ggml_add(C, y, ggml_reshape_4d(C, cw.b, 1, 1, cw.b->ne[0], 1)); } From 11861b347695c09d26e4460ca85af9406f49390c Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 17:09:51 +0700 Subject: [PATCH 28/38] docs: ACT on the IQ-10 Hexagon NPU, benchmark at 0f3e010 hexagon.md: pads that read a weight run on the CPU; ACT's ResNet convs pad their patch matrices for the HMX; ACT in the fidelity (6.1e-4) and serving (1.0e-3) tables. qualcomm-iq10.md: ACT's configuration, latency (152.9 ms mean, 170.8 ms on the CPU) and flag screen at 0f3e010, and the 7c76ea9 run's date given in UTC+07 (the board's clock is UTC). CHANGELOG: ACT. --- CHANGELOG.md | 8 ++++++-- docs/backend/hexagon.md | 26 ++++++++++++++++++++------ docs/benchmark/qualcomm-iq10.md | 15 +++++++++++---- 3 files changed, 37 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9c2b7eaa..48f547e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,8 +31,12 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke observation at 13% more latency. VLA-Adapter's LM and action head and Octo's block transformer pad their softmax rows to a multiple of 32, and VLA-Adapter's and OpenVLA-OFT's DINOv2 build GELU (erf) from ops the NPU runs: - VLA-Adapter 17% faster, Octo 4%. GR00T N1.5 and N1.6 are above the bar on a - served LIBERO observation, as the CPU is at F16. See + VLA-Adapter 17% faster, Octo 4%. ACT runs within 6.1e-4 of the CPU F32 + reference (1.0e-3 served): pads that read a weight, also answered + `NO_SUPPORT`, run on the CPU, and its ResNet's im2col matrices are + zero-padded to a multiple of 32 so the convs' matmuls take the HMX (152.9 ms, + 170.8 ms on the CPU). GR00T N1.5 and N1.6 are above the bar on a served LIBERO + observation, as the CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install and the numbers. - **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200` diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 7ebb82fd..14a4abd8 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -5,7 +5,7 @@ to take once it ran. > **Status: running on Windows on Arm and on a Dragonwing IQ-10 under Linux.** > vla.cpp builds with `GGML_HEXAGON=ON` and runs eleven checkpoints on a -> Snapdragon X laptop's v73 NPU and twelve on an IQ-10's v81 NPU under Linux, +> Snapdragon X laptop's v73 NPU and thirteen on an IQ-10's v81 NPU under Linux, > each checked against a CPU reference. The laptop build, its measurements and > the upstream bugs found along the way are in > [hexagon-windows.md](hexagon-windows.md); the Linux board is in [Linux @@ -129,9 +129,9 @@ Two more ops are claimed and then refused at run time. On buffers mapped above 4 GB of DSP address space, which the IQ-10 under Linux hands out, the NPU runs `CONCAT` only as the dim-1 contiguous DMA copy and answers any other concat with `NO_SUPPORT`, and the host aborts on the first predict. The wrapper sends those -concats to the CPU. A `CPY` that reads a weight gets the same answer -(VLA-Adapter casts its F16 action queries to F32 in the graph), and also runs on -the CPU. +concats to the CPU. A `CPY` or a `PAD` that reads a weight gets the same answer +(VLA-Adapter casts its F16 action queries to F32 in the graph, ACT zero-pads its +stem kernel), and also runs on the CPU. The experts' KV concats (π0, π0.5, SmolVLA) were such concats: along the token dim of contiguous [head dim, heads, tokens] tensors, once per layer per denoise @@ -178,7 +178,7 @@ less than the trip to the CPU. The matrix unit (HMX) takes a matmul only when its inner dim K and its output dim N are multiples of 32, and with a batch only an F16 src0. Anything else runs on the vector units, at about 1/20 of the speed. `GGML_HEXAGON_PROFILE=1` on the -IQ-10 showed three such matmuls dominating, and vla.cpp now avoids them in +IQ-10 showed such matmuls dominating, and vla.cpp now avoids them in Hexagon builds: - **SigLIP-so400m's MLP is 4304 wide**: 440 ms of GR00T N1.5's 565 ms tower. @@ -196,6 +196,15 @@ Hexagon builds: 311 ms. - **Evo-1's LM attention multiplied F32 K and V against 14 query heads.** K and V now go to F16 (`fa_kv()`); the scores still accumulate in F32. +- **ACT's ResNet convs are im2col and one matmul** with the patches as its + first operand. The stem's K of 147 and layer3's and layer4's 1200 and 300 + output positions (640x480) are not multiples of 32: 80 ms of the 101 ms spent + on the NPU. On Hexagon `conv_2d_padded()` in `act.cpp` zero-pads the patch + matrix and the stem's kernel to `gemm_width()` and drops the extra rows: + 228.5 to 155.6 ms (mean, 5 reps). Its im2col (padded and overlapping), bias + adds (a gapped broadcast) and max pool still run on the CPU; under the + benchmark's protocol ACT takes 152.9 ms on the NPU against 170.8 ms on the + CPU. What is left on the vector units: GR00T's DiT attention (head dim 48) and Evo-1's DiT cross-attention (head dim 112, 1025 keys), whose shapes the HMX does not @@ -366,9 +375,11 @@ differ from the benchmark's for VLA-JEPA, VLA-Adapter, OpenVLA-OFT and Octo: | OpenVLA-OFT | 2 | 3.2e-4 | | Octo-Small | 1 | 5.3e-4 | | TurboVLA | 2 | 3.7e-4 | +| ACT | 1 | 6.1e-4 | GR00T N1.6 with two views is 8.8e-4. At the benchmark's single view VLA-JEPA is -1.9e-3, VLA-Adapter 5.7e-4 and OpenVLA-OFT 2.1e-4. +1.9e-3, VLA-Adapter 5.7e-4 and OpenVLA-OFT 2.1e-4. ACT takes its one 640x480 +view, the benchmark's input. ### Serving @@ -379,6 +390,8 @@ state), with fixed noise for the archs that draw it. The same request goes to a Hexagon server and to a CPU server with `--weight-dtype f32 --flash-attn 0`. Every model returns a finite chunk, the same chunk for a repeated request, an error reply to a malformed or an invalid request, and keeps serving afterwards. +ACT has no `eval/client` preset; it gets the same checks from raw `vla.proto` +requests carrying one fixed 640x480 view and a fixed 14-dim state. | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 | |---|--:|--:| @@ -394,6 +407,7 @@ error reply to a malformed or an invalid request, and keeps serving afterwards. | OpenVLA-OFT | 5.9e-4 | | | Octo-Small | 2.4e-4 | | | TurboVLA | 8.5e-4 | | +| ACT | 1.0e-3 | | GR00T N1.5 and N1.6 are above the bar on this input, and so is the CPU itself with F16 weights: N1.5 lands closer to the reference than the CPU at F16, N1.6 diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 1e153881..39d1ef14 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `7c76ea9` | +| Commit | `7c76ea9`; ACT at `0f3e010` | | llama.cpp | `b11223` | -| Date | 2026-11-05 07:26 to 07:57 (UTC+07) | +| Date | 2026-11-05 14:26 to 14:57 (UTC+07); ACT 17:00 to 17:01 | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -31,8 +31,9 @@ buffers. Nothing else ran on the board. ## Model configuration Each model runs at its native view count and input size, with weights as -shipped. Checkpoints are the `vrfai/*` GGUFs; where a repo ships several, we use -the `libero_object` variant. Settings not listed are the defaults. +shipped. Checkpoints are the `vrfai/*` GGUFs, and ACT's is +`ravediamond/act-aloha-sim-transfer-cube-gguf`; where a repo ships several, we +use the `libero_object` variant. Settings not listed are the defaults. | Model | Checkpoint | Views | Input | Lang tokens | Action chunk | Action dim | Action head | Steps | |---|---|--:|--:|--:|--:|--:|---|--:| @@ -49,6 +50,7 @@ the `libero_object` variant. Settings not listed are the defaults. | OpenVLA-OFT | `openvla-oft-libero-gguf` | 1 | 224 | 16 | 8 | 7 | L1 regression head | 1 | | Octo-Small | `octo-small-libero-gguf` | 2 | 256 + 128 | 16 | 4 | 7 | DDPM diffusion | 20 | | TurboVLA | `turbovla-libero-gguf` | 2 | 256 | 16 | 12 | 7 | single-pass decoder | 1 | +| ACT | `act-aloha-sim-transfer-cube-gguf` | 1 | 640 x 480 | - | 100 | 14 | transformer decoder | 1 | - **Lang tokens** is the synthetic prompt length. VLA-JEPA adds its 32 `` tokens with `--extra-token 151697 --extra-count 32`. @@ -57,6 +59,8 @@ the `libero_object` variant. Settings not listed are the defaults. `VLA_GR00T_EMBODIMENT=new_embodiment` for N1.5, `libero_panda` for N1.6 and `libero_sim` for N1.7. **Octo** needs `VLA_OCTO_UNNORM_DATASET=libero_object`. - **Octo** takes a 256 px primary view and a 128 px wrist view. +- **ACT** is an ALOHA checkpoint (sim transfer cube): no language, a 14-dim + bimanual state and action, run with `--images 1 --size 640 --height 480`. ## Latency and memory @@ -66,6 +70,7 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| | Octo-Small | 2 | 256 + 128 | 117.6 | 119.4 | 118.4 | 120.7 | 21.2 | 253 | 723 | +| ACT | 1 | 640 x 480 | 148.5 | 152.9 | 152.5 | 157.3 | - | 194 | 467 | | TurboVLA | 2 | 256 | 188.1 | 190.8 | 188.6 | 194.9 | - | 249 | 1075 | | VLA-JEPA | 1 | 256 | 505.4 | 512.6 | 510.1 | 522.2 | 196.4 | 184 | 3969 | | SmolVLA | 2 | 512 | 583.6 | 584.4 | 584.4 | 585.1 | 229.9 | 145 | 1196 | @@ -104,6 +109,7 @@ which the defaults leave off because it is outside the fidelity bar (VLA-JEPA | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| | Octo-Small | *(defaults)* | 117.6 | 119.4 | 118.4 | 120.7 | 21.2 | 253 | 723 | - | +| ACT | *(defaults)* | 148.5 | 152.9 | 152.5 | 157.3 | - | 194 | 467 | - | | TurboVLA | *(defaults)* | 188.1 | 190.8 | 188.6 | 194.9 | - | 249 | 1075 | - | | VLA-JEPA | `--flash-attn --mm-prec default` | 442.1 | 452.4 | 450.2 | 460.6 | 198.3 | 184 | 3985 | -11.7% | | SmolVLA | *(defaults)* | 583.6 | 584.4 | 584.4 | 585.1 | 229.9 | 145 | 1196 | - | @@ -121,6 +127,7 @@ which the defaults leave off because it is outside the fidelity bar (VLA-JEPA | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| | Octo-Small | 121.8 | 120.4 | 121.8 | 123.2 | 123.8 | +| ACT | 143.4 | 153.4 | 157.5 | 155.8 | 174.8 | | TurboVLA | 196.5 | 189.3 | 192.3 | 190.9 | 257.4 | | VLA-JEPA | 514.8 | 454.3 | 512.0 | 452.9 | 1662.4 | | SmolVLA | 584.6 | 585.0 | 585.2 | 584.6 | 2147.2 | From 07b4ca98f4d7d70730c21102d2d41785068af9d2 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 17:14:12 +0700 Subject: [PATCH 29/38] openvla_oft: the LM's attention on the HMX on Hexagon The LM's explicit attention multiplied F32 K and V against 32 heads on the NPU's vector units, 626 ms per predict on the Dragonwing IQ-10, and 330 keys are not a width the HMX takes. On Hexagon K and V now go to F16 (fa_kv()) and the keys are padded with zeros to gemm_width(SEQ), masked out in the softmax; the queries keep SEQ rows. Scores accumulate in F32 and the softmax stays F32. Other backends keep F32 K/V and SEQ keys. OpenVLA-OFT goes from 1729 to 1127 ms (mean, vla-bench, one view), 3.1e-4 from the CPU F32 reference on the fixed input and 7.5e-4 served. --- src/models/openvla_oft.cpp | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/src/models/openvla_oft.cpp b/src/models/openvla_oft.cpp index 4a7ecc77..9d460a5e 100644 --- a/src/models/openvla_oft.cpp +++ b/src/models/openvla_oft.cpp @@ -28,6 +28,7 @@ #include "layers/linear.h" #include "layers/norm.h" +#include #include #include #include @@ -68,6 +69,7 @@ struct OpenVlaOftModelArch : public ModelArchBase { }; struct MainIO { ggml_tensor *t_ids=nullptr,*t_state=nullptr,*t_proj=nullptr,*act0=nullptr,*t_pos=nullptr,*norm_actions=nullptr; + ggml_tensor *kv_mask=nullptr; bool consts=false; }; graph_cache main_graph; @@ -255,6 +257,9 @@ std::vector OpenVlaOftModelArch::predict(const Inputs& in) { seq=ggml_concat(C,seq,stop,1); ggml_tensor*t_pos=ggml_new_tensor_1d(C,GGML_TYPE_I32,SEQ); ggml_set_input(t_pos); ggml_set_output(t_pos); + const int64_t KP=gemm_width(SEQ); + ggml_tensor*kv_mask=nullptr; + if(KP>SEQ){ kv_mask=ggml_new_tensor_2d(C,GGML_TYPE_F32,KP,SEQ); ggml_set_input(kv_mask); ggml_set_output(kv_mask); } const float lsc=1.0f/std::sqrt((float)lm_head_dim); ggml_tensor*x=seq; const bool fused_lm=fused_ops && cublas_rows(SEQ) && SEQ<=4096; @@ -278,11 +283,21 @@ std::vector OpenVlaOftModelArch::predict(const Inputs& in) { ggml_tensor*qh=ggml_reshape_3d(C,qp,lm_head_dim,n_q,SEQ),*kh=ggml_reshape_3d(C,kp,lm_head_dim,n_kv,SEQ),*vh=ggml_reshape_3d(C,vp,lm_head_dim,n_kv,SEQ); ggml_tensor*qr=ggml_rope_ext(C,qh,t_pos,nullptr,(int)lm_head_dim,GGML_ROPE_TYPE_NEOX,0,lm_rope_base,1.0f,0.0f,1.0f,32.0f,1.0f); ggml_tensor*kr=ggml_rope_ext(C,kh,t_pos,nullptr,(int)lm_head_dim,GGML_ROPE_TYPE_NEOX,0,lm_rope_base,1.0f,0.0f,1.0f,32.0f,1.0f); + // fa_kv: on Hexagon K and V go to F16, and the keys are padded with zeros + // to gemm_width(SEQ), masked out, so both matmuls run on the HMX rather + // than as F32 x F32 on the vector units. Scores accumulate in F32 and the + // softmax stays F32. Other backends keep F32 K/V and SEQ keys. ggml_tensor*Q=ggml_cont(C,ggml_permute(C,qr,0,2,1,3)),*K=ggml_cont(C,ggml_permute(C,kr,0,2,1,3)),*V=ggml_cont(C,ggml_permute(C,vh,1,2,0,3)); + if(kv_mask){ + K=ggml_pad(C,K,0,(int)(KP-SEQ),0,0); + V=ggml_pad(C,V,(int)(KP-SEQ),0,0,0); + } + K=fa_kv(C,K); V=fa_kv(C,V); ggml_tensor*kq=ggml_mul_mat(C,K,Q); ggml_prec_set_acc(kq,GGML_PREC_F32); - // Unmasked on purpose: OpenVLA-OFT patches transformers to replace the - // causal mask across the whole sequence (modeling_llama.py:719-723). - ggml_tensor*aw=ggml_soft_max_ext(C,kq,nullptr,lsc,0.0f); + // No causal mask on purpose: OpenVLA-OFT patches transformers to replace + // it across the whole sequence (modeling_llama.py:719-723). kv_mask only + // hides the key padding. + ggml_tensor*aw=ggml_soft_max_ext(C,kq,kv_mask,lsc,0.0f); ggml_tensor*kqv=ggml_mul_mat(C,V,aw); ggml_tensor*att=ggml_reshape_2d(C,ggml_cont(C,ggml_permute(C,kqv,0,2,1,3)),HC,SEQ); x=ggml_add(C,x,ggml_mul_mat(C,l.Wo,att)); @@ -306,7 +321,7 @@ std::vector OpenVlaOftModelArch::predict(const Inputs& in) { ggml_tensor*norm_actions=linear(C,h_fc2w,h_fc2b,hh); ggml_set_output(norm_actions); gio.t_ids=t_ids; gio.t_state=t_state; gio.t_proj=t_proj; gio.act0=act0; - gio.t_pos=t_pos; gio.norm_actions=norm_actions; + gio.t_pos=t_pos; gio.norm_actions=norm_actions; gio.kv_mask=kv_mask; ggml_cgraph*gf=ggml_new_graph_custom(C,16384,false); ggml_build_forward_expand(gf,norm_actions); return gf; @@ -330,6 +345,13 @@ std::vector OpenVlaOftModelArch::predict(const Inputs& in) { for(int64_t i=0;ine[0]; + std::vector mk((size_t)(KP*SEQ),0.0f); + for(int64_t q=0;q z((size_t)HC*n_act,0.0f); ggml_backend_tensor_set(act0,z.data(),0,ggml_nbytes(act0)); gio.consts=true; From f7ad69822694c6f6f37f820dc61e06b48962e230 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 17:34:58 +0700 Subject: [PATCH 30/38] docs: Hexagon support matrix and an ACT column for the IQ-10 results MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README: the Hexagon column follows the Dragonwing IQ-10 under Linux, where π0, VLA-JEPA, OpenVLA-OFT and ACT run within the 2.9e-3 bar and are benchmarked; a note gives the Snapdragon X laptop's state, and the column and the backend list link docs/backend/hexagon.md, which covers both. docs/benchmark/README.md: an ACT column, 148.5 ms on the IQ-10 at 0f3e010 and n/r elsewhere. --- README.md | 18 +++++++++----- docs/benchmark/README.md | 52 +++++++++++++++++++++------------------- 2 files changed, 39 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index 0cacbd2e..bdd9e87b 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ cmake --build build -j$(nproc) For a GPU or NPU, follow the backend's page in [docs/backend/](docs/backend): [CUDA](docs/backend/cuda.md), [ROCm](docs/backend/rocm.md), [SYCL](docs/backend/sycl.md), [OpenVINO](docs/backend/ov.md), -[Metal](docs/backend/metal.md), [Hexagon](docs/backend/hexagon-windows.md). +[Metal](docs/backend/metal.md), [Hexagon](docs/backend/hexagon.md). For containers, see [docs/DOCKER.md](docs/DOCKER.md). --- @@ -86,10 +86,10 @@ Per-device latency and memory are in [docs/benchmark/](docs/benchmark). Use [ggml-compiler](https://github.com/khanhnd61-vr/ggml-compiler) and follow [the instruction](docs/ARCHITECTURE.md#adding-an-architecture) to add a new model. -| Model | CPU (x86-64 / ARM) | [CUDA (NVIDIA)](docs/backend/cuda.md) | [ROCm (AMD)](docs/backend/rocm.md) | [SYCL (Intel)](docs/backend/sycl.md) | [Metal (Apple)](docs/backend/metal.md) | [OpenVINO (Intel)](docs/backend/ov.md) | [Hexagon (Qualcomm)](docs/backend/hexagon-windows.md) | +| Model | CPU (x86-64 / ARM) | [CUDA (NVIDIA)](docs/backend/cuda.md) | [ROCm (AMD)](docs/backend/rocm.md) | [SYCL (Intel)](docs/backend/sycl.md) | [Metal (Apple)](docs/backend/metal.md) | [OpenVINO (Intel)](docs/backend/ov.md) | [Hexagon (Qualcomm)](docs/backend/hexagon.md) | |---|:--:|:--:|:--:|:--:|:--:|:--:|:--:| | [SmolVLA](https://hf.co/vrfai/smolvla-libero-gguf) | Y | Y | Y | Y | Y | Y | Y | -| [π0](https://hf.co/vrfai/pi0-libero-finetuned-v044-gguf) | Y | Y | Y | Y | Y | Y | ~ | +| [π0](https://hf.co/vrfai/pi0-libero-finetuned-v044-gguf) | Y | Y | Y | Y | Y | Y | Y | | [π0.5](https://hf.co/vrfai/pi05-libero-gguf) | Y | Y | Y | Y | Y | Y | Y | | [GR00T N1.5](https://hf.co/vrfai/gr00tn1d5-libero-object-gguf) | Y | Y | Y | Y | Y | Y | Y | | [GR00T N1.6](https://hf.co/vrfai/gr00tn1d6-libero-gguf) | Y | Y | Y | Y | Y | Y | Y | @@ -97,14 +97,20 @@ Use [ggml-compiler](https://github.com/khanhnd61-vr/ggml-compiler) and follow [t | [BitVLA](https://hf.co/vrfai/bitvla-libero-gguf) | Y | Y | - | - | - | - | - | | [Evo-1](https://hf.co/vrfai/evo1-libero-gguf) | Y | Y | Y | Y | Y | Y | Y | | [VLA-Adapter](https://hf.co/vrfai/vla-adapter-libero-gguf) | Y | Y | Y | Y | Y | Y | Y | -| [OpenVLA-OFT](https://hf.co/vrfai/openvla-oft-libero-gguf) | Y | Y | Y | Y | Y | Y | - | -| [VLA-JEPA](https://hf.co/vrfai/vla-jepa-libero) | Y | Y | Y | Y | Y | Y | ~ | +| [OpenVLA-OFT](https://hf.co/vrfai/openvla-oft-libero-gguf) | Y | Y | Y | Y | Y | Y | Y | +| [VLA-JEPA](https://hf.co/vrfai/vla-jepa-libero) | Y | Y | Y | Y | Y | Y | Y | | [Octo-Small](https://hf.co/vrfai/octo-small-libero-gguf) | Y | Y | Y | Y | Y | - | Y | | [TurboVLA](https://hf.co/vrfai/turbovla-libero-gguf) | Y | Y | Y | Y | Y | Y | Y | -| [ACT](https://hf.co/ravediamond/act-aloha-sim-transfer-cube-gguf) | Y | Y | Y | ~ | Y | ~ | ~ | +| [ACT](https://hf.co/ravediamond/act-aloha-sim-transfer-cube-gguf) | Y | Y | Y | ~ | Y | ~ | Y | Legend: `Y` = supported (released and benchmarked), `~` = in progress, `-` = planned. +The Hexagon column is the Dragonwing IQ-10 under Linux +([docs/backend/hexagon.md](docs/backend/hexagon.md)). On a Snapdragon X laptop +under Windows, π0 and VLA-JEPA land above the 2.9e-3 fidelity bar, OpenVLA-OFT +was not attempted (about 14 GB at F16, over its 8 GB budget) and ACT was not run +([docs/backend/hexagon-windows.md](docs/backend/hexagon-windows.md)). + --- ## Rollout on a real robot diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index 1d0a42a7..50c184ea 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`7c76ea9` (same llama.cpp pin). Each report +`7c76ea9`, its ACT cell at `0f3e010` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -21,33 +21,35 @@ every report has. `N/A` means the arch does not run on that backend, and `-` means the model does not fit or fails there; each report says which and why. `n/r` means not run. -| Device | Octo-Small | TurboVLA | VLA-JEPA | VLA-Adapter | GR00T N1.5 | GR00T N1.6 | GR00T N1.7 | BitVLA | SmolVLA | π0 | π0.5 | OpenVLA-OFT | Evo-1 | -|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| [RTX 5090 (CUDA), author's numbers](rtx-5090.md) | 2.69 | 4.88 | 14.3 | 16.0 | 16.1 | 19.4 | 19.5 | 20.8 | 38.2 | 28.8 | 29.2 | 34.5 | 49.1 | -| [RTX 3090 (CUDA)](rtx-3090.md) | 4.3 | 9.0 | 27.3 | 34.2 | 36.6 | 36.3 | 35.3 | 26.9 | 61.0 | 97.5 | 97.8 | 134.2 | 108.6 | -| [RTX 3060 (CUDA)](rtx-3060.md) | 8.3 | 21.6 | 57.1 | 74.7 | 86.8 | 78.1 | 82.8 | 59.1 | 125.1 | 228.2 | 229.9 | - | 230.8 | -| [RTX 5070 Laptop (CUDA)](rtx-5070-laptop.md) | 4.7 | 14.1 | 38.0 | 56.3 | 73.0 | 65.5 | 63.3 | 54.4 | 89.3 | 177.1 | 180.5 | - | 184.6 | -| [Radeon RX 9060 XT (ROCm)](rx-9060-xt.md) | 14.0 | 33.8 | 65.9 | 70.1 | 79.5 | 94.6 | 97.1 | N/A | 207.7 | 217.9 | 220.6 | 182.2 | 233.6 | -| [Jetson AGX Orin (CUDA)](jetson-agx-orin.md) | 14.3 | 28.0 | 60.6 | 79.1 | 96.7 | 101.2 | 93.0 | 76.8 | 164.3 | 269.0 | 266.6 | 297.4 | 298.5 | -| [Jetson Orin NX 16 GB (CUDA)](jetson-orin-nx.md) | 27.4 | 79.7 | 176.7 | 270.7 | 260.8 | 252.1 | 251.4 | 269.1 | 411.2 | 739.6 | 742.9 | - | 893.2 | -| [Jetson Orin Nano Super (CUDA)](jetson-orin-nano.md) | 31.9 | 89.3 | 193.9 | 297.5 | 293.4 | 272.3 | 271.7 | 335.0 | 462.8 | 845.2 | 849.7 | - | 1011.5 | -| [Apple M4 (Metal)](apple-m4.md) | 22.6 | 65.2 | 248.7 | 315.7 | 401.5 | 358.9 | 350.8 | N/A | 357.6 | 1141.1 | 1153.8 | 1650.7 | 829.3 | -| [Intel Arc B390 iGPU (SYCL)](core-ultra-x7-358h.md) | 15.5 | 25.0 | 102.1 | 62.6 | 101.2 | 106.1 | 178.4 | N/A | 61.7 | 269.3 | 275.2 | 434.9 | 270.4 | -| [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 28.5 | 116.3 | 170.2 | 106.0 | 122.8 | 131.4 | N/A | 213.1 | 620.5 | 232.5 | 1262.5 | 566.0 | -| [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | -| [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 117.6 | 188.1 | 505.4 | 703.2 | 591.6 | 696.5 | 759.5 | N/A | 583.6 | 1114.0 | 1113.0 | 1720.3 | 2333.6 | -| [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 23.5 | 90.7 | 126.7 | 134.0 | 161.1 | 144.9 | N/A | 161.3 | 438.3 | 430.2 | 405.8 | 256.3 | -| [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | -| [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | -| [Intel Core Ultra X7 358H (CPU)](core-ultra-x7-358h.md) | 48.0 | 199.0 | 1258.1 | 1645.4 | 1979.1 | 1697.3 | 1532.5 | N/A | 1758.0 | 6083.9 | 6120.3 | 9894.3 | 4025.8 | -| [Intel Core Ultra X7 358H (OpenVINO CPU)](core-ultra-x7-358h.md) | N/A | 238.4 | 1407.2 | 1831.2 | 2456.6 | 2300.4 | 2076.5 | N/A | 1899.5 | 7537.1 | 7522.7 | 11651.3 | 4626.5 | -| [Intel Core i5-12400F (CPU)](core-i5-12400f.md) | 78.1 | 392.7 | 1668.5 | 2165.7 | 2681.4 | 2216.8 | 1973.6 | N/A | 2288.4 | 8953.5 | 8637.9 | - | 5435.0 | -| [AMD Ryzen 5 5500 (CPU)](ryzen-5-5500.md) | 96.6 | 475.7 | 2133.0 | 2781.7 | 3447.0 | 2826.0 | 2518.8 | N/A | 3124.3 | 11321.3 | 11225.6 | - | 7347.2 | -| [Snapdragon X Oryon (CPU)](snapdragon-x-hexagon.md) | 109.5 | 548.9 | 1656.5 | 2013.1 | 2736.6 | 2226.0 | 1988.1 | N/A | 2080.9 | 9224.5 | 9140.6 | - | 5518.9 | +| Device | Octo-Small | ACT | TurboVLA | VLA-JEPA | VLA-Adapter | GR00T N1.5 | GR00T N1.6 | GR00T N1.7 | BitVLA | SmolVLA | π0 | π0.5 | OpenVLA-OFT | Evo-1 | +|---|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:| +| [RTX 5090 (CUDA), author's numbers](rtx-5090.md) | 2.69 | n/r | 4.88 | 14.3 | 16.0 | 16.1 | 19.4 | 19.5 | 20.8 | 38.2 | 28.8 | 29.2 | 34.5 | 49.1 | +| [RTX 3090 (CUDA)](rtx-3090.md) | 4.3 | n/r | 9.0 | 27.3 | 34.2 | 36.6 | 36.3 | 35.3 | 26.9 | 61.0 | 97.5 | 97.8 | 134.2 | 108.6 | +| [RTX 3060 (CUDA)](rtx-3060.md) | 8.3 | n/r | 21.6 | 57.1 | 74.7 | 86.8 | 78.1 | 82.8 | 59.1 | 125.1 | 228.2 | 229.9 | - | 230.8 | +| [RTX 5070 Laptop (CUDA)](rtx-5070-laptop.md) | 4.7 | n/r | 14.1 | 38.0 | 56.3 | 73.0 | 65.5 | 63.3 | 54.4 | 89.3 | 177.1 | 180.5 | - | 184.6 | +| [Radeon RX 9060 XT (ROCm)](rx-9060-xt.md) | 14.0 | n/r | 33.8 | 65.9 | 70.1 | 79.5 | 94.6 | 97.1 | N/A | 207.7 | 217.9 | 220.6 | 182.2 | 233.6 | +| [Jetson AGX Orin (CUDA)](jetson-agx-orin.md) | 14.3 | n/r | 28.0 | 60.6 | 79.1 | 96.7 | 101.2 | 93.0 | 76.8 | 164.3 | 269.0 | 266.6 | 297.4 | 298.5 | +| [Jetson Orin NX 16 GB (CUDA)](jetson-orin-nx.md) | 27.4 | n/r | 79.7 | 176.7 | 270.7 | 260.8 | 252.1 | 251.4 | 269.1 | 411.2 | 739.6 | 742.9 | - | 893.2 | +| [Jetson Orin Nano Super (CUDA)](jetson-orin-nano.md) | 31.9 | n/r | 89.3 | 193.9 | 297.5 | 293.4 | 272.3 | 271.7 | 335.0 | 462.8 | 845.2 | 849.7 | - | 1011.5 | +| [Apple M4 (Metal)](apple-m4.md) | 22.6 | n/r | 65.2 | 248.7 | 315.7 | 401.5 | 358.9 | 350.8 | N/A | 357.6 | 1141.1 | 1153.8 | 1650.7 | 829.3 | +| [Intel Arc B390 iGPU (SYCL)](core-ultra-x7-358h.md) | 15.5 | n/r | 25.0 | 102.1 | 62.6 | 101.2 | 106.1 | 178.4 | N/A | 61.7 | 269.3 | 275.2 | 434.9 | 270.4 | +| [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | n/r | 28.5 | 116.3 | 170.2 | 106.0 | 122.8 | 131.4 | N/A | 213.1 | 620.5 | 232.5 | 1262.5 | 566.0 | +| [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | n/r | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | +| [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | n/r | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 117.6 | 148.5 | 188.1 | 505.4 | 703.2 | 591.6 | 696.5 | 759.5 | N/A | 583.6 | 1114.0 | 1113.0 | 1720.3 | 2333.6 | +| [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | n/r | 23.5 | 90.7 | 126.7 | 134.0 | 161.1 | 144.9 | N/A | 161.3 | 438.3 | 430.2 | 405.8 | 256.3 | +| [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | n/r | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | +| [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | n/r | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | +| [Intel Core Ultra X7 358H (CPU)](core-ultra-x7-358h.md) | 48.0 | n/r | 199.0 | 1258.1 | 1645.4 | 1979.1 | 1697.3 | 1532.5 | N/A | 1758.0 | 6083.9 | 6120.3 | 9894.3 | 4025.8 | +| [Intel Core Ultra X7 358H (OpenVINO CPU)](core-ultra-x7-358h.md) | N/A | n/r | 238.4 | 1407.2 | 1831.2 | 2456.6 | 2300.4 | 2076.5 | N/A | 1899.5 | 7537.1 | 7522.7 | 11651.3 | 4626.5 | +| [Intel Core i5-12400F (CPU)](core-i5-12400f.md) | 78.1 | n/r | 392.7 | 1668.5 | 2165.7 | 2681.4 | 2216.8 | 1973.6 | N/A | 2288.4 | 8953.5 | 8637.9 | - | 5435.0 | +| [AMD Ryzen 5 5500 (CPU)](ryzen-5-5500.md) | 96.6 | n/r | 475.7 | 2133.0 | 2781.7 | 3447.0 | 2826.0 | 2518.8 | N/A | 3124.3 | 11321.3 | 11225.6 | - | 7347.2 | +| [Snapdragon X Oryon (CPU)](snapdragon-x-hexagon.md) | 109.5 | n/r | 548.9 | 1656.5 | 2013.1 | 2736.6 | 2226.0 | 1988.1 | N/A | 2080.9 | 9224.5 | 9140.6 | - | 5518.9 | Notes on the table: +- ACT runs its ALOHA checkpoint on one 640x480 view. Only the IQ-10 report + measures it; the other rows are `n/r`. - The RTX 5090 row comes from the PR #32 author and was not re-measured. Its numbers are a minimum over 5 alternating rounds, not the protocol the other reports use (3 warmups + 20 timed reps, best of 3 processes). From b23efd960d9620bbae8b61a06b2ff8bb1294d18d Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 17:44:32 +0700 Subject: [PATCH 31/38] vla_adapter: the LM's PV matmul on the HMX on Hexagon The LM's explicit attention multiplied F32 K and V on the NPU's vector units. On Hexagon V now goes to F16 (fa_kv()), so the PV matmul runs on the HMX. K stays F32: F16 scores put the action chunk 3.9e-3 from the CPU F32 reference on the Dragonwing IQ-10 (1.2e-2 served). Other backends keep F32 V. VLA-Adapter goes from 710 to 672 ms (mean, vla-bench, one view), 4.5e-4 from the CPU F32 reference on the fixed input and 3.9e-4 served. --- src/models/vla_adapter.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/models/vla_adapter.cpp b/src/models/vla_adapter.cpp index 25c65a13..b34a17ec 100644 --- a/src/models/vla_adapter.cpp +++ b/src/models/vla_adapter.cpp @@ -317,7 +317,10 @@ std::vector VlaAdapterModelArch::predict(const Inputs& in) { ggml_tensor*qh=ggml_reshape_3d(C,qp,lm_head_dim,n_q,LSEQ),*kh=ggml_reshape_3d(C,kp,lm_head_dim,n_kv,LSEQ),*vh=ggml_reshape_3d(C,vp,lm_head_dim,n_kv,LSEQ); ggml_tensor*qr=ggml_rope_ext(C,qh,t_pos,nullptr,(int)lm_head_dim,GGML_ROPE_TYPE_NEOX,0,lm_rope_base,1.0f,0.0f,1.0f,32.0f,1.0f); ggml_tensor*kr=ggml_rope_ext(C,kh,t_pos,nullptr,(int)lm_head_dim,GGML_ROPE_TYPE_NEOX,0,lm_rope_base,1.0f,0.0f,1.0f,32.0f,1.0f); - ggml_tensor*Q=ggml_cont(C,ggml_permute(C,qr,0,2,1,3)),*K=ggml_cont(C,ggml_permute(C,kr,0,2,1,3)),*V=ggml_cont(C,ggml_permute(C,vh,1,2,0,3)); + // fa_kv on V only: on Hexagon the PV matmul then runs on the HMX. K stays + // F32, since F16 scores put this LM 3.9e-3 from the CPU F32 reference on + // the IQ-10. Other backends keep F32 V. + ggml_tensor*Q=ggml_cont(C,ggml_permute(C,qr,0,2,1,3)),*K=ggml_cont(C,ggml_permute(C,kr,0,2,1,3)),*V=fa_kv(C,ggml_cont(C,ggml_permute(C,vh,1,2,0,3))); ggml_tensor*kq=ggml_mul_mat(C,K,Q); ggml_prec_set_acc(kq,GGML_PREC_F32); ggml_tensor*aw=ggml_soft_max_ext(C,kq,t_mask,lsc,0.0f); ggml_tensor*kqv=ggml_mul_mat(C,V,aw); From 0f684e23d18341169a248f7e922bd169fe3fa1bd Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Thu, 5 Nov 2026 18:03:06 +0700 Subject: [PATCH 32/38] octo: the block transformer's attention on the HMX on Hexagon The block transformer's explicit attention multiplied F32 K and V on the NPU's vector units. On Hexagon K and V now go to F16 (fa_kv()), so both matmuls run on the HMX; the sequence is already padded to a multiple of 32. Scores accumulate in F32 and the softmax stays F32. Other backends keep F32 K/V. On the Dragonwing IQ-10, Octo-Small goes from 120.1 to 86.9 ms (mean, vla-bench, two views), 5.0e-4 from the CPU F32 reference on the fixed input and 1.4e-4 served. --- src/models/octo.cpp | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/models/octo.cpp b/src/models/octo.cpp index 4774c722..4e6cf65d 100644 --- a/src/models/octo.cpp +++ b/src/models/octo.cpp @@ -1149,8 +1149,11 @@ bool run_transformer_graph(OctoRuntime& rt, ggml_tensor * n1 = layer_norm(C, x, blk_w[i][0], blk_w[i][1], ln_eps); ggml_tensor * qkv = linear(C, blk_w[i][2], blk_w[i][3], n1); ggml_tensor * Q = ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, pseq, kHidden, 3, 0), 0, 2, 1, 3)); - ggml_tensor * K = ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, pseq, kHidden, 3, 1), 0, 2, 1, 3)); - ggml_tensor * V = ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, pseq, kHidden, 3, 2), 1, 2, 0, 3)); + // fa_kv: on Hexagon K and V go to F16, so both matmuls run on the HMX + // rather than as F32 x F32 on the vector units. Scores accumulate in F32 + // and the softmax stays F32. Other backends keep F32 K/V. + ggml_tensor * K = fa_kv(C, ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, pseq, kHidden, 3, 1), 0, 2, 1, 3))); + ggml_tensor * V = fa_kv(C, ggml_cont(C, ggml_permute(C, head_view(C, qkv, head_dim, heads, pseq, kHidden, 3, 2), 1, 2, 0, 3))); ggml_tensor * merged = attention(C, Q, K, V, mask, attn_scale, kHidden, pseq); ggml_tensor * residual = ggml_add(C, x, linear(C, blk_w[i][4], blk_w[i][5], merged)); From a51c789c196e43ee62f9685ee631ae8bb9b2ab0c Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Fri, 6 Nov 2026 09:26:50 +0700 Subject: [PATCH 33/38] evo1: the DiT's cross-attention on the HMX on Hexagon Each of the 32 denoising steps scored 50 action queries against the 1025-row context as F32 x F32 matmuls on the NPU's vector units: head dim 112 and 1025 keys are not widths the HMX takes, 574 ms per predict on the Dragonwing IQ-10. On Hexagon the head dim and the keys are now padded with zeros to gemm_width() (128, 1056), the padded keys masked out, and K and V go to F16 once per predict. Scores accumulate in F32 and the softmax stays F32. Other backends build the same graph. Evo-1 goes from 2346 to 1660 ms (mean, vla-bench), 5.9e-4 from the CPU F32 reference on the fixed input and 2.1e-3 served (2.1e-3 before). --- src/models/evo1.cpp | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/src/models/evo1.cpp b/src/models/evo1.cpp index ce31ffab..547927ea 100644 --- a/src/models/evo1.cpp +++ b/src/models/evo1.cpp @@ -85,6 +85,7 @@ struct Evo1ModelArch : public ModelArchBase { struct MainIO { ggml_tensor *t_embeds=nullptr,*t_pos=nullptr,*t_lmmask=nullptr,*t_qmask=nullptr; ggml_tensor *t_state=nullptr,*t_x=nullptr,*t_amask=nullptr,*x_action=nullptr; + ggml_tensor *t_dmask=nullptr; std::vector lm_ok; }; graph_cache main_graph; @@ -870,6 +871,14 @@ std::vector Evo1ModelArch::predict(const Inputs& in) { const int64_t E = embed_dim, hd_dit = E/dit_heads; const float scale_dit = 1.0f/std::sqrt((float) hd_dit); const int64_t Nctx = SEQ+1; + // On Hexagon the DiT's cross-attention pads its head dim and its keys with + // zeros to gemm_width(), the keys masked out, and takes F16 K/V: both + // matmuls then run on the HMX instead of as F32 x F32 on the vector units. + const int64_t HP = gemm_width(hd_dit), NK = gemm_width(Nctx); + ggml_tensor * t_dmask = nullptr; + if (NK > Nctx) { + t_dmask = ggml_new_tensor_2d(C, GGML_TYPE_F32, NK, horizon); ggml_set_input(t_dmask); ggml_set_output(t_dmask); + } ggml_tensor * t_embeds = ggml_new_tensor_2d(C, GGML_TYPE_F32, lm_hidden, SEQ); ggml_set_input(t_embeds); ggml_tensor * t_pos = ggml_new_tensor_1d(C, GGML_TYPE_I32, SEQ); ggml_set_input(t_pos); @@ -935,6 +944,12 @@ std::vector Evo1ModelArch::predict(const Inputs& in) { } else { dc[i].K = ggml_cont(C, ggml_permute(C, ggml_reshape_3d(C, kp, hd_dit, dit_heads, Nctx), 0, 2, 1, 3)); dc[i].V = ggml_cont(C, ggml_permute(C, ggml_reshape_3d(C, vp, hd_dit, dit_heads, Nctx), 1, 2, 0, 3)); + if (HP > hd_dit || NK > Nctx) { + dc[i].K = ggml_pad(C, dc[i].K, (int) (HP-hd_dit), (int) (NK-Nctx), 0, 0); + dc[i].V = ggml_pad(C, dc[i].V, (int) (NK-Nctx), (int) (HP-hd_dit), 0, 0); + } + dc[i].K = fa_kv(C, dc[i].K); + dc[i].V = fa_kv(C, dc[i].V); } } @@ -1003,11 +1018,16 @@ std::vector Evo1ModelArch::predict(const Inputs& in) { } else { ggml_tensor * qp = as_type(C, ggml_add(C, mm_act(C, c.Wq, x_q, at), c.bq), GGML_TYPE_F32); ggml_tensor * Q = ggml_cont(C, ggml_permute(C, ggml_reshape_3d(C, qp, hd_dit, dit_heads, horizon), 0, 2, 1, 3)); + if (HP > hd_dit) + Q = ggml_pad(C, Q, (int) (HP-hd_dit), 0, 0, 0); ggml_tensor * kq = ggml_mul_mat(C, c.K, Q); ggml_prec_set_acc(kq, GGML_PREC_F32); // The Evo-1 reference cross-attends over the full padded context - // (no key mask), so the action queries see every LM position. - ggml_tensor * aw = ggml_soft_max_ext(C, kq, nullptr, scale_dit, 0.0f); + // (no key mask), so the action queries see every LM position; + // t_dmask hides only the keys padded on Hexagon. + ggml_tensor * aw = ggml_soft_max_ext(C, kq, t_dmask, scale_dit, 0.0f); ggml_tensor * kqv = ggml_mul_mat(C, c.V, aw); + if (HP > hd_dit) + kqv = ggml_view_3d(C, kqv, hd_dit, horizon, dit_heads, kqv->nb[1], kqv->nb[2], 0); att_pre = ggml_reshape_2d(C, ggml_cont(C, ggml_permute(C, kqv, 0, 2, 1, 3)), E, horizon); } ggml_tensor * attn_out = ggml_add(C, mm_act(C, w.Wo, as_type(C, att_pre, at), at), w.bo); @@ -1040,7 +1060,7 @@ std::vector Evo1ModelArch::predict(const Inputs& in) { ggml_set_output(x_action); gio.t_embeds=t_embeds; gio.t_pos=t_pos; gio.t_lmmask=t_lmmask; gio.t_qmask=t_qmask; - gio.t_state=t_state; gio.t_x=t_x; gio.t_amask=t_amask; gio.x_action=x_action; + gio.t_state=t_state; gio.t_x=t_x; gio.t_amask=t_amask; gio.x_action=x_action; gio.t_dmask=t_dmask; ggml_cgraph * gf = ggml_new_graph_custom(C, main_nodes, false); ggml_build_forward_expand(gf, x_action); @@ -1067,6 +1087,14 @@ std::vector Evo1ModelArch::predict(const Inputs& in) { ggml_backend_tensor_set(t_amask, am.data(), 0, ggml_nbytes(t_amask)); } { std::vector qm(SEQ, 0.0f); for (int64_t p=0; pne[0], rows = gio.t_dmask->ne[1], Nctx = SEQ+1; + std::vector dm((size_t) (NK*rows), 0.0f); + for (int64_t r=0; r::infinity(); + ggml_backend_tensor_set(gio.t_dmask, dm.data(), 0, ggml_nbytes(gio.t_dmask)); + } gio.lm_ok = attn_ok; } ggml_backend_tensor_set(t_state, state_norm.data(), 0, ggml_nbytes(t_state)); From f0a670dacc8df2b8d5e9c00014cd41edd81d0972 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Fri, 6 Nov 2026 10:32:34 +0700 Subject: [PATCH 34/38] dit_head: the DiT's attention on the HMX on Hexagon GR00T's DiT attention multiplied F32 K and V on the NPU's vector units: head dim 48 and 41-272 keys are not widths the HMX takes. On Hexagon hmx_kv() pads the head dim and the keys with zeros to gemm_width() and casts K and V to F16; the queries take the head-dim padding, the padded keys are masked out (one mask per shape per graph) and the output drops the padded columns. DitCfg::cross_kv_f16 keeps cross-attention K/V F32 where they overflow F16: VLA-JEPA's give NaN actions otherwise. Other backends build the same graph. On the Dragonwing IQ-10 (mean, vla-bench): GR00T N1.5 598 to 525 ms, N1.6 700 to 678 ms, N1.7 756 to 746 ms, VLA-JEPA 515 to 499 ms; against a CPU-only F32 build at the same commit, 1.9e-3, 1.3e-3, 2.0e-3 and 2.1e-3 on fixed inputs, and N1.7 2.0e-3, VLA-JEPA 2.2e-3 served. --- src/models/vla_jepa.cpp | 3 +++ src/modules/dit_head.cpp | 57 +++++++++++++++++++++++++++++++++++++--- src/modules/dit_head.h | 9 ++++++- 3 files changed, 64 insertions(+), 5 deletions(-) diff --git a/src/models/vla_jepa.cpp b/src/models/vla_jepa.cpp index ff42d56d..f2c20073 100644 --- a/src/models/vla_jepa.cpp +++ b/src/models/vla_jepa.cpp @@ -148,6 +148,9 @@ bool load_config(const gguf_reader & g, const Options & opts, VlaJepaModelArch & m.dit.cfg.layers = m.dit_layers; m.dit.cfg.ln_eps = m.dit_ln_eps; m.dit.cfg.norm_out_eps = m.dit_norm_out_eps; + // Its cross-attention K/V overflow F16 on the IQ-10 (NaN actions); they + // stay F32 there, and only the self-attention takes the HMX. + m.dit.cfg.cross_kv_f16 = false; cfg = Config{}; cfg.n_img = m.vit.n_tokens(); diff --git a/src/modules/dit_head.cpp b/src/modules/dit_head.cpp index bb4250ad..29de7d33 100644 --- a/src/modules/dit_head.cpp +++ b/src/modules/dit_head.cpp @@ -152,8 +152,25 @@ void DitHead::declare(WeightLoader & L, const char * prefix, bool fuse_qkv, bool po2b = L.f32 ("%s.proj_out2.bias", outer); } +namespace { + +// K [hd, T, heads] and V^T [T, hd, heads] in F16, the head dim and the keys +// zero-padded to gemm_width(), as the HMX takes them (Hexagon). Elsewhere both +// come back unchanged. +void hmx_kv(ggml_context * C, ggml_tensor ** K, ggml_tensor ** V) { + const int64_t hd = (*K)->ne[0], T = (*K)->ne[1], HP = gemm_width(hd), NK = gemm_width(T); + if (HP > hd || NK > T) { + *K = ggml_pad(C, *K, (int) (HP-hd), (int) (NK-T), 0, 0); + *V = ggml_pad(C, *V, (int) (NK-T), (int) (HP-hd), 0, 0); + } + *K = fa_kv(C, *K); + *V = fa_kv(C, *V); +} + +} + void DitHead::kv(ggml_context * C, const DitLayerW & w, ggml_tensor * src, - ggml_tensor ** K_out, ggml_tensor ** V_out, ggml_tensor * xq_pre) const { + ggml_tensor ** K_out, ggml_tensor ** V_out, ggml_tensor * xq_pre, bool cross) const { const int64_t hd = cfg.head_dim; const int64_t heads = cfg.heads; const int64_t Tkv = src->ne[1]; @@ -187,6 +204,8 @@ void DitHead::kv(ggml_context * C, const DitLayerW & w, ggml_tensor * src, } *K_out = ggml_cont(C, ggml_permute(C, head_view(C, kvp, hd, heads, Tkv, cfg.hidden, 2, 0), 0, 2, 1, 3)); *V_out = ggml_cont(C, ggml_permute(C, head_view(C, kvp, hd, heads, Tkv, cfg.hidden, 2, 1), 1, 2, 0, 3)); + if (!cfg.fused_ops && (!cross || cfg.cross_kv_f16)) + hmx_kv(C, K_out, V_out); return; } if (w.fq_k) { @@ -198,6 +217,20 @@ void DitHead::kv(ggml_context * C, const DitLayerW & w, ggml_tensor * src, } *K_out = to_heads (C, linear(C, w.Wk, w.bk, src), hd, heads, Tkv); *V_out = to_heads_v(C, linear(C, w.Wv, w.bv, src), hd, heads, Tkv); + if (!cfg.fused_ops && (!cross || cfg.cross_kv_f16)) + hmx_kv(C, K_out, V_out); +} + +ggml_tensor * DitHead::key_mask(ggml_context * C, int64_t NK, int64_t Nk, int64_t Tq) const { + char name[GGML_MAX_NAME]; + std::snprintf(name, sizeof(name), "dit.key_mask.%lld.%lld.%lld", (long long) NK, (long long) Nk, (long long) Tq); + if (ggml_tensor * m = ggml_get_tensor(C, name)) + return m; + // 0 for the Nk real keys, -1e30 for the padding: step(i - Nk + 0.5) * -1e30. + ggml_tensor * m = ggml_step(C, ggml_scale_bias(C, ggml_arange(C, 0.0f, (float) NK, 1.0f), 1.0f, 0.5f - (float) Nk)); + m = ggml_repeat_4d(C, ggml_scale(C, m, -1e30f), NK, Tq, 1, 1); + ggml_set_name(m, name); + return m; } ggml_tensor * DitHead::block(ggml_context * C, const DitLayerW & w, ggml_tensor * h, ggml_tensor * mod, @@ -278,6 +311,8 @@ ggml_tensor * DitHead::block(ggml_context * C, const DitLayerW & w, ggml_tensor Q = ggml_cont(C, ggml_permute(C, head_view(C, qkv, hd, heads, Tk, dim, 3, 0), 0, 2, 1, 3)); K = ggml_cont(C, ggml_permute(C, head_view(C, qkv, hd, heads, Tk, dim, 3, 1), 0, 2, 1, 3)); V = ggml_cont(C, ggml_permute(C, head_view(C, qkv, hd, heads, Tk, dim, 3, 2), 1, 2, 0, 3)); + if (!cfg.fused_ops) + hmx_kv(C, &K, &V); } } else { // Self-attention with separate FoldQuant q/k/v: one activation blob of n @@ -294,7 +329,7 @@ ggml_tensor * DitHead::block(ggml_context * C, const DitLayerW & w, ggml_tensor V = V_pre; } else { - kv(C, w, enc ? enc : n, &K, &V, enc ? nullptr : xq); + kv(C, w, enc ? enc : n, &K, &V, enc ? nullptr : xq, enc != nullptr); } } @@ -310,8 +345,22 @@ ggml_tensor * DitHead::block(ggml_context * C, const DitLayerW & w, ggml_tensor ggml_tensor * g = fused_bias_gelu(C, mm_exact(C, w.Wff0, n3), w.bff0, bf16_operand(w.Wff2, Tk)); return fused_bias_res(C, mm_exact(C, w.Wff2, g), w.bff2, h1); } - ggml_tensor * att = exact ? fused_attn_exact(C, Q, K, V, scale, false) - : attention(C, Q, K, V, nullptr, scale, dim, Tk); + ggml_tensor * att; + const int64_t Nk = enc ? enc->ne[1] : Tk; + if (!exact && (K->ne[0] > hd || K->ne[1] > Nk)) { + // hmx_kv padded K/V: the queries take the head-dim padding too, the + // padded keys are masked out and the output drops the padded columns. + if (K->ne[0] > hd) + Q = ggml_pad(C, Q, (int) (K->ne[0]-hd), 0, 0, 0); + ggml_tensor * kq = ggml_mul_mat(C, K, Q); ggml_prec_set_acc(kq, GGML_PREC_F32); + ggml_tensor * mask = K->ne[1] > Nk ? key_mask(C, K->ne[1], Nk, Tk) : nullptr; + ggml_tensor * kqv = ggml_mul_mat(C, V, ggml_soft_max_ext(C, kq, mask, scale, 0.0f)); + kqv = ggml_view_3d(C, kqv, hd, Tk, heads, kqv->nb[1], kqv->nb[2], 0); + att = ggml_reshape_2d(C, ggml_cont(C, ggml_permute(C, kqv, 0, 2, 1, 3)), dim, Tk); + } else { + att = exact ? fused_attn_exact(C, Q, K, V, scale, false) + : attention(C, Q, K, V, nullptr, scale, dim, Tk); + } // FoldQuant: both residual adds ride in the GEMM epilogues. ggml_tensor * h1 = w.fq_o ? fq_linear(C, w.fq_o, att, h) : ggml_add(C, h, linear(C, w.Wo, w.bo, att)); ggml_tensor * n3 = cfg.fused_ops ? fused_ln(C, h1, nullptr, nullptr, cfg.ln_eps, false) : ggml_norm(C, h1, cfg.ln_eps); diff --git a/src/modules/dit_head.h b/src/modules/dit_head.h index 5eb573ed..cff7ff5c 100644 --- a/src/modules/dit_head.h +++ b/src/modules/dit_head.h @@ -50,6 +50,9 @@ struct DitCfg { float ln_eps = 1e-5f; float norm_out_eps = 1e-6f; bool fused_ops = false; // layers/fused_ops.h nodes (CUDA only) + // Cross-attention K/V in F16 where the HMX wants it (Hexagon). Off for a + // conditioning whose projections overflow F16. + bool cross_kv_f16 = true; }; struct DitHead { @@ -72,14 +75,18 @@ struct DitHead { const char * outer = nullptr, const FqModuleSpec * fq = nullptr); // xq_pre: an activation blob already computed from src (shared with q). + // cross: src is the conditioning rather than the block's own tokens. void kv(ggml_context * C, const DitLayerW & w, ggml_tensor * src, - ggml_tensor ** K_out, ggml_tensor ** V_out, ggml_tensor * xq_pre = nullptr) const; + ggml_tensor ** K_out, ggml_tensor ** V_out, ggml_tensor * xq_pre = nullptr, bool cross = true) const; ggml_tensor * block(ggml_context * C, const DitLayerW & w, ggml_tensor * h, ggml_tensor * mod, ggml_tensor * enc, ggml_tensor * K_pre = nullptr, ggml_tensor * V_pre = nullptr) const; ggml_tensor * time_emb(ggml_context * C, ggml_tensor * tproj) const; + // [NK, Tq] additive mask hiding keys Nk.. of NK, made once per graph (by name). + ggml_tensor * key_mask(ggml_context * C, int64_t NK, int64_t Nk, int64_t Tq) const; + // (shift, scale), opposite to the blocks. ggml_tensor * proj_out(ggml_context * C, ggml_tensor * h, ggml_tensor * mod) const; }; From bc10b41e8d7731f21d0a8f3d393e4912e7757fb9 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Fri, 6 Nov 2026 15:01:54 +0700 Subject: [PATCH 35/38] docs: explicit attention on the HMX, CPU-only reference, IQ-10 benchmark at bd166b8 hexagon.md: OpenVLA-OFT's LM, Octo's block transformer, Evo-1's DiT cross-attention and GR00T's DiT take F16 K/V with the head dim and keys padded to gemm_width(); VLA-Adapter's LM only for V, VLA-JEPA's cross-attention stays F32. Fidelity and serving are re-measured against a CPU-only build of the same commit, since a Hexagon build's CPU run carries the NPU-only changes. qualcomm-iq10.md and the summary row at bd166b8: OpenVLA-OFT 1725.5 to 1120.2 ms, Evo-1 2348.3 to 1661.4 ms, Octo-Small 119.4 to 84.6 ms, GR00T N1.5 594.9 to 532.0 ms. --- CHANGELOG.md | 9 +++- docs/backend/hexagon.md | 72 ++++++++++++++++------------ docs/benchmark/README.md | 4 +- docs/benchmark/qualcomm-iq10.md | 85 +++++++++++++++++---------------- 4 files changed, 95 insertions(+), 75 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 48f547e9..94a58861 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,8 +35,13 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke reference (1.0e-3 served): pads that read a weight, also answered `NO_SUPPORT`, run on the CPU, and its ResNet's im2col matrices are zero-padded to a multiple of 32 so the convs' matmuls take the HMX (152.9 ms, - 170.8 ms on the CPU). GR00T N1.5 and N1.6 are above the bar on a served LIBERO - observation, as the CPU is at F16. See + 170.8 ms on the CPU). Explicit attention takes F16 K/V with its head dim and + keys zero-padded to a multiple of 32, the padding masked out, so both matmuls + run on the HMX: OpenVLA-OFT's LM, Octo's block transformer, Evo-1's DiT + cross-attention and GR00T's DiT (OpenVLA-OFT 35% faster, Evo-1 29%, Octo + 29%, GR00T N1.5 11%); VLA-Adapter's LM only for V. Fidelity is measured + against a CPU-only build of the same commit. GR00T N1.5 and N1.6 are above + the bar on a served LIBERO observation, as the CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install and the numbers. - **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200` diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 14a4abd8..9d61d75b 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -196,6 +196,16 @@ Hexagon builds: 311 ms. - **Evo-1's LM attention multiplied F32 K and V against 14 query heads.** K and V now go to F16 (`fa_kv()`); the scores still accumulate in F32. +- **Explicit attention elsewhere multiplied F32 K and V too**, and its key + counts and head dims are often not multiples of 32. OpenVLA-OFT's LM (330 + keys), Octo's block transformer, Evo-1's DiT cross-attention (head dim 112, + 1025 keys) and GR00T's DiT (head dim 48) now take F16 K/V with the head dim + and the keys zero-padded to `gemm_width()` and the padded keys masked out + (`hmx_kv()` in `dit_head.cpp`): OpenVLA-OFT 1725 to 1120 ms, Evo-1 2348 to + 1661 ms, Octo-Small 119 to 85 ms, GR00T N1.5 595 to 532 ms. Two LMs keep part + of it in F32: in VLA-Adapter's LM only V goes to F16, since F16 scores put the + chunk 3.9e-3 off, and VLA-JEPA's cross-attention K/V overflow F16 (NaN + actions), so only its DiT self-attention takes the HMX. - **ACT's ResNet convs are im2col and one matmul** with the patches as its first operand. The stem's K of 147 and layer3's and layer4's 1200 and 300 output positions (640x480) are not multiples of 32: 80 ms of the 101 ms spent @@ -206,10 +216,10 @@ Hexagon builds: benchmark's protocol ACT takes 152.9 ms on the NPU against 170.8 ms on the CPU. -What is left on the vector units: GR00T's DiT attention (head dim 48) and Evo-1's -DiT cross-attention (head dim 112, 1025 keys), whose shapes the HMX does not -take, and Evo-1's InternViT attention, which stays on the CPU because F16 K/V -there measured 3.9e-2. +What is left on the vector units: VLA-Adapter's QK scores and VLA-JEPA's LM and +DiT cross-attention scores, which stay F32 for the reasons above, and Evo-1's +InternViT attention, which stays on the CPU because F16 K/V there measured +3.9e-2. ## What changed in `vla.cpp` @@ -355,30 +365,31 @@ export LD_LIBRARY_PATH=$PWD/pkg/lib ### Fidelity -`vla_predict_check` at the Hexagon defaults, against the same binary with -`VLA_DEVICE=cpu --weight-dtype f32 --flash-attn 0`. A Hexagon build turns flash -attention on by default even on the CPU, so the reference asks for the explicit -path that other builds run. Views are the ones the LIBERO client sends, so they +`vla_predict_check` at the Hexagon defaults, against a CPU-only build +(`-DGGML_HEXAGON=OFF`) of the same commit with `--weight-dtype f32 +--flash-attn 0`. The Hexagon build's own CPU run is not a reference: what it +changes for the NPU (F16 K/V, padding to `gemm_width()`) is compiled in and runs +on the CPU too. Views are the ones the LIBERO client sends, so they differ from the benchmark's for VLA-JEPA, VLA-Adapter, OpenVLA-OFT and Octo: | Model | Views | max\|Δ\| | |---|--:|--:| -| SmolVLA | 2 | 7.2e-4 | -| π0 | 2 | 1.2e-3 | +| SmolVLA | 2 | 7.4e-4 | +| π0 | 2 | 7.7e-4 | | π0.5 | 2 | 4.8e-4 | -| Evo-1 | 1 | 5.5e-4 | -| GR00T N1.5 | 1 | 2.4e-3 | -| GR00T N1.6 | 1 | 1.4e-3 | +| Evo-1 | 1 | 6.2e-4 | +| GR00T N1.5 | 1 | 1.9e-3 | +| GR00T N1.6 | 1 | 1.3e-3 | | GR00T N1.7 | 1 | 2.0e-3 | -| VLA-JEPA | 2 | 2.2e-3 | -| VLA-Adapter | 2 | 5.4e-4 | -| OpenVLA-OFT | 2 | 3.2e-4 | -| Octo-Small | 1 | 5.3e-4 | +| VLA-JEPA | 2 | 2.1e-3 | +| VLA-Adapter | 2 | 3.9e-4 | +| OpenVLA-OFT | 2 | 3.1e-4 | +| Octo-Small | 1 | 5.0e-4 | | TurboVLA | 2 | 3.7e-4 | | ACT | 1 | 6.1e-4 | -GR00T N1.6 with two views is 8.8e-4. At the benchmark's single view VLA-JEPA is -1.9e-3, VLA-Adapter 5.7e-4 and OpenVLA-OFT 2.1e-4. ACT takes its one 640x480 +GR00T N1.6 with two views is 7.5e-4. At the benchmark's single view VLA-JEPA is +1.8e-3, VLA-Adapter 4.8e-4 and OpenVLA-OFT 1.8e-4. ACT takes its one 640x480 view, the benchmark's input. ### Serving @@ -387,7 +398,8 @@ view, the benchmark's input. `eval/client/vla_cpp_client.py` and the adapters `run_sim_client_direct.py` uses, on one fixed LIBERO observation (two views, the task prompt, a fixed robot state), with fixed noise for the archs that draw it. The same request goes to a -Hexagon server and to a CPU server with `--weight-dtype f32 --flash-attn 0`. +Hexagon server and to a server from the CPU-only build with `--weight-dtype f32 +--flash-attn 0`. Every model returns a finite chunk, the same chunk for a repeated request, an error reply to a malformed or an invalid request, and keeps serving afterwards. ACT has no `eval/client` preset; it gets the same checks from raw `vla.proto` @@ -395,23 +407,23 @@ requests carrying one fixed 640x480 view and a fixed 14-dim state. | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 | |---|--:|--:| -| SmolVLA | 1.3e-3 | | -| π0.5 | 3.5e-4 | | +| SmolVLA | 1.9e-3 | | +| π0.5 | 3.7e-4 | | | Evo-1 | 1.1e-3 | | -| GR00T N1.5 | 1.1e-2 | 4.3e-2 | -| GR00T N1.6 | 1.3e-2 | 1.1e-2 | -| GR00T N1.7 | 2.5e-3 | | +| GR00T N1.5 | 1.2e-2 | 2.8e-2 | +| GR00T N1.6 | 1.0e-2 | 9.0e-3 | +| GR00T N1.7 | 2.0e-3 | | | π0 | 5.8e-4 | | -| VLA-JEPA | 2.6e-3 | 2.1e-2 | -| VLA-Adapter | 5.4e-4 | | -| OpenVLA-OFT | 5.9e-4 | | -| Octo-Small | 2.4e-4 | | +| VLA-JEPA | 2.2e-3 | | +| VLA-Adapter | 3.9e-4 | | +| OpenVLA-OFT | 7.5e-4 | | +| Octo-Small | 1.4e-4 | | | TurboVLA | 8.5e-4 | | | ACT | 1.0e-3 | | GR00T N1.5 and N1.6 are above the bar on this input, and so is the CPU itself with F16 weights: N1.5 lands closer to the reference than the CPU at F16, N1.6 -slightly further (1.3e-2 against 1.1e-2). They are sensitive to 16-bit precision +slightly further (1.0e-2 against 9.0e-3). They are sensitive to 16-bit precision rather than miscomputed. Running them with F32 weights on the NPU has not been tried yet. diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index 50c184ea..50791593 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`7c76ea9`, its ACT cell at `0f3e010` (same llama.cpp pin). Each report +`bd166b8` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | n/r | 28.5 | 116.3 | 170.2 | 106.0 | 122.8 | 131.4 | N/A | 213.1 | 620.5 | 232.5 | 1262.5 | 566.0 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | n/r | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | n/r | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 117.6 | 148.5 | 188.1 | 505.4 | 703.2 | 591.6 | 696.5 | 759.5 | N/A | 583.6 | 1114.0 | 1113.0 | 1720.3 | 2333.6 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 81.7 | 147.8 | 184.6 | 489.7 | 664.0 | 530.2 | 680.9 | 739.6 | N/A | 583.3 | 1117.6 | 1114.0 | 1117.7 | 1653.5 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | n/r | 23.5 | 90.7 | 126.7 | 134.0 | 161.1 | 144.9 | N/A | 161.3 | 438.3 | 430.2 | 405.8 | 256.3 | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | n/r | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | n/r | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 39d1ef14..54231994 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `7c76ea9`; ACT at `0f3e010` | +| Commit | `bd166b8` | | llama.cpp | `b11223` | -| Date | 2026-11-05 14:26 to 14:57 (UTC+07); ACT 17:00 to 17:01 | +| Date | 2026-11-06 10:46 to 11:16 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -69,19 +69,19 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| Octo-Small | 2 | 256 + 128 | 117.6 | 119.4 | 118.4 | 120.7 | 21.2 | 253 | 723 | -| ACT | 1 | 640 x 480 | 148.5 | 152.9 | 152.5 | 157.3 | - | 194 | 467 | -| TurboVLA | 2 | 256 | 188.1 | 190.8 | 188.6 | 194.9 | - | 249 | 1075 | -| VLA-JEPA | 1 | 256 | 505.4 | 512.6 | 510.1 | 522.2 | 196.4 | 184 | 3969 | -| SmolVLA | 2 | 512 | 583.6 | 584.4 | 584.4 | 585.1 | 229.9 | 145 | 1196 | -| GR00T N1.5 | 1 | 224 | 591.6 | 594.9 | 593.4 | 597.1 | 138.9 | 162 | 3639 | -| GR00T N1.6 | 1 | 224 | 696.5 | 699.3 | 699.2 | 702.2 | 145.1 | 147 | 4692 | -| VLA-Adapter | 1 | 224 | 703.2 | 708.4 | 708.2 | 715.3 | 258.1 | 872 | 3566 | -| GR00T N1.7 | 1 | 256 | 759.5 | 766.3 | 766.0 | 773.4 | 197.6 | 191 | 5113 | -| π0 | 2 | 224 | 1114.0 | 1116.6 | 1116.4 | 1118.2 | 277.1 | 284 | 5537 | -| π0.5 | 2 | 224 | 1113.0 | 1117.3 | 1116.6 | 1120.7 | 277.9 | 285 | 5826 | -| OpenVLA-OFT | 1 | 224 | 1720.3 | 1725.5 | 1724.7 | 1731.0 | 263.4 | 844 | 15451 | -| Evo-1 | 1 | 448 | 2333.6 | 2348.3 | 2349.9 | 2360.1 | 820.6 | 319 | 1757 | +| Octo-Small | 2 | 256 + 128 | 81.7 | 84.6 | 82.3 | 89.6 | 21.9 | 253 | 728 | +| ACT | 1 | 640 x 480 | 147.8 | 151.2 | 150.7 | 153.6 | - | 193 | 471 | +| TurboVLA | 2 | 256 | 184.6 | 191.8 | 190.3 | 199.4 | - | 249 | 1118 | +| VLA-JEPA | 1 | 256 | 489.7 | 495.8 | 494.7 | 501.6 | 196.4 | 184 | 4023 | +| GR00T N1.5 | 1 | 224 | 530.2 | 532.0 | 531.3 | 534.3 | 138.5 | 162 | 3734 | +| SmolVLA | 2 | 512 | 583.3 | 584.0 | 583.9 | 584.7 | 229.8 | 146 | 1194 | +| VLA-Adapter | 1 | 224 | 664.0 | 668.6 | 666.6 | 673.7 | 257.8 | 872 | 3584 | +| GR00T N1.6 | 1 | 224 | 680.9 | 682.1 | 681.6 | 683.9 | 145.3 | 146 | 4889 | +| GR00T N1.7 | 1 | 256 | 739.6 | 746.4 | 745.1 | 753.8 | 196.4 | 190 | 5215 | +| π0.5 | 2 | 224 | 1114.0 | 1116.1 | 1115.5 | 1117.8 | 278.0 | 285 | 5802 | +| OpenVLA-OFT | 1 | 224 | 1117.7 | 1120.2 | 1120.4 | 1121.8 | 262.3 | 844 | 15321 | +| π0 | 2 | 224 | 1117.6 | 1120.5 | 1119.9 | 1123.8 | 277.8 | 284 | 5547 | +| Evo-1 | 1 | 448 | 1653.5 | 1661.4 | 1661.2 | 1665.6 | 815.6 | 319 | 1556 | ### Fastest configuration @@ -108,37 +108,40 @@ which the defaults leave off because it is outside the fidelity bar (VLA-JEPA | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| Octo-Small | *(defaults)* | 117.6 | 119.4 | 118.4 | 120.7 | 21.2 | 253 | 723 | - | -| ACT | *(defaults)* | 148.5 | 152.9 | 152.5 | 157.3 | - | 194 | 467 | - | -| TurboVLA | *(defaults)* | 188.1 | 190.8 | 188.6 | 194.9 | - | 249 | 1075 | - | -| VLA-JEPA | `--flash-attn --mm-prec default` | 442.1 | 452.4 | 450.2 | 460.6 | 198.3 | 184 | 3985 | -11.7% | -| SmolVLA | *(defaults)* | 583.6 | 584.4 | 584.4 | 585.1 | 229.9 | 145 | 1196 | - | -| GR00T N1.5 | *(defaults)* | 591.6 | 594.9 | 593.4 | 597.1 | 138.9 | 162 | 3639 | - | -| GR00T N1.6 | *(defaults)* | 696.5 | 699.3 | 699.2 | 702.2 | 145.1 | 147 | 4692 | - | -| VLA-Adapter | *(defaults)* | 703.2 | 708.4 | 708.2 | 715.3 | 258.1 | 872 | 3566 | - | -| GR00T N1.7 | `--flash-attn --mm-prec default` | 727.8 | 734.1 | 733.2 | 742.4 | 196.2 | 191 | 5081 | -4.2% | -| π0 | *(defaults)* | 1114.0 | 1116.6 | 1116.4 | 1118.2 | 277.1 | 284 | 5537 | - | -| π0.5 | *(defaults)* | 1113.0 | 1117.3 | 1116.6 | 1120.7 | 277.9 | 285 | 5826 | - | -| OpenVLA-OFT | *(defaults)* | 1720.3 | 1725.5 | 1724.7 | 1731.0 | 263.4 | 844 | 15451 | - | -| Evo-1 | *(defaults)* | 2333.6 | 2348.3 | 2349.9 | 2360.1 | 820.6 | 319 | 1757 | - | +| Octo-Small | *(defaults)* | 81.7 | 84.6 | 82.3 | 89.6 | 21.9 | 253 | 728 | - | +| ACT | *(defaults)* | 147.8 | 151.2 | 150.7 | 153.6 | - | 193 | 471 | - | +| TurboVLA | *(defaults)* | 184.6 | 191.8 | 190.3 | 199.4 | - | 249 | 1118 | - | +| VLA-JEPA | `--flash-attn` | 427.8 | 434.2 | 432.6 | 441.7 | 196.1 | 184 | 3971 | -12.4% | +| GR00T N1.5 | *(defaults)* | 530.2 | 532.0 | 531.3 | 534.3 | 138.5 | 162 | 3734 | - | +| SmolVLA | *(defaults)* | 583.3 | 584.0 | 583.9 | 584.7 | 229.8 | 146 | 1194 | - | +| VLA-Adapter | *(defaults)* | 664.0 | 668.6 | 666.6 | 673.7 | 257.8 | 872 | 3584 | - | +| GR00T N1.6 | *(defaults)* | 680.9 | 682.1 | 681.6 | 683.9 | 145.3 | 146 | 4889 | - | +| GR00T N1.7 | `--flash-attn --mm-prec default` | 715.1 | 719.2 | 718.7 | 722.9 | 195.5 | 190 | 5074 | -3.6% | +| π0.5 | *(defaults)* | 1114.0 | 1116.1 | 1115.5 | 1117.8 | 278.0 | 285 | 5802 | - | +| OpenVLA-OFT | *(defaults)* | 1117.7 | 1120.2 | 1120.4 | 1121.8 | 262.3 | 844 | 15321 | - | +| π0 | *(defaults)* | 1117.6 | 1120.5 | 1119.9 | 1123.8 | 277.8 | 284 | 5547 | - | +| Evo-1 | *(defaults)* | 1653.5 | 1661.4 | 1661.2 | 1665.6 | 815.6 | 319 | 1556 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| Octo-Small | 121.8 | 120.4 | 121.8 | 123.2 | 123.8 | -| ACT | 143.4 | 153.4 | 157.5 | 155.8 | 174.8 | -| TurboVLA | 196.5 | 189.3 | 192.3 | 190.9 | 257.4 | -| VLA-JEPA | 514.8 | 454.3 | 512.0 | 452.9 | 1662.4 | -| SmolVLA | 584.6 | 585.0 | 585.2 | 584.6 | 2147.2 | -| GR00T N1.5 | 594.7 | 592.6 | 600.1 | 601.0 | 877.2 | -| GR00T N1.6 | 709.6 | 714.5 | 711.1 | 713.1 | 875.3 | -| VLA-Adapter | 712.4 | 711.1 | 713.7 | 712.8 | 989.2 | -| GR00T N1.7 | 765.5 | 739.0 | 758.3 | 733.1 | 865.3 | -| π0 | 1131.6 | 1138.1 | 1118.0 | 1128.8 | 2051.6 | -| π0.5 | 1125.8 | 1109.1 | 1134.2 | 1134.6 | 2036.0 | -| OpenVLA-OFT | 1737.9 | 1722.4 | 1715.7 | 1718.5 | 2005.8 | -| Evo-1 | 2345.2 | 2352.4 | 2355.4 | 2361.4 | 3877.1 | +| Octo-Small | 90.8* | 83.0 | 83.2 | 85.3 | 88.1 | +| ACT | 154.4 | 158.4 | 154.7 | 153.2 | 179.9 | +| TurboVLA | 192.0 | 192.6 | 194.0 | 191.9 | 257.3 | +| VLA-JEPA | 498.2 | 431.3 | 494.0 | 439.8 | 602.4 | +| GR00T N1.5 | 534.0 | 526.4 | 527.0 | 526.4 | 811.6 | +| SmolVLA | 584.4 | 584.7 | 585.2 | 585.4 | 2145.2 | +| VLA-Adapter | 671.6 | 670.4 | 669.4 | 671.9 | 950.8 | +| GR00T N1.6 | 689.2 | 685.8 | 687.3 | 683.0 | 851.0 | +| GR00T N1.7 | 758.3 | 733.1 | 752.8 | 723.2 | 858.2 | +| π0.5 | 1125.2 | 1117.6 | 1134.2 | 1134.2 | 2039.2 | +| OpenVLA-OFT | 1125.5 | 1127.8 | 1123.4 | 1123.5 | 1402.2 | +| π0 | 1113.2 | 1126.1 | 1114.8 | 1126.1 | 2056.3 | +| Evo-1 | 1668.8 | 1669.2 | 1669.8 | 1657.4 | 3116.8 | + +\* Octo-Small's defaults screen was re-run: the first run (672.8 ms, with a +241 ms vision stage) overlapped a transient slowdown on the board.
From 53de2fc59851505cb48225f8bbbd3c3aff48dc6f Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Fri, 6 Nov 2026 17:03:49 +0700 Subject: [PATCH 36/38] docs: IQ-10 tables sorted by min, fastest-flag fidelity re-measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TEMPLATE.md sorts the latency and fastest-flag tables by min; they were sorted by mean (π0 and OpenVLA-OFT swapped). The fidelity quoted for the two non-default fastest sets is re-measured against the CPU-only build: VLA-JEPA with --flash-attn 6.2e-3 served, GR00T N1.7 with --flash-attn 6.2e-3 on fixed inputs, both outside the bar. The BF16 timing (SmolVLA 18.6 s against 0.58 s), N1.6's two-view fidelity (7.5e-4) and its CPU fallback note are updated to bd166b8. --- docs/backend/hexagon.md | 8 ++++---- docs/benchmark/qualcomm-iq10.md | 16 +++++++++------- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 9d61d75b..8c9e395f 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -143,11 +143,11 @@ the same path. Flash attention on the HMX (v81) keeps scores and probabilities in F16. That is a precision limit rather than a wrong op, but in GR00T N1.7's LM it moves the -action chunk 5.4e-3 from the CPU reference. N1.7's LM therefore takes the +action chunk 6.2e-3 from the CPU reference. N1.7's LM therefore takes the explicit path unless `--flash-attn` is given: 2.0e-3, at the same latency. VLA-JEPA's LM does the same: with flash attention a served LIBERO observation -lands 6.7e-3 from the reference (the CPU with flash attention 5.5e-3), without -it 2.6e-3, at a cost of 683 to 810 ms with two views. +lands 6.2e-3 from the reference, without it 2.2e-3, at a cost of 683 to 810 ms +with two views. The softmax takes rows of up to 32 elements or a multiple of 32. Where a sequence can grow without changing its result, vla.cpp pads it to @@ -165,7 +165,7 @@ tower takes two camera views that way: its action chunk lands 0.16 from the CPU reference. The wrapper runs those matmuls on the CPU. On Hexagon `linear()` in `src/layers/linear.h` multiplies a contiguous batch as one 2D matrix of rows instead, the same product, so most of them never reach the rule: GR00T N1.6 -with two views lands 8.8e-4 from the reference, and the matmuls of TurboVLA's +with two views lands 7.5e-4 from the reference, and the matmuls of TurboVLA's DINOv3, which takes its two views as a batch, run on the NPU. A copy that transposes the innermost dim, such as the `CONT` of a permuted V diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 54231994..d583607b 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -79,8 +79,8 @@ vision stage separately. | GR00T N1.6 | 1 | 224 | 680.9 | 682.1 | 681.6 | 683.9 | 145.3 | 146 | 4889 | | GR00T N1.7 | 1 | 256 | 739.6 | 746.4 | 745.1 | 753.8 | 196.4 | 190 | 5215 | | π0.5 | 2 | 224 | 1114.0 | 1116.1 | 1115.5 | 1117.8 | 278.0 | 285 | 5802 | -| OpenVLA-OFT | 1 | 224 | 1117.7 | 1120.2 | 1120.4 | 1121.8 | 262.3 | 844 | 15321 | | π0 | 2 | 224 | 1117.6 | 1120.5 | 1119.9 | 1123.8 | 277.8 | 284 | 5547 | +| OpenVLA-OFT | 1 | 224 | 1117.7 | 1120.2 | 1120.4 | 1121.8 | 262.3 | 844 | 15321 | | Evo-1 | 1 | 448 | 1653.5 | 1661.4 | 1661.2 | 1665.6 | 815.6 | 319 | 1556 | ### Fastest configuration @@ -94,7 +94,7 @@ under the full protocol: The weight dtype stays F16 in every set. `--weight-dtype f16` is the default, and `--weight-dtype bf16` is left out: the NPU has no BF16 kernels, so every -matmul would run on the CPU (SmolVLA: 18.7 s against 0.82 s). +matmul would run on the CPU (SmolVLA: 18.6 s against 0.58 s, mean, 5 reps). The screen runs 2 warmups and 5 reps in one process per flag set. A flag set replaces the defaults only if its mean is more than 3% lower, both in the screen @@ -103,8 +103,9 @@ and in the full re-run. Flash attention and the matmul precision can move the action chunk. Success rates measured at the defaults therefore do not carry over to these rows. For VLA-JEPA and GR00T N1.7, `--flash-attn` puts their LM back on flash attention, -which the defaults leave off because it is outside the fidelity bar (VLA-JEPA -6.7e-3 on a served LIBERO observation, N1.7 5.4e-3 on fixed inputs). +which the defaults leave off because it is outside the fidelity bar against a +CPU-only build: VLA-JEPA 6.2e-3 on a served LIBERO observation (2.7e-3 on fixed +inputs), N1.7 6.2e-3 on fixed inputs (2.7e-3 served). | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| @@ -118,8 +119,8 @@ which the defaults leave off because it is outside the fidelity bar (VLA-JEPA | GR00T N1.6 | *(defaults)* | 680.9 | 682.1 | 681.6 | 683.9 | 145.3 | 146 | 4889 | - | | GR00T N1.7 | `--flash-attn --mm-prec default` | 715.1 | 719.2 | 718.7 | 722.9 | 195.5 | 190 | 5074 | -3.6% | | π0.5 | *(defaults)* | 1114.0 | 1116.1 | 1115.5 | 1117.8 | 278.0 | 285 | 5802 | - | -| OpenVLA-OFT | *(defaults)* | 1117.7 | 1120.2 | 1120.4 | 1121.8 | 262.3 | 844 | 15321 | - | | π0 | *(defaults)* | 1117.6 | 1120.5 | 1119.9 | 1123.8 | 277.8 | 284 | 5547 | - | +| OpenVLA-OFT | *(defaults)* | 1117.7 | 1120.2 | 1120.4 | 1121.8 | 262.3 | 844 | 15321 | - | | Evo-1 | *(defaults)* | 1653.5 | 1661.4 | 1661.2 | 1665.6 | 815.6 | 319 | 1556 | - |
Screen means (ms) per flag set @@ -136,8 +137,8 @@ which the defaults leave off because it is outside the fidelity bar (VLA-JEPA | GR00T N1.6 | 689.2 | 685.8 | 687.3 | 683.0 | 851.0 | | GR00T N1.7 | 758.3 | 733.1 | 752.8 | 723.2 | 858.2 | | π0.5 | 1125.2 | 1117.6 | 1134.2 | 1134.2 | 2039.2 | -| OpenVLA-OFT | 1125.5 | 1127.8 | 1123.4 | 1123.5 | 1402.2 | | π0 | 1113.2 | 1126.1 | 1114.8 | 1126.1 | 2056.3 | +| OpenVLA-OFT | 1125.5 | 1127.8 | 1123.4 | 1123.5 | 1402.2 | | Evo-1 | 1668.8 | 1669.2 | 1669.8 | 1657.4 | 3116.8 | \* Octo-Small's defaults screen was re-run: the first run (672.8 ms, with a @@ -157,7 +158,8 @@ The HTP skel has to be processed for this board and installed in [backend/hexagon.md](../backend/hexagon.md#linux-iq-10). GR00T N1.6 runs one view here, as the configuration above fixes. With two views -one batched `MUL_MAT` of its SigLIP2 tower still runs on the CPU; see +its SigLIP2 tower's patch matmul and flash attention, batched over the views, +still run on the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). These rows time the defaults; they do not vouch for accuracy. On a served LIBERO From f60d925420f75288a6446d2dd12f47cd969f9f7d Mon Sep 17 00:00:00 2001 From: Khanh Nguyen Date: Mon, 9 Nov 2026 08:49:57 +0700 Subject: [PATCH 37/38] hexagon: gate DitHead's padded attention on gemm_width() and drop the stale VLA_DEVICE=cpu reference --- docs/backend/hexagon-windows.md | 2 +- docs/backend/hexagon.md | 11 +++++++---- src/modules/dit_head.cpp | 12 ++++++++---- 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/docs/backend/hexagon-windows.md b/docs/backend/hexagon-windows.md index 0241be4c..c259e8d2 100644 --- a/docs/backend/hexagon-windows.md +++ b/docs/backend/hexagon-windows.md @@ -130,7 +130,7 @@ PyTorch and LIBERO; the smoke test used a 20-line pyzmq client instead. | Variable / flag | Effect in the Hexagon and OpenCL builds | |---|---| -| `VLA_DEVICE=cpu` | Skip the accelerator. The same binary then produces the CPU reference. | +| `VLA_DEVICE=cpu` | Skip the accelerator. Not a fidelity reference: the Hexagon build's NPU changes (F16 K/V, padding) run on the CPU too, so compare against a CPU-only build with `--weight-dtype f32 --flash-attn 0`. | | `VLA_DEVICE=` | Registry device index: an HTP session, or an OpenCL GPU | | `--weight-dtype f16\|f32\|bf16` | Resident dtype for GEMM weights. New: `f16`, which is also the default on both accelerators. | | `--flash-attn on\|off` | On by default in Hexagon builds, off elsewhere | diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 8c9e395f..d87a1ee2 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -230,7 +230,8 @@ The shape of the diff was predicted correctly, apart from the scheduler: 2. **A rung in [src/backend.h](../../src/backend.h).** - It goes through the registry, because `ggml_backend_hexagon_init` is declared but not defined upstream. - - `VLA_DEVICE` indexes HTP sessions, and `VLA_DEVICE=cpu` gives the reference. + - `VLA_DEVICE` indexes HTP sessions, and `VLA_DEVICE=cpu` skips them. That + run is not a fidelity reference (see [Fidelity](#fidelity)). - It sets the defaults above and points `ADSP_LIBRARY_PATH` at the executable's folder. 3. **Weights.** @@ -264,9 +265,11 @@ found. to the CPU and how much was copied (`VLA_FALLBACK_STATS=1` for per-graph detail). 4. **Fidelity before latency.** - - Compare `vla_predict_check` output against the same binary with - `VLA_DEVICE=cpu`, and against `--weight-dtype f32`. Add `--flash-attn 0` - to the reference: a Hexagon build defaults to flash attention on the CPU too. + - Compare `vla_predict_check` output against a CPU-only build + (`-DGGML_HEXAGON=OFF`) of the same commit, run with `--weight-dtype f32 + --flash-attn 0`. The Hexagon binary with `VLA_DEVICE=cpu` is not a + reference: its NPU changes (flash attention, F16 K/V, padding to + `gemm_width()`) run on the CPU too. - The bar is 2.9e-3. - A run with a wrong kernel still prints a latency, and it is usually a good one. diff --git a/src/modules/dit_head.cpp b/src/modules/dit_head.cpp index 29de7d33..1d37bb7e 100644 --- a/src/modules/dit_head.cpp +++ b/src/modules/dit_head.cpp @@ -346,14 +346,18 @@ ggml_tensor * DitHead::block(ggml_context * C, const DitLayerW & w, ggml_tensor return fused_bias_res(C, mm_exact(C, w.Wff2, g), w.bff2, h1); } ggml_tensor * att; - const int64_t Nk = enc ? enc->ne[1] : Tk; - if (!exact && (K->ne[0] > hd || K->ne[1] > Nk)) { + // hmx_kv pads to gemm_width(), which is the identity off Hexagon, so these + // stay false there whatever K/V the caller passes. + const int64_t Nk = enc ? enc->ne[1] : Tk; + const bool hpad = K->ne[0] > hd && K->ne[0] == gemm_width(hd); + const bool kpad = K->ne[1] > Nk && K->ne[1] == gemm_width(Nk); + if (!exact && (hpad || kpad)) { // hmx_kv padded K/V: the queries take the head-dim padding too, the // padded keys are masked out and the output drops the padded columns. - if (K->ne[0] > hd) + if (hpad) Q = ggml_pad(C, Q, (int) (K->ne[0]-hd), 0, 0, 0); ggml_tensor * kq = ggml_mul_mat(C, K, Q); ggml_prec_set_acc(kq, GGML_PREC_F32); - ggml_tensor * mask = K->ne[1] > Nk ? key_mask(C, K->ne[1], Nk, Tk) : nullptr; + ggml_tensor * mask = kpad ? key_mask(C, K->ne[1], Nk, Tk) : nullptr; ggml_tensor * kqv = ggml_mul_mat(C, V, ggml_soft_max_ext(C, kq, mask, scale, 0.0f)); kqv = ggml_view_3d(C, kqv, hd, Tk, heads, kqv->nb[1], kqv->nb[2], 0); att = ggml_reshape_2d(C, ggml_cont(C, ggml_permute(C, kqv, 0, 2, 1, 3)), dim, Tk); From 287e2007d95a81a95e95309f7da46b53f84d2887 Mon Sep 17 00:00:00 2001 From: Khanh Nguyen Date: Mon, 9 Nov 2026 08:52:32 +0700 Subject: [PATCH 38/38] docs: drop the Hexagon note under the README support matrix --- README.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/README.md b/README.md index a6197876..7b3e6873 100644 --- a/README.md +++ b/README.md @@ -105,12 +105,6 @@ Use [ggml-compiler](https://github.com/khanhnd61-vr/ggml-compiler) and follow [t Legend: `Y` = supported (released and benchmarked), `~` = in progress, `-` = planned. -The Hexagon column is the Dragonwing IQ-10 under Linux -([docs/backend/hexagon.md](docs/backend/hexagon.md)). On a Snapdragon X laptop -under Windows, π0 and VLA-JEPA land above the 2.9e-3 fidelity bar, OpenVLA-OFT -was not attempted (about 14 GB at F16, over its 8 GB budget) and ACT was not run -([docs/backend/hexagon-windows.md](docs/backend/hexagon-windows.md)). - --- ## Rollout on a real robot