From 2a66f9faf30d45a5b8df74294bb249b59c4f1d9c Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Mon, 2 Nov 2026 15:04:25 +0700 Subject: [PATCH 01/14] docs: benchmark the Qualcomm Dragonwing IQ-10 Hexagon NPU vla-bench under TEMPLATE.md's protocol at e8dfa34, for the six models whose GGUFs are on the board. The flag screen keeps F16 weights and varies only --flash-attn and --mm-prec: the NPU has no BF16 kernels, and F16 is the default. Memory adds the drop in system MemAvailable, since the weights sit in FastRPC buffers that peak RSS leaves out. The CPU section and the archs without GGUFs on the board are left for a later round. --- docs/benchmark/qualcomm-iq10.md | 148 ++++++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 docs/benchmark/qualcomm-iq10.md diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md new file mode 100644 index 00000000..b511afd0 --- /dev/null +++ b/docs/benchmark/qualcomm-iq10.md @@ -0,0 +1,148 @@ +# Qualcomm Dragonwing IQ-10: Hexagon NPU + +`vla-bench` times `predict()` in-process on synthetic inputs. It measures the +engine only: no transport, no simulator, and no claim about task success. + +The graph runs on the Hexagon NPU, and ops the NPU rejects run on the CPU +through vla.cpp's fallback wrapper. + +## Test setup + +| | | +|---|---| +| Device | Qualcomm Dragonwing IQ-10 (IQX EVK): QCS8797, 18 Oryon cores, Hexagon v81 NPU (one of its four CDSP sessions in use), 60.8 GiB RAM visible | +| Threads | 16 (the default: all hardware threads, capped at 16), for the NPU's CPU fallback | +| Power | CPU governor `performance` | +| OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | +| Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | +| Commit | `e8dfa34` (PR #69, on top of #68) | +| llama.cpp | `b11223` | +| Date | 2026-11-02 11:02 to 11:48 (UTC+07) | +| Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | +| Runtime flags | defaults; the fastest flags per model are in the second table | +| Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | + +**Peak RSS** is the process's peak resident set (`getrusage`, via +`/usr/bin/time`). It leaves out the weights, which live in FastRPC buffers +mapped for the NPU. **System RAM** is the largest drop in the system's +`MemAvailable` while the process ran, sampled every 0.5 s; it counts those +buffers. Nothing else ran on the board. + +## Model configuration + +Each model runs at its native view count and input size, with weights as +shipped. Checkpoints are the `vrfai/*` GGUFs; where a repo ships several, we use +the `libero_object` variant. Settings not listed are the defaults. + +| Model | Checkpoint | Views | Input | Lang tokens | Action chunk | Action dim | Action head | Steps | +|---|---|--:|--:|--:|--:|--:|---|--:| +| SmolVLA | `smolvla-libero-gguf` | 2 | 512 | 16 | 50 | 7 | flow matching | 10 | +| π0 | `pi0-libero-finetuned-v044-gguf` | 2 | 224 | 16 | 50 | 7 | flow matching | 10 | +| π0.5 | `pi05-libero-gguf` | 2 | 224 | 16 | 50 | 7 | flow matching | 10 | +| GR00T N1.5 | `gr00tn1d5-libero-object-gguf` | 1 | 224 | 16 | 16 | 32 | flow matching (DiT) | 4 | +| GR00T N1.6 | `gr00tn1d6-libero-gguf` | 1 | 224 | 16 | 50 | 128 | flow matching (DiT) | 4 | +| GR00T N1.7 | `gr00tn1d7-libero-gguf` | 1 | 256 | 16 | 40 | 132 | flow matching (DiT) | 4 | +| VLA-JEPA | `vla-jepa-libero` | 1 | 256 | 16 + 32 | 7 | 7 | flow matching (DiT) | 4 | +| Evo-1 | `evo1-libero-gguf` | 1 | 448 | 16 | 50 | 7 | flow matching | 32 | +| BitVLA | `bitvla-libero-gguf` (int2) | 1 | 224 | 16 | 8 | 7 | parallel decoding | 1 | +| VLA-Adapter | `vla-adapter-libero-gguf` | 1 | 224 | 16 | 8 | 7 | bridge-attention policy | 1 | +| OpenVLA-OFT | `openvla-oft-libero-gguf` | 1 | 224 | 16 | 8 | 7 | L1 regression head | 1 | +| Octo-Small | `octo-small-libero-gguf` | 2 | 256 + 128 | 16 | 4 | 7 | DDPM diffusion | 20 | +| TurboVLA | `turbovla-libero-gguf` | 2 | 256 | 16 | 12 | 7 | single-pass decoder | 1 | + +- **Lang tokens** is the synthetic prompt length. VLA-JEPA adds its 32 + `` tokens with `--extra-token 151697 --extra-count 32`. +- **Action dim** is the checkpoint's padded width. LIBERO uses the first 7 dims. +- **GR00T** needs its embodiment set in the environment: + `VLA_GR00T_EMBODIMENT=new_embodiment` for N1.5, `libero_panda` for N1.6 and + `libero_sim` for N1.7. **Octo** needs `VLA_OCTO_UNNORM_DATASET=libero_object`. +- **Octo** takes a 256 px primary view and a 128 px wrist view. + +## Latency and memory + +Rows are sorted by latency. `vision ms` is `-` for archs that do not time their +vision stage separately. + +| Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | +|---|--:|--:|--:|--:|--:|--:|--:|--:|--:| +| GR00T N1.7 | 1 | 256 | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | +| SmolVLA | 2 | 512 | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | +| GR00T N1.5 | 1 | 224 | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | +| GR00T N1.6 | 1 | 224 | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | +| π0.5 | 2 | 224 | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | +| Evo-1 | 1 | 448 | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | + +### Fastest configuration + +We screen each model over these runtime flag sets, then re-run the fastest set +under the full protocol: + +- the defaults (F16 weights, the Hexagon build's default); +- `--flash-attn` and `--mm-prec default`, alone and together; +- `--flash-attn 0`, because flash attention is on by default in Hexagon builds. + +The weight dtype stays F16 in every set. `--weight-dtype f16` is the default, +and `--weight-dtype bf16` is left out: the NPU has no BF16 kernels, so every +matmul would run on the CPU (SmolVLA: 18.7 s against 0.82 s). + +The screen runs 2 warmups and 5 reps in one process per flag set. A flag set +replaces the defaults only if its mean is more than 3% lower, both in the screen +and in the full re-run. + +Flash attention and the matmul precision can move the action chunk. Success +rates measured at the defaults therefore do not carry over to these rows. + +| Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | +|---|---|--:|--:|--:|--:|--:|--:|--:|--:| +| GR00T N1.7 | *(defaults)* | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | - | +| SmolVLA | *(defaults)* | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | - | +| GR00T N1.5 | *(defaults)* | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | - | +| GR00T N1.6 | *(defaults)* | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | - | +| π0.5 | *(defaults)* | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | - | +| Evo-1 | *(defaults)* | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | - | + +
Screen means (ms) per flag set + +| Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | +|---|--:|--:|--:|--:|--:| +| GR00T N1.7 | 768.7 | 748.5 | 761.3 | 744.5 | 1086.6 | +| SmolVLA | 823.1 | 818.9 | 819.5 | 818.5 | 3018.8 | +| GR00T N1.5 | 1026.1 | 1023.0 | 1025.3 | 1026.1 | 1501.2 | +| GR00T N1.6 | 1114.4 | 1115.1 | 1114.4 | 1113.3 | 1290.3 | +| π0.5 | 2939.0 | 2956.9 | 2959.0 | 2962.7 | 2959.6 | +| Evo-1 | 3583.7 | 3593.9 | 3591.8 | 3582.6 | 5828.5 | + +
+ +## Not run + +- **BitVLA**: N/A off CUDA. The only published GGUF is int2-packed, and only + CUDA builds load it. +- **π0, VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small, TurboVLA**: not run in + this round; their GGUFs were not on the board. + +## Notes + +The HTP skel has to be processed for this board and installed in +`/vendor/dsp/cdsp0..3`: its FastRPC ignores `ADSP_LIBRARY_PATH`. See +[backend/hexagon.md](../backend/hexagon.md#linux-iq-10). + +GR00T N1.6 runs one view here, as the configuration above fixes. With two views +its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see +[backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). + +These rows time the defaults; they do not vouch for accuracy. On a served LIBERO +observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar, about as +far off as the CPU at F16; see +[backend/hexagon.md](../backend/hexagon.md#serving). + +## Reproducing + +```bash +scripts/build_linux_hexagon.sh +# copy build-lnx-htp/pkg to the board, install its HTP skel (see backend/hexagon.md), then on the board: +export LD_LIBRARY_PATH=$PWD/pkg/lib + +# one row, for example SmolVLA +./pkg/bin/vla-bench --ckpt smolvla-libero.gguf --images 2 --size 512 --warmup 3 --reps 20 +``` From cc3c2bb1ae1064516e10dba4baec8cca0145e230 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Mon, 2 Nov 2026 15:46:39 +0700 Subject: [PATCH 02/14] docs: add the IQ-10 Hexagon row to the benchmark table Minimum latency at the defaults from qualcomm-iq10.md, measured at e8dfa34. Models whose GGUFs were not on the board are marked n/r; BitVLA is N/A off CUDA. --- docs/benchmark/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index 6dfc44ae..c75ebc14 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -11,7 +11,8 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path `1012c05`, before that path, for comparison. The Radeon RX 9060 XT ROCm row ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in -[backend/rocm.md](../backend/rocm.md). Each report +[backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at +`e8dfa34` (PR #69, same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -35,6 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 1020.9 | 1110.3 | 747.5 | N/A | 810.3 | n/r | 2914.6 | n/r | 3519.4 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | From b33ead8ac6adf79789e882e2f4793503bea68c47 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 09:42:06 +0700 Subject: [PATCH 03/14] hexagon: pad SigLIP's MLP to a multiple of 32 so it runs on the HMX ggml-hexagon takes an F16 weight on the matrix unit only when both dims are multiples of 32. SigLIP-so400m's MLP is 4304 wide, so fc1 and fc2 ran on the vector units: 440 ms of GR00T N1.5's 565 ms vision tower on the IQ-10. WeightLoader::pad() declares a float weight larger than in the file, with the extra elements zero, and EncStack pads fc1, its bias and fc2 to gemm_width(), which rounds up to 32 in Hexagon builds and is the identity elsewhere. The padded units are zero through GELU and fc2, so the block's output is the same. On the IQ-10 against the unpadded CPU F32 reference: pi0.5 2926 -> 2122 ms (3.1e-4), GR00T N1.5 1024 -> 610 ms (2.4e-3), N1.6 1117 -> 709 ms (1.4e-3). SmolVLA, Evo-1 and N1.7 do not use this MLP and are unchanged. --- src/backend.h | 16 ++++++++++++++++ src/loader.cpp | 35 +++++++++++++++++++++++++++++++++-- src/loader.h | 5 +++++ src/modules/encoder.cpp | 24 ++++++++++++++++++++++++ 4 files changed, 78 insertions(+), 2 deletions(-) diff --git a/src/backend.h b/src/backend.h index bd574c0d..894f4004 100644 --- a/src/backend.h +++ b/src/backend.h @@ -191,6 +191,22 @@ inline ggml_tensor * fa_kv([[maybe_unused]] ggml_context * ctx, ggml_tensor * t) #endif } +/** + * @brief A GEMM's inner width as the compiled-in backend's matmul takes it. + * + * Hexagon's matrix unit (HMX) takes an F16 weight only when both of its dims + * are multiples of 32; anything else runs on the vector units. SigLIP's + * 4304-wide MLP spends 440 ms there per tower on the IQ-10, against about + * 0.5 ms a matmul on the HMX. Everywhere else this returns @p n. + */ +inline int64_t gemm_width(int64_t n) { +#ifdef GGML_USE_HEXAGON + return (n + 31) / 32 * 32; +#else + return n; +#endif +} + /** * @brief `ggml_gelu` (the tanh form), in ops the compiled-in backend gets right. * diff --git a/src/loader.cpp b/src/loader.cpp index 54826859..fc6d95f7 100644 --- a/src/loader.cpp +++ b/src/loader.cpp @@ -20,6 +20,7 @@ #include #include #include +#include namespace vla { @@ -39,11 +40,25 @@ void WeightLoader::as_float(const std::string & name, int64_t K, int64_t N, Floa as_float_[name] = AsFloat{K, N, std::move(make)}; } +void WeightLoader::pad(const std::string & name, int64_t ne0, int64_t ne1) { + pad_[name] = {ne0, ne1}; +} + bool WeightLoader::read_resident(const std::string & name, ggml_type type, bool gemma_norm, std::vector & out) { const auto it = as_float_.find(name); if (it == as_float_.end()) { out = g_.read_convert(name.c_str(), type, gemma_norm); - return !out.empty(); + const auto p = pad_.find(name); + if (p == pad_.end() || out.empty()) + return !out.empty(); + const ggml_tensor * src = g_.meta(name.c_str()); + const size_t rs = ggml_row_size(type, src->ne[0]); + const size_t rp = ggml_row_size(type, p->second.first); + std::vector padded(rp * (size_t) p->second.second, 0); + for (int64_t r = 0; r < src->ne[1]; ++r) + std::memcpy(padded.data() + r * rp, out.data() + r * rs, rs); + out.swap(padded); + return true; } const AsFloat & a = it->second; std::vector f; @@ -97,7 +112,23 @@ ggml_tensor * WeightLoader::declare(ggml_type want, bool required, bool gemma_no return nullptr; } - ggml_tensor * t = ggml_new_tensor(ctx_, g_.resident_type(src, want), ggml_n_dims(src), src->ne); + const ggml_type rt = g_.resident_type(src, want); + int64_t ne[GGML_MAX_DIMS]; + std::memcpy(ne, src->ne, sizeof(ne)); + int n_dims = ggml_n_dims(src); + if (const auto p = pad_.find(name); p != pad_.end()) { + const bool float_type = rt == GGML_TYPE_F32 || rt == GGML_TYPE_F16 || rt == GGML_TYPE_BF16; + if (!float_type || n_dims > 2 || p->second.first < ne[0] || p->second.second < ne[1]) { + std::fprintf(stderr, "vla(%s): cannot pad %s to [%lld, %lld]\n", arch_, name, + (long long) p->second.first, (long long) p->second.second); + ok_ = false; + return nullptr; + } + ne[0] = p->second.first; + ne[1] = p->second.second; + n_dims = 2; + } + ggml_tensor * t = ggml_new_tensor(ctx_, rt, n_dims, ne); if (!t) { std::fprintf(stderr, "vla(%s): ggml_new_tensor failed for %s (weight context too small?)\n", arch_, name); ok_ = false; diff --git a/src/loader.h b/src/loader.h index 2f71d099..e44d01e6 100644 --- a/src/loader.h +++ b/src/loader.h @@ -82,6 +82,10 @@ class WeightLoader { using FloatMaker = std::function &)>; void as_float(const std::string & name, int64_t K, int64_t N, FloatMaker make); + // Declare 2D (or 1D) float tensor `name` as [ne0, ne1], at least its shape in + // the file, with the extra elements zero. Call before declaring it. + void pad(const std::string & name, int64_t ne0, int64_t ne1); + // For helpers that inspect the file before declaring (foldquant.cpp). gguf_reader & reader() { return g_; @@ -111,6 +115,7 @@ class WeightLoader { FloatMaker make; }; std::map as_float_; + std::map> pad_; const char * arch_; gguf_reader & g_; diff --git a/src/modules/encoder.cpp b/src/modules/encoder.cpp index 8dbb8b89..5c6fa428 100644 --- a/src/modules/encoder.cpp +++ b/src/modules/encoder.cpp @@ -14,6 +14,7 @@ #include "modules/encoder.h" +#include "backend.h" #include "layers/attn.h" #include "layers/ffn.h" #include "layers/fused_ops.h" @@ -25,9 +26,30 @@ #include "layers/qlinear.h" #include +#include namespace vla { +namespace { + +// Widens the MLP to gemm_width() with zeros. The extra fc1 rows and bias are 0, +// GELU(0) is 0 and the extra fc2 columns are 0, so the block's output is the same. +void pad_mlp(WeightLoader & L, const std::string & fc1, const std::string & fc2) { + const ggml_tensor * w1 = L.reader().meta((fc1 + ".weight").c_str()); + const ggml_tensor * w2 = L.reader().meta((fc2 + ".weight").c_str()); + if (!w1 || !w2 || ggml_is_quantized(L.gemm_type())) + return; + const int64_t ff = w1->ne[1]; + const int64_t fp = gemm_width(ff); + if (fp == ff || w2->ne[0] != ff) + return; + L.pad(fc1 + ".weight", w1->ne[0], fp); + L.pad(fc1 + ".bias", fp, 1); + L.pad(fc2 + ".weight", fp, w2->ne[1]); +} + +} + void EncStack::declare(WeightLoader & L, const char * prefix, int64_t layers, const EncNames & n) { blk.resize(layers); ln_spec.eps = cfg.ln_eps; @@ -61,6 +83,8 @@ void EncStack::declare(WeightLoader & L, const char * prefix, int64_t layers, co w.q_fc1 = L.quant_i8(b + n.fc1 + ".q8", {b + n.fc1 + ".weight"}); w.q_fc2 = L.quant_i8(b + n.fc2 + ".q8", {b + n.fc2 + ".weight"}); } else { + const std::string b = std::string(prefix) + "." + std::to_string(i) + "."; + pad_mlp(L, b + n.fc1, b + n.fc2); w.Wo = L.gemm("%s.%lld.attn_o.weight", prefix, (long long)i); w.Wfc1 = L.gemm("%s.%lld.%s.weight", prefix, (long long)i, n.fc1); w.Wfc2 = L.gemm("%s.%lld.%s.weight", prefix, (long long)i, n.fc2); From b85e1746a860ea5ef4d7b43379604aaeded24b37 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 10:00:45 +0700 Subject: [PATCH 04/14] pi05: flash attention in SigLIP on Hexagon pi0.5 ties SigLIP's flash attention to a view-taking kernel, which only SYCL has, so on Hexagon its tower ran the explicit scores: an F32 x F32 matmul at head dim 72 that the HMX does not take, 266 ms on the vector units. Where flash attention is on by default (Hexagon), SigLIP now takes the plain flash attention that GR00T's identical tower already uses. Other backends are unchanged. IQ-10: 2118 -> 1833 ms, 2.8e-4 from the CPU F32 reference. --- src/models/pi05.cpp | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/models/pi05.cpp b/src/models/pi05.cpp index 1f4faa31..a73e916a 100644 --- a/src/models/pi05.cpp +++ b/src/models/pi05.cpp @@ -544,7 +544,10 @@ std::unique_ptr pi05_create(const std::string& mmproj_path, std::printf("vla(pi05): expert weights INT4 (W4A8), %s\n", aex_w4_group > 0 ? ("group " + std::to_string(aex_w4_group)).c_str() : "per channel"); } - m->vis.vit.enc.cfg.flash_attn = use_fused_attention(); + // On Hexagon (flash attention on by default) SigLIP takes the plain flash + // attention too: its explicit scores, head dim 72 against 256 keys, are an + // F32 x F32 matmul the HMX does not take, 266 ms of vector-unit time. + m->vis.vit.enc.cfg.flash_attn = use_fused_attention() || (vla::default_flash_attn() && vla::flash_attn_enabled()); // Fused SigLIP blocks (modules/encoder.h, EncCfg::fused): SYCL, BF16, fused // attention. A FoldQuant GGUF takes them too: its integer sites are the LM's // and the expert's, and its vision tower is BF16 like any other checkpoint's. From 74e145de94673e6628b20c12dbc7b217ba99a387 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 10:04:51 +0700 Subject: [PATCH 05/14] evo1: F16 K/V in the LM's attention on Hexagon The LM's explicit attention multiplies F32 K and V against a batch of 14 query heads. ggml-hexagon's HMX takes no batched F32 src0, so both matmuls ran on the vector units, about 1 s per chunk on the IQ-10. On Hexagon K and V now go to F16 through fa_kv(); the scores still accumulate in F32 and the softmax stays F32. Other backends keep F32 K/V. IQ-10: 3261 -> 2336 ms, 4.3e-4 from the CPU F32 reference (5.2e-4 before). The InternViT flash attention stays on the CPU: F16 K/V there measured 3.9e-2. --- src/models/evo1.cpp | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/models/evo1.cpp b/src/models/evo1.cpp index f6e56563..c61adc1d 100644 --- a/src/models/evo1.cpp +++ b/src/models/evo1.cpp @@ -162,8 +162,12 @@ ggml_tensor * build_qwen2_layer(ggml_context * C, const Evo1ModelArch & m, const ggml_tensor * q_rope = ggml_rope_ext(C, ggml_reshape_3d(C, qp, hd, n_q, seq), positions, nullptr, (int) hd, GGML_ROPE_TYPE_NEOX, 0, m.lm_rope_base, 1.0f, 0.0f, 1.0f, 32.0f, 1.0f); ggml_tensor * k_rope = ggml_rope_ext(C, ggml_reshape_3d(C, kp, hd, n_kv, seq), positions, nullptr, (int) hd, GGML_ROPE_TYPE_NEOX, 0, m.lm_rope_base, 1.0f, 0.0f, 1.0f, 32.0f, 1.0f); ggml_tensor * Q = ggml_cont(C, ggml_permute(C, q_rope, 0, 2, 1, 3)); - ggml_tensor * K = ggml_cont(C, ggml_permute(C, k_rope, 0, 2, 1, 3)); - ggml_tensor * V = ggml_cont(C, ggml_permute(C, ggml_reshape_3d(C, vp, hd, n_kv, seq), 1, 2, 0, 3)); + // fa_kv: on Hexagon K and V go to F16, so both matmuls run on the HMX, which + // takes no batched F32 src0 (1 s of vector-unit time per chunk on the IQ-10). + // The scores still accumulate in F32 and the softmax stays F32: 4.3e-4 from + // the CPU F32 reference there. Other backends keep F32 K/V. + ggml_tensor * K = fa_kv(C, ggml_cont(C, ggml_permute(C, k_rope, 0, 2, 1, 3))); + ggml_tensor * V = fa_kv(C, ggml_cont(C, ggml_permute(C, ggml_reshape_3d(C, vp, hd, n_kv, seq), 1, 2, 0, 3))); ggml_tensor * kq = ggml_mul_mat(C, K, Q); ggml_prec_set_acc(kq, GGML_PREC_F32); ggml_tensor * aw = ggml_soft_max_ext(C, kq, mask, scale, 0.0f); ggml_tensor * kqv = ggml_mul_mat(C, V, aw); From bdacd9e54f5e1d43c22cc9eb7a0f059aef991831 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 12:05:15 +0700 Subject: [PATCH 06/14] docs: Hexagon matmuls on the HMX, IQ-10 benchmark at 74e145d hexagon.md: which matmuls the HMX takes, the three that now move onto it, what is left on the vector units, and why the GELU_ERF kernel was left out. The fidelity and serving tables are re-measured at this commit. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 74e145d: GR00T N1.5 613.6 ms, N1.6 706.9, pi0.5 1789.4, Evo-1 2589.2; SmolVLA and N1.7 unchanged. The defaults stay the fastest configuration. --- CHANGELOG.md | 4 +++ docs/backend/hexagon.md | 44 +++++++++++++++++++++++------- docs/benchmark/README.md | 4 +-- docs/benchmark/qualcomm-iq10.md | 47 ++++++++++++++++++--------------- 4 files changed, 65 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d2436c7..3be3ac7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,10 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke space and matmuls of a 2D weight against a batch, which it computes wrong (GR00T N1.6 with two views), run on the CPU, and GR00T N1.7's LM leaves flash attention (F16 scores on the HMX) unless `--flash-attn` is given. + Three matmuls that the HMX does not take move onto it: SigLIP's 4304-wide + MLP is zero-padded to 4320 at load, π0.5's SigLIP takes flash attention, and + Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, π0.5 and Evo-1 run + 27-40% faster, within the bar on fixed inputs. GR00T N1.5 and N1.6 are above the bar on a served LIBERO observation, as the CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install and the numbers. diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 6df93b30..c664eb90 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -94,7 +94,10 @@ changed all three conclusions: it. - **`GELU_ERF` is still missing.** It now runs on the CPU (next point) in every DINOv2 and SigLIP-so400m tower, and costs what that implies: Evo-1 spends more - on copies than the NPU saves. + on copies than the NPU saves. An HTP kernel was written and checked on the + IQ-10 (b11223): more accurate than the CPU's (relative error 1.5e-6 against + 5.1e-5), and Evo-1 7% faster. It was left out: GR00T N1.7 sits close to the + bar, and the kernel moved its fixed-input chunk from 2.0e-3 to 3.0e-3. - **Absent is no longer fatal.** `src/backend_fallback.cpp` wraps the NPU in a backend that runs whatever it rejects on the CPU, without touching an arch. That also covers ops refused for shape rather than type: softmax rows that are @@ -133,6 +136,26 @@ of 1, activations above 1) is claimed and comes out wrong. GR00T N1.6's SigLIP2 tower takes two camera views that way: its action chunk lands 0.16 from the CPU reference. The wrapper runs those matmuls on the CPU: 1.9e-3, and 25% faster. +The matrix unit (HMX) takes a matmul only when its inner dim K and its output +dim N are multiples of 32, and with a batch only an F16 src0. Anything else runs +on the vector units, at about 1/20 of the speed. `GGML_HEXAGON_PROFILE=1` on the +IQ-10 showed three such matmuls dominating, and vla.cpp now avoids them in +Hexagon builds: + +- **SigLIP-so400m's MLP is 4304 wide**: 440 ms of GR00T N1.5's 565 ms tower. + The loader zero-pads fc1, its bias and fc2 to 4320 (`WeightLoader::pad`, + `gemm_width()` in `src/backend.h`), which leaves the output unchanged. +- **π0.5's SigLIP ran explicit attention** (head dim 72, F32 x F32), because + its flash attention was tied to SYCL's view-taking kernel. It now takes the + plain flash attention, as GR00T's tower does. +- **Evo-1's LM attention multiplied F32 K and V against 14 query heads.** K and + V now go to F16 (`fa_kv()`); the scores still accumulate in F32. + +What is left on the vector units: GR00T's DiT attention (head dim 48) and Evo-1's +DiT cross-attention (head dim 112, 1025 keys), whose shapes the HMX does not +take, and Evo-1's InternViT attention, which stays on the CPU because F16 K/V +there measured 3.9e-2. + ## What changed in `vla.cpp` The shape of the diff was predicted correctly, apart from the scheduler: @@ -285,9 +308,9 @@ path that other builds run: | Model | max\|Δ\| | |---|--:| | SmolVLA | 7.2e-4 | -| π0.5 | 2.7e-4 | -| Evo-1 | 4.1e-4 | -| GR00T N1.5 | 1.7e-3 | +| π0.5 | 2.8e-4 | +| Evo-1 | 5.0e-4 | +| GR00T N1.5 | 2.4e-3 | | GR00T N1.6 | 1.4e-3 | | GR00T N1.7 | 2.0e-3 | @@ -307,15 +330,16 @@ malformed or an invalid request, and keeps serving afterwards. | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 | |---|--:|--:| | SmolVLA | 1.3e-3 | | -| π0.5 | 4.3e-4 | | -| Evo-1 | 4.3e-4 | | -| GR00T N1.5 | 8.3e-3 | 4.3e-2 | +| π0.5 | 4.2e-4 | | +| Evo-1 | 1.1e-3 | | +| GR00T N1.5 | 1.1e-2 | 4.3e-2 | | GR00T N1.6 | 1.3e-2 | 1.1e-2 | | GR00T N1.7 | 2.5e-3 | | -GR00T N1.5 and N1.6 are above the bar on this input, by about what the CPU -itself gives with F16 weights: they are sensitive to 16-bit precision rather -than miscomputed. Running them with F32 weights on the NPU has not been tried +GR00T N1.5 and N1.6 are above the bar on this input, and so is the CPU itself +with F16 weights: N1.5 lands closer to the reference than the CPU at F16, N1.6 +slightly further (1.3e-2 against 1.1e-2). They are sensitive to 16-bit +precision rather than miscomputed. Running them with F32 weights on the NPU has not been tried yet. ## IQ-10, imagined diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index c75ebc14..a2befb19 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`e8dfa34` (PR #69, same llama.cpp pin). Each report +`74e145d` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 1020.9 | 1110.3 | 747.5 | N/A | 810.3 | n/r | 2914.6 | n/r | 3519.4 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 613.6 | 706.9 | 746.5 | N/A | 811.5 | n/r | 1789.4 | n/r | 2589.2 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index b511afd0..9415b204 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `e8dfa34` (PR #69, on top of #68) | +| Commit | `74e145d` (branch `hexagon-b11223-ops`, on top of #68 to #70) | | llama.cpp | `b11223` | -| Date | 2026-11-02 11:02 to 11:48 (UTC+07) | +| Date | 2026-11-03 10:43 to 11:02 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -65,12 +65,12 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.7 | 1 | 256 | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | -| SmolVLA | 2 | 512 | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | -| GR00T N1.5 | 1 | 224 | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | -| GR00T N1.6 | 1 | 224 | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | -| π0.5 | 2 | 224 | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | -| Evo-1 | 1 | 448 | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | +| GR00T N1.5 | 1 | 224 | 613.6 | 615.7 | 614.8 | 618.5 | 148.2 | 162 | 3644 | +| GR00T N1.6 | 1 | 224 | 706.9 | 710.9 | 710.3 | 715.0 | 145.3 | 147 | 4688 | +| GR00T N1.7 | 1 | 256 | 746.5 | 754.0 | 753.4 | 760.6 | 195.7 | 191 | 5047 | +| SmolVLA | 2 | 512 | 811.5 | 816.3 | 815.9 | 820.2 | 283.1 | 146 | 1234 | +| π0.5 | 2 | 224 | 1789.4 | 1820.3 | 1818.5 | 1850.2 | 296.7 | 285 | 5783 | +| Evo-1 | 1 | 448 | 2589.2 | 2602.0 | 2598.8 | 2613.3 | 853.9 | 319 | 1588 | ### Fastest configuration @@ -94,23 +94,23 @@ rates measured at the defaults therefore do not carry over to these rows. | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.7 | *(defaults)* | 747.5 | 754.7 | 753.0 | 762.7 | 196.1 | 191 | 5083 | - | -| SmolVLA | *(defaults)* | 810.3 | 818.6 | 820.5 | 822.6 | 283.0 | 146 | 1187 | - | -| GR00T N1.5 | *(defaults)* | 1020.9 | 1024.2 | 1024.2 | 1027.0 | 563.2 | 162 | 3650 | - | -| GR00T N1.6 | *(defaults)* | 1110.3 | 1114.5 | 1113.6 | 1118.5 | 559.8 | 147 | 4687 | - | -| π0.5 | *(defaults)* | 2914.6 | 2942.6 | 2934.4 | 2967.2 | 1429.6 | 285 | 5517 | - | -| Evo-1 | *(defaults)* | 3519.4 | 3572.4 | 3576.7 | 3588.2 | 854.5 | 319 | 1579 | - | +| GR00T N1.5 | *(defaults)* | 613.6 | 615.7 | 614.8 | 618.5 | 148.2 | 162 | 3644 | - | +| GR00T N1.6 | *(defaults)* | 706.9 | 710.9 | 710.3 | 715.0 | 145.3 | 147 | 4688 | - | +| GR00T N1.7 | *(defaults)* | 746.5 | 754.0 | 753.4 | 760.6 | 195.7 | 191 | 5047 | - | +| SmolVLA | *(defaults)* | 811.5 | 816.3 | 815.9 | 820.2 | 283.1 | 146 | 1234 | - | +| π0.5 | *(defaults)* | 1789.4 | 1820.3 | 1818.5 | 1850.2 | 296.7 | 285 | 5783 | - | +| Evo-1 | *(defaults)* | 2589.2 | 2602.0 | 2598.8 | 2613.3 | 853.9 | 319 | 1588 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| GR00T N1.7 | 768.7 | 748.5 | 761.3 | 744.5 | 1086.6 | -| SmolVLA | 823.1 | 818.9 | 819.5 | 818.5 | 3018.8 | -| GR00T N1.5 | 1026.1 | 1023.0 | 1025.3 | 1026.1 | 1501.2 | -| GR00T N1.6 | 1114.4 | 1115.1 | 1114.4 | 1113.3 | 1290.3 | -| π0.5 | 2939.0 | 2956.9 | 2959.0 | 2962.7 | 2959.6 | -| Evo-1 | 3583.7 | 3593.9 | 3591.8 | 3582.6 | 5828.5 | +| GR00T N1.5 | 616.1 | 615.0 | 619.5 | 615.9 | 1096.1 | +| GR00T N1.6 | 716.2 | 698.0 | 700.7 | 696.3 | 873.4 | +| GR00T N1.7 | 753.6 | 733.7 | 764.7 | 731.8 | 1070.6 | +| SmolVLA | 828.8 | 818.5 | 812.5 | 822.8 | 3012.6 | +| π0.5 | 1815.6 | 1816.9 | 11581.7 | 1822.6 | 2144.2 | +| Evo-1 | 2614.6 | 2599.1 | 2606.5 | 2605.6 | 4870.0 |
@@ -131,9 +131,12 @@ GR00T N1.6 runs one view here, as the configuration above fixes. With two views its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). +In the screen, π0.5 with `--mm-prec default` took 11.6 s against 1.8 s at the +defaults. That was not investigated. + These rows time the defaults; they do not vouch for accuracy. On a served LIBERO -observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar, about as -far off as the CPU at F16; see +observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar: N1.5 +closer to the reference than the CPU at F16, N1.6 slightly further; see [backend/hexagon.md](../backend/hexagon.md#serving). ## Reproducing From c10cb805342c95f1215acc6f1de2a5609d809884 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 17:00:49 +0700 Subject: [PATCH 07/14] pi05: flash attention in the VLM prefix and action expert on Hexagon The VLM's and the expert's attention took flash attention only on a view-taking kernel (SYCL). On Hexagon they ran explicit scores against a single KV head, which the fallback wrapper sends to the CPU. Where flash attention is on by default (Hexagon), all three sites now take the plain flash attention; the view-only paths keep use_fused_attention(). IQ-10: 1773 -> 1320 ms, 4.8e-4 from the CPU F32 reference, 3.5e-4 on a served LIBERO observation. Other models unchanged. --- src/models/pi05.cpp | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/src/models/pi05.cpp b/src/models/pi05.cpp index a73e916a..6eeece50 100644 --- a/src/models/pi05.cpp +++ b/src/models/pi05.cpp @@ -146,6 +146,14 @@ namespace { // straight off the [hd, H, S] views. Otherwise the unfused graph is used. bool use_fused_attention() { return vla::flash_attn_enabled() && vla::fa_takes_views(); } +// Where flash attention is on by default (Hexagon), the three sites also take the +// plain flash attention when the kernel does not take views. Their explicit +// scores are matmuls the NPU does not run: SigLIP's at head dim 72, and the VLM's +// and the expert's, whose single KV head the fallback wrapper sends to the CPU. +bool use_flash_attention() { + return use_fused_attention() || (vla::default_flash_attn() && vla::flash_attn_enabled()); +} + ggml_tensor * build_adarms( ggml_context * ctx, ggml_tensor * x, ggml_tensor * mod, int64_t h, float eps, ggml_tensor ** gate_out) { @@ -185,7 +193,7 @@ ggml_tensor * build_expert_layer( ggml_tensor * gate_ffn = ggml_view_1d(ctx, mod_ffn, h, (size_t) 2*h * sizeof(float)); ggml_tensor * xq_attn = fq_act_mod(ctx, &w.act_mod_qkv, w.fq_qkv, x_in, fqx->scale1_attn, fqx->shift_attn); ggml_tensor * h1 = gemma_attn(ctx, w, nullptr, positions, cfg, seq, cached_K, cached_V, mask, nullptr, nullptr, - GGML_TYPE_F32, use_fused_attention(), nullptr, x_in, nullptr, fqx->K16, fqx->V16, + GGML_TYPE_F32, use_flash_attention(), nullptr, x_in, nullptr, fqx->K16, fqx->V16, xq_attn, gate_attn); ggml_tensor * xq_ffn = fq_act_mod(ctx, &w.act_mod_gu, w.fq_gu, h1, fqx->scale1_ffn, fqx->shift_ffn); return gemma_mlp(ctx, w, nullptr, GGML_TYPE_F32, nullptr, h1, xq_ffn, gate_ffn); @@ -207,13 +215,13 @@ ggml_tensor * build_expert_layer( // out * gate + x reading the GEMM output in whatever type it has. ggml_tensor * xn = fused_adarms(ctx, x_in, mod_attn, at, *fa); ggml_tensor * o = gemma_attn(ctx, w, xn, positions, cfg, seq, cached_K, cached_V, mask, - nullptr, nullptr, at, use_fused_attention(), nullptr, nullptr, kv); + nullptr, nullptr, at, use_flash_attention(), nullptr, nullptr, kv); ggml_tensor * h1 = fused_gateres(ctx, x_in, o, mod_attn, *fg); ggml_tensor * xm = fused_adarms(ctx, h1, mod_ffn, at, *fa); return fused_gateres(ctx, h1, gemma_mlp(ctx, w, xm, at), mod_ffn, *fg); } - if (!mask && !kv && gemma_fused(w, at, use_fused_attention(), seq, seq + (cached_K ? cached_K->ne[2] : 0))) { + if (!mask && !kv && gemma_fused(w, at, use_flash_attention(), seq, seq + (cached_K ? cached_K->ne[2] : 0))) { auto part = [&](ggml_tensor * mod, int j) { return ggml_view_1d(ctx, mod, h, (size_t) j*h*sizeof(float)); }; @@ -237,7 +245,7 @@ ggml_tensor * build_expert_layer( // mm_act does not convert its input: a BF16 GEMM must be handed BF16 rows, // but adaRMS produces F32, whose bits would be misread as BF16. ggml_tensor * o_out = gemma_attn(ctx, w, as_type(ctx, x_norm, at), positions, cfg, seq, cached_K, cached_V, mask, - nullptr, nullptr, at, use_fused_attention(), x_norm, nullptr, kv, + nullptr, nullptr, at, use_flash_attention(), x_norm, nullptr, kv, fqx ? fqx->K16 : nullptr, fqx ? fqx->V16 : nullptr); o_out = as_type(ctx, o_out, GGML_TYPE_F32); @@ -544,10 +552,7 @@ std::unique_ptr pi05_create(const std::string& mmproj_path, std::printf("vla(pi05): expert weights INT4 (W4A8), %s\n", aex_w4_group > 0 ? ("group " + std::to_string(aex_w4_group)).c_str() : "per channel"); } - // On Hexagon (flash attention on by default) SigLIP takes the plain flash - // attention too: its explicit scores, head dim 72 against 256 keys, are an - // F32 x F32 matmul the HMX does not take, 266 ms of vector-unit time. - m->vis.vit.enc.cfg.flash_attn = use_fused_attention() || (vla::default_flash_attn() && vla::flash_attn_enabled()); + m->vis.vit.enc.cfg.flash_attn = use_flash_attention(); // Fused SigLIP blocks (modules/encoder.h, EncCfg::fused): SYCL, BF16, fused // attention. A FoldQuant GGUF takes them too: its integer sites are the LM's // and the expert's, and its vision tower is BF16 like any other checkpoint's. @@ -840,7 +845,7 @@ std::vector Pi05ModelArch::predict(const Inputs& in) { pl.rms_spec); else h = gemma_layer(C, pl.blk[i], h, t_prefix_pos, cfg, n_prefix_g, - nullptr, nullptr, t_prefix_mask, &cK[i], &cV[i], act_type, use_fused_attention(), + nullptr, nullptr, t_prefix_mask, &cK[i], &cV[i], act_type, use_flash_attention(), kv_inplace ? &kv[i] : nullptr); } (void) h; From 0b78eb9726dc1254832d81a67020b592d13f875f Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 17:20:22 +0700 Subject: [PATCH 08/14] hexagon: KV concats as row copies so they stay on the NPU MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The NPU runs a concat only as a dim-1 copy of whole rows, so the expert's dim-2 KV concats (π0, π0.5, SmolVLA) went to the CPU fallback every step. concat_dim2() in backend.h does them on Hexagon as a dim-1 concat of the [d0*d1, n] views, which writes the same bytes; other backends keep ggml_concat(a, b, 2). IQ-10: π0.5 1379 -> 1133 ms, SmolVLA 810 -> 638 ms, both with outputs identical to before on fixed inputs and on a served observation. --- src/backend.h | 22 ++++++++++++++++++++++ src/models/smolvla.cpp | 4 ++-- src/modules/gemma_expert.h | 4 ++-- 3 files changed, 26 insertions(+), 4 deletions(-) diff --git a/src/backend.h b/src/backend.h index 894f4004..3b50682a 100644 --- a/src/backend.h +++ b/src/backend.h @@ -191,6 +191,28 @@ inline ggml_tensor * fa_kv([[maybe_unused]] ggml_context * ctx, ggml_tensor * t) #endif } +/** + * @brief `ggml_concat(a, b, 2)` of [d0, d1, n] tensors, as the compiled-in + * backend runs it. + * + * Hexagon runs a concat on the NPU only as a dim-1 copy of whole rows; any other + * goes to the CPU fallback (src/backend_fallback.cpp). For contiguous inputs a + * dim-2 concat appends whole d0*d1 rows, so it is done as a dim-1 concat of the + * [d0*d1, n] views, which writes the same bytes. Everywhere else, and for + * non-contiguous inputs, this is ggml_concat(a, b, 2). + */ +inline ggml_tensor * concat_dim2(ggml_context * ctx, ggml_tensor * a, ggml_tensor * b) { +#ifdef GGML_USE_HEXAGON + if (ggml_is_contiguous(a) && ggml_is_contiguous(b) && a->ne[3] == 1 && b->ne[3] == 1 && + a->ne[0] == b->ne[0] && a->ne[1] == b->ne[1]) { + ggml_tensor * rows = ggml_concat(ctx, ggml_reshape_2d(ctx, a, a->ne[0] * a->ne[1], a->ne[2]), + ggml_reshape_2d(ctx, b, b->ne[0] * b->ne[1], b->ne[2]), 1); + return ggml_reshape_3d(ctx, rows, a->ne[0], a->ne[1], a->ne[2] + b->ne[2]); + } +#endif + return ggml_concat(ctx, a, b, 2); +} + /** * @brief A GEMM's inner width as the compiled-in backend's matmul takes it. * diff --git a/src/models/smolvla.cpp b/src/models/smolvla.cpp index 91059d62..95764fd8 100644 --- a/src/models/smolvla.cpp +++ b/src/models/smolvla.cpp @@ -708,8 +708,8 @@ ggml_tensor * build_expert_self_attn_layer( ggml_tensor * k_h = ggml_reshape_3d(ctx, mm_w(ctx, w.Wk, x_norm), cfg.head_dim, cfg.n_kv_heads, cfg.n_suffix); ggml_tensor * v_h = ggml_reshape_3d(ctx, mm_w(ctx, w.Wv, x_norm), cfg.head_dim, cfg.n_kv_heads, cfg.n_suffix); - ggml_tensor * K_full = ggml_concat(ctx, cached_K, rope(ctx, rope_spec(cfg), k_h, positions_full), 2); - ggml_tensor * V_full = ggml_concat(ctx, cached_V, v_h, 2); + ggml_tensor * K_full = concat_dim2(ctx, cached_K, rope(ctx, rope_spec(cfg), k_h, positions_full)); + ggml_tensor * V_full = concat_dim2(ctx, cached_V, v_h); return attn_mlp(ctx, w, x_in, rope(ctx, rope_spec(cfg), q_h, positions_full), K_full, V_full, mask_full, cfg, cfg.n_suffix); } diff --git a/src/modules/gemma_expert.h b/src/modules/gemma_expert.h index 1c66fe3f..e7641166 100644 --- a/src/modules/gemma_expert.h +++ b/src/modules/gemma_expert.h @@ -525,8 +525,8 @@ inline ggml_tensor * gemma_attn( ggml_tensor * K_full = k_rope; ggml_tensor * V_full = v_h; if (cached_K && cached_V) { - K_full = ggml_concat(ctx, cached_K, k_rope, 2); - V_full = ggml_concat(ctx, cached_V, v_h, 2); + K_full = concat_dim2(ctx, cached_K, k_rope); + V_full = concat_dim2(ctx, cached_V, v_h); } const float scale = 1.f/std::sqrt((float) hd); From 26ad84b88bb388c605b3f2e38e893f2a0b7ac332 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Tue, 3 Nov 2026 17:42:54 +0700 Subject: [PATCH 09/14] =?UTF-8?q?docs:=20=CF=800.5=20attention=20and=20KV?= =?UTF-8?q?=20concats=20on=20the=20Hexagon=20NPU,=20IQ-10=20benchmark=20at?= =?UTF-8?q?=200b78eb9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hexagon.md: π0.5's VLM and action expert take flash attention on Hexagon, and the experts' KV concats run as row copies that stay on the NPU (π0 shares the code but has not been run on the board). π0.5's fidelity is re-measured: 4.8e-4 on fixed inputs, 3.5e-4 on a served observation. The IQ-10 benchmark page and its README row are re-measured under TEMPLATE.md at 0b78eb9: π0.5 1117.2 ms, SmolVLA 633.0 ms; the other models are unchanged. The defaults stay the fastest configuration. --- CHANGELOG.md | 3 +++ docs/backend/hexagon.md | 20 +++++++++++---- docs/benchmark/README.md | 4 +-- docs/benchmark/qualcomm-iq10.md | 43 +++++++++++++++------------------ 4 files changed, 40 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3be3ac7b..7a0a0d1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,9 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke MLP is zero-padded to 4320 at load, π0.5's SigLIP takes flash attention, and Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, π0.5 and Evo-1 run 27-40% faster, within the bar on fixed inputs. + π0.5's VLM and action expert take flash attention too, and the experts' KV + concats run as row copies that stay on the NPU (π0.5, SmolVLA; π0 shares the + code, not yet run on the board). GR00T N1.5 and N1.6 are above the bar on a served LIBERO observation, as the CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install and the numbers. diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index c664eb90..2112f061 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -126,6 +126,14 @@ One more op is claimed and then refused at run time. On buffers mapped above `NO_SUPPORT`, and the host aborts on the first predict. The wrapper sends those concats to the CPU. +The experts' KV concats (π0, π0.5, SmolVLA) were such concats: along the token +dim of contiguous [head dim, heads, tokens] tensors, once per layer per denoise +step. Appending along that dim appends whole rows, so on Hexagon +`concat_dim2()` in `src/backend.h` does them as a dim-1 concat of the +[head dim x heads, tokens] views, which writes the same bytes and stays on the +NPU: π0.5 1379 to 1133 ms, SmolVLA 810 to 638 ms, outputs unchanged. π0 shares +the code but has not been run on the board yet. + Flash attention on the HMX (v81) keeps scores and probabilities in F16. That is a precision limit rather than a wrong op, but in GR00T N1.7's LM it moves the action chunk 5.4e-3 from the CPU reference. N1.7's LM therefore takes the @@ -145,9 +153,11 @@ Hexagon builds: - **SigLIP-so400m's MLP is 4304 wide**: 440 ms of GR00T N1.5's 565 ms tower. The loader zero-pads fc1, its bias and fc2 to 4320 (`WeightLoader::pad`, `gemm_width()` in `src/backend.h`), which leaves the output unchanged. -- **π0.5's SigLIP ran explicit attention** (head dim 72, F32 x F32), because - its flash attention was tied to SYCL's view-taking kernel. It now takes the - plain flash attention, as GR00T's tower does. +- **π0.5's attention ran explicit scores** in SigLIP (head dim 72, F32 x F32), + the VLM and the action expert, because its flash attention was tied to SYCL's + view-taking kernel. The VLM's and the expert's single KV head also sent their + matmuls to the CPU (the batched `MUL_MAT` rule above). All three now take the + plain flash attention on Hexagon (`use_flash_attention()` in `pi05.cpp`). - **Evo-1's LM attention multiplied F32 K and V against 14 query heads.** K and V now go to F16 (`fa_kv()`); the scores still accumulate in F32. @@ -308,7 +318,7 @@ path that other builds run: | Model | max\|Δ\| | |---|--:| | SmolVLA | 7.2e-4 | -| π0.5 | 2.8e-4 | +| π0.5 | 4.8e-4 | | Evo-1 | 5.0e-4 | | GR00T N1.5 | 2.4e-3 | | GR00T N1.6 | 1.4e-3 | @@ -330,7 +340,7 @@ malformed or an invalid request, and keeps serving afterwards. | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 | |---|--:|--:| | SmolVLA | 1.3e-3 | | -| π0.5 | 4.2e-4 | | +| π0.5 | 3.5e-4 | | | Evo-1 | 1.1e-3 | | | GR00T N1.5 | 1.1e-2 | 4.3e-2 | | GR00T N1.6 | 1.3e-2 | 1.1e-2 | diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index a2befb19..b3825840 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`74e145d` (same llama.cpp pin). Each report +`0b78eb9` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 613.6 | 706.9 | 746.5 | N/A | 811.5 | n/r | 1789.4 | n/r | 2589.2 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 614.3 | 696.4 | 750.3 | N/A | 633.0 | n/r | 1117.2 | n/r | 2588.6 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 | diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 9415b204..4d843293 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `74e145d` (branch `hexagon-b11223-ops`, on top of #68 to #70) | +| Commit | `0b78eb9` (branch `hexagon-pi05-attn`, on top of #71) | | llama.cpp | `b11223` | -| Date | 2026-11-03 10:43 to 11:02 (UTC+07) | +| Date | 2026-11-03 17:20 to 17:37 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -65,12 +65,12 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.5 | 1 | 224 | 613.6 | 615.7 | 614.8 | 618.5 | 148.2 | 162 | 3644 | -| GR00T N1.6 | 1 | 224 | 706.9 | 710.9 | 710.3 | 715.0 | 145.3 | 147 | 4688 | -| GR00T N1.7 | 1 | 256 | 746.5 | 754.0 | 753.4 | 760.6 | 195.7 | 191 | 5047 | -| SmolVLA | 2 | 512 | 811.5 | 816.3 | 815.9 | 820.2 | 283.1 | 146 | 1234 | -| π0.5 | 2 | 224 | 1789.4 | 1820.3 | 1818.5 | 1850.2 | 296.7 | 285 | 5783 | -| Evo-1 | 1 | 448 | 2589.2 | 2602.0 | 2598.8 | 2613.3 | 853.9 | 319 | 1588 | +| GR00T N1.5 | 1 | 224 | 614.3 | 616.7 | 616.1 | 619.6 | 148.5 | 162 | 3661 | +| SmolVLA | 2 | 512 | 633.0 | 633.5 | 633.4 | 634.0 | 282.1 | 146 | 1193 | +| GR00T N1.6 | 1 | 224 | 696.4 | 701.7 | 700.9 | 706.0 | 144.5 | 147 | 4700 | +| GR00T N1.7 | 1 | 256 | 750.3 | 756.2 | 755.2 | 763.3 | 196.9 | 191 | 5093 | +| π0.5 | 2 | 224 | 1117.2 | 1124.3 | 1123.8 | 1127.1 | 295.3 | 285 | 5786 | +| Evo-1 | 1 | 448 | 2588.6 | 2602.2 | 2600.9 | 2616.2 | 857.1 | 319 | 1573 | ### Fastest configuration @@ -94,23 +94,23 @@ rates measured at the defaults therefore do not carry over to these rows. | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.5 | *(defaults)* | 613.6 | 615.7 | 614.8 | 618.5 | 148.2 | 162 | 3644 | - | -| GR00T N1.6 | *(defaults)* | 706.9 | 710.9 | 710.3 | 715.0 | 145.3 | 147 | 4688 | - | -| GR00T N1.7 | *(defaults)* | 746.5 | 754.0 | 753.4 | 760.6 | 195.7 | 191 | 5047 | - | -| SmolVLA | *(defaults)* | 811.5 | 816.3 | 815.9 | 820.2 | 283.1 | 146 | 1234 | - | -| π0.5 | *(defaults)* | 1789.4 | 1820.3 | 1818.5 | 1850.2 | 296.7 | 285 | 5783 | - | -| Evo-1 | *(defaults)* | 2589.2 | 2602.0 | 2598.8 | 2613.3 | 853.9 | 319 | 1588 | - | +| GR00T N1.5 | *(defaults)* | 614.3 | 616.7 | 616.1 | 619.6 | 148.5 | 162 | 3661 | - | +| SmolVLA | *(defaults)* | 633.0 | 633.5 | 633.4 | 634.0 | 282.1 | 146 | 1193 | - | +| GR00T N1.6 | *(defaults)* | 696.4 | 701.7 | 700.9 | 706.0 | 144.5 | 147 | 4700 | - | +| GR00T N1.7 | *(defaults)* | 750.3 | 756.2 | 755.2 | 763.3 | 196.9 | 191 | 5093 | - | +| π0.5 | *(defaults)* | 1117.2 | 1124.3 | 1123.8 | 1127.1 | 295.3 | 285 | 5786 | - | +| Evo-1 | *(defaults)* | 2588.6 | 2602.2 | 2600.9 | 2616.2 | 857.1 | 319 | 1573 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| GR00T N1.5 | 616.1 | 615.0 | 619.5 | 615.9 | 1096.1 | -| GR00T N1.6 | 716.2 | 698.0 | 700.7 | 696.3 | 873.4 | -| GR00T N1.7 | 753.6 | 733.7 | 764.7 | 731.8 | 1070.6 | -| SmolVLA | 828.8 | 818.5 | 812.5 | 822.8 | 3012.6 | -| π0.5 | 1815.6 | 1816.9 | 11581.7 | 1822.6 | 2144.2 | -| Evo-1 | 2614.6 | 2599.1 | 2606.5 | 2605.6 | 4870.0 | +| GR00T N1.5 | 616.7 | 620.3 | 616.1 | 616.4 | 1095.2 | +| SmolVLA | 634.6 | 634.2 | 634.9 | 634.6 | 2830.8 | +| GR00T N1.6 | 699.9 | 701.5 | 702.9 | 698.8 | 872.8 | +| GR00T N1.7 | 758.4 | 743.5 | 761.2 | 732.3 | 1073.6 | +| π0.5 | 1152.2 | 1151.7 | 1153.3 | 1152.4 | 2127.4 | +| Evo-1 | 2612.8 | 2611.6 | 2600.5 | 2600.4 | 4832.9 |
@@ -131,9 +131,6 @@ GR00T N1.6 runs one view here, as the configuration above fixes. With two views its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). -In the screen, π0.5 with `--mm-prec default` took 11.6 s against 1.8 s at the -defaults. That was not investigated. - These rows time the defaults; they do not vouch for accuracy. On a served LIBERO observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar: N1.5 closer to the reference than the CPU at F16, N1.6 slightly further; see From 1ca86332a973a80f8463701b7593017f2186aa14 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 11:42:34 +0700 Subject: [PATCH 10/14] hexagon: run weight-reading copies on the CPU ggml-hexagon claims a CPY whose source is a weight, then answers the batch with NO_SUPPORT and the host aborts. VLA-Adapter casts its F16 action queries to F32 in the graph and aborted on every predict on the Dragonwing IQ-10. With the cast on the CPU it is within 4.2e-4 of the CPU F32 reference, 4.5e-4 when served. --- src/backend_fallback.cpp | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/backend_fallback.cpp b/src/backend_fallback.cpp index 7e0d4035..a7629eaf 100644 --- a/src/backend_fallback.cpp +++ b/src/backend_fallback.cpp @@ -351,6 +351,14 @@ bool accel_runs(FallbackCtx * fc, const ggml_tensor * t, const char ** why) { *why = "miscomputes a 2D weight against a batch in"; return false; } + if (fc->hexagon && t->op == GGML_OP_CPY && is_weight(t->src[0])) { + // Claimed, but a copy that reads a weight is answered with NO_SUPPORT + // and the host aborts. VLA-Adapter casts its F16 action queries + // [896,64] to F32 in the graph; with the cast on the CPU it is within + // 4.2e-4 of the CPU F32 reference. + *why = "cannot run (NO_SUPPORT) a weight-reading"; + return false; + } return ggml_backend_supports_op(fc->accel, t); } From 1cab1c3c733099b63c0b150dcc7743fd39137da6 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 11:42:34 +0700 Subject: [PATCH 11/14] hexagon: run REPEAT of a computed tensor on the CPU The NPU's REPEAT reads its source through the data cache, which is invalidated only between op batches, while ops write their output by DMA. When the source is written earlier in the same batch, REPEAT can read stale lines. TurboVLA's ViT prefix, repeated over two views, came out 0.19 off the CPU F32 reference and different on every run; with the repeat on the CPU it is 3.1e-4 and repeatable, 2.2e-4 when served. --- src/backend_fallback.cpp | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/backend_fallback.cpp b/src/backend_fallback.cpp index a7629eaf..151692b9 100644 --- a/src/backend_fallback.cpp +++ b/src/backend_fallback.cpp @@ -359,6 +359,15 @@ bool accel_runs(FallbackCtx * fc, const ggml_tensor * t, const char ** why) { *why = "cannot run (NO_SUPPORT) a weight-reading"; return false; } + if (fc->hexagon && t->op == GGML_OP_REPEAT && !is_weight(t->src[0])) { + // The NPU's REPEAT reads its source through the data cache, which is + // invalidated only between op batches, while the op that wrote the + // source used DMA, so within one batch it can read stale lines. + // TurboVLA repeats its ViT prefix [768,5] over two views: max|delta| + // 0.19 and different on every run, 3.1e-4 with the repeat on the CPU. + *why = "can read stale data in a"; + return false; + } return ggml_backend_supports_op(fc->accel, t); } From 6aec07081b6bdcf079b3562a04f6207ab428a925 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 13:50:56 +0700 Subject: [PATCH 12/14] docs: six more archs on the Hexagon NPU, IQ-10 benchmark at 1cab1c3 --- CHANGELOG.md | 33 ++++++++-------- docs/backend/hexagon.md | 70 ++++++++++++++++++++------------- docs/benchmark/qualcomm-iq10.md | 66 +++++++++++++++++++------------ 3 files changed, 101 insertions(+), 68 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7a0a0d1d..5a91d22f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,22 +9,23 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke - **Hexagon NPU on the Qualcomm Dragonwing IQ-10 (Linux).** `scripts/build_linux_hexagon.sh` cross-builds vla.cpp with `GGML_HEXAGON=ON` in Qualcomm's toolchain container, `vla-server` included. On the IQ-10 (v81, - llama.cpp `b11223`) SmolVLA, π0.5, Evo-1 and GR00T N1.5 / N1.6 / N1.7 run - within 2.9e-3 of the CPU build on fixed inputs and serve through - `eval/client`. Concats the NPU refuses on buffers above 4 GB of DSP address - space and matmuls of a 2D weight against a batch, which it computes wrong - (GR00T N1.6 with two views), run on the CPU, and GR00T N1.7's LM leaves - flash attention (F16 scores on the HMX) unless `--flash-attn` is given. - Three matmuls that the HMX does not take move onto it: SigLIP's 4304-wide - MLP is zero-padded to 4320 at load, π0.5's SigLIP takes flash attention, and - Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, π0.5 and Evo-1 run - 27-40% faster, within the bar on fixed inputs. - π0.5's VLM and action expert take flash attention too, and the experts' KV - concats run as row copies that stay on the NPU (π0.5, SmolVLA; π0 shares the - code, not yet run on the board). - GR00T N1.5 and N1.6 are above the bar on a served LIBERO observation, as the - CPU is at F16. See `docs/backend/hexagon.md` for the build, the skel install - and the numbers. + llama.cpp `b11223`) SmolVLA, π0, π0.5, Evo-1, GR00T N1.5 / N1.6 / N1.7, + VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small and TurboVLA run within 2.9e-3 + of the CPU build on fixed inputs and serve through `eval/client`. Concats the + NPU refuses on buffers above 4 GB of DSP address space, copies that read a + weight (also answered `NO_SUPPORT`; VLA-Adapter), matmuls of a 2D weight + against a batch, which it computes wrong (GR00T N1.6 with two views), and + repeats of a computed tensor, which can read stale cache lines (TurboVLA), run + on the CPU, and GR00T N1.7's LM leaves flash attention (F16 scores on the + HMX) unless `--flash-attn` is given. Three matmuls that the HMX does not take + move onto it: SigLIP's 4304-wide MLP is zero-padded to 4320 at load, π0.5's + SigLIP takes flash attention, and Evo-1's LM attention takes F16 K/V. GR00T N1.5, N1.6, + π0.5 and Evo-1 run 27-40% faster, within the bar on fixed inputs. π0.5's VLM + and action expert take flash attention too, and the experts' KV concats run as + row copies that stay on the NPU (π0, π0.5, SmolVLA). GR00T N1.5, N1.6 and + VLA-JEPA are above the bar on a served LIBERO observation, as the CPU is at + F16. See `docs/backend/hexagon.md` for the build, the skel install and the + numbers. - **ROCm/HIP backend for VLA inference.** `-DGGML_HIP=ON` selects ggml's HIP backend without compiling vla.cpp's CUDA-only kernels. On Linux `gfx1200` (Radeon RX 9060 XT) with ROCm 7.14 and llama.cpp `b11223`, every arch except diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 2112f061..213e22e4 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -112,6 +112,7 @@ these, and they compute something else: | `ADD`/`MUL`/... | `src1` broadcast over one dim but not a higher one | Refused, runs on the CPU | | `IM2COL` | Wrong unless kernel = stride > 1 and no padding | Refused, runs on the CPU | | `CPY` F32→F16 | Reads back wrong on the host | Copies feeding a CPU op run on the CPU | +| `REPEAT` | Reads stale cache lines of a source written by DMA earlier in the same op batch | Refused unless the source is a weight, runs on the CPU | | fused `MUL_MAT+ADD`, `RMS_NORM+MUL` | ~10x the error of the unfused ops | `GGML_HEXAGON_OPFUSION=0` by default | Each one flips the gripper channel, or comes close, on some checkpoint. The @@ -120,19 +121,21 @@ was right: a missing op fails at load, and these only show in the actions. [hexagon-windows.md](hexagon-windows.md#hexagon-issues-found-and-worked-around) has the numbers and how each was bisected. They belong upstream. -One more op is claimed and then refused at run time. On buffers mapped above +Two more ops are claimed and then refused at run time. On buffers mapped above 4 GB of DSP address space, which the IQ-10 under Linux hands out, the NPU runs `CONCAT` only as the dim-1 contiguous DMA copy and answers any other concat with `NO_SUPPORT`, and the host aborts on the first predict. The wrapper sends those -concats to the CPU. +concats to the CPU. A `CPY` that reads a weight gets the same answer +(VLA-Adapter casts its F16 action queries to F32 in the graph), and also runs on +the CPU. The experts' KV concats (π0, π0.5, SmolVLA) were such concats: along the token dim of contiguous [head dim, heads, tokens] tensors, once per layer per denoise step. Appending along that dim appends whole rows, so on Hexagon `concat_dim2()` in `src/backend.h` does them as a dim-1 concat of the [head dim x heads, tokens] views, which writes the same bytes and stays on the -NPU: π0.5 1379 to 1133 ms, SmolVLA 810 to 638 ms, outputs unchanged. π0 shares -the code but has not been run on the board yet. +NPU: π0.5 1379 to 1133 ms, SmolVLA 810 to 638 ms, outputs unchanged. π0 takes +the same path. Flash attention on the HMX (v81) keeps scores and probabilities in F16. That is a precision limit rather than a wrong op, but in GR00T N1.7's LM it moves the @@ -313,29 +316,36 @@ export LD_LIBRARY_PATH=$PWD/pkg/lib `vla_predict_check` at the Hexagon defaults, against the same binary with `VLA_DEVICE=cpu --weight-dtype f32 --flash-attn 0`. A Hexagon build turns flash attention on by default even on the CPU, so the reference asks for the explicit -path that other builds run: +path that other builds run. Views are the ones the LIBERO client sends, so they +differ from the benchmark's for VLA-JEPA, VLA-Adapter, OpenVLA-OFT and Octo: -| Model | max\|Δ\| | -|---|--:| -| SmolVLA | 7.2e-4 | -| π0.5 | 4.8e-4 | -| Evo-1 | 5.0e-4 | -| GR00T N1.5 | 2.4e-3 | -| GR00T N1.6 | 1.4e-3 | -| GR00T N1.7 | 2.0e-3 | - -GR00T N1.6 here takes one view; with two it is 1.9e-3. The other archs have -not been run on the board yet. +| Model | Views | max\|Δ\| | +|---|--:|--:| +| SmolVLA | 2 | 7.2e-4 | +| π0 | 2 | 1.2e-3 | +| π0.5 | 2 | 4.8e-4 | +| Evo-1 | 1 | 5.5e-4 | +| GR00T N1.5 | 1 | 2.4e-3 | +| GR00T N1.6 | 1 | 1.4e-3 | +| GR00T N1.7 | 1 | 2.0e-3 | +| VLA-JEPA | 2 | 2.8e-3 | +| VLA-Adapter | 2 | 4.2e-4 | +| OpenVLA-OFT | 2 | 3.5e-4 | +| Octo-Small | 1 | 4.1e-4 | +| TurboVLA | 2 | 3.1e-4 | + +GR00T N1.6 with two views is 1.9e-3. At the benchmark's single view VLA-JEPA is +1.9e-3, VLA-Adapter 4.8e-4 and OpenVLA-OFT 2.2e-4. ### Serving `vla-server` on the board, driven from the host by `eval/client/vla_cpp_client.py` and the adapters `run_sim_client_direct.py` -uses, on one fixed LIBERO observation (two views, the task prompt, a fixed -robot state) with fixed noise. The same request goes to a Hexagon server and to -a CPU server with `--weight-dtype f32 --flash-attn 0`. Every model returns a -finite chunk, the same chunk for a repeated request, an error reply to a -malformed or an invalid request, and keeps serving afterwards. +uses, on one fixed LIBERO observation (two views, the task prompt, a fixed robot +state), with fixed noise for the archs that draw it. The same request goes to a +Hexagon server and to a CPU server with `--weight-dtype f32 --flash-attn 0`. +Every model returns a finite chunk, the same chunk for a repeated request, an +error reply to a malformed or an invalid request, and keeps serving afterwards. | Model | max\|Δ\| vs CPU F32 | CPU F16 vs CPU F32 | |---|--:|--:| @@ -345,12 +355,18 @@ malformed or an invalid request, and keeps serving afterwards. | GR00T N1.5 | 1.1e-2 | 4.3e-2 | | GR00T N1.6 | 1.3e-2 | 1.1e-2 | | GR00T N1.7 | 2.5e-3 | | - -GR00T N1.5 and N1.6 are above the bar on this input, and so is the CPU itself -with F16 weights: N1.5 lands closer to the reference than the CPU at F16, N1.6 -slightly further (1.3e-2 against 1.1e-2). They are sensitive to 16-bit -precision rather than miscomputed. Running them with F32 weights on the NPU has not been tried -yet. +| π0 | 5.8e-4 | | +| VLA-JEPA | 6.7e-3 | 2.1e-2 | +| VLA-Adapter | 4.5e-4 | | +| OpenVLA-OFT | 5.0e-4 | | +| Octo-Small | 9.3e-5 | | +| TurboVLA | 2.2e-4 | | + +GR00T N1.5, N1.6 and VLA-JEPA are above the bar on this input, and so is the CPU +itself with F16 weights: N1.5 and VLA-JEPA land closer to the reference than the +CPU at F16, N1.6 slightly further (1.3e-2 against 1.1e-2). They are sensitive to +16-bit precision rather than miscomputed. Running them with F32 weights on the +NPU has not been tried yet. ## IQ-10, imagined diff --git a/docs/benchmark/qualcomm-iq10.md b/docs/benchmark/qualcomm-iq10.md index 4d843293..6dc1d5c8 100644 --- a/docs/benchmark/qualcomm-iq10.md +++ b/docs/benchmark/qualcomm-iq10.md @@ -15,9 +15,9 @@ through vla.cpp's fallback wrapper. | Power | CPU governor `performance` | | OS | Ubuntu 24.04.4 LTS, kernel 6.6.110 | | Toolchain | `ghcr.io/snapdragon-toolchain/arm64-linux:v0.7`: clang 21.1.8, Hexagon SDK 6.6.0.0 (tools 19.0.07) | -| Commit | `0b78eb9` (branch `hexagon-pi05-attn`, on top of #71) | +| Commit | `1cab1c3` | | llama.cpp | `b11223` | -| Date | 2026-11-03 17:20 to 17:37 (UTC+07) | +| Date | 2026-11-04 11:52 to 12:30 (UTC+07) | | Build | `scripts/build_linux_hexagon.sh` (servers included); the HTP skel processed for the board and installed in `/vendor/dsp/cdsp0..3` | | Runtime flags | defaults; the fastest flags per model are in the second table | | Method | 3 warmups + 20 timed reps per process, 3 processes. The process with the lowest mean is reported, and memory is the peak over all three. | @@ -65,12 +65,18 @@ vision stage separately. | Model | Views | Input | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | |---|--:|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.5 | 1 | 224 | 614.3 | 616.7 | 616.1 | 619.6 | 148.5 | 162 | 3661 | -| SmolVLA | 2 | 512 | 633.0 | 633.5 | 633.4 | 634.0 | 282.1 | 146 | 1193 | -| GR00T N1.6 | 1 | 224 | 696.4 | 701.7 | 700.9 | 706.0 | 144.5 | 147 | 4700 | -| GR00T N1.7 | 1 | 256 | 750.3 | 756.2 | 755.2 | 763.3 | 196.9 | 191 | 5093 | -| π0.5 | 2 | 224 | 1117.2 | 1124.3 | 1123.8 | 1127.1 | 295.3 | 285 | 5786 | -| Evo-1 | 1 | 448 | 2588.6 | 2602.2 | 2600.9 | 2616.2 | 857.1 | 319 | 1573 | +| Octo-Small | 2 | 256 + 128 | 121.6 | 123.0 | 122.3 | 125.8 | 20.5 | 253 | 804 | +| TurboVLA | 2 | 256 | 428.1 | 433.0 | 431.6 | 436.8 | - | 438 | 1327 | +| VLA-JEPA | 1 | 256 | 440.2 | 446.3 | 444.7 | 450.2 | 195.2 | 185 | 3971 | +| GR00T N1.5 | 1 | 224 | 613.7 | 615.7 | 615.7 | 618.0 | 148.3 | 162 | 3636 | +| SmolVLA | 2 | 512 | 631.0 | 632.4 | 631.7 | 634.6 | 281.8 | 146 | 1190 | +| GR00T N1.6 | 1 | 224 | 705.8 | 708.5 | 708.0 | 710.7 | 145.0 | 146 | 4869 | +| GR00T N1.7 | 1 | 256 | 747.0 | 756.0 | 755.0 | 764.5 | 197.2 | 191 | 5090 | +| π0.5 | 2 | 224 | 1117.4 | 1121.0 | 1120.8 | 1123.3 | 292.5 | 285 | 5790 | +| π0 | 2 | 224 | 1124.4 | 1128.7 | 1128.6 | 1130.5 | 293.0 | 285 | 5534 | +| VLA-Adapter | 1 | 224 | 1733.6 | 1752.3 | 1749.5 | 1765.6 | 1187.2 | 872 | 3566 | +| Evo-1 | 1 | 448 | 2579.1 | 2595.5 | 2592.1 | 2611.9 | 854.0 | 319 | 1551 | +| OpenVLA-OFT | 1 | 224 | 4097.7 | 4107.8 | 4107.8 | 4114.2 | 1187.3 | 844 | 15327 | ### Fastest configuration @@ -94,23 +100,35 @@ rates measured at the defaults therefore do not carry over to these rows. | Model | Fastest flags | min ms | mean ms | p50 ms | p90 ms | vision ms | Peak RSS MiB | System RAM MiB | vs defaults | |---|---|--:|--:|--:|--:|--:|--:|--:|--:| -| GR00T N1.5 | *(defaults)* | 614.3 | 616.7 | 616.1 | 619.6 | 148.5 | 162 | 3661 | - | -| SmolVLA | *(defaults)* | 633.0 | 633.5 | 633.4 | 634.0 | 282.1 | 146 | 1193 | - | -| GR00T N1.6 | *(defaults)* | 696.4 | 701.7 | 700.9 | 706.0 | 144.5 | 147 | 4700 | - | -| GR00T N1.7 | *(defaults)* | 750.3 | 756.2 | 755.2 | 763.3 | 196.9 | 191 | 5093 | - | -| π0.5 | *(defaults)* | 1117.2 | 1124.3 | 1123.8 | 1127.1 | 295.3 | 285 | 5786 | - | -| Evo-1 | *(defaults)* | 2588.6 | 2602.2 | 2600.9 | 2616.2 | 857.1 | 319 | 1573 | - | +| Octo-Small | *(defaults)* | 121.6 | 123.0 | 122.3 | 125.8 | 20.5 | 253 | 804 | - | +| TurboVLA | *(defaults)* | 428.1 | 433.0 | 431.6 | 436.8 | - | 438 | 1327 | - | +| VLA-JEPA | *(defaults)* | 440.2 | 446.3 | 444.7 | 450.2 | 195.2 | 185 | 3971 | - | +| GR00T N1.5 | *(defaults)* | 613.7 | 615.7 | 615.7 | 618.0 | 148.3 | 162 | 3636 | - | +| SmolVLA | *(defaults)* | 631.0 | 632.4 | 631.7 | 634.6 | 281.8 | 146 | 1190 | - | +| GR00T N1.6 | *(defaults)* | 705.8 | 708.5 | 708.0 | 710.7 | 145.0 | 146 | 4869 | - | +| GR00T N1.7 | *(defaults)* | 747.0 | 756.0 | 755.0 | 764.5 | 197.2 | 191 | 5090 | - | +| π0.5 | *(defaults)* | 1117.4 | 1121.0 | 1120.8 | 1123.3 | 292.5 | 285 | 5790 | - | +| π0 | *(defaults)* | 1124.4 | 1128.7 | 1128.6 | 1130.5 | 293.0 | 285 | 5534 | - | +| VLA-Adapter | *(defaults)* | 1733.6 | 1752.3 | 1749.5 | 1765.6 | 1187.2 | 872 | 3566 | - | +| Evo-1 | *(defaults)* | 2579.1 | 2595.5 | 2592.1 | 2611.9 | 854.0 | 319 | 1551 | - | +| OpenVLA-OFT | *(defaults)* | 4097.7 | 4107.8 | 4107.8 | 4114.2 | 1187.3 | 844 | 15327 | - |
Screen means (ms) per flag set | Model | defaults | `--flash-attn` | `--mm-prec default` | `--flash-attn --mm-prec default` | `--flash-attn 0` | |---|--:|--:|--:|--:|--:| -| GR00T N1.5 | 616.7 | 620.3 | 616.1 | 616.4 | 1095.2 | -| SmolVLA | 634.6 | 634.2 | 634.9 | 634.6 | 2830.8 | -| GR00T N1.6 | 699.9 | 701.5 | 702.9 | 698.8 | 872.8 | -| GR00T N1.7 | 758.4 | 743.5 | 761.2 | 732.3 | 1073.6 | -| π0.5 | 1152.2 | 1151.7 | 1153.3 | 1152.4 | 2127.4 | -| Evo-1 | 2612.8 | 2611.6 | 2600.5 | 2600.4 | 4832.9 | +| Octo-Small | 125.3 | 128.8 | 124.4 | 123.5 | 124.6 | +| TurboVLA | 438.8 | 436.5 | 439.3 | 438.2 | 535.1 | +| VLA-JEPA | 446.7 | 445.8 | 444.8 | 445.2 | 817.8 | +| GR00T N1.5 | 615.8 | 619.3 | 615.3 | 618.4 | 1096.5 | +| SmolVLA | 632.1 | 633.0 | 632.5 | 632.6 | 2834.8 | +| GR00T N1.6 | 714.2 | 712.1 | 718.7 | 714.9 | 885.7 | +| GR00T N1.7 | 765.2 | 742.0 | 761.5 | 743.6 | 1081.0 | +| π0.5 | 1121.1 | 1121.9 | 1120.5 | 1121.4 | 2062.9 | +| π0 | 1129.8 | 1128.8 | 1131.8 | 1128.5 | 2090.5 | +| VLA-Adapter | 1766.1 | 1760.1 | 1756.1 | 1757.2 | 1767.1 | +| Evo-1 | 2613.0 | 2613.7 | 2617.3 | 2617.9 | 4840.1 | +| OpenVLA-OFT | 4110.3 | 4101.9 | 4101.7 | 4108.2 | 4107.9 |
@@ -118,8 +136,6 @@ rates measured at the defaults therefore do not carry over to these rows. - **BitVLA**: N/A off CUDA. The only published GGUF is int2-packed, and only CUDA builds load it. -- **π0, VLA-JEPA, VLA-Adapter, OpenVLA-OFT, Octo-Small, TurboVLA**: not run in - this round; their GGUFs were not on the board. ## Notes @@ -132,9 +148,9 @@ its SigLIP2 tower hits the batched `MUL_MAT` the wrapper sends to the CPU; see [backend/hexagon.md](../backend/hexagon.md#op-coverage-measured). These rows time the defaults; they do not vouch for accuracy. On a served LIBERO -observation, GR00T N1.5 and N1.6 land above the 2.9e-3 fidelity bar: N1.5 -closer to the reference than the CPU at F16, N1.6 slightly further; see -[backend/hexagon.md](../backend/hexagon.md#serving). +observation, GR00T N1.5, N1.6 and VLA-JEPA land above the 2.9e-3 fidelity bar: +N1.5 and VLA-JEPA closer to the reference than the CPU at F16, N1.6 slightly +further; see [backend/hexagon.md](../backend/hexagon.md#serving). ## Reproducing From 9d49a3690972030ee6adce283f8a903cef4ef523 Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 14:09:52 +0700 Subject: [PATCH 13/14] encoder: leave an MLP unpadded when its weights are quantized in the file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pad_mlp() checked only the loader's GEMM type. scripts/quantize_gguf.py --vision packs SigLIP's fc1 [1152, 4304] to Q8_0 while fc2, 4304 wide, stays float; the loader keeps fc1 packed and declare() refused to pad it, so such a GGUF failed to load in Hexagon builds. An MLP with either weight quantized in the file now loads unpadded, as it did before the padding. Float checkpoints are unchanged: π0, π0.5, GR00T N1.5/N1.6/N1.7, VLA-Adapter and OpenVLA-OFT give bit-identical actions on the Dragonwing IQ-10. --- src/modules/encoder.cpp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/modules/encoder.cpp b/src/modules/encoder.cpp index 5c6fa428..458436ed 100644 --- a/src/modules/encoder.cpp +++ b/src/modules/encoder.cpp @@ -37,7 +37,8 @@ namespace { void pad_mlp(WeightLoader & L, const std::string & fc1, const std::string & fc2) { const ggml_tensor * w1 = L.reader().meta((fc1 + ".weight").c_str()); const ggml_tensor * w2 = L.reader().meta((fc2 + ".weight").c_str()); - if (!w1 || !w2 || ggml_is_quantized(L.gemm_type())) + if (!w1 || !w2 || ggml_is_quantized(L.gemm_type()) || ggml_is_quantized(w1->type) || + ggml_is_quantized(w2->type)) return; const int64_t ff = w1->ne[1]; const int64_t fp = gemm_width(ff); From 7dc357119ca684b9c094be1e5d80106ea7cd6abc Mon Sep 17 00:00:00 2001 From: thanhdq1o2 Date: Wed, 4 Nov 2026 16:32:41 +0700 Subject: [PATCH 14/14] docs: list the twelve IQ-10 archs in the Hexagon status and benchmark summary The previous docs commit added six archs to the IQ-10 pages but left two summaries behind: the status note in docs/backend/hexagon.md still said six checkpoints run on the IQ-10, and the IQ-10 row of docs/benchmark/README.md still carried the 0b78eb9 numbers with the six new archs marked not run. Both now match docs/benchmark/qualcomm-iq10.md at 1cab1c3. --- docs/backend/hexagon.md | 6 +++--- docs/benchmark/README.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/backend/hexagon.md b/docs/backend/hexagon.md index 213e22e4..d9196bb4 100644 --- a/docs/backend/hexagon.md +++ b/docs/backend/hexagon.md @@ -5,9 +5,9 @@ to take once it ran. > **Status: running on Windows on Arm and on a Dragonwing IQ-10 under Linux.** > vla.cpp builds with `GGML_HEXAGON=ON` and runs eleven checkpoints on a -> Snapdragon X laptop's v73 NPU and six on an IQ-10's v81 NPU under Linux, each -> checked against a CPU reference. The laptop build, its measurements and the -> upstream bugs found along the way are in +> Snapdragon X laptop's v73 NPU and twelve on an IQ-10's v81 NPU under Linux, +> each checked against a CPU reference. The laptop build, its measurements and +> the upstream bugs found along the way are in > [hexagon-windows.md](hexagon-windows.md); the Linux board is in [Linux > (IQ-10)](#linux-iq-10). What follows is the backend in general: the parts that > hold on any Snapdragon part, and the IQ-9 and multi-NPU notes from before diff --git a/docs/benchmark/README.md b/docs/benchmark/README.md index b3825840..54f0f3c5 100644 --- a/docs/benchmark/README.md +++ b/docs/benchmark/README.md @@ -12,7 +12,7 @@ AGX Orin row ran at `e6653e8`, on the bit-exact CUDA path ran at `d0e5d00`, PR #35 merged with main `f464653` (same llama.cpp pin), with fixed-input checks against the CPU build in [backend/rocm.md](../backend/rocm.md). The Qualcomm Dragonwing IQ-10 row ran at -`0b78eb9` (same llama.cpp pin). Each report +`1cab1c3` (same llama.cpp pin). Each report lists the device, the build, each model's configuration, latency and peak memory at the defaults, and the fastest runtime flags for that device. @@ -36,7 +36,7 @@ means the model does not fit or fails there; each report says which and why. | [Intel Arc B390 iGPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 35.6 | 152.4 | 232.6 | 145.5 | 225.1 | 230.8 | N/A | 499.8 | 795.8 | 581.5 | 1494.4 | 841.5 | | [Intel Arc A380 (SYCL)](arc-a380.md) | 60.5 | 103.5 | 355.9 | 408.9 | 472.0 | 646.0 | 615.3 | N/A | 734.5 | 921.9 | 945.6 | - | 1047.3 | | [Snapdragon X Hexagon NPU](snapdragon-x-hexagon.md) | 239.7 | 2280.8 | 821.0 | 4072.1 | 2755.7 | 2940.0 | 1400.2 | N/A | 1239.3 | 5730.0 | 9321.7 | - | 7208.6 | -| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | n/r | n/r | n/r | n/r | 614.3 | 696.4 | 750.3 | N/A | 633.0 | n/r | 1117.2 | n/r | 2588.6 | +| [Qualcomm Dragonwing IQ-10 Hexagon NPU](qualcomm-iq10.md) | 121.6 | 428.1 | 440.2 | 1733.6 | 613.7 | 705.8 | 747.0 | N/A | 631.0 | 1124.4 | 1117.4 | 4097.7 | 2579.1 | | [Intel AI Boost NPU (OpenVINO)](core-ultra-x7-358h.md) | N/A | 63.8 | - | 316.0 | - | - | - | N/A | 1656.5 | 836.6 | - | 468.7 | - | | [Intel Core i7-14700F (CPU)](core-i7-14700f.md) | 39.8 | 195.1 | 825.5 | 1140.6 | 1913.8 | 1627.9 | 1453.4 | N/A | 1689.2 | 6455.7 | 6068.1 | 9869.0 | 4159.6 | | [Intel Core i9-14900HX (CPU)](core-i9-14900hx.md) | 67.0 | 314.7 | 1227.2 | 1659.3 | 1935.4 | 1688.9 | 1505.5 | N/A | 1822.6 | 6182.1 | 5735.0 | 9126.4 | 4118.2 |