Repository navigation
hexagon: GR00T N1.6 within the fidelity bar when served - #80
Open
thanhdq1o2 wants to merge 5 commits into
Open
thanhdq1o2 wants to merge 5 commits into
thanhdq1o2 wants to merge 5 commits into
Conversation
GEMM weights declared while a GemmScope lives take its type instead of the loader's, and the loader's type comes back when it ends. An arch can then keep one module in a different resident type from the rest.
fallback_keep_on_cpu() registers a weight whose products the accelerator computes too coarsely for an arch; accel_runs() then leaves every MUL_MAT that reads it to the CPU. Hexagon's matrix unit rounds the activations to F16 whatever the weight type, so a weight registered in F32 gets the CPU F32 product. backend.h adds hmx_f16_activations() and keep_matmul_on_cpu(), which does nothing in other builds.
Hexagon's matrix unit multiplies with F16 activations, and in SigLIP2's first blocks that rounding grows through the tower: a served LIBERO observation lands 1.0e-2 from a CPU-only F32 build on the Dragonwing IQ-10, above the 2.9e-3 bar. F32 weights alone leave it at 5.4e-3. Unless --weight-dtype is given, the tower and its projector now keep F32 weights on Hexagon and the first three blocks' matmuls run on the CPU. Other backends are unchanged. GR00T N1.6 goes from 1.0e-2 to 8.9e-4 served, 9.4e-4 and 5.0e-4 on fixed inputs (one and two views), at 689.5 to 739.7 ms (mean, vla-bench). The other archs give the same actions.
hexagon.md: the HMX rounds activations to F16 whatever the weight type; GR00T N1.6's SigLIP2 tower keeps F32 weights and its first three blocks' matmuls on the CPU, 1.0e-2 to 8.9e-4 served. GR00T N1.5 spreads the same rounding over the model: no single module on the CPU brings it under the bar, only every matmul does. Fidelity and serving tables updated. qualcomm-iq10.md and the summary row at f0b12ae: GR00T N1.6 682.1 to 733.7 ms, the other archs within run-to-run noise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hexagon's matrix unit (HMX) multiplies with F16 activations whatever the weight type. In GR00T N1.6's SigLIP2 tower that rounding grows from the first blocks, and a served LIBERO observation landed 1.0e-2 from a CPU-only F32 build of the same commit, above the 2.9e-3 bar. This PR lets an arch keep chosen weights' matmuls on the CPU, and uses it for N1.6: on Hexagon the tower and its projector keep F32 weights and the first three blocks' matmuls run on the CPU. Other backends are unchanged.
Changes
c6277edWeightLoader::GemmScope: GEMM weights declared while it lives take another resident typefd3d02bfallback_keep_on_cpu(): the fallback wrapper leaves everyMUL_MATthat reads a registered weight to the CPU.backend.haddshmx_f16_activations()andkeep_matmul_on_cpu(), a no-op in other buildsf0b12ae--weight-dtypeis given634e8a1hexagon.md, CHANGELOG, IQ-10 benchmark atf0b12aeGR00T N1.6 on the Dragonwing IQ-10
vla-benchmean, TEMPLATE.mdThe reference is a CPU-only build (
-DGGML_HEXAGON=OFF) of the same commit with--weight-dtype f32 --flash-attn 0.Verified
vla-server(Async serving: ROUTER socket, predict thread and latest-wins request queue #34): a finite chunk, the same chunk for a repeated request, an error reply to a malformed and to an invalid request, and serving continues. GR00T N1.5 stays above the bar there (1.2e-2).ctestpasses (11/11).Tried and left out