feat(compiled): serve GGUFs written by ggml-compiler through their vla manifest

.github/ISSUE_TEMPLATE/model_request.md

@@ -16,5 +16,5 @@ Where the weights live and under what license.
 
 Benchmark numbers, or what it does that the supported archs do not.
 
-CONTRIBUTING.md has the six-site walkthrough if you want to send the port
-yourself.
+docs/COMPILED.md shows how to serve a model compiled by ggml-compiler without
+a port; docs/ARCHITECTURE.md covers a hand-written one.

CHANGELOG.md

@@ -14,6 +14,15 @@ Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://ke
   `scripts/convert_cosmos3_policy_to_gguf.py` converts the checkpoint. Agrees with lerobot's policy in bf16 to
   2e-3 to 2e-2 on the action chunk; the client preset `cosmos3_policy` and `eval/run_libero.sh -m cosmos3_policy`
   serve it. CPU and CUDA only.
+- **Models compiled by ggml-compiler.** A GGUF written by `ggml-compiler
+  compile` that carries a `vla` serving manifest loads as the `compiled` arch
+  and serves through `vla-server`, `vla-cli`, `vla-bench` and `libvla` with no
+  model-specific code: the manifest names each program input's role (camera,
+  instruction, state, noise) and preprocessing, and the actions output with its
+  statistics. ggml-compiler's runtime is built in-tree against vla.cpp's own
+  ggml (`VLA_COMPILED`, default ON), so compiled models run on every backend
+  vla.cpp builds. First version: fixed shapes, one entry point, token ids from
+  the client. [docs/COMPILED.md](docs/COMPILED.md).
 - **Hexagon NPU on the Qualcomm Dragonwing IQ-10 (Linux).**
   `scripts/build_linux_hexagon.sh` cross-builds vla.cpp with `GGML_HEXAGON=ON`
   in Qualcomm's toolchain container, `vla-server` included. On the IQ-10 (v81,

CMakeLists.txt

@@ -159,6 +159,25 @@ function(vla_exclude_fetched_targets dir)
 endfunction()
 vla_exclude_fetched_targets(${llama_SOURCE_DIR})
 
+# Models compiled by ggml-compiler run from their GGUF through its runtime,
+# built here against this tree's ggml so one process holds one ggml (its
+# backends included). The runtime is pinned to a ggml-compiler commit and
+# checks that our llama.cpp tag is the one it was tested with.
+option(VLA_COMPILED "Serve GGUFs written by ggml-compiler (fetches its runtime)" ON)
+if(VLA_COMPILED)
+    set(VLA_GGML_COMPILER_TAG "f72f90189dff555fca70b3a43c1a6d5aa89dac16" CACHE STRING "ggml-compiler git ref providing the runtime")
+    set(GGML_COMPILER_EXTERNAL_GGML     ON                 CACHE BOOL   "" FORCE)
+    set(GGML_COMPILER_GGML_SOURCE_DIR   ${llama_SOURCE_DIR} CACHE PATH   "" FORCE)
+    set(GGML_COMPILER_EXTERNAL_GGML_TAG ${VLA_LLAMA_TAG}    CACHE STRING "" FORCE)
+    set(GGML_COMPILER_BUILD_TOOLS       OFF                CACHE BOOL   "" FORCE)
+    FetchContent_Declare(ggml_compiler
+        GIT_REPOSITORY https://github.com/khanhnd61-vr/ggml-compiler
+        GIT_TAG        ${VLA_GGML_COMPILER_TAG}
+        SOURCE_SUBDIR  runtime
+    )
+    FetchContent_MakeAvailable(ggml_compiler)
+endif()
+
 # SentencePiece, and through it a protobuf, is only needed to tokenize --text with
 # a tokenizer stored in the GGUF. Optional so a build without it has no protobuf.
 option(VLA_SPM "Link SentencePiece for in-GGUF tokenizers (needs system protobuf)" ON)
@@ -269,6 +288,11 @@ target_include_directories(vla_core
 )
 # The VLA archs call no llama_* API; only vlm_core needs llama.
 target_link_libraries(vla_core PUBLIC ggml)
+if(VLA_COMPILED)
+    target_sources(vla_core PRIVATE src/models/compiled.cpp)
+    target_link_libraries(vla_core PUBLIC ggml_compiler_rt)
+    target_compile_definitions(vla_core PUBLIC VLA_COMPILED)
+endif()
 # The FoldQuant CPU reference must match the CUDA and SYCL kernels bit for bit,
 # so no FMA contraction on either side (aarch64 GCC contracts by default; under
 # icpx, after the project-wide -fp-model=precise, which would allow it).
@@ -630,6 +654,9 @@ install(FILES LICENSE.md ${llama_SOURCE_DIR}/licenses/LICENSE-jsonhpp
 install(FILES ${llama_SOURCE_DIR}/LICENSE DESTINATION ${CMAKE_INSTALL_DOCDIR}
     RENAME LICENSE.llama.cpp COMPONENT libvla)
 install(TARGETS vla_core vla-cli vla-bench)
+if(VLA_COMPILED)
+    install(TARGETS ggml_compiler_rt)
+endif()
 install(FILES scripts/tokenize_prompt.py DESTINATION ${CMAKE_INSTALL_DATADIR}/vla)
 if(VLA_BUILD_SERVER)
     install(TARGETS vlm_core vla-server vlm-server)

CONTRIBUTING.md

@@ -42,35 +42,76 @@ the binaries fetch them:
 ./build/vla-cli -hf vrfai/smolvla-libero-gguf --image assets/front.jpg --tokens 1,100,2
 ```
 
-## Adding an architecture
-
-Seven sites, all mechanical. Every arch loads one GGUF with its vision tower
-bundled; `mmproj_path` is accepted and ignored.
-
-1. `src/arch.h` - add to `enum class Arch`.
-2. `src/arch.h` - declare `<name>_create(mmproj_path, ckpt_path, config_path, opts)`.
-3. `src/model.cpp` - add `<name>.architecture` to the `try_str` list in
-   `detect_arch_gguf`.
-4. `src/model.cpp` - map the string to the enum in the same function.
-5. `src/model.cpp` - add a `case` to the `model_load` switch.
-6. `src/model.cpp` - if the arch has no flow-matching solver, add it to the
-   `opts.num_steps` refusal list in `model_load`; if it has one, read the step
-   count with `resolve_num_steps`.
-7. `CMakeLists.txt` - add `src/models/<name>.cpp` to `vla_core`.
-
-Then write `src/models/<name>.cpp`. Before adding a helper, check `src/`:
-`gguf_reader.h` (tensor and KV reads), `loader.h` (weight upload and fusion),
-`modules/preprocess.h` (view checks, CHW preprocessing), `modules/dual_tower.h`
-(DINOv2 + SigLIP), `modules/qwen3vl_vit.h` (Qwen3-VL tower), `layers/` (attention,
-FFN, norms, RoPE, time embeddings, causal mask), `scratch_ctx.h` (compute context
-and graph reuse), `backend.h` (accelerator selection).
-
-Your loader must fail rather than return a half-built model: check every tensor
-lookup, and check `real_*_dim <= max_*_dim` (`config_is_sane` in `src/model.cpp`
-does this for all archs).
-
-A converter goes in `scripts/convert_<name>_to_gguf.py`, and its tensor-name
-remap should be covered by `tests/py/test_converters.py`.
+## Benchmarking new hardware
+
+New models are not hand-written into this tree (see
+[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md#adding-an-architecture)); new
+devices are. A benchmark report is one file,
+`docs/benchmark/<device>.md`, written from
+[docs/benchmark/TEMPLATE.md](docs/benchmark/TEMPLATE.md). The template fixes
+the section and column order so that reports compare cell for cell; this is
+the procedure that fills it.
+
+1. **Build once, at one commit.** Configure a Release build of `main` with the
+   backend's flags and run the whole round from it. If the code changes, start
+   the round over. The report records the short SHA, the llama.cpp tag
+   (`scripts/llama_tag.sh`), the cmake flags including `GGML_NATIVE`, the
+   driver or toolkit version and the compiler (`scripts/print_versions.sh`
+   prints a block), the OS and kernel, and the power state: limits, governor
+   or profile, AC or battery, `nvpmodel` on Jetson.
+
+2. **Fetch the checkpoints.** Every report runs the `vrfai/*` GGUFs in the
+   template's "Model configuration" table, at the views, input size, token
+   count and steps given there. `-hf vrfai/<repo>` fetches them; check each
+   file's sha256 against the one Hugging Face lists, so every device runs
+   identical weights.
+
+3. **Measure the defaults.** One process per model, run three times:
+
+   ```bash
+   ./build/vla-bench --ckpt smolvla-libero.gguf --images 2 --size 512 --warmup 3 --reps 20
+   ```
+
+   Report the process with the lowest mean, copying `min`, `mean`, `p50`,
+   `p90` and `vision` as printed (`--markdown` prints the row). Memory is the
+   peak over the three processes, taken with the probe the template names
+   for the backend: `nvidia-smi --query-compute-apps` sampled every 0.5 s on
+   discrete CUDA, DRM `fdinfo` on Intel GPUs and NPUs, peak RSS where the
+   device shares the process's memory (Jetson, Apple silicon, CPU). Per-model
+   extras: `--extra-token 151697 --extra-count 32` for VLA-JEPA,
+   `VLA_GR00T_EMBODIMENT` for GR00T (`new_embodiment`, `libero_panda`,
+   `libero_sim` for N1.5, N1.6, N1.7), `VLA_OCTO_UNNORM_DATASET=libero_object`
+   for Octo, `--size 640 --height 480` for ACT.
+
+4. **Screen the runtime flags.** For each model, run 2 warmups and 5 reps
+   per flag set from the template's list (`--flash-attn`, `--mm-prec default`,
+   `--weight-dtype bf16|f16`, their combinations, `--act-dtype bf16` on CUDA
+   only, plus any backend-specific candidate). A set replaces the defaults
+   only if its mean is more than 3% lower in the screen and again under the
+   full protocol of step 3. The screen means go in the collapsed table; the
+   winner goes in "Fastest configuration".
+
+5. **Account for every model.** A model with no latency row gets a bullet
+   under "Not run": `N/A` when the arch has no path on the backend, "does
+   not fit" with the quoted allocation failure, "not attempted" with both
+   sizes when the weights plainly exceed the device's memory, or the error
+   it printed. Anything a reader needs to interpret the numbers (a power
+   limit that puts `min` well below `p50`, a required environment variable,
+   a row whose output is known to be wrong) goes under "Notes".
+
+6. **Write it up.** Copy the template, fill every `<placeholder>`, delete the
+   HTML comments, and end with a "Reproducing" block that gives the cmake
+   line and one `vla-bench` row. Never name a host, a user or an IP address.
+   Add the device to the summary table in
+   [docs/benchmark/README.md](docs/benchmark/README.md): the `min ms` at the
+   defaults per model, `N/A`, `-` or `n/r` with the meanings the README
+   gives, and a sentence in its intro if the round ran at a commit the other
+   reports did not.
+
+A new backend, as opposed to a new device on a backend the tree already
+supports, also needs a numeric check: run `vla_predict_check` on fixed inputs
+against the CPU build and report the differences, as
+[docs/backend/rocm.md](docs/backend/rocm.md) does.
 
 ## Ports of upstream quirks
 

README.md

@@ -83,7 +83,7 @@ variables are in [docs/USAGE.md](docs/USAGE.md).
 
 Models (rows) against platforms (columns).
 Per-device latency and memory are in [docs/benchmark/](docs/benchmark).
-Use [ggml-compiler](https://github.com/khanhnd61-vr/ggml-compiler) and follow [the instruction](docs/ARCHITECTURE.md#adding-an-architecture) to add a new model.
+A model not listed here runs without a vla.cpp change: compile it with [ggml-compiler](https://github.com/khanhnd61-vr/ggml-compiler) and serve the GGUF ([docs/COMPILED.md](docs/COMPILED.md)).
 
 
 | Model | CPU (x86-64 / ARM) | [CUDA (NVIDIA)](docs/backend/cuda.md) | [ROCm (AMD)](docs/backend/rocm.md) | [SYCL (Intel)](docs/backend/sycl.md) | [Metal (Apple)](docs/backend/metal.md) | [OpenVINO (Intel)](docs/backend/ov.md) | [Hexagon (Qualcomm)](docs/backend/hexagon.md) |
@@ -151,7 +151,7 @@ Installing, wiring and running the client are in the fork's
 | [docs/ADOPTION.md](docs/ADOPTION.md) | C ABI, Python bindings, release packaging, and what is left |
 | [docs/UPSTREAMING.md](docs/UPSTREAMING.md) | The ggml-openvino fixes to send upstream to llama.cpp |
 | [CHANGELOG.md](CHANGELOG.md) | Per-release changes, with the fastest configuration and success rate per model |
-| [CONTRIBUTING.md](CONTRIBUTING.md) | Proving a change is numerically neutral, adding an architecture |
+| [CONTRIBUTING.md](CONTRIBUTING.md) | Proving a change is numerically neutral, benchmarking new hardware |
 | [bindings/python/](bindings/python/README.md) | Python bindings |
 
 ---

docs/ADOPTION.md

@@ -22,8 +22,9 @@ Done:
    `vla-cli`, `vla-server` and `vla-bench`, cached under `$VLA_CACHE`. A repo
    with several GGUFs lists them instead of guessing.
 6. **Reproducible benchmarks.** `vla-bench` emits the README table rows.
-7. **Contributor path.** `CONTRIBUTING.md` has the six-site walkthrough for
-   adding an architecture, plus issue and PR templates.
+7. **Contributor path.** `CONTRIBUTING.md` has the benchmark protocol for a
+   new device, plus issue and PR templates; new models go through
+   ggml-compiler (`docs/COMPILED.md`).
 8. **Instruction in, action out.** `vla-cli --text` builds each arch's real
    prompt. Octo carries its tokenizer in the GGUF, and
    `scripts/add_tokenizer_to_gguf.py` adds one to pi0, pi0.5 and OpenVLA-OFT, so

docs/ARCHITECTURE.md

@@ -13,7 +13,9 @@ map; the source is the detail.
 - `src/model.cpp` - loads a checkpoint, detects the architecture from its GGUF keys
   (or safetensors namespace), and dispatches to the matching factory.
 - `src/models/*.cpp` - one translation unit per architecture. Each owns its ggml
-  contexts, vision tower, weights, and compute graph.
+  contexts, vision tower, weights, and compute graph. `compiled.cpp` is the one
+  that owns no graph: it runs a GGUF written by ggml-compiler through its
+  runtime, from the `vla` manifest in `compiled_manifest.h` ([COMPILED.md](COMPILED.md)).
 - `src/gguf_reader.h` - the shared GGUF reader (metadata, tensor bytes,
   on-demand embedding rows). `src/loader.h` uploads weights at the resident dtype.
 - `src/layers/` - ggml building blocks: linear, attention, FFN, norms, RoPE,
@@ -92,40 +94,34 @@ reproduce ggml's output bit for bit ([backend/cuda.md](backend/cuda.md#the-bit-e
 
 ## Adding an architecture
 
-[ggml-compiler](https://github.com/khanhnd61-vr/ggml-compiler) makes the graph
-implementation easier: compile the model's PyTorch inference path into ggml C++
-source, with one graph builder per stage and a GGUF containing weights and the
-compiled computation. Its
+New models are not hand-written into this tree. Compile the policy with
+[ggml-compiler](https://github.com/khanhnd61-vr/ggml-compiler) into a GGUF that
+carries its weights, its op graphs and a `vla` serving manifest, and the
+`compiled` arch (`src/models/compiled.cpp`) serves it through ggml-compiler's
+runtime, built here against this tree's ggml. `vla-server`, `vla-cli`,
+`vla-bench` and `libvla` run it unchanged; every backend vla.cpp builds is
+available to it. [COMPILED.md](COMPILED.md) is the guide; ggml-compiler's
 [export guide](https://github.com/khanhnd61-vr/ggml-compiler/blob/main/docs/EXPORT.md)
 and [examples](https://github.com/khanhnd61-vr/ggml-compiler/tree/main/examples)
 cover ACT, SmolVLA, pi0/pi0.5, GR00T N1.7 and Diffusion Policy.
 
 1. Wrap the reference model in an `export.py:program` function returning a
-   `ggml_compiler.Program`. Define tensor inputs and outputs, fix deployment
-   shapes, and split iterative policies into an encoder/prefix stage and a
-   reusable denoising step. Record preprocessing, normalization, noise inputs,
-   and any prompt or embodiment specialization.
-2. Generate the ggml C++ source and GGUF:
+   `ggml_compiler.Program`: tensor inputs and outputs, fixed deployment
+   shapes, iterative policies split into a prefix stage and a reusable
+   denoising step, and the manifest (`ggml_compiler.vla`) naming each input's
+   role and preprocessing.
+2. Compile and verify:
 
    ```bash
-   ggml-compiler compile export.py:program --out out/new_model \
-     --emit gguf,cpp --name new_model
+   ggml-compiler compile export.py:program --out out/new_model --emit gguf
    ggml-compiler verify out/new_model/model.gguf --backend cpu
-   ggml-compiler verify out/new_model/model.gguf --backend cuda
    ```
 
-   Also compare the staged PyTorch inference with the original policy. The
-   compiler's `verify` checks against the staged reference saved at compile
-   time; build the emitted C++ checker to compare its graph builders with the
-   compiler runtime.
-3. Integrate the validated computation into a vla.cpp architecture. Adapt the
-   generated graph builders to vla.cpp's weight loader, graph lifecycle, and
-   prediction interface, including the policy's preprocessing and action units.
-   The generated C++ uses ggml-compiler's runtime/build API; its GGUF is not
-   automatically loadable by `vla-cli`. Check ggml version compatibility and
-   validate each target backend before marking it supported.
-
-For registration, extend the `Arch` enum, declare a `*_create` factory in `arch.h`, implement it under
-`src/models/`, wire detection and dispatch in `src/model.cpp`, and add a converter in
-`scripts/`. Reuse `gguf_reader.h`, `loader.h`, `layers/` and `modules/` rather than
-copying them. [CONTRIBUTING.md](../CONTRIBUTING.md) lists the exact sites.
+3. `vla-server out/new_model/model.gguf`.
+
+A hand-written arch is still how a model gets vla.cpp's own kernels (fused
+CUDA nodes, FoldQuant) and per-backend tuning. For one, extend the `Arch` enum,
+declare a `*_create` factory in `arch.h`, implement it under `src/models/`, wire
+detection and dispatch in `src/model.cpp`, and add a converter in `scripts/`.
+Reuse `gguf_reader.h`, `loader.h`, `layers/` and `modules/` rather than copying
+them; `src/models/act.cpp` is the smallest complete example.