Repository navigation
Add the Cosmos3-Edge world-action model (LIBERO) - #79
Merged
Merged
Conversation
Cosmos3-Edge-Policy-LIBERO: a 28-layer Mixture-of-Transformers that denoises 4 latent video frames and a 16-step action chunk together, 30 UniPC steps. The five stage graphs (Wan2.2 VAE encoder, prompt prefix, init, one UniPC step, de-normalization) were generated by ggml-compiler and are used as generated: src/models/cosmos3_edge_graphs.cpp, ported by scripts/port_cosmos3_graphs.py. cosmos3_edge_build.h gives them the five runtime calls they use, and cosmos3_edge_ops.h is ggml-compiler's ops header, copied unchanged. cosmos3_edge.cpp is the engine: weights, the stage graphs with a persistent buffer for what crosses stages, the 30-step loop, and the canvas the model expects (two views side by side, bicubic with antialias, reflection padding; or one canvas the caller made). cosmos3_edge_prompt.cpp builds the prompt from the instruction (JSON caption, chat template, a byte-level BPE read from the GGUF; ASCII text only, anything else is refused) and decodes rot6d to LIBERO commands. vla-cli --text and --libero use them. scripts/convert_cosmos3_edge_to_gguf.py turns a `ggml-compiler compile` output into the vla.cpp GGUF, with the tensors copied byte for byte; --policy-dir adds the tokenizer and the caption settings. Verified, bf16 GGUF, CPU and CUDA (RTX 3090) only; no other arch or backend touched, no existing output moved: - given lerobot's canvas, bit-identical to ggml-compiler's runtime: CPU on three prompts of 150, 153 and 161 tokens, CUDA on one (tests/py/cosmos3_edge_parity.py); - captions byte-identical and ids identical to lerobot and Hugging Face for the 40 LIBERO tasks and five edge cases, and for 550 random ASCII and special-token texts; llama.cpp b11223's BPE gave the same ids on 607 of 607 (test_cosmos3_edge_prompt, data in tests/data); rot6d decode within 7e-7 of lerobot's on 62 rows; - libero_spatial, 2 episodes per task, seed 0, CUDA: 18/20 with lerobot's canvas and tokens, 19/20 with the C++ canvas, tokenizer and decode in the loop (0 token mismatches in 168 calls). Twenty episodes do not separate these from each other or from the PyTorch policy; - ctest: 11/11. Not verified: ROCm, SYCL, Metal, OpenVINO, Hexagon, vla-server, a full 50-episode-per-task run through vla.cpp.
README support-matrix row (CPU and CUDA marked in progress: no GGUF is released), CHANGELOG entry, the 256-wide two-view size in CONTRIBUTING's predict_check list, and docs/MODELS.md#cosmos3-edge-world-action-model with the four build steps, the inputs and outputs, the limits (ASCII text, one fixed setup) and the numbers behind each claim, including the ones that do not settle anything (20 closed-loop episodes).
…heck vla-server needs no change for this model: it passes the images, token ids and noise of a request to predict() for any architecture, and Cosmos3-Edge fits its checks (no state, a 16x10 noise and response, a warm-up its engine accepts). The server has no text field, so the client sends token ids. eval/client/vla_cpp_client.py gets a `cosmos3_edge` preset: it builds the JSON caption of lerobot's Cosmos3PromptProcessorStep and the chat template of Cosmos3TokenizerProcessorStep with the policy's tokenizer, sends the two 256x256 views as float RGB, and decodes the 16x10 rot6d chunk to 16 LIBERO commands (numpy, float64, as lerobot's Cosmos3LiberoActionProcessorStep does). Use it with --n-action-steps 16. tests/py/cosmos3_edge_server_check.py checks a running server, bit for bit against a direct libvla call, and the client's tokens, rot6d decode and get_action. Verified, bf16 GGUF of revision 0a81054, CPU only: a request returns bit-identical actions to a direct call (after a 62 s warm-up); the client builds the same 150 token ids as the case's; its rot6d decode is within 7e-7 of lerobot's on 62 rows; get_action returns a chunk of 16 finite LIBERO commands. No other arch touched; ctest 11/11. Not verified: the server on a GPU, a closed-loop LIBERO run through the server, eval/run_libero.sh (no Cosmos3-Edge block until there is a released model directory).
run_libero.sh -m cosmos3_edge starts vla-server on cosmos3-edge-libero-gguf/cosmos3-edge-libero-bf16.gguf under the models directory, runs the LIBERO client with 16 action steps per chunk, and stops the server, like the other models. A missing model directory is skipped, so -m all is unaffected. The client needs a Python with LIBERO and a transformers that reads the policy's tokenizer, which the LIBERO venv of setup_libero.sh (transformers 4.51) may not: COSMOS3_EDGE_PYTHON picks another, as VLA_JEPA_PYTHON does for vla_jepa, and the LIBERO venv check is skipped when it is set. COSMOS3_EDGE_TOKENIZER names a local tokenizer/ directory, and the client waits 300 s per chunk by default (a chunk takes about a minute on a CPU). Verified, bf16 GGUF of revision 0a81054, CPU server, no other arch touched: libero_spatial task 0, one episode, through this script: solved in 79 steps (reward 1.00; server RSS 6.5 GB). One episode shows that the path works end to end, not a success rate. Not verified: the server on a GPU (both GPUs of the test machine were full), more than one episode.
…e 10 scalars as one tensor, size the stage arenas from the graph cap and drop the tensor-count pin
…start with, not only '<' and '['
…kenizer_config.json
… prompt before loading the model
…ver's GGUF (--gguf)
…s3 and a converter from the checkpoint's safetensors, in place of the ggml-compiler output
…nt, with what was checked against lerobot
… the converter, the prompt test and the docs remain
…edge.cpp and drop the test code
…, GGUF keys, client preset, docs)
khanhnd61-vr
force-pushed
the
cosmos3-edge
branch
from
November 9, 2026 10:56
fda76b0 to
2c01050
Compare
…ter changelog entry; no WAM tag in the README
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Written for: vla.cpp maintainers reviewing the PR.
Add the Cosmos3-Edge world-action model (LIBERO)
lerobot/Cosmos3-Edge-Policy-LIBERO(28-layer Mixture-of-Transformers, 30 UniPC steps). The five stage graphs are generated by ggml-compiler and used as generated.scripts/port_cosmos3_graphs.pyregenerates them.vla-cli --textand--libero;cosmos3_edgeclient preset and aneval/run_libero.shblock;vla-serverand the public C API.vla-serverreturns the same bits as a direct call, and the harness solved one episode through it (CPU).ctest: 11/11.--weight-dtype,--act-dtype,--flash-attn,--mm-prec-f32and--num-steps.cosmos3_edge_graphs.cppis about 5,000 generated lines, so review the generator and the engine.~, notY.