Repository navigation
Add the stream API: pluggable head types for single-token streams - #172
Open
AlonMalach wants to merge 23 commits into
Open
AlonMalach wants to merge 23 commits into
AlonMalach wants to merge 23 commits into
Conversation
A new kind of stream, like the activation probe, now plugs in with three things and no change to the model code: a config entry, a module for HF and for vLLM, and a weights file. Before, the probe was wired in by hand across the config, the composer and both backends, and a second head type would have repeated all of it. - config.streams: one entry per slot. "lora" for an adapter; otherwise a registered head type with its read layer, its row in its type's bank, its output (select or merge, the token ids, a merge weight) and type-specific settings. A checkpoint composed before streams has no such field; its entries are rebuilt from the probe fields it carries, so it loads unchanged. - granite_switch.streams: the HeadType contract, a registry that also loads plugins from the "granite_switch.streams" entry-point group, the routing tables both backends build at load, and the two output modes. The probe is the first built-in type and keeps its weight names. - Both backends find head markers, score each type at its read layer and write the scores into the marker's logits row through the same generic code. On vLLM the head runs branch-free on every token, as the probe did, so it stays safe under CUDA graphs. - Compact banks: head slots take no rows in the LoRA tensors, and each type's bank holds only its own slots, so slot order is free. A checkpoint with heads only builds no LoRA layers at all. - Compose writes the stream entries from the manifest and loads each head's weights through its type. A head reads the base model's state, as the probe does today. tests/shared/stream_plugins/mlp_head.py is an example plugin type (a two-layer MLP, merge mode) written against the API from outside the package; the tests compose it, run it and reload it. Signed-off-by: AlonMalach <alonmalach@gmail.com>
The head-output step looped over head rows in Python (.tolist(), .any(), nonzero, a host-to-device copy per row), so every logits step on a checkpoint with heads waited on the GPU. Each row now gathers its slot's output ids, score columns, mode and merge weight from tables built once at load, for both output modes; a non-head row writes back what it had. It reads nothing back to the host, and it traces as one graph under fullgraph compile. The vLLM probe head gathered a copy of its weights for every token, [tokens, labels, hidden]: 120 MiB at 8,192 tokens with 3 labels. It now scores every slot's head with one matmul and gathers each token's own labels. The matmul reads the activation as served (bf16) and sums into fp32 (mm out_dtype, CUDA); the bias is added in fp32 and each score is rounded once. Two simpler forms failed the exact two-slot check under full CUDA graphs on feature/streams-config: a bf16 matmul rounds inside its reduction (one bf16 step off), and upcasting the activation first lets the compiler fuse the final norm into the head and drop the activation's own bf16 rounding, so the head scored a different input than it served. Both come from feature/streams-config (af7c34e, b029cc7), where they passed the vLLM 0.30 GPU checks; here the rewrite also covers merge. Signed-off-by: AlonMalach <alonmalach@gmail.com>
With free slot order, a head may come before an adapter, and a streams checkpoint's LoRA tensors hold one row per adapter slot. Two compose checks still indexed those tensors by slot: - the cross-stream check read row "slot" and skipped probe slots by position, so with [probe, SR adapter] it checked nothing: an adapter whose cross_stream never loaded passed as OK; - the compose report gave the population table every discovered folder, so a probe folder took the adapter's column, and the table either read the adapter's row under the probe's name or failed on the probe folder's missing adapter_config.json. Both now walk the LoRA rows (streams routing lora_rows), and the report gets the LoRA folders only, in slot (= row) order. Tests compose [probe, SR adapter] and check both; they fail on the old code. Signed-off-by: AlonMalach <alonmalach@gmail.com>
A head entry, or its output, with a field this version doesn't know is now refused with the field's name, instead of being skipped. Such a field may come from a newer Granite Switch: skipping it would run the slot as something else, with no error. For example, a head that names an adapter would run on the base model. Signed-off-by: AlonMalach <alonmalach@gmail.com>
On top of the stream API: a head's stream entry may name an adapter slot, and the head then reads the state that adapter computes, not the base model's. One request names one slot, as before: each pair of a head and an adapter is its own slot, fixed at compose time, so MultiSwitch is unchanged. - Stream entry and manifest: an optional "adapter" naming an adapter slot of the same checkpoint, checked when the model loads. It joins the fields a head entry may carry, so a version without option B refuses such an entry instead of running the head on the base model. - Both backends: a table built from the streams at load sends a head marker's token through its adapter's LoRA row. It runs branch-free on vLLM, as the rest of the routing does. - Chat template: when the adapter would also run on tokens before the marker (a LoRA from the sequence start, an aLoRA invoked inside the user message), the template places the adapter's own token where it goes alone, and the head's marker last. Those tokens then carry the adapter's K/V, so the head's read is exact. Where the adapter would start at the marker's position (an aLoRA invoked at the assistant header) or writes no K/V (Shadow Residual), the marker alone gives the same state, and the template places nothing more. Tests: the template renders for each adapter kind; the head on an adapter reads exactly the adapter-alone state, on Shadow Residual from the marker alone and on single-stream LoRA with the adapter's token (and not without it); the same on vLLM. Signed-off-by: AlonMalach <alonmalach@gmail.com>
…core - A head's read layer is checked for every head type, when the model is built (so at compose too): it must be a decoder layer, or null for the final norm. Before, only the probe's legacy field was range-checked, and another type with a layer the model doesn't have silently scored nothing. - The models no longer keep a probe_head attribute when a checkpoint has no probe: the core names no type. - The composer skips every head bank by its module prefix, not by the probe's name, so a plugin's bank no longer looks like an uninitialized base weight. - Three derived config fields nothing reads are gone: the probe's label counts, its bank width and its marker ids (the marker ids left out every other head type). The routing tables already give all three. - HeadType declares module_name, the optional fixed name of a type's module in checkpoints, which the probe uses to keep model.probe_head.*. Signed-off-by: AlonMalach <alonmalach@gmail.com>
…type - The switch's split helpers move from probe_routing to streams.split, as build_head_slot_lut and split_head_indices. HF names its head stream head_indices, as vLLM already does. - The probe's HF and vLLM modules move from the core packages to the probe type (streams/types/probe/), with their bank rows named as rows. The probe's file constants live only there. - The chat template's head placement mode is "head", not "probe", and so is its head_mode variable. The composer's manifest metadata is head_meta, and the marker's substitute head_substitute_id. - Docstrings and comments that said "probe" where any head type is meant say so, and the config documents its streams field. No behavior changes. The probe's checkpoint names (model.probe_head.*) and the switch's adapter_kind_lut buffer, an HF checkpoint key, keep their names. A checkpoint composed before this change keeps its own chat template. Signed-off-by: AlonMalach <alonmalach@gmail.com>
A head stream answers a question about the conversation with one token, read at its marker, the prompt's last token. Conversation treated it as an ordinary turn, which went wrong under PRESERVE_MIXED_HISTORY in two ways: - A control token kept in the history runs its adapter over the history, and a LoRA or aLoRA writes that history's K/V, so a head on the base model read a state that adapter had changed. - Recording a head turn kept its marker inside every later prompt, where it routed the tokens after it through the head's slot. HF then refuses the prompt; vLLM serves it without a check. Now, which slots are heads comes from the config (adapter_names and adapter_kinds): - A head turn whose kept ids hold any control token is rendered from text, as RE_PREFILL renders it, and the kept ids are not touched. With none kept, the turn appends as before. - Recording a head turn adds its verdict to the transcript as text; the kept ids stay where they were, and the next turn's delta carries the head turn. - completion_payload asks for one token for a head turn, unless max_tokens is given. A head on an adapter renders two control tokens in one turn (the adapter's, then the marker); the earliest still decides where the turn's routing starts. Signed-off-by: AlonMalach <alonmalach@gmail.com>
…ifest - test_stream_serving: the first engine test of the stream API's layout. A tiny Shadow Residual checkpoint composed with stream entries (compact LoRA rows), with an identity probe on the base model, the same probe on the SR adapter (a head on an adapter), and the example mlp_head plugin in merge mode, which every process finds only through its granite_switch.streams entry point. Served eager and under FULL CUDA graphs, and with tensor or pipeline parallelism on two GPUs. Each head is checked against the HF fp32 reference, against the gap a routing bug would leave. - test_head_manifest_compose: the compose CLI on a manifest with every head key (a probe with no type, a probe on an adapter, a plugin in merge mode with settings), on a tiny base with the real tokenizer: the stream entries, the chat template's placement, and the pair's routing in HF. - The engine-side recorder skips a tensor-parallel rank that holds no logits. - TestProbeSlotRouting now runs: test_model_forward had no wrapper for it, so pytest never collected it. Signed-off-by: AlonMalach <alonmalach@gmail.com>
- docs/STREAM_API.md: what a stream is; the stream entry and its fields, output modes, refusals and older checkpoints; writing a head type (the HeadType contract, the module's call, the rules a vLLM module follows, what a head costs, registering a plugin); composing heads from a manifest; heads on adapters, with what the chat template places per adapter kind; Conversation's head turns; limits. - The activation probe guide: manifest keys without type, a probe on an adapter, the stream entry that compose now writes (and the older fields it keeps for older versions), and compose's new output. - The composer README documents a head's manifest keys; CLAUDE.md's tree has the streams package; CHANGELOG lists the stream API, its moved internals, and Conversation's head turns. Signed-off-by: AlonMalach <alonmalach@gmail.com>
The registry is how a new head type plugs in, so it has to be safe with several plugins installed: - A different class under a name that is taken is refused, so a plugin can't silently replace the built-in probe, or another plugin. Registering the same class again (a re-import) is still a no-op. - A lookup imports only the entry point with that name. A plugin that fails to import now breaks only the checkpoints that use it, not every lookup. - An entry point must load a HeadType subclass named like the entry point; either mistake is refused with the entry point's name and value. - head_type_names() lists installed plugins from their entry points without importing them. - HeadType's docstring now says which members a type must define, the module's call on both backends, and that load_weights' row is the 0-based head_row while the module's rows are head_row + 1. The hooks are typed. docs/STREAM_API.md states the three registration rules. Signed-off-by: AlonMalach <alonmalach@gmail.com>
CLAUDE.md asks every model a test runs to come from GraniteSwitchComposer, so the compose pipeline itself is tested. The stream API's model tests built theirs from a hand-assembled GraniteSwitchConfig and then set weights by hand. Each model is now composed from folders on disk: a tiny base, a Shadow Residual or single-stream LoRA adapter, and head folders (an identity probe, a probe with a fixed verdict, the example mlp_head). The checks are the same: a head on an adapter reads that adapter's state, head slots take no LoRA row and may sit in any order, a heads-only checkpoint has no LoRA, and a pre-streams layout is unchanged. A bad settings value and misshaped head weights now fail inside a real compose. With two probe folders, the compact row test also checks that each probe slot reads its own bank row. A config is still built directly only by the entry and routing tests, which test the config itself. Signed-off-by: AlonMalach <alonmalach@gmail.com>
The HF model re-read the stream entries on every decoder layer of every forward: the read groups and each type's module attribute (a registry lookup), plus the marker ids and the score width once per forward, and it copied each read layer's slot ids to the device each time. These are fixed when the model is built. - __init__ now keeps the score width and, per read layer, the head types that read it with their module attributes. - _row_lut_cached is now _stream_table(build, device, *args): it takes the routing function itself instead of its name as a string, so a rename fails at import, and it caches every device table the forward reads (the marker ids and each read layer's slots too). Tables are still built on first use, not in __init__, because from_pretrained builds the model on the meta device. - routing.read_slots gives the slots of one type that read one layer. - routing.adapter_lut parses the entries once instead of inside its loop. Signed-off-by: AlonMalach <alonmalach@gmail.com>
The module contract lets a head type return any float dtype and at least as many columns as its outputs, but neither backend handled all of it: - vLLM merged the module's output into the score buffer without casting it. An fp32 module on a bf16 model promoted the buffer, and with it the hidden state the scores are packed onto, to fp32. HF casts, so such a plugin passed the HF tests and broke only on vLLM. - A module wider than the checkpoint's widest head failed on both: vLLM's torch.where and HF's row write both need the exact width. Both backends now cut the module's output to the score row's width and cast it to the row's dtype. A test type in HF returns float64 with two extra columns, and a vLLM test returns fp32 with two extra columns from a bf16 model; each label still gets its own score. Signed-off-by: AlonMalach <alonmalach@gmail.com>
…ettings An entry's output tokens were checked to be token ids but not to be distinct. A repeated id (two label words that encode to one token, or an explicit list) makes the output step scatter two scores into one logit, and which one wins is undefined on GPU, so the verdict could differ between backends or runs. Now refused. The merge weight must be a finite number (not a bool or a string), and settings must be a dict or null; before, a bad settings value failed inside dict() with an unrelated message. Signed-off-by: AlonMalach <alonmalach@gmail.com>
Both comments said the LoRA weight transfer needs the LoRA slots first. With stream entries it doesn't: a streams compose keeps LoRA rows for the adapter slots only, in slot order, so LoRA path i fills row i in any order. - The compose CLI's sort is kept because it gives the slot layout of checkpoints composed before streams, and says so. - The transfer step's comment says why path i is row i in both layouts, and the alphas slice that did nothing is gone: adapter_alphas already holds one alpha per LoRA path. Signed-off-by: AlonMalach <alonmalach@gmail.com>
Validate stream and manifest inputs, prevent module collisions, and enforce the shared score contract. Load plugin parameters and buffers strictly and preserve vLLM slot IDs exactly in bf16. Add composed regression fixtures and record the revised review, CPU evidence, and deferred GPU release gates. Signed-off-by: AlonMalach <alonmalach@gmail.com>
Signed-off-by: AlonMalach <alonmalach@gmail.com>
Heads are decoupled from adapters. A head reads the base model's state at its marker, as the probe did before the stream API. Coupling a head with an adapter will come back later, in a different form, so the pairs (option B) are taken out instead of being kept half-used: - The stream entry's `adapter` field and the manifest's `adapter:` key. An entry or manifest that still has one is refused as an unknown field, so a checkpoint from the earlier build fails loudly instead of quietly reading the base model. - The routing table that sent a pair's marker through its adapter's LoRA, in HF and vLLM. - The chat template's per-adapter placement for pairs: the template code is main's again, with only the probe-to-head rename. - Conversation's handling of a head with two control tokens; the head prompt builder is simpler. - The pair tests and the "Heads on adapters" docs. Two tests replace what the pairs covered. An HF test checks that a head reads the same state with or without a Shadow Residual adapter's tokens before it. The engine test's probe on the SR adapter becomes a base probe reading decoder layer 0, so the test keeps a routing-bug gap and now serves an intermediate read layer. Signed-off-by: AlonMalach <alonmalach@gmail.com>
…ctly vLLM's MultiSwitch writes each control token's slot number into its memory head as a value in the model dtype, and rounds what it reads back. A float type holds every integer exactly only up to 2 / eps: 256 in bf16, 2048 in fp16. bf16 holds 257 as 256, so in a bf16 checkpoint with more than 256 slots slot 257's tokens ran as slot 256's, with no error, for adapters and heads alike. The HF switch computes the memory head in fp32 and is exact. The GPU run of the slot-257 test found it: the head's row held slot 256's verdict, so the slot was already wrong before the head ran. The byte-valued slot transport after the last layer was fine. - max_exact_slot_number(dtype) in hf/switch/codes, beside the counting head's numerics. - The vLLM switch refuses, when it is built, a checkpoint with more slots than its dtype numbers exactly, and says to serve with --dtype float16 or compose fewer slots. - The slot-257 test now checks the refusal; a CPU test pins the limit for bf16, fp16 and fp32, both ways. - MULTISWITCH_EXPLAINED.md, CLAUDE.md, STREAM_API.md, the changelog and the review record name the limit; the changelog no longer says slots above 256 are exact end to end. Signed-off-by: AlonMalach <alonmalach@gmail.com>
The slot transport packed each token's slot id into byte columns, so that ids above 256 survived bf16 on their way to the logits step. The vLLM switch now refuses a checkpoint with more slots than the model dtype numbers exactly, so one column of that same dtype holds every slot a checkpoint can have, and the transport has nothing left to do. The packing is main's single column again. The transport's contract tests go; the output-fit check keeps its graph-compile test. docs/STREAM_API_REVIEW.md, the second review's record, is removed: it is a process record, not product docs. Signed-off-by: AlonMalach <alonmalach@gmail.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Reverts the load-time refusal of fb45ceb. vLLM's MultiSwitch writes each slot number in the model dtype, which bf16 holds exactly only up to 256. That limit is on main for adapters, with no refusal, and heads now follow the same rule: a checkpoint with more slots loads as it does on main. The refusal, its helper, its unit and vLLM tests, and the docs, CLAUDE.md gotcha and changelog lines that named it go back to main's state. The single slot column from 9c4cefe stays: the switch writes slot numbers in the same dtype, so the column carries any number the switch produces. Signed-off-by: AlonMalach <alonmalach@gmail.com>
CI's coverage run executes only tests/unit/, and the streams package is not in the coverage omit list, but its tests lived in tests/hf/: Codecov reported 57% patch coverage, with 187 changed lines missing. - Move the tests that need no model (the registry, stream entries and their refusals, the output modes) from tests/hf/test_stream_api.py to tests/unit/test_stream_entries.py, unchanged, plus three small cases: the defaults a head type inherits, and streams that are not a list. - Add tests/unit/test_stream_heads.py: the routing tables, one module per type and backend and attaching it, the probe type's checks, bank and weight loading, the probe's HF module, and strict vLLM head loading (moved from the HF buffered-head test, which keeps its save/reload). - Omit the probe's vLLM module from CI coverage, like vllm/*: its forward needs CUDA (torch.mm with out_dtype has no CPU kernel), and the GPU tests cover it. docs/CICD.md says so. Locally, CI's unit run now covers every changed line it measures. Signed-off-by: AlonMalach <alonmalach@gmail.com>
AlonMalach
marked this pull request as ready for review
November 17, 2026 14:36
AlonMalach
requested review from
antonpibm,
aviv1ron1,
freunda and
yairallouche
as code owners
November 17, 2026 14:36
Collaborator
Author
|
/gpu-test-multi |
❌ GPU tests failed —
|
❌ GPU tests failed —
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A slot can now be a head stream, not only an adapter. A head reads the base model's hidden state at its marker token (its control token, placed last in the prompt) and writes scores into that row's logits. A new head type plugs in with three things, and no change to Granite Switch:
config.json(streams), written by compose from the manifest;HeadTypesubclass that builds the type's module for HF and vLLM;The activation probe from #134 is the first head type on this API. Its checkpoint names stay (
model.probe_head.*), so probe checkpoints composed onmainload unchanged. An example plugin type (tests/shared/stream_plugins/mlp_head.py) is found only through thegranite_switch.streamsentry-point group, and runs inmergemode. The guide isdocs/STREAM_API.md.What changes
granite_switch.streams(new): the stream entry and its parser; theHeadTypecontract, its registry and plugin entry points; the routing tables both models build; the output modes (selectreplaces the marker's row,mergeadds to it); the probe type and its modules.kind,labels,layer,output,settings) become stream entries; each head's weights load through its type; the cross-stream check and the population table read LoRA rows.Conversation: a head turn reads only itself, and its verdict joins the transcript as text.Compatibility
streams(adapters only, or a probe checkpoint frommain) loads as before: its entries are rebuilt from the older fields.streamsis meant for this version and later. Compact banks and plugin types need a runtime that understands streams.Not in this PR
mainfor adapters; heads follow the same rule.Conversationalways do this. It is listed under known limitations in the changelog anddocs/STREAM_API.md.Testing
mergemode found only through its entry point) eager and under full CUDA graphs against the HF reference, and the same with tensor and pipeline parallelism on two GPUs.main; no slot-number refusal): the vLLM files they touch pass on 0.32 and 0.26 (model forward, stream serving, probe capture and chunked prefill, MultiSwitch), on two GPUs on 0.32 too, and so does the real-model plugin check.