Skip to content

Merge the HF real-checkpoint MultiSwitch tests and mark the real-model tests release - #174

Open
noaakl wants to merge 1 commit into
mainfrom
feature/release-tests
Open

noaakl wants to merge 1 commit into
mainfrom
feature/release-tests

Conversation

@noaakl

@noaakl noaakl commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #167. Puts the tests gated on GRANITE_SWITCH_E2E_MODELS into the release tier, following the per-file audit of those tests.

Changes

Merge the 4 HF real-checkpoint files into one. test_multi_switch_{e2e,alora,mixed_tech,realistic}.py -> tests/hf/test_multi_switch_real_checkpoint.py (31 tests). They overlapped and carried three near-identical compose helpers; _realistic borrowed _e2e's model. One helper now builds multi, multi-alora and multi-mixed (same output dir names, so warm reuse still works). No checks dropped.

Fix: repeated --adapters kept only the last repo list. multi-mixed was being built from guardian only, so the "mixed LoRA + aLoRA" tests never had a mix. All repos now follow a single --adapters. Pipeline-parallel equivalence uses the shared helper instead of its own copy.

Fix: conversation adapter-map regex matched no aLoRA entry. aLoRA entries in the composed template now carry 'invocation_drop_chars': N after invocation_text; the regex required } right after it, so the prefix-cache and concurrency tests found 0 aLoRA adapters and bailed out. New CPU test tests/unit/test_conversation_adapter_map_parse.py parses a template built by the real composer, so the next format change fails on PRs.

Release markers on: the merged HF file, test_multi_switch_vllm_generate, test_multi_switch_serving_e2e, test_multi_switch_alora_cache, test_conversation_prefix_cache, test_pipeline_parallelism_equivalence. Listed in docs/CICD.md. test_conversation_concurrency stays out: it never composes a checkpoint and needs one pre-built at GRANITE_SWITCH_CONV_DIR.

Not in this PR: converting these files to fakes for PR runs.

Testing

Vela, 2x A100, vLLM 0.26 (uv sync --frozen --group test), main 756218a + this branch:

Scope Result
tests/unit 1797 passed
tests/hf/test_multi_switch.py 297 passed
Default run of the release files 55 deselected
-m release on the 7 files 54 passed, 1 failed, 0 skipped

The failure: test_batched_generation_matches_solo (vLLM generate), request 1 differs from solo only at the 16th and last token (653 vs 3392). It passed on a rerun on clean main. The compose command and the test body are unchanged by this PR (only its import path), so this looks like a batch-size near-tie flip (vLLM is not batch-invariant) rather than a regression, but one fail vs one pass doesn't prove that.

Note for anyone with a persistent GRANITE_SWITCH_E2E_DIR: delete multi-mixed there once. Checkpoints are reused whenever config.json exists, so the old guardian-only build would keep being used.

…e real-model tests release

- Replace test_multi_switch_{e2e,alora,mixed_tech,realistic}.py with
  test_multi_switch_real_checkpoint.py: one compose helper for the multi,
  multi-alora and multi-mixed checkpoints instead of three copies. Point
  vllm_generate's import, comments and MULTISWITCH_EXPLAINED.md at it.
- Pass all repos after a single --adapters flag (a repeated flag keeps only
  the last list), so multi-mixed really gets rag + guardian; pipeline-parallel
  equivalence now uses the shared helper instead of its own copy.
- Let the conversation adapter-map regex accept trailing fields: aLoRA entries
  now carry invocation_drop_chars, and the old regex matched no aLoRA entry.
  Add a CPU unit test that parses a template built by the real composer.
- Mark the real-checkpoint HF, vLLM-generate, serving, aLoRA-cache, prefix-cache
  and pipeline-parallel tests release, and list them in docs/CICD.md.
  test_conversation_concurrency stays out: it never composes a checkpoint.

Signed-off-by: noaa <noaa.kless@gmail.com>
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@aviv1ron1 aviv1ron1 self-assigned this Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants