[egeozkoc]

On MPS, the Hub GEMV kernel indexes quantization scales per row, but bitsandbytes quantizes the flattened weight. When K is not divisible by blocksize, those layouts disagree: single-token matmul can fail reshaping absmax, and direct GEMV can silently return incorrect values.

Gate both MPS GEMV call sites on K % blocksize == 0 so unsupported shapes use the existing dequantize + linear fallback. Aligned shapes retain the Hub kernel.

Coverage adds K=320 to the existing matmul matrix and checks direct GEMV plus its out variant against dequantize + linear for blocksizes 128/256, FP4/NF4, fp16/bf16/fp32, and uint8/bf16 packed storage.

Validation

Refreshed November 3 on Apple M5, macOS 26.0, Python 3.12.13, torch 2.12.1, kernels 0.17.1. Actual Hub build: c3cadfafa7f693bbabbe65479be68f23ffb1c880, torch212-metal-aarch64-darwin.

• Unfixed MPS backend: all 24 new direct-GEMV cases fail; 12 NF4 cases of the K=320 matmul regression also fail. The same cases pass with the guards.
• Hub loaded: BNB_TEST_DEVICE=mps python -m pytest tests/test_functional.py tests/test_ops.py -k "(test_matmul_4bit and nf4) or test_gemv_4bit_unaligned_mps" — 648 passed.
• Hub loaded: BNB_TEST_DEVICE=mps python -m pytest tests/test_ops.py -k "gemv_4bit or gemm_4bit" — 172 passed, 12 skipped; includes existing opcheck coverage.
• MPS fallback explicitly selected by setting ops._kernel = None; ops._kernel_load_failed = True before pytest.main: functional/ops selection test_matmul_4bit or test_gemv_4bit_unaligned_mps — 1,273 passed.
• Rebuilt native CPU library. CPU tests/test_ops.py -k "gemv_4bit or gemm_4bit" — 148 passed, 36 skipped; tests/test_functional.py -k test_matmul_4bit — 289 passed, 960 skipped under the existing CPU matrix restrictions.
• Traced the actual Hub GEMV entry point for NF4 and FP4: aligned K=256 calls it for direct GEMV, out, and public matmul; K=320 calls it zero times. All outputs match dequantize + linear.
• Full pre-commit run --all-files (all 10 hooks) and git diff --check pass.

Earlier CUDA validation on an RTX 5090 Laptop (torch 2.11.0+cu128) passed all 96 new-shape cases and 1,249 matmul cases. CUDA was not rerun in this refresh; the additional test is MPS-specific.

Separate limitation and CI

The published Hub FP4 quantizer still chooses code 5 instead of 2 for normalized 0.7; tracked in huggingface/kernels-community#1193. The broad Hub matmul accuracy matrix above therefore selects NF4. Direct GEMV tests include FP4 and compare against the same dequantized weights, isolating multiplication correctness.

Current macOS CI selects the CPU backend on macos-15, so it does not exercise the macOS-26+ MPS Hub path. The previous workflow runs expired awaiting maintainer approval; upstream CI still needs to run before merge.

AI assistance: original patch prepared with Cursor; validation and direct-GEMV coverage refreshed with Codex.

[egeozkoc]

@matthewdouglas, could you approve the pending workflows and review this when you have a chance? The November 3 validation covers the real Hub kernel, MPS fallback, and CPU, including checks that aligned shapes still use Hub GEMV. Results are in the PR description.