Skip to content

Preserve INT8 row scales when parameter conversions omit a device - #2138

Draft
Afloat16 wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
Afloat16:fix/int8params-device-unspecified-scales-20261113
Draft

Afloat16 wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
Afloat16:fix/int8params-device-unspecified-scales-20261113

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

For a quantized Int8Params, to() creates a new parameter and preserves its packed data, but only copies SCB when the caller explicitly supplies a device. Valid no-device calls such as weight.to(), weight.to(dtype=torch.int8), and weight.to(non_blocking=True) therefore discard the row scales.

Replacing an embedding or a linear layer's weight with that result breaks its first quantized forward pass.

Change

Carry the existing SCB into the new parameter before handling an explicit device move. Continue binding CB to the returned parameter's data, and forward non_blocking when moving the scales.

Validation

  • 15 new cases cover no-argument, dtype-only, non-blocking-only, and explicit-device conversions, including meta-device scale preservation.
  • Native Embedding8bit and Linear8bitLt cases replace the weight before the first forward, so cached matmul statistics cannot hide lost metadata. Their outputs, downstream gradients, and two SGD updates match the unchanged models exactly.
  • Original source: 12 failures and 3 passing controls.
  • Patched source: all 15 new cases pass; 62 existing embedding/Int8Params cases pass and one existing unsupported CPU case is skipped.
  • Source-built AVX2 CPU kernels, PyTorch 2.14.0, Ruff check and formatting.

Exact production commit: eeaa2f804eda7d04d4326097e6140429b01e247a.
Validation: https://github.com/Afloat16/bitsandbytes/actions/runs/43401310219

Keep the SCB metadata when generic parameter conversion omits a target device.

Signed-off-by: Afloat <ShuffleMonkey@163.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant