Skip to content

fix(texture): avoid norm16 for 3D textures that cannot be linearly filtered - #3707

Open
GMrrc wants to merge 4 commits into
Kitware:masterfrom
GMrrc:fix/norm16-linear-filtering
Open

GMrrc wants to merge 4 commits into
Kitware:masterfrom
GMrrc:fix/norm16-linear-filtering

Conversation

@GMrrc

@GMrrc GMrrc commented Oct 30, 2026 •

Copy link
Copy Markdown

The volume, reslice and CPR mappers set the scalar texture filter only at render time, so the norm16 linear filtering guard never applied. Also make preferSizeOverAccuracy pick half float over float when norm16 is not usable.

Context

Fixes #3706

On browsers that expose EXT_texture_norm16 but cannot linearly filter norm16 textures (e.g. Firefox on Windows), vtkVolumeMapper, vtkImageResliceMapper and vtkImageCPRMapper render nothing for Int16/Uint16 scalars with linear interpolation. supportsNorm16Linear() correctly returns false there, but the guard added for #3194 only applies when the texture filter is already LINEAR when the format is chosen. These mappers create the scalar texture with the default NEAREST filter and switch to LINEAR afterwards, so the texture is uploaded as norm16 and samples as 0. The texture is also cached per scalars array, so it is never rebuilt when the interpolation changes.

Results

Before: on an affected browser, Int16 data with linear interpolation renders nothing (scalar texture R16_SNORM).
After: the scalar texture falls back to R32F, or to half float with preferSizeOverAccuracy, and the volume, reslice and CPR render correctly with both linear and nearest interpolation.
Browsers where norm16 linear filtering works (Chrome, Edge) are unchanged and keep exact norm16, with or without preferSizeOverAccuracy.

Changes

  • VolumeMapper, ImageResliceMapper, ImageCPRMapper (OpenGL): only use EXT_texture_norm16 for the scalar texture when supportsNorm16Linear() is true, since the filter is set at render time and the texture is shared between mappers.
  • Texture (OpenGL): with preferSizeOverAccuracy, Int16/Uint16 data uses half float instead of R32F when norm16 is not usable. Uint16 values above 65504 are clamped at upload instead of becoming infinity.
  • Texture (OpenGL): partial updates (updatedExtents) keep the data type the texture was allocated with.
  • VolumeMapper, ImageCPRMapper (OpenGL): pass preferSizeOverAccuracy on partial updates too.
  • Tests: testNorm16LinearFiltering (volume and reslice mappers) and testPreferSizeOverAccuracy (Texture).

No API added, removed or changed.

PR Checklist

  • GitHub Actions CI passed: semantic-release commit messages, lint, and tests
  • Test coverage added
  • Documentation and TypeScript definitions are updated to match these changes (no API change)

…ltered

The volume, reslice and CPR mappers set the scalar texture filter only at
render time, so the norm16 linear filtering guard never applied. Also make
preferSizeOverAccuracy override norm16 as documented.

Kitware#3706
preferSizeOverAccuracy forced half float regardless of the data range,
so Uint16 values above 65504 (e.g. 65535) became infinity and linear
sampling returned NaN. Fall back to norm16 or float in that case.
@PaulHax
PaulHax requested a review from sankhesh November 11, 2026 15:38
@PaulHax

PaulHax commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

One concern with the new overflow handling when preferSizeOverAccuracy is set.

When a value exceeds 65504, setCanUseHalfFloat now rejects half float. On devices where norm16 linear filtering is not usable, that means falling back to R32F. So the flag silently stops doing anything in this case: the texture takes twice the memory the app asked for, and it also depends on the device having OES_texture_float_linear.

An alternative is what #3750 does on WebGPU side. Keep half float and clamp at upload. In updateArrayDataTypeForGL, the toHalf(src[i]) loop is the only conversion site, so clamping there to ±65504 covers full and partial uploads. Then hasFiniteHalfFloat can go away and preferSizeOverAccuracy keeps its meaning.

The cost is that 65505..65535 read as 65504, an error of at most 31. Half float already rounds to steps of 32 in that range, so this is the same kind of loss preferSizeOverAccuracy already opts into.

Here is the change as a diff on top of this branch: PaulHax#72

Not a blocker, the norm16 probe is the important part of this PR.

@GMrrc

GMrrc commented Nov 12, 2026

Copy link
Copy Markdown
Author

One concern with the new overflow handling when preferSizeOverAccuracy is set.

When a value exceeds 65504, setCanUseHalfFloat now rejects half float. On devices where norm16 linear filtering is not usable, that means falling back to R32F. So the flag silently stops doing anything in this case: the texture takes twice the memory the app asked for, and it also depends on the device having OES_texture_float_linear.

An alternative is what #3750 does on WebGPU side. Keep half float and clamp at upload. In updateArrayDataTypeForGL, the toHalf(src[i]) loop is the only conversion site, so clamping there to ±65504 covers full and partial uploads. Then hasFiniteHalfFloat can go away and preferSizeOverAccuracy keeps its meaning.

The cost is that 65505..65535 read as 65504, an error of at most 31. Half float already rounds to steps of 32 in that range, so this is the same kind of loss preferSizeOverAccuracy already opts into.

Here is the change as a diff on top of this branch: PaulHax#72

Not a blocker, the norm16 probe is the important part of this PR.

I hadn't considered this alternative. I agree that clamping the values during conversion is probably a better approach than my previous change, as it preserves the intended behavior of preferSizeOverAccuracy while avoiding unnecessary memory overhead. I'll update the implementation accordingly. Thanks for the suggestion!

Rejecting half float on overflow made preferSizeOverAccuracy silently
fall back to R32F when norm16 cannot be linearly filtered, doubling
memory and requiring OES_texture_float_linear. Keep half float and clamp
values to the finite half float range (+-65504) at upload instead.
Comment on lines +1395 to +1398
const overrideNorm16 =
preferSizeOverAccuracy &&
(model.openGLDataType === model.context.SHORT ||
model.openGLDataType === model.context.UNSIGNED_SHORT);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overrideNorm16 checks model.openGLDataType, so it only fires when norm16 was picked first. On the browsers this PR targets, the mappers now pass a null norm16 ext, the type starts as FLOAT, and Int16 data with preferSizeOverAccuracy still ends up in R32F (same on master; checked in headless Chromium with a CT range). Could we test dataType === VtkDataTypes.SHORT || dataType === VtkDataTypes.UNSIGNED_SHORT here instead, and always call getOpenGLDataType(dataType, true) in processDataArray? With that change the case picks HALF_FLOAT, partial updates stay consistent, and the OpenGL texture/mapper tests still pass.

Comment on lines 1399 to +1402
const useHalfFloat =
forceHalfFloat || model.openGLDataType === model.context.HALF_FLOAT;
forceHalfFloat ||
overrideNorm16 ||
model.openGLDataType === model.context.HALF_FLOAT;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where norm16 filters fine (Chrome, Edge), this moves preferSizeOverAccuracy users from exact norm16 to half float at the same 2 bytes per texel, so CT values above 2048 lose precision with no memory saving. Is the crbug 1408247 workaround still needed, given the separate Mac workaround in useTexStorage? If we keep it, could the commit message call out the behavior change? The description says those browsers are unchanged, and also that exact ranges fall back to half float, but they still get FLOAT without the flag.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the probe, norm16 is only picked where linear filtering works, so I'd drop the crbug 1408247 workaround: Chrome/Edge keep exact norm16, and preferSizeOverAccuracy only prefers half float over float when norm16 isn't usable. I'll fix the description.

Comment on lines +19 to +30
// Prevent finite inputs such as Uint16 65535 from becoming infinite texels,
// which turn linear interpolation with neighboring values into NaN.
function toHalfForUpload(value) {
let uploadValue = value;
if (Number.isFinite(value)) {
uploadValue = Math.max(
-MAX_FINITE_HALF_FLOAT,
Math.min(value, MAX_FINITE_HALF_FLOAT)
);
}
return toHalf(uploadValue);
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only UNSIGNED_SHORT can exceed 65504 on this path (HALF_FLOAT is only chosen for Int16/Uint16), so could we clamp only when dataType is UNSIGNED_SHORT and keep the plain toHalf loop otherwise? In a quick Node run over 32M Uint16 values, the clamp added about 17% to this loop (409 ms to 480 ms).

height: dims[1],
depth: dims[2],
dataArray: scalars,
preferSizeOverAccuracy: volumeProperty.getPreferSizeOverAccuracy(),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This keeps the volume mapper consistent, but ImageResliceMapper's partial update (its lines 399-406) still re-derives the type without the flag, and both mappers share this texture through getGraphicsResourceForObject(scalars). Since the storage is immutable after texStorage3D, would it be safer for create3DFilterableFromDataArray to keep the allocated openGLDataType when updatedExtents is non-empty? That would also cover toggling preferSizeOverAccuracy after the first build, which isn't part of scalarsHash.

Comment on lines +108 to +109
vi.restoreAllMocks();
gc.releaseResources();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If an expect above fails, this cleanup is skipped and the getExtension spy leaks into the next test. Could we move vi.restoreAllMocks() and gc.releaseResources() into afterEach or a try/finally (same for the reslice test)?

const gc = testUtils.createGarbageCollector();
const { view } = createTrackedRenderView(gc);
const gl = view.getContext();
expect(gl.getExtension('EXT_color_buffer_float')).not.toBeNull();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fails the test on a GPU without EXT_color_buffer_float instead of skipping it. Could we call ctx.skip() when the extension is missing?

preferSizeOverAccuracy only took effect when norm16 had been picked
first, so Int16/Uint16 data still ended up in R32F on browsers where
norm16 is not usable. Decide from the data type instead, and keep exact
norm16 where it works since it has the same size as half float.

Partial updates now keep the data type the texture was allocated with,
and only Uint16 data is clamped to the finite half float range.
@GMrrc

GMrrc commented Nov 20, 2026

Copy link
Copy Markdown
Author

While testing it I found a related edge case that this PR doesn't fix. When a volume (with preferSizeOverAccuracy) and an ImageResliceMapper slice share the same scalars from the first render, the slice is rendered first and creates the shared texture. The OpenGL reslice mapper has no preferSizeOverAccuracy option, so the texture is allocated as R32F and the volume reuses it: the flag has no effect. With the volume alone, it correctly gets half float. This is the same on master.
A fix would probably mean adding preferSizeOverAccuracy to vtkImageResliceMapper (the WebGPU reslice mapper already reads getPreferSizeOverAccuracy?.()), which is a new API. Should I open a separate PR for that, or would you rather have it here?

@sankhesh

Copy link
Copy Markdown
Collaborator

While testing it I found a related edge case that this PR doesn't fix. When a volume (with preferSizeOverAccuracy) and an ImageResliceMapper slice share the same scalars from the first render, the slice is rendered first and creates the shared texture. The OpenGL reslice mapper has no preferSizeOverAccuracy option, so the texture is allocated as R32F and the volume reuses it: the flag has no effect. With the volume alone, it correctly gets half float. This is the same on master. A fix would probably mean adding preferSizeOverAccuracy to vtkImageResliceMapper (the WebGPU reslice mapper already reads getPreferSizeOverAccuracy?.()), which is a new API. Should I open a separate PR for that, or would you rather have it here?

Yes, please make a separate pr for that. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants