Skip to content

BUG: Clamp the quaternion scalar part before acos() in conversions - #80

Closed
imikejackson wants to merge 1 commit into
BlueQuartzSoftware:developfrom
imikejackson:fix/orientation-acos-clamp
Closed

imikejackson wants to merge 1 commit into
BlueQuartzSoftware:developfrom
imikejackson:fix/orientation-acos-clamp

Conversation

@imikejackson

@imikejackson imikejackson commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Quaternion::toAxisAngle(), toRodrigues() and toHomochoric() passed the scalar part w straight to acos(). Float rounding can leave a unit quaternion with |w| one ULP above 1.0. acos() then returns NaN, and the angle, the Rodrigues vector or the homochoric vector is NaN.

  • Example: (-0.000147104263, 0.00024650991, 0.000268757343, 1.00000012) is the misorientation a^-1 * (a * b) of a small rotation b, computed in float. isValid() accepts it, but all three conversions returned NaN.
  • w is now clamped to [-1, 1] before acos(). A value already inside that range does not change, so results for all other input stay the same.

Other inverse-trigonometric calls I checked:

  • OrientationMatrix::toEuler() calls acos(om[8]) only when |om[8]| is more than 1e-6 away from 1, so it is already safe.
  • Quaternion::getMisorientationVector() already clamps.
  • No asin() calls exist in the orientation classes.
  • Homochoric::toAxisAngle() has a related NaN: the constant term of its fit (tfit[0] = 1.0000000000018852) is above 1.0, so in double precision every rotation smaller than about 4e-6 rad gives a NaN angle. Commit e101edf on BUG: Correct ODF sampling and MDF normalization #61 (topic/synthetic_changes, also on ENH: Standard ODF section images and in-ball ODF sampling #60) already fixes it, so this PR does not change Homochoric.hpp. The input HomochoricDType(1.0e-7, 2.0e-7, 0.0) gives a NaN angle from toAxisAngle() and a NaN quaternion from toQuaternion() on develop, so it could be added as a regression test there.
  • The legacy Core/OrientationTransformation.hpp has the same unclamped pattern in qu2ax, qu2ro, qu2ho and ho2ax. I did not change it: it includes EbsdLib/Math/EbsdMatrixMath.h, which now exists only under Math/deprecated/, so no translation unit can compile it.

Test Plan

  • New test EbsdLib::ebsdlib::OrientationTransformationTest::Quaternion_ScalarPartRoundingError fails on develop (NaN axis-angle, Rodrigues and homochoric results) and passes with this change.
  • Linux x64, Release: full ctest suite passes, 430/430.

Related Issue

Related to BlueQuartzSoftware/simplnx#1875 (the unclamped acos class, item ComputeTwinBoundariesFilter#4). This PR fixes the same pattern in the EbsdLib quaternion conversions that simplnx filters call. The acos call in simplnx ComputeTwinBoundaries.cpp still needs its own clamp.

🤖 Generated with Claude Code

* Quaternion::toAxisAngle(), toRodrigues() and toHomochoric() passed the
  scalar part straight to acos(). Float rounding can leave a unit
  quaternion with |w| one ULP above 1.0 (for example the misorientation
  a^-1 * (a * b) for a small rotation b), and acos() then returns NaN, so
  the angle, Rodrigues vector or homochoric vector was NaN.
* The scalar part is now clamped to [-1, 1] first. Values inside that
  range are unchanged, so results for other input do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@imikejackson
imikejackson force-pushed the fix/orientation-acos-clamp branch from 95cbdcc to b257d8f Compare November 13, 2026 02:47
@imikejackson imikejackson changed the title BUG: Clamp the cosine before acos() in quaternion and homochoric conversions BUG: Clamp the quaternion scalar part before acos() in conversions Nov 13, 2026
@imikejackson

Copy link
Copy Markdown
Collaborator Author

Closed in favor of #86, which merges this branch (with its history) together with the other open EbsdLib fixes for the next release, so simplnx can build against one EbsdLib branch. The branch is kept.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant