Repository navigation
BUG: Fix copy-paste index drift in 15 filters - #1919
imikejackson merged 27 commits into
Conversation
1eaed2c to
7d8a44d
Compare
7d8a44d to
ee5159b
Compare
There was a problem hiding this comment.
PR #1919 Code Review
Reviewed head: 78d48c23051f21800f79ea87d635180dd15bafe0.
Base: aacedb4dc5c009d21045d715ac8f09a0d213d42e (develop).
Coverage: all 81 changed files, including production code, tests, CMake, documentation, and V&V records.
Current architectural problem
-
[Important, CONFIRMED] Keep results saveable when preserving supplied connectivity.
File: LabelTriangleGeometry.cpp:24,
CleanupCreatedSupport. The same integration concern applies to the retention changes inFeatureFaceCurvatureandVerifyTriangleWinding.The filters now retain supplied
DynamicListArrayconnectivity, but the writer has no factory for that type. A later save returns-5,Could not find IO factory for datatype: DynamicListArray, and can leave a partial destination. The new save-compatibility tests assert this failure rather than a successful round trip.An isolated MSVC Release probe supplied connectivity, ran
LabelTriangleGeometry, and saved the result. Saving succeeded with the algorithm from the PR base and failed with the algorithm from this head. At this head, saving succeeded after explicit support removal.The serializer limitation predates this PR, but it directly blocks the new retention contract. Preserve caller-owned objects in memory and add serialization support, or safely omit recomputable connectivity and its associations during serialization. Add a successful save/reload regression with supplied support. Documenting manual deletion does not preserve the previous filter-to-save behavior.
Lenses/checks: design, ownership and lifetime, data contracts, cross-layer integration. This is a current, reproduced problem, not a future risk. It blocks approval.
Implementation findings
-
[Minor, CONFIRMED] Write newly captured blocking stderr only once.
File: ExecuteProcess.cpp:95,
ExecuteProcess::operator().Both streams now drain into
output. The blocking branch writes that string at line 101 and again afterprocess.wait(). The probe recorded two stderr markers in blocking mode and one in nonblocking mode. Write the captured output once and assert marker occurrence counts in the regression test. Stdout duplication predates this PR; this finding concerns the newly captured stderr.Lenses/checks: correctness and defensive programming.
-
[Nit, CONFIRMED] Complete the new histogram API documentation.
File: StatisticsCalculations.hpp:21,
HistogramBinPolicyandCalculateBin.Document the policy enum and its values. Add
@tparam T, tags forvalue,minimum, andincrement, and a return description. State the nonzero, finite increment and representable-arithmetic/conversion preconditions. The comment audit confirms the missing classifier tags.Lenses/checks: API contracts, maintainability, and C++ documentation.
-
[Nit, CONFIRMED] Correct the knowledge-base examples.
File: Coding_Knowledge_Base.md:1222.
The statement that
ComputeShapesFilterTesthas no valid YZ/XZ expected output becomes stale with these new tests. Replace it with: "Before PR #1919,ComputeShapesFilterTesthad no valid YZ or XZ expected output."At line 1228, replace "used the wrong ray origin" with "used cos(phi3) instead of sin(phi3) in the local Y rotation basis." The Ray change corrects a basis term, not the origin.
Lenses/checks: documentation accuracy and maintainability.
-
[Nit, CONFIRMED] Declare the required special members for the new structs.
Files: ComputeShapesTriangleGeom.hpp:32,
AxialLengths; ComputeShapesTriangleGeomTest.cpp:27,ShapeMesh.Both new structs omit the explicit copy/move constructors, copy/move assignments, and destructor required by the project policy. Declare those members and supply value construction where aggregate initialization must be replaced, or use an existing suitable value type.
Lens: language practice.
Confirmed correct
- Source inspection supports the histogram lookup's stored lower/upper pairs and integer offsets. Its tests contain independent literal and arithmetic expectations.
- The cell-array shape corrections use the owning matrix's ZYX tuple contract while geometry dimensions remain XYZ.
- The principal-frame conversion corrects handedness while preserving ordered axis lines. The analytic geometry and explicit-frame tests address the changed contract.
All seven review lenses were applied to implementation and architecture. No additional actionable concern was found through CPU performance, memory, language practice, design, defensive programming, maintainability, or generated-code failure checks. This is a scoped review result, not proof that no defects remain.
Validation and limits
- Local clang-format 19.1.1 passed on all 63 changed C/C++ files using the repository-root style.
- Local x64 clang-tidy 19.1.5 completed on all 26 changed production translation units using the repository checks and changed-line filters. It reported no parse errors or changed-line diagnostics. Test translation units were not analyzed; configured header and line filters limit this result. The initial x86 analyzer crashed and supplied no passing evidence.
- The comment audit ran on all 63 changed C/C++ files. Its changed-line classifier-tag findings are listed above; source and comments were also inspected manually.
- Isolated MSVC Release CTest probes reproduced the blocking stderr duplication and the supplied-connectivity save regression. They compiled the two affected algorithms from the PR base and reviewed head, reusing existing local core libraries. These were component reproductions, not full PR builds.
- Reused Linux CI evidence passed 3,582 tests with GCC 13 and 3,582 with Clang 16. Both jobs tested merge
85a4d76, combining the reviewed head with the recorded base. - Full local builds, Debug execution, real out-of-core execution, and performance measurements were not performed in this review. No runtime claim is made for those checks.
Verdict
REQUEST CHANGES — the serialization regression must be resolved before approval.
* Document HistogramBinPolicy and CalculateBin, including their arithmetic, conversion and overflow preconditions. * Correct two knowledge-base examples: the YZ/XZ shape test note now dates from before PR BlueQuartzSoftware#1919, and the Ray fix is described as the cos(phi3)/sin(phi3) basis term. * Declare the special members of AxialLengths and the test ShapeMesh. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
jmarquisbq
left a comment
There was a problem hiding this comment.
PR #1919 Follow-up Review
Reviewed head: 0ad53d0a80243e2386113ca2fbee47587663b711.
Previous reviewed head: 78d48c23051f21800f79ea87d635180dd15bafe0.
The three new commits (fde38b70, 3225a21d, and 0ad53d0a) address all five findings in the posted Request Changes review. This follow-up covers all 22 changed files and the affected I/O callers. The earlier review is reused for unchanged code.
Disposition of posted findings
| Finding | Result |
|---|---|
| Supplied connectivity prevents saving | Addressed. Save/reload succeeds. The writer preserves live connectivity while omitting the caches and their geometry associations from the file. Tests cover shared aliases, append ID remapping, stale attributes, and attribute-removal failures. |
| Blocking stderr is duplicated | Addressed. Each output stream appears exactly once in both blocking and nonblocking modes. |
| Histogram API documentation is incomplete | Addressed. The policy, parameters, return type, and arithmetic/conversion preconditions are documented. |
| Knowledge-base examples are incorrect or stale | Addressed. The YZ/XZ test history and Ray rotation-basis explanation are corrected. |
| New structs omit required special members | Addressed. Explicit special members and value constructors are present. |
The architectural and implementation checks found no new actionable findings in the follow-up scope. The checks covered ownership and lifetime, file data contracts, dependency/API changes, concurrency, CPU and memory behavior, defensive programming, language practice, maintainability, and generated-code failure modes.
Validation
- A fresh MSVC core configuration built successfully in Release and Debug at the reviewed head.
- Release core tests passed 34/34: nine writer/attribute checks and 25 existing I/O regression tests.
- Focused Debug core tests passed 11/11, including the writer/attribute checks, node-geometry I/O, and append behavior.
- Independent component probes passed in both Release and Debug. They compiled the current
ExecuteProcessandLabelTriangleGeometryalgorithms against the freshly built core library. Both process modes logged one stdout marker and one stderr marker. Saving and reloading after labeling succeeded, preserved the live connectivity objects, and omitted connectivity from the reloaded geometry. - clang-format 19.1.1 passed on all 16 C/C++ files changed since the prior review, using the repository-root style.
- Scoped clang-tidy 19.1.5 completed on five changed production translation units without parse errors or changed-line diagnostics. Configured header and line filters limit this result; test translation units were not analyzed.
Validation limits
The core configuration did not build the production filter plugins or the DREAM3D-NX application. Full application/filter suites and real out-of-core suites were not rerun. The component probes do not replace those suites. No performance measurements were made.
Full build/test CI was not verified during this re-review; the check reported at that time was formatting. Historical full CI results for the previous head do not validate the new writer changes.
This approval applies to the stated review scope and evidence. It does not assert that unrun validation passed or that no defects remain.
Verdict
APPROVE — all five posted findings are addressed, with no new actionable findings in the reviewed follow-up scope and the validation limits above.
|
@joeykleingers This PR is in merge train 2, so I rebased it onto the current 1. On if(auto outputResult = printOutput(outputFile, " # Bounding Box is xmin xmax ymin ymax zmin zmax\n"); outputResult.invalid())
{
return outputResult;
}
const std::string boundingBox = fmt::format(" BoundingBox {} {} {} {} {} {}\n", origin[0] + 0.5 * res[0], origin[0] + (static_cast<float64>(dims[0]) - 0.5) * res[0], origin[1] + 0.5 * res[1],
origin[1] + (static_cast<float64>(dims[1]) - 0.5) * res[1], origin[2] + 0.5 * res[2], origin[2] + (static_cast<float64>(dims[2]) - 0.5) * res[2]);
if(auto outputResult = printOutput(outputFile, "%s", boundingBox.c_str()); outputResult.invalid())
{
return outputResult;
}2. The mesh slice (#1938) rewrote this file and added the bounded out-of-core helpers. // Cleanup removes only invocation-created connectivity before volume assessment.
Result<> windingResult = {};
{
const std::array entryIds = {triGeom.getElementNeighborsId(), triGeom.getElementContainingVertId()};
const std::array childExisted = {m_DataStructure.getData(m_InputValues->TargetGeometryPath.createChildPath(TriangleGeom::k_EltNeighbors)) != nullptr,
m_DataStructure.getData(m_InputValues->TargetGeometryPath.createChildPath(TriangleGeom::k_EltsContainingVert)) != nullptr};
// Generate Connectivity
m_MessageHandler.sendInfoMessage("Generating Connectivity and Triangle Neighbors...");
triGeom.findElementNeighbors(true);
const auto optionalId = triGeom.getElementNeighborsId();
if(!optionalId.has_value())
{
CleanupCreatedSupport(m_DataStructure, triGeom, m_InputValues->TargetGeometryPath, entryIds, childExisted);
return MakeErrorResult(-56321, fmt::format("Unable to generate the connectivity list for {} geometry.", triGeom.getName()));
}
const auto& connectivity = m_DataStructure.getDataRefAs<IGeometry::ElementDynamicList>(optionalId.value());
m_MessageHandler.sendInfoMessage("Repairing Windings...");
// This is reused since it may contain warnings
windingResult = MeshingUtilities::RepairTriangleWinding(triangles, connectivity, idsStore, m_ShouldCancel, m_MessageHandler);
if(windingResult.invalid())
{
CleanupCreatedSupport(m_DataStructure, triGeom, m_InputValues->TargetGeometryPath, entryIds, childExisted);
return windingResult;
}
CleanupCreatedSupport(m_DataStructure, triGeom, m_InputValues->TargetGeometryPath, entryIds, childExisted);
}I also added your new 3. These writers now collect errors with AppendWriteResult(result, WriteConnectivityId(dataStructureWriter, groupWriter, geometry.getElementContainingVertId(), IOConstants::k_ElementContainingVertTag));
if(result.invalid())
{
return result;
}
AppendWriteResult(result, WriteConnectivityId(dataStructureWriter, groupWriter, geometry.getElementNeighborsId(), IOConstants::k_ElementNeighborsTag));
if(result.invalid())
{
return result;
}Other changes made during the rebase:
|
* Document HistogramBinPolicy and CalculateBin, including their arithmetic, conversion and overflow preconditions. * Correct two knowledge-base examples: the YZ/XZ shape test note now dates from before PR BlueQuartzSoftware#1919, and the Ray fix is described as the cos(phi3)/sin(phi3) basis term. * Declare the special members of AxialLengths and the test ShapeMesh. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
0ad53d0 to
b3f3aa6
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Median-only requests checked the disabled mean output and failed with -69315. Check the median output in both algorithms so these requests produce the singleton and even-count medians. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Modal ranges used a bin index as a scalar offset and returned the wrong bounds. Use two values per bin in Direct and Scanline calculations. Correct the range layout comments and analytical test expectations. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The reduced face matrix took its name from the edge matrix or the fallback. Use the input face matrix name instead. Custom face matrices now keep their output paths and geometry association. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
YZ and XZ planes compared centroids and spacing from the wrong axes. Use the active centroid components and spacing for the planar bounds. Test both defects on XY, YZ, and XZ planes. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Phase-number checks skipped eligible cubic pairs and warned for phase 2. Compute metrics when both positive phases have Cubic_High structures. Return -94742 for invalid positive phase indices before eligibility. Unsupported structures retain zero metrics and produce warning -94739. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
USRNOD variables read token zero and later records overwrote tuple zero. Use each variable token and record tuple. Stop at the first failed value. Short records now return -2023; conversion failures return -2008 with variable and record context. Earlier completed values remain written. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The local Y basis used cos(phi3) where sin(phi3) is required. Use the correct term so Ray and CachedRay rotate the diagonal correctly. Keep the existing unnormalized length convention. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Execute Process never wrote the child's standard error to its log. reproc sends a child's stderr to the parent process unless asked otherwise, and the blocking branch also drained stderr into a null sink, so error messages went to the console of the application instead of the log file in both modes. Request a stderr pipe when starting the process and drain it into the same sink as stdout. Both blocking and non-blocking runs now record the child's standard error in the log. A small test child writes fixed markers to stdout and stderr so the regression runs the same way on every platform. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Compute Feature Shapes produced wrong planar results for YZ and XZ images and wrong 3D axis orientations for about half of all features. - The planar moments mapped the image dimensions to the active plane but kept X/Y spacing, origin and centroid components, so YZ and XZ images used the wrong axes. Area also used the X/Y spacing. - Planar samples were taken around the cell corner instead of the cell center, which shifted every moment. - A feature whose second moment lies along its second axis was given an angle of pi/180 (1 degree) instead of pi/2. - The principal axes form the rows of a frame matrix. Their signs are arbitrary, so the frame is mirrored (determinant -1) for about half of the features. The handedness check compared EbsdLib's validity code with 0, a value it never returns, so mirrored frames were converted to Euler angles as if they were rotations, which mislabels the axes. The planar code now selects the active axes once and uses them for every quantity, samples around cell centers and writes pi/2. A shared helper negates the third axis of a mirrored frame before converting it, so the Euler angles describe a rotation with the same ordered axis lines. AxisEulerAngles change for every feature whose principal frame was mirrored. The exemplar test now checks those features against the old conversion of the unmirrored frame. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Triangle axis lengths used one ray, so an eigenvector sign change altered lengths and aspect ratios. Improper principal frames also changed axis assignments during Euler conversion. Measure each length as half the sum of the farthest hits in both directions. Keep distances and ratios in float64 until output writes. Use the shared proper-frame conversion to retain ordered principal-axis lines in the Euler output. This changes the length definition and Euler angles for mirrored frames. Replace the triangle exemplar comparison with in-memory analytic tests. The 7_compute_triangle_shapes_test archive is no longer used by registered tests. Remove its download registration and documentation value table. Closes BlueQuartzSoftware#1802 Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Boolean group means used a whole-array majority comparison. Use the fraction of selected true values in each group and its population standard deviation. Remove the unreachable resident Boolean mean branches. Count Boolean values in uint64 and divide in float64 before conversion. This also corrects grouped summation above 2^24 and ungrouped mean rounding. Standardized Boolean outputs use the corrected statistics, including NaN for constant groups. Keep the input and masked outputs. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Smoothing deleted input edges and cleared the Edge Attribute Matrix association. Preserve existing edge topology and edge data instead. Remove only the temporary edge list that this invocation creates. Use one cleanup block for success, cancellation, and returned errors. Find a temporary array by its child name if failed edge creation clears its association. Keep shared geometry code and error codes unchanged. The partial edge-creation failure path has source review coverage only; no deterministic fault seam is added. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The bounds used voxel corners, which changed the implied sample spacing and gave singleton axes two coordinates. Write the first and last cell centers on each axis instead. Replace the Uniform exemplar comparison with an independent six-label ASCII and binary file oracle. Check all 75000 labels across transfer blocks, ASCII wrapping, payload sizes, and explicit little-endian bytes. Keep the shared Rectilinear archive unchanged. Closes BlueQuartzSoftware#1823 Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The SignedMaurer finding is withdrawn. ITK leaves +FLT_MAX for squared images without boundaries. For non-squared output, ITK applies the sign from foreground or background classification and InsideIsPositive. The existing production behavior agrees with these rules. Test the literal truth table under both dispatch scenarios, with low and high memory budgets, singleton, 2D, and 3D inputs. Replace the engine expectation derived from MaurerSignedValue with the same independent table. These tests pass without a production change. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
One of the four Case-3 problem-voxel checks required both f1 == f7 and f1 != f7, so it could never match and the v2-v7 body diagonal was never corrected. Replace it in Direct and Scanline with the condition that matches the other three diagonals. Test all four diagonals with allowed FeatureIds outcomes and an independent 54-triangle oracle. The v2 test archive was generated with the missed corrections, so its two Problem Voxels exemplar meshes are wrong. quick_surface_mesh_test_v3 holds the same dataset with only those two meshes regenerated. Every test that read v2 (QuickSurfaceMesh and M3C Surface Meshing) now reads v3, and the v2 download is no longer registered. Independent validation explains 8 corrected voxels: 6 direct Case-3 sites plus 2 follow-on corrections. Both groups have 30 fewer triangles. Direct/Scanline outputs are identical, and old code reproduces v2 exactly. The corrected FeatureIds independently imply 63,410 triangles per group. Every other DataObject in the archive equals v2. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The ungrouped histogram read stored lower/upper pairs as bin edges. It also rebuilt bin widths from rounded bounds. Modal ranges now use the stored pair selected by the counting descriptor. Out-of-bin modes add no pair. Degenerate counting widths retain their existing outputs. All four histogram paths use StatisticsCalculations::findModalBinRange. HistogramBinPolicy preserves Raw and LegacyInt32 classification and caller conversions. The lookup allocates no memory and performs no I/O. Callers reuse bin-sized buffers or check one bulk range read. StatisticsCalculations.hpp replaces the old source/ranges/mode public signature with an optional pair lookup and adds the policy classifier. HistogramUtilities.hpp keeps CalculateBin as a forwarding wrapper and removes the unused modal classes. Both interfaces are header templates. Add direct core tests and document rounded bounds and retained exceptions. Classification and range problems remain with BlueQuartzSoftware#1874, BlueQuartzSoftware#1873, and BlueQuartzSoftware#1867. Histogram counts and existing By Feature modal expectations stay unchanged. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
The BoundingBox line used %f, which keeps six decimal places. An image
with spacing 1e-7 was written with bounds 0.000000 in both the ASCII
and the binary file. Write the six float64 bounds with fmt {} so each
reads back to the same value.
Add a tiny-spacing test that parses the header in both encodings and
compares each bound with the independent cell-center formula. Build the
expected header text of the existing test from its oracle values.
Add the knowledge-base entry for this class of defect and the places
where it occurs; the other places are tracked in BlueQuartzSoftware#1927.
Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Extend the axis-and-plane and final-output-path entries, and add rules for geometry support data ownership, image-cell tuple shapes, symmetry support from Crystal Structures, and paired histogram bin ranges, from the defects fixed for BlueQuartzSoftware#1861. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Preserve GridMontage resize stride and decode plane and column remainders with their correct populations. * Construct Vec7 during seven-component type conversion. * Select the second BackTopEdge and RightBackEdge endpoints in MMCellMap and export the class for direct shared-library regression tests. * Associate the selected cell matrix and use ZYX label tuple shapes in UncertainRegularGridSampleSurfaceMeshFilter. * Create later-phase WritePoleFigure intensity arrays with the shape of their Cell Attribute Matrix, and record the fix in its V&V package. * Cover both BlueQuartzSoftware#1800 defects and BlueQuartzSoftware#1894. Issue BlueQuartzSoftware#1800 was closed as a duplicate of BlueQuartzSoftware#1861. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Use the phase's crystal structure for the ComputeBoundaryStrengths unsupported-symmetry warning and preserve every numerical output. * Remove the warning's unproven promise of zero numerical outputs. * Cover the ComputeBoundaryStrengths warning finding from BlueQuartzSoftware#1880, now tracked in BlueQuartzSoftware#1861. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Preserve supplied vertex incidence in FeatureFaceCurvature and return a support-creation failure before workers use missing data. * Preserve supplied incidence and neighbors in LabelTriangleGeometry. * Preserve supplied incidence and neighbors in VerifyTriangleWinding. A failed recalculation still clears their associations, and the winding repair is unchanged. * Remove only invocation-created support on applicable exits, including partial support left by failed creation. * Cover the support-cleanup entries in BlueQuartzSoftware#1861, including the LabelTriangleGeometry cleanup moved from BlueQuartzSoftware#1917. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
…x shape
* Create the numeric, NeighborList and StringArray outputs of
ResampleRectGridToImageGeomFilter with the {Z,Y,X} tuple shape of the
Cell Attribute Matrix that CreateImageGeometryAction creates. Before,
they used the {X,Y,Z} geometry dimensions.
* Flat values are unchanged.
Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
Broaden the image-cell tuple-shape entry to every array of every Attribute Matrix, including Rectilinear Grid Geometry cell data, and add the Write Pole Figure and rectilinear-grid resampling examples. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* The .dream3d writer leaves out geometry connectivity: the vertex incidence and element neighbor lists (DynamicListArray) that an Edge, Triangle, Quad, Tetrahedral or Hexahedral geometry references. The geometry's ID attributes for them are left out too, and a stale attribute in an existing group is removed. The lists in memory are unchanged; a reloaded geometry has none, and filters rebuild them. * Saving after FeatureFaceCurvature, LabelTriangleGeometry or VerifyTriangleWinding kept supplied connectivity no longer fails with error -5. The three filter tests save and reload with supplied lists. * A DynamicListArray that no geometry references this way still fails with -5. An association to an ordinary array is still written. * AppendFile removes connectivity from its copy before pruning and renumbering, so a list outside the appended object cannot leave a reference to another object. Appending a list itself writes nothing. * Add GroupIO::removeAttribute, which reports HDF5 failures (-18801, -18802). * EdgeGeom writes now return the error of the base geometry write instead of reporting success with missing child objects. * Document the omission in the writer, the file specification, Write DREAM3D-NX File and the three filter pages. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* The blocking branch wrote the captured standard output and standard error before and after waiting for the process. Write it once. * The standard-error test now requires each marker exactly once. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
* Document HistogramBinPolicy and CalculateBin, including their arithmetic, conversion and overflow preconditions. * Correct two knowledge-base examples: the YZ/XZ shape test note now dates from before PR BlueQuartzSoftware#1919, and the Ray fix is described as the cos(phi3)/sin(phi3) basis term. * Declare the special members of AxialLengths and the test ShapeMesh. Signed-off-by: Joey Kleingers <joey.kleingers@bluequartz.net>
b3f3aa6 to
539ebbf
Compare
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
…case * The "real OOC tuple masks" case of ComputeArrayHistogramTest.cpp came from BlueQuartzSoftware#1942 and still expected the old modal ranges {4,5,6,7}. * With this branch, each mode reports the stored pair of the bin that counted it. The custom range [0,8) puts modes 4 and 6 in bins 2 and 3, so the expected ranges are {4,6,6,8}. This matches the in-core case "tuple masks" above it. * Only the out-of-core build runs this case, through the SimplnxCoreOocStoreContracts test, so the in-core CI did not find it. Signed-off-by: Michael Jackson <mike.jackson@bluequartz.net>
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
d62b36b
into
BlueQuartzSoftware:develop
Summary
This change fixes the copy-paste index defects in #1861. In each case, code used the wrong index, axis, component, offset or variable. Each fix has a regression test with an independent oracle (closed-form arithmetic, analytic geometry or literal tables). The tests run on the in-core and out-of-core algorithm paths. Shared code in
Ray.hppandStatisticsCalculations.hppis also corrected. 91 files changed, +5734 / −762 lines.Out-of-core coverage for the two histogram filters is in the companion PR BlueQuartzSoftware/SimplnxOoc#18. That PR has only test changes: corrected modal-range expectations and a new real-store modal case.
1. Statistics and histograms
src/simplnx/Utilities/Math/StatisticsCalculations.hpp: the shared bin classifier and modal-range lookup.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayHistogram.cppandComputeArrayHistogramByFeature.cpp: modal ranges come from the shared lookup.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeArrayStatistics.cpp: Boolean statistics.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ComputeBoundingBoxStatsDirect.cppandComputeBoundingBoxStatsScanline.cpp: median check.Compute Array Histogram and Compute Array Histogram by Feature write each modal range as the stored lower/upper pair of the bin that counted the mode. Before, the code read a bin index as a scalar offset, read stored pairs as bin edges and rebuilt bin widths from rounded bounds. All four histogram paths (Direct and Scanline, with and without features) now call one function,
StatisticsCalculations::findModalBinRange.HistogramBinPolicykeeps the existing classification and conversion of each caller.Public header changes:
StatisticsCalculations::findModalBinRangereplaces the old(source, binRanges, mode)signature with an optional-pair lookup.StatisticsCalculations::CalculateBinis added with the policy.HistogramUtilities.hppkeepsCalculateBinas a forwarding wrapper.HistogramUtilities.hppremoves the unusedCalculateModalBinRangesImplandCalculateModalBinRangesImplFunctor.Histogram bin classification and range construction are not changed. Those problems are tracked in BUG: C15: Unsafe numeric conversion and range computation in 39 filters #1874, BUG: C14: Parameter values not validated in preflight in 65 filters #1873 and BUG: C07: Preflight/execute or Direct/Scanline disagree in 53 filters #1867.
Compute Attribute Array Statistics gives the mean of a Boolean group as the fraction of selected true values, with the population standard deviation. Before, it compared the sum with the size of the whole array.
Compute Bounding Box Statistics checks the median output when the median is requested. Before, it checked the mean output, so a median-only request failed with -69315.
2. Shape filters
src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeShapes.cpp: planar moments and principal-axis Euler angles.src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeShapesTriangleGeom.cpp: triangle axis lengths and Euler angles.src/Plugins/OrientationAnalysis/src/OrientationAnalysis/utilities/OrientationUtilities.cpp: the shared proper-frame conversion.Compute Feature Shapes (Image):
Compute Feature Shapes (Triangle Geometry) measures each axis length as half the sum of the farthest hits in both directions. Lengths and aspect ratios no longer change when an eigenvector sign changes. Distances and ratios stay float64 until the output is written. Mirrored frames use the same proper-frame conversion. The triangle exemplar comparison is replaced by analytic in-memory tests, and the
7_compute_triangle_shapes_testdownload is no longer registered.3. Geometry and meshing
src/simplnx/Common/Ray.hpp: the ZXZ rotation basis.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/QuickSurfaceMeshDirect.cppandQuickSurfaceMeshScanline.cpp: problem-voxel correction.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/LaplacianSmoothing.cpp: edge cleanup.src/simplnx/DataStructure/Montage/GridMontage.cppandsrc/simplnx/Common/Array.hpp: montage indexing and seven-component conversion.src/Plugins/SimplnxCore/src/SimplnxCore/SurfaceNets/MMCellMap.cpp: edge endpoints.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/UncertainRegularGridSampleSurfaceMeshFilter.cpp: cell matrix association and label tuple shape.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/ResampleRectGridToImageGeomFilter.cpp: copied cell array tuple shape.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/FeatureFaceCurvature.cpp,LabelTriangleGeometry.cppandVerifyTriangleWinding.cpp: connectivity ownership.RayandCachedRayuse sin(phi3) in the local Y basis, as the ZXZ rotation requires. Before, they used cos(phi3). The existing unnormalized length convention is kept. The newtest/RayTest.cppchecks the basis.Quick Surface Mesh corrects the v2–v7 body diagonal. One of the four Case-3 checks required both
f1 == f7andf1 != f7, so it never matched. Meshes with that problem-voxel pattern now have fewer triangles. The tests readquick_surface_mesh_test_v3. It is the same dataset as v2, with only the two Problem Voxels meshes regenerated.Laplacian Smoothing keeps the existing edge list and Edge Attribute Matrix of the input geometry. It removes only the temporary edge list that it creates. The cleanup also runs after cancellation and returned errors.
Remove Flagged Triangles names the reduced Face Attribute Matrix after the input face matrix. Before, it used the edge matrix name or the fallback.
Compute Biased Features uses the active centroid components and spacing for the bounds of YZ and XZ planes.
GridMontageuses the column count as the resize stride. Reverse lookup uses the plane size for the plane remainder and the column count for the column remainder. The first plane no longer uses a zero divisor.Vec7::convertTypeconstructs aVec7of the requested type and preserves all seven components. Before, it tried to construct aVec6.MMCellMap::getEdgeLabelscorrects the second endpoints ofRightBackEdgeandBackTopEdge. Both endpoints now lie on the selected edge.MMCellMapalso hasSIMPLNXCORE_EXPORTfor shared-library consumers.Sample Triangle Geometry on Uncertain Regular Grid associates the selected Cell Attribute Matrix with the output image cells. It creates that matrix once, including when the matrix and geometry names match. Geometry dimensions stay
{X,Y,Z}; the matrix and label array use tuple shape{Z,Y,X}. These fixes cover both defects in BUG: Uncertain grid sampling misassociates cell data and tuple shapes #1800, which is closed as a duplicate of BUG: C01: Copy-paste index drift in 15 filters #1861.Resample Rectilinear Grid to Image Geometry creates the copied numeric, NeighborList and StringArray cell arrays with the
{Z,Y,X}tuple shape of the new Cell Attribute Matrix. Before, they used the{X,Y,Z}geometry dimensions. Flat values are unchanged.Compute Feature Face Curvature, Label Triangle Geometry and Verify Triangle Winding keep supplied Element Neighbors and Elements Containing Vert. They delete only connectivity that the current invocation creates, including partial children on returned errors.
.dream3dwriter leaves the connectivity lists out (section 7).4. Readers, writers and processes
src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ReadDeformKeyFileV12.cpp: USRNOD variables.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/WriteAvizoUniformCoordinate.cpp: bounding box and its number format.src/Plugins/SimplnxCore/src/SimplnxCore/Filters/Algorithms/ExecuteProcess.cpp: stderr capture and one write of blocking output.src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeSlipTransmissionMetrics.cpp: pair selection.src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/ComputeBoundaryStrengths.cpp: unsupported-symmetry warning.src/Plugins/OrientationAnalysis/src/OrientationAnalysis/Filters/Algorithms/WritePoleFigure.cpp: later-phase intensity array shapes.Read DEFORM Key File (v12) maps each USRNOD variable to its own token and each record to its own tuple. It stops at the first failed value.
Write Avizo Uniform Coordinate writes the first and last cell centers on each axis as the bounding box. Before, it wrote voxel corners, which changed the implied spacing and gave singleton axes two coordinates. The Uniform exemplar comparison is replaced by an independent ASCII and binary file oracle. The shared Rectilinear archive is not changed.
{}, so each one reads back to the same value. Before,%fkept six decimal places, so an image with spacing1e-7was written with bounds0.000000. The bounds now appear as, for example,11 15 -3.5instead of11.000000 15.000000 -3.500000.Execute Process requests a stderr pipe and drains it into the same log as stdout, in blocking and non-blocking modes. Before, the child's standard error went to the console of the application. The new
ExecuteProcessTestChild.cppwrites fixed markers to both streams.Compute Slip Transmission Metrics computes metrics when both positive phases have Cubic_High structures. Before, it checked phase numbers, so it skipped eligible pairs and gave a warning for phase 2.
Compute Boundary Strengths checks the Crystal Structures value for its same-phase unsupported-symmetry warning, rather than the phase number. Cubic phase 2 no longer causes a false warning; noncubic phase 1 now causes the warning. The rule also covers accepted phase 0. Warning
-94739now identifies the selected Crystal Structures array and requests a check of that array. Numerical outputs remain bit-identical; the warning does not force unsupported structures to zero. This warning defect is reported in BUG: C21: Wrong or misleading diagnostics in 10 filters #1880.Write Pole Figure takes the tuple shape of later-phase intensity arrays from their owning Cell Attribute Matrix, fixing BUG: Correct tuple shapes for later-phase pole-figure arrays #1894. All three arrays inherit
{1,N,N}, as the phase-one arrays already do. Flat values, tuple counts, component counts and intensity generation are unchanged. The V&V report and provenance package record the synthetic two-phase shape regressions and their results; the historical validation and sign-off are unchanged.5. Coding knowledge base
docs/Coding_Knowledge_Base.md: new entry "Write floating-point data as text that reads back to the same value".The review of this change found that many writers save floating-point data as text with too few digits. The entry states the rule and lists those places. They are tracked in #1927. This change fixes only the Avizo uniform bounding box.
6. Signed Maurer distance map (test only)
The Signed Maurer finding in #1861 is withdrawn. The production code already agrees with ITK:
New tests assert this literal truth table under both dispatch scenarios, with low and high memory budgets and on singleton, 2D and 3D inputs.
7. Saving geometries that hold connectivity
src/simplnx/DataStructure/IO/HDF5/DataStructureWriter.cppand.hpp: connectivity omission andAppendFile.src/simplnx/DataStructure/IO/HDF5/INodeGeom1dIO.cpp,.hppandEdgeGeomIO.cpp: connectivity ID attributes and the edge base-write result.src/simplnx/Utilities/Parsing/HDF5/IO/GroupIO.cppand.hpp: checked attribute removal.docs/dream3d_file_spec.md,WriteDREAM3DFilter.mdand the three filter pages: the omission.The
.dream3dwriter leaves out geometry connectivity: the vertex-incidence and element-neighbor lists (DynamicListArray) that an Edge, Triangle, Quad, Tetrahedral or Hexahedral geometry references.-5("Could not find IO factory for datatype: DynamicListArray").A
DynamicListArraythat no geometry references this way still fails with-5. An association to an ordinary array is still written.AppendFileremoves connectivity from its copy before it prunes and renumbers the copy, so a list outside the appended object cannot leave a reference to another object. Appending a list itself writes nothing.New
GroupIO::removeAttributereports HDF5 failures (-18801,-18802).AppendFilereturns-18803if a connectivity list cannot be removed from its copy.An Edge Geometry write now returns the error of the base geometry write. Before, it discarded that error and reported success with missing child objects.
8. Related findings that this change does not fix
AppendFilestill saves a reference to an object outside the appended object (for example a vertex array that Move Data moved out of its geometry) under its old ID, which renumbering can give to another object. This change handles only connectivity lists. Tracked in BUG: AppendFile saves references to objects outside the appended object under the wrong ID #1960.GenerateHistogramandGenerateHistogramAtComponentinsrc/simplnx/Utilities/HistogramUtilities.hppread the input store one value at a time (inputStore[i]). For an out-of-core store this is one store access per value. A bulk read in chunks would remove this cost. This change does not touch those loops.Test plan
StatisticsCalculationscore tests pass: classifier types for all element types, legacy int32 fallback conversions, omitted pairs, zero increment, and a modal offset above 2^24.PY::OrientationAnalysis_ComputeBiasedFeaturesreads a file written by an earlier example pipeline; the 17-test Small IN100 Python chain passes 17/17 when run in order.OrientationAnalysisOocStoreContractsstill fails on the existing H5OINA raster-grid case, which develop also fails; this change does not touch that reader.[ooc]set passes in a DREAM3D-NX out-of-core build with the companion SimplnxOoc test changes: 714/717. The three failures also fail on develop:ReadH5OimDataFilter: Multi Scan actual OOC output parity(requests an archive that simplnx no longer registers),HDF5ChunkedStore deferred zero fill leaves chunks unallocated until written, and the H5OINA store-contract case above.{2,3,4}grid and checks the{4,3,2}tuple shape and coordinate-encoded values of numeric, string and NeighborList outputs (Release, Linux and macOS, in-core and out-of-core builds).H5Testwriter cases: Triangle and Edge round trips with rebuilt connectivity compared with known lists, connectivity shared by two parents, append (including a connectivity list outside the appended geometry whose old ID overlaps a renumbered object, and appending a list itself), stale attributes in an existing group, an ordinary-array association, a standaloneDynamicListArray(still-5), the Edge base-write error andGroupIO::removeAttributefailures.HDF5ChunkedStore deferred zero fill leaves chunks unallocated until written, both listed above.Related Issue
Closes #1802
Closes #1823
Closes #1896
Closes #1894
Closes #1861