Repository navigation
Conversation
BinaryStatistics.updateStats(Binary) compares each value with the current minimum and maximum. When the value is the same instance as one of these bounds, the result of the comparison is always equal. This occurs when a writer sends the same constant Binary instance again, for example values decoded from a dictionary. Update BinaryStatistics in the parquet-column module to skip the comparisons when the value is the same instance as the minimum or the maximum. The resulting statistics do not change. Add tests in TestStatistics that make sure the behavior does not change: a single constant instance that repeats, bound instances that repeat, and other instances with equal, lower, or higher values. Unit tests cannot detect the skipped comparisons, because the result is the same with or without them.
Contributor
Author
|
@Fokko this is an improvement where I recently stumbled upon. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
BinaryStatistics.updateStats(Binary)compares each value with the current minimum and the current maximum.Binary.copy()keeps the same instance when the backing bytes are not reused. Thus, the minimum or the maximum can be the same instance as a value that the writer sends again. For example, values that come from a dictionary repeat the same constantBinaryinstance. For these values, the comparisons always return "equal" and do not change the statistics. Each comparison reads the bytes of the two values, so the writer does work that it does not need.What changes are included in this PR?
BinaryStatistics.updateStats(Binary), return early when the value is the same instance asminormax. In this case, the comparator is not called.elseblock.The subclasses
Float16StatisticsandIEEE754Float16StatisticsoverrideupdateStats(Binary). Thus, this change does not apply to them.Are these changes tested?
Yes. The new tests in
TestStatisticsmake sure that the statistics do not change:testBinaryMinMaxForRepeatedSingleConstantInstance: The same constant instance repeats, and it is both the minimum and the maximum.testBinaryMinMaxKeepsBoundInstancesForConstantValues: The bound instances repeat. A different instance with equal content does not replace a bound. Lower and higher values change the bounds.The existing test
testBinaryMinMaxForReusedBackingByteArraycontinues to make sure that a reused byte array that changes between updates gives correct statistics.These tests also pass without the change, because the skipped comparisons do not change the result. The tests make sure that the behavior stays the same.
Are there any user-facing changes?
No. The statistics are the same as before. There are no API changes.
Closes #3913