Repository navigation
Conversation
A multi-value IN list on an INT32 or INT64 field converted its literals without the range and decimal handling that "=" already has. Out-of-range integers were narrowed with an unchecked int cast and matched unrelated rows, and decimal literals were cast to LongLiteral, which failed with a ClassCastException. Both the pushed-down filter and the projection evaluator were affected, and "x = a OR x = b" hit the same code after it is normalized into IN. Convert each literal through a shared helper that keeps integers inside the column range and integral decimals, and drops literals that can never match. An impossible list becomes an empty IN set, so NOT IN still excludes NULL rows. Closes apache#19276
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #19276
Bug
A multi-value
INlist on an INT32 or INT64 field converted its literals without the range and decimal handling that=already has. The bug only shows up when the list has two or more values; a single value is rewritten to=.On a field
i INT32holding 1, 2, 3, INT32_MIN, INT32_MAX and NULL:where i in (1, 4294967298)where i in (1, -4294967293)where i in (1, 2147483648)where i = 1 or i = 4294967298(normalized to IN)where i not in (1, 4294967298)where i in (2, 2.5),where l in (2, 2.5)(INT64)301: DoubleLiteral cannot be cast to LongLiteralselect i in (2, 2.5),select l in (2, 2.5)ClassCastExceptiontruefor 2,nullfor NULLwhere i not in (2.5, 4294967298)ClassCastExceptionThe "Before" column comes from a standalone server built from master. The "After" column comes from a standalone server built from this branch, running the same statements as #19276.
Cause
There were two paths with the same problem:
ConvertPredicateToFilterVisitor.constructInSetconverted every literal throughRELATIONAL_CONVERT_PREDICATE_VALUE_PARSER_SERVICE. That is(int) ((LongLiteral) v).getParsedValue()for INT32. Out-of-range values wrapped, and anyDoubleLiteralfailed the cast.calc-commonsTypeServices.IN_MULTI_COLUMN_TRANSFORMER_SERVICE, already skipped out-of-range INT32 values. It still cast every literal toLongLiteral, though, so decimals failed for both INT32 and INT64.The analyzer does not insert casts for IN lists, so these literals reach both paths unchanged.
Fix
TypeServices.getIntegralInListValue(Literal, Type)in calc-commons converts each literal for an INT32/INT64 column. It keeps the value when it is an integer within the column range, or an integral decimal within that range. It returnsnullwhen the literal can never match. This is the same ruleconstructCompareFilteruses for=.constructInSetuses it for INT32/INT64 columns and drops literals that can never match. Other column types keep the existing conversion.IN_MULTI_COLUMN_TRANSFORMER_SERVICEuse the same helper. The INT32try/catch IllegalArgumentExceptionaroundgetParsedValue()is removed; that method cannot throw, so the block was dead code.FalseLiteralFilter.NOT INthen still goes through the normal value filter and excludes NULL rows; the IT covers this case.(double) Long.MAX_VALUErounds up to 2^63, so the bound is checked asv < (double) max + 1. That keeps9223372036854775808.0from saturating ontoLong.MAX_VALUE.CANNOT_CAST_TO_TYPEmessage instead of aClassCastException.Tests
TablePredicateConversionTest(new tests:testMultiValueInPredicateAndInt32Column,testMultiValueInPredicateAndInt64Column):2147483648at the boundary);TypeServicesTest.testIntegralInListValue: the shared conversion used by the projection path.IoTDBPredicateConversionTableIT(new tests:testMultiValueInPredicateAndInt32Column,testMultiValueInPredicateAndInt64Column):Local runs (JDK 25, macOS arm64):
Without the main-code change, both new
TablePredicateConversionTestcases fail withClassCastException: ... DoubleLiteral cannot be cast to ... LongLiteral.Not run locally:
TableClusterIT(I relied on the CI run for the cluster environment) and the full unit-test suites of the two modules.This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
org.apache.iotdb.calc.utils.TypeServicesorg.apache.iotdb.db.queryengine.plan.relational.analyzer.predicate.ConvertPredicateToFilterVisitor