Skip to content

Fix multi-value IN on INT32/INT64 fields in the table model - #19277

Open
Karman580 wants to merge 1 commit into
apache:masterfrom
Karman580:fix-table-multi-value-in-predicates
Open

Karman580 wants to merge 1 commit into
apache:masterfrom
Karman580:fix-table-multi-value-in-predicates

Conversation

@Karman580

Copy link
Copy Markdown
Contributor

Description

Closes #19276

Bug

A multi-value IN list on an INT32 or INT64 field converted its literals without the range and decimal handling that = already has. The bug only shows up when the list has two or more values; a single value is rewritten to =.

On a field i INT32 holding 1, 2, 3, INT32_MIN, INT32_MAX and NULL:

Query Before After
where i in (1, 4294967298) rows 1, 2 row 1
where i in (1, -4294967293) rows 1, 3 row 1
where i in (1, 2147483648) rows 1, INT32_MIN row 1
where i = 1 or i = 4294967298 (normalized to IN) rows 1, 2 row 1
where i not in (1, 4294967298) row 2 missing rows 2, 3, INT32_MIN, INT32_MAX
where i in (2, 2.5), where l in (2, 2.5) (INT64) 301: DoubleLiteral cannot be cast to LongLiteral row 2
select i in (2, 2.5), select l in (2, 2.5) same ClassCastException true for 2, null for NULL
where i not in (2.5, 4294967298) ClassCastException every non-NULL row, NULL row excluded

The "Before" column comes from a standalone server built from master. The "After" column comes from a standalone server built from this branch, running the same statements as #19276.

Cause

There were two paths with the same problem:

  • Filter pushdown: ConvertPredicateToFilterVisitor.constructInSet converted every literal through RELATIONAL_CONVERT_PREDICATE_VALUE_PARSER_SERVICE. That is (int) ((LongLiteral) v).getParsedValue() for INT32. Out-of-range values wrapped, and any DoubleLiteral failed the cast.
  • Projection: the column-transformer path, calc-commons TypeServices.IN_MULTI_COLUMN_TRANSFORMER_SERVICE, already skipped out-of-range INT32 values. It still cast every literal to LongLiteral, though, so decimals failed for both INT32 and INT64.

The analyzer does not insert casts for IN lists, so these literals reach both paths unchanged.

Fix

  • Shared conversion: a new TypeServices.getIntegralInListValue(Literal, Type) in calc-commons converts each literal for an INT32/INT64 column. It keeps the value when it is an integer within the column range, or an integral decimal within that range. It returns null when the literal can never match. This is the same rule constructCompareFilter uses for =.
  • Filter pushdown: constructInSet uses it for INT32/INT64 columns and drops literals that can never match. Other column types keep the existing conversion.
  • Projection: the INT32 and INT64 branches of IN_MULTI_COLUMN_TRANSFORMER_SERVICE use the same helper. The INT32 try/catch IllegalArgumentException around getParsedValue() is removed; that method cannot throw, so the block was dead code.
  • Impossible lists and NULLs: if no literal can match, the result is an empty IN set rather than FalseLiteralFilter. NOT IN then still goes through the normal value filter and excludes NULL rows; the IT covers this case.
  • INT64 upper bound: (double) Long.MAX_VALUE rounds up to 2^63, so the bound is checked as v < (double) max + 1. That keeps 9223372036854775808.0 from saturating onto Long.MAX_VALUE.
  • Non-numeric literals: these still fail, now with the existing CANNOT_CAST_TO_TYPE message instead of a ClassCastException.

Tests

  • TablePredicateConversionTest (new tests: testMultiValueInPredicateAndInt32Column, testMultiValueInPredicateAndInt64Column):
    • INT32 out-of-range literals (positive, negative, and 2147483648 at the boundary);
    • integral and non-integral decimals, and duplicate values;
    • NOT IN, and a list where no literal can match;
    • INT64 decimals and the 2^63 boundary.
  • calc-commons TypeServicesTest.testIntegralInListValue: the shared conversion used by the projection path.
  • IoTDBPredicateConversionTableIT (new tests: testMultiValueInPredicateAndInt32Column, testMultiValueInPredicateAndInt64Column):
    • filter pushdown, and the OR form normalized into IN;
    • NOT IN, including that NULL rows stay excluded;
    • duplicates;
    • projection for INT32 and INT64.

Local runs (JDK 25, macOS arm64):

./mvnw test -pl iotdb-core/calc-commons,iotdb-core/datanode \
  -Dtest='TypeServicesTest,TablePredicateConversionTest' -Dsurefire.failIfNoSpecifiedTests=false
Tests run: 13, Failures: 0, Errors: 0  -- calc-commons TypeServicesTest
Tests run: 9,  Failures: 0, Errors: 0  -- TablePredicateConversionTest
Tests run: 11, Failures: 0, Errors: 0  -- datanode TypeServicesTest
BUILD SUCCESS

./mvnw verify -DskipUTs -pl integration-test -am -PTableSimpleIT -P with-integration-tests \
  -Dit.test=IoTDBPredicateConversionTableIT -Dfailsafe.failIfNoSpecifiedTests=false
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 -- IoTDBPredicateConversionTableIT
BUILD SUCCESS

Without the main-code change, both new TablePredicateConversionTest cases fail with ClassCastException: ... DoubleLiteral cannot be cast to ... LongLiteral.

Not run locally: TableClusterIT (I relied on the CI run for the cluster environment) and the full unit-test suites of the two modules.


This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage.
  • added integration tests.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.calc.utils.TypeServices
  • org.apache.iotdb.db.queryengine.plan.relational.analyzer.predicate.ConvertPredicateToFilterVisitor

A multi-value IN list on an INT32 or INT64 field converted its literals
without the range and decimal handling that "=" already has. Out-of-range
integers were narrowed with an unchecked int cast and matched unrelated
rows, and decimal literals were cast to LongLiteral, which failed with a
ClassCastException. Both the pushed-down filter and the projection
evaluator were affected, and "x = a OR x = b" hit the same code after it
is normalized into IN.

Convert each literal through a shared helper that keeps integers inside
the column range and integral decimals, and drops literals that can never
match. An impossible list becomes an empty IN set, so NOT IN still
excludes NULL rows.

Closes apache#19276
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] [Table model] Multi-value IN on INT32/INT64 fields returns wrong rows for out-of-range literals and throws ClassCastException for decimal literals

1 participant