[Bug] date_bin month bins add the origin's intra-second milliseconds twice #19264

Search before asking

• I searched in the issues and found nothing similar.

Version

master @ 8a2f85adf4a37b4a70a90bf456bd8e98e37eb28d (2.0.11-SNAPSHOT).

DateBinFunctionColumnTransformer.convertToTimestamp and AstBuilder.visitDateBin are the same on the 2.0.11 release commit 7c0de06. The bug is in the month-bin conversion itself, so it does not depend on the operating system. The numbers below use the default timestamp_precision=ms and TZ=UTC.

Describe the bug and provide the minimal reproduce step

date_bin with a month or year interval returns a bin start that is later than the real start, by exactly the milliseconds already present inside that second. date_bin_gapfill with a month interval uses the same conversion when it steps to the next bin.

convertToTimestamp adds toEpochMilli() and getNano() together. toEpochMilli() already contains the whole milliseconds of that second. getNano() is the full nanosecond-of-second (0–999_999_999), not the remainder below one millisecond, so those milliseconds are added a second time.

long epochMilliSecond = dateTime.atZone(zoneId).toInstant().toEpochMilli();
long nanoAdjustment = dateTime.getNano();
// ms:
return epochMilliSecond + nanoAdjustment / NANOSECONDS_IN_MILLISECOND;
// us:
return TimeUnit.MILLISECONDS.toMicros(epochMilliSecond)
    + nanoAdjustment / NANOSECONDS_IN_MICROSECOND;

iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/unary/scalar/DateBinFunctionColumnTransformer.java, convertToTimestamp (lines 92–110). Callers are dateBin, dateBinStartEnd, and nextDateBin(int, ZoneId, long).

Hour and day bins do not use this method. They stay correct. A month bin whose start is an exact second also stays correct, because getNano() is 0. The default origin is 0, so date_bin(1mo, time) without an origin is not affected.

AstBuilder.visitDateBin passes an integer origin through unchanged, so this SQL reaches the conversion:

CREATE DATABASE testdb;
USE testdb;
CREATE TABLE t (device_id STRING TAG, s INT32 FIELD);
INSERT INTO t (time, device_id, s) VALUES
  (500, 'd1', 1),
  (2678400500, 'd1', 2);
SELECT time, date_bin(1mo, time, 500) FROM t ORDER BY time;

500 is 1970-01-01T00:00:00.500Z. 2678400500 is that instant plus one calendar month (1970-02-01T00:00:00.500Z). Both points sit on the origin, so each bin start should be the point itself.

The same shift happens for any later year. An origin of 2024-03-15T00:00:00.500Z moves the March and April bins half a second later.

With timestamp_precision=us or ns, the extra amount is still the whole milliseconds inside the second. For an origin of 1.5 ms the us result is 2500 instead of 1500.

What did you expect to see?

The bin start is the origin plus a whole number of calendar months, and the sub-second part of the origin is kept.

What did you see instead?

Each start is 500 ms late. A point that belongs in the earlier bin can fall into the next one.

Anything else?

Only this private method needs to change. date_bin, the month-bin end time, and the month step in GapFillWGroupWMoOperator / GapFillWoGroupWMoOperator all call it. The fix is to add only the nanoseconds below one millisecond (getNano() % 1_000_000), or to convert from Instant.getEpochSecond() and getNano() the way DateTimeUtils.getInstantWithPrecision does. Existing DateBinFunctionTest cases use whole milliseconds and should stay the same.

On the same class, convertToLocalDateTime builds an Instant with timestamp % 1_000_000 (us) or timestamp % 1_000_000_000 (ns). For a negative timestamp that is not an exact second, the remainder is negative and Instant.ofEpochSecond throws DateTimeException. That is a separate failure of month date_bin before 1970 under us/ns.

Are you willing to submit a PR?

• I'm willing to submit a PR.