Skip to content

Decode binary local timestamps like PostgreSQL around DST changes (0 mismatches over 2.45M values, 1 object instead of 2) - #763

Open
czj wants to merge 5 commits into
ged:masterfrom
czj:fix/binary-timestamp-db-local
Open

czj wants to merge 5 commits into
ged:masterfrom
czj:fix/binary-timestamp-db-local

Conversation

@czj

@czj czj commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

The review of #760 turned up two bugs in the TIMESTAMP_DB_LOCAL branch of PG::BinaryDecoder::Timestamp, both older than any recent change. Neither shows up in CI because every job runs with TZ=UTC.

TimestampLocal subtracted the UTC offset found at the wall clock read as UTC, instead of the one at the real instant, so values near a DST change came back one hour off. With TIMESTAMP_DB_LOCAL alone, the Time was already in UTC mode, its utc_offset was 0, and the wall clock came back labelled UTC (12:00 UTC instead of 10:00 UTC for a Paris summer noon), unlike the text decoder with the same flags.

This version follows PostgreSQL's own rule (DetermineTimeZoneOffset) in C: it reads the offsets one day before and one day after the wall clock with localtime_r, uses that offset when they agree, and otherwise keeps the candidate that maps back to the wall clock, or the later instant when both or neither do (overlaps and gaps). I compared it with wall_clock::timestamp AT TIME ZONE zone for every quarter hour of 1950, 1990, 2011, 2024, 2038, 2040 and 2100 in 10 zones (Paris, London, New York, Santiago, St John's, Cairo, Kathmandu, Sydney, Lord Howe, Apia): 0 mismatches out of 2,454,720 values, for both flag combinations. With 1.7.0, TimestampLocal had 28 (London) to 2,968 (Apia) mismatches, and TIMESTAMP_DB_LOCAL alone was wrong whenever the offset wasn't 0. The new specs hardcode PostgreSQL's answers under TZ=Europe/Paris, including a gap and an overlap in 2024 and both beyond 2038, and fail on master.

An earlier version used Time.local, which I dropped after a review: on macOS, Ruby refuses localtime outside 32-bit time_t, so it was wrong after 2038, and TruffleRuby resolves overlaps differently. tzset() is called per value because glibc's localtime_r doesn't re-read TZ (macOS does). Before 1901-12-14, Ruby on macOS renders local times without the C library's LMT offsets, so those keep master's subtraction, which already round-trips there.

This is a behavior change, so I would rather say it upfront. Anyone who worked around the hour shift or the UTC label will see different values. Speed depends on the platform. These are medians of 7 runs over 10,000 values with Ruby 4.0.9, ranges over 3 alternating rounds in TZ=Europe/Paris and TZ=UTC. TimestampLocal allocates 1 object instead of 2 everywhere. On Linux it gets faster: from 210–320 to 130–160 ns per value on x86_64, and from 370–420 to 245–315 ns on aarch64 (GitHub runners, ubuntu-24.04). On arm64-darwin it gets slower: from ~490 to ~895 ns, because macOS localtime_r is much more expensive than glibc's. TIMESTAMP_DB_LOCAL alone used to skip the work, so it costs more everywhere: from ~110 to ~130–160 ns on x86_64, from ~230 to ~250–320 ns on aarch64, and from ~145 to ~900 ns on macOS. On the same Linux runners, TimestampLocal also matched PostgreSQL for every quarter hour of 1950, 2011, 2024, 2038, 2040 and 2100 in Paris, New York, Sydney, Apia and Lord Howe, against 60 to 430 mismatches for 1.7.0, and the type specs pass under glibc.

The text decoder still differs on overlaps: mktime() picks the DST side on macOS, where PostgreSQL picks standard time. I left it out of this one. Siblings: #759 (the text decoder's timegm(), merged) and #760. The comparison and benchmark scripts are in https://github.com/czj/ruby-pg/tree/bench/binary-local-timestamp if you'd like to rerun them.

Does the per-value tzset() look acceptable to you, or would you rather leave TZ changes at runtime to the caller?


🤖 Generated with Claude Code, but prompted, read, and verified by @czj, a caring human :)

czj added 2 commits October 26, 2026 21:38
TimestampLocal subtracted the UTC offset of the wall clock read as UTC, which
is one hour off within the offset's distance of a DST change. With
TIMESTAMP_DB_LOCAL alone, the Time was already in UTC mode, so its utc_offset
was 0 and the wall clock came back labelled UTC. Time.local resolves both, and
handles DST gaps and overlaps like PostgreSQL.
The UTC Time was built then thrown away: 2 objects per value instead of 1.
The specs reuse BinaryEncoder::TimestampUtc to build the wire values.
@czj
czj marked this pull request as draft October 27, 2026 09:18
@czj

czj commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

A deeper review found that relying on Time.local trades the 2024 bugs for new ones: on 64-bit macOS it resolves gaps and overlaps the opposite way to PostgreSQL after 2038, and TruffleRuby picks the summer offset on every overlap, so some values master decoded right come back an hour off. I've moved this back to draft and will rework it with PostgreSQL's own rule (the offsets a day before and after the wall clock, taking the later instant when they differ), with specs pinned to PostgreSQL's answers rather than Time.local's. Sorry for the noise!

czj added 3 commits October 27, 2026 12:18
Time.local resolves DST gaps and overlaps the opposite way to PostgreSQL on
64-bit macOS after 2038, and picks the summer offset on every overlap on
TruffleRuby. Instead, take the local UTC offsets a day before and after the
wall clock with localtime_r(), and when they differ, keep the interpretation
that gives back the wall clock, or the later instant in a gap or an overlap,
like DetermineTimeZoneOffset(). The Time is then built like the UTC one,
so its local fields stay lazy, and Time.local is only a fallback when the C
library can't convert the instant.

The specs now compare with PostgreSQL's answers instead of Time.local, up to
2040, and skip when Europe/Paris isn't available as TZ.
Before 1901-12-14, Ruby on macOS renders local times without the LMT offsets
the C library gives, so decoded wall clocks didn't read back as written. Those
dates now keep master's subtraction of Ruby's offset, which has no DST to get
wrong there and covers the whole int64 range, unlike j2date(). The local and
UTC dates are at most a day apart, so their weekdays give the offset without
calendar arithmetic.
@czj czj changed the title Fix binary local timestamps one hour off around DST changes, and TIMESTAMP_DB_LOCAL alone ignored Decode binary local timestamps like PostgreSQL around DST changes (0 mismatches over 2.45M values, 1 object instead of 2) Oct 27, 2026
@czj
czj marked this pull request as ready for review October 27, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant