Repository navigation
Decode text timestamps ~40x faster on macOS, ~15x on Windows and ~1.4x on Linux, without timegm() - #759
Conversation
macOS implements timegm() as a binary search over gmtime(), taking ~3.4µs per call. Counting days arithmetically decodes timestamps ~40x faster there, keeps nanoseconds for years the libc can't convert, and gives Windows the fast path.
|
Good catch! |
- Windows' mktime() fails on certain dates (be07045), so local timestamps keep using Time.new there, as before dropping the timegm() check - The slow path keeps nanoseconds, so 32 bit time_t builds decode the same values as the fast path - The UTC and explicit offset branches share one computation, and struct tm is only filled for mktime() - Specs cover february, leap years, centuries and BC dates
On Windows long is 32 bit, so Fixnums stop at ~1.07 billion and sec * 1000000000 + nsec became a Bignum: TimestampLocal, which still takes Time.new there, decoded 12% slower with 17.8 objects per value instead of 10. Microseconds, all PostgreSQL sends, keep the numerator a Fixnum again.
|
You're right, thanks! Windows has no |
|
I think this is OK how it is. Thank you very much! I'll check if the Update: PR is here #764 |
|
Thanks @larskanis, and nice to see #764 simplify the Windows path! |
Following up on #758: while profiling our Rails app on macOS, decoding a
timestampturned out to cost ~3.6 µs per value, 50 times an integer. Nearly all of it istimegm(), which macOS implements as a binary search overgmtime()(~3.4 µs per call, against 33–55 ns with glibc).This counts the seconds since the epoch arithmetically instead (Howard Hinnant's
days_from_civil), for timestamps with an explicit offset and for timestamps without one read as UTC, which is what Rails does by default. Windows gets the fast path for these too: it has notimegm(), so it always went throughTime.new.TIMESTAMP_DB_LOCALstill goes throughmktime(), soTimestampLocal(and Rails withdefault_timezone = :local) stays as slow as before on macOS, and keeps usingTime.newon Windows, where be07045 explains whymktime()must stay off. Values a 32-bittime_tcan't hold still take the slow path, which now keeps nanoseconds too, so both paths return the sameTime.Each figure is the median of 7 passes, each averaging 10,000 values, with Ruby 4.0.9. On GitHub Actions, both versions were compiled from source with the same flags and run alternately, and the table shows the median of 3 rounds (run). macOS was measured locally:
TimestampUtcTimestampWithTimeZoneTimestampLocalOn Windows, the fast path also allocates 1 object per value instead of 10.
In our app, an e-commerce page decodes 750 to 3,100 timestamps (
created_at/updated_atadd up). That's 2.6 to 11 ms per page in development on a Mac, and 0.03 to 0.2 ms in production on Linux.To check equivalence, I decoded 200,000 random timestamps (years 1 to 294276, BC, offsets in hours, minutes and seconds) with the 4 text timestamp decoders, with pg 1.7.0 and with this branch: under 3 time zones on macOS and Linux (x86_64 and arm64), and under the default one on Windows. Output is identical for every value with up to 6 fractional digits, i.e. everything PostgreSQL sends, and 100,000 values with 9 fractional digits match
Time.utcat nanosecond precision on every platform. Beyond microseconds, pg 1.7.0 truncated whenever it fell back to the slow path (macOS before 1900 or BC, Windows, 32-bittime_t); the new spec covers that and fails on macOS with pg 1.7.0. I also ran the type specs with the fast path disabled, to cover 32-bit builds, and a new spec walks february, leap years, centuries and BC dates.One trade-off: with a leap-second zone such as
TZ=right/UTC, glibc'stimegm()counted leap seconds, as Ruby'sTimedoes there, while the arithmetic doesn't, likePG::BinaryDecoder::Timestampalready. On glibc under such a zone, UTC timestamps would now come out 27 seconds apart fromTime.utc. These zones are rare, but I'd rather mention it. Is it acceptable for you, or should the decoder keeptimegm()on glibc?Full suite: 1,166 examples, 0 failures except
tells about the libpq library path, which only comes from my build setup (compiled withextconf.rb+makerather thanrake compile).🤖 Generated with Claude Code, but prompted, read, and verified by @czj, a caring human :)