Skip to content

Add a stringDecoder option to customise lazy string decoding - #57

Open
scottsue wants to merge 1 commit into
streamnative:masterfrom
scottsue:string-decoder-hook
Open

scottsue wants to merge 1 commit into
streamnative:masterfrom
scottsue:string-decoder-hook

Conversation

@scottsue

Copy link
Copy Markdown

Fixes #56

Motivation

A parsed message creates a new String each time it decodes a lazy string field. Applications that read the same names, codes, or keys in every message cannot intern them, because the raw bytes are private. See #56 for numbers.

Change

  • A new generator option, stringDecoder: a fully qualified class name with public static String decode(ByteBuf buffer, int index, int length).
    • Maven: <stringDecoder>com.example.StringInterner</stringDecoder>
    • Gradle: lightproto { stringDecoder = 'com.example.StringInterner' }
    • API: a new LightProtoGenerator.generate(...) overload. The old overload passes null.
  • Every lazy decode in generated code already goes through LightProtoCodec.readString(ByteBuf, int, int), and the codec is copied into each generated package. When the option is set, the generator replaces the body of that method with a call to the user's decode(). This covers singular and repeated string fields, map keys and values, materialize(), copyFrom(), and equals(). JSON and TextFormat parsing use a different reader, so they do not use it.
  • When the option is not set, the generated code does not change. I generated the tests module protos with master and with this branch, and diff -r found no difference.
  • README: a new section, "Custom string decoding (interning)", with the contract.

Tests

  • tests: a second plugin execution generates string_decoder.proto with the option set. TestStringInterner is a small reference decoder: a bounded, lock-free, direct-mapped table keyed by a hash of the bytes.
  • StringDecoderTest (6 tests):
    • values equal the plain decode for ASCII and non-ASCII strings;
    • the same instance comes back across parses for singular, repeated, and map fields;
    • materialize() and copyFrom() use the decoder, and the results survive a zeroed buffer;
    • clear() and reuse work;
    • a steady-state allocation check: parse plus five string reads is 0 B/op with the interner, compared with 240 B/op with the built-in decode.
  • mvn clean install on JDK 21 passes: 605 tests in each tests run (default and -XX:-CompactStrings), 0 failures.

Open points for review

  • Mechanism: I replaced the body of readString() and did not pass the option to each field class. That keeps the generator change to 6 lines. If you prefer the option to appear at each call site, I can change it.
  • Call count: materialize() decodes a singular string again after its getter has already decoded it. It checks the buffer index and not the cached value. So the README says that decode() is "usually" called once for each value. A fix would change the default output, so I left it for a separate change.
  • Field context: the decoder does not know the field. A user can choose what to cache only from the bytes, for example by length.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WUshTFQ4hEUxEQAySamFiV

A parsed message decodes a string field on the first getter call and
creates a new String each time. An application that reads the same
names, codes or keys in every message cannot intern them, because the
raw bytes are private. The new stringDecoder option names a class whose
static decode(ByteBuf, int, int) method then decodes every lazy string.
If the option is not set, the generated code does not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WUshTFQ4hEUxEQAySamFiV
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow lazy string fields to be interned (custom string decoder)

1 participant