Repository navigation
Prevent out-of-bounds write when generating a keypair - #41
Merged
Merged
Conversation
jimouris
approved these changes
Nov 18, 2026
jimouris
left a comment
Member
There was a problem hiding this comment.
LGTM, thanks for the quick fix!
The Signature and KeyEncapsulation constructors that take a secret key store it at whatever length the caller passes. generate_keypair() then handed that array to liboqs, which writes length_secret_key bytes into it, so a secret key shorter than that overflowed the native buffer and corrupted the heap. KeyEncapsulation.generate_keypair(seed) had the same problem. generate_keypair() now replaces a wrong-length secret key with one of the right length before generating, since the old key is overwritten anyway. The native keypair functions also refuse key buffers and seeds that are too short, so liboqs is never handed a buffer it would overrun. Fixes #40 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Douglas Stebila <dstebila@uwaterloo.ca>
dstebila
force-pushed
the
ds-fix-keygen-short-secret-key
branch
from
November 18, 2026 16:00
8310e02 to
74940b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #40.
Problem
Signature(alg, secret_key)andKeyEncapsulation(alg, secret_key)store the secret key at whatever length the caller passes.generate_keypair()then hands that array to liboqs, which writeslength_secret_keybytes into it. liboqs can't check the buffer size (sig.handkem.hmake the caller responsible), so a too-short secret key overflows the native buffer and corrupts the heap.KeyEncapsulation.generate_keypair(seed)has the same problem. Before this fix, the new tests crash the test JVM on macOS (SIGABRT / SIGTRAP from the allocator's heap-corruption check).sign()anddecap_secret()already check the secret key length; key generation was the only unchecked path.Changes
generate_keypair()andgenerate_keypair(seed)replace a wrong-length secret key with one of the right length (wiping the old one). The old key is overwritten by key generation anyway, so a wrong-length key passed to the constructor now just works.-1if a key buffer is too short for liboqs to write into, or if the seed is too short for it to read from. The check runs before any array elements are fetched, so the error path has nothing to release. It also returns-1on a null handle instead of dereferencing it.Tests
SigTest/KEMTesttestGenerateKeypair*WithWrongLengthSecretKey: construct with secret keys of length 0, 32 and 4096, generate a key pair, and check that sign/verify or encaps/decaps works.testNativeGenerateKeypair*RejectsShortBuffers: call the private native methods through reflection with each buffer one byte short (expect-1) and with exact sizes (expect0).mvn packagepasses locally on macOS: 290 tests, all passing.🤖 Generated with Claude Code