Skip to content

Prevent out-of-bounds write when generating a keypair - #41

Merged
dstebila merged 1 commit into
masterfrom
ds-fix-keygen-short-secret-key
Nov 18, 2026
Merged

dstebila merged 1 commit into
masterfrom
ds-fix-keygen-short-secret-key

Conversation

@dstebila

Copy link
Copy Markdown
Member

Fixes #40.

Problem

Signature(alg, secret_key) and KeyEncapsulation(alg, secret_key) store the secret key at whatever length the caller passes. generate_keypair() then hands that array to liboqs, which writes length_secret_key bytes into it. liboqs can't check the buffer size (sig.h and kem.h make the caller responsible), so a too-short secret key overflows the native buffer and corrupts the heap. KeyEncapsulation.generate_keypair(seed) has the same problem. Before this fix, the new tests crash the test JVM on macOS (SIGABRT / SIGTRAP from the allocator's heap-corruption check).

sign() and decap_secret() already check the secret key length; key generation was the only unchecked path.

Changes

  • Java: before generating, generate_keypair() and generate_keypair(seed) replace a wrong-length secret key with one of the right length (wiping the old one). The old key is overwritten by key generation anyway, so a wrong-length key passed to the constructor now just works.
  • JNI (backstop): the three native keypair functions return -1 if a key buffer is too short for liboqs to write into, or if the seed is too short for it to read from. The check runs before any array elements are fetched, so the error path has nothing to release. It also returns -1 on a null handle instead of dereferencing it.

Tests

  • SigTest / KEMTest testGenerateKeypair*WithWrongLengthSecretKey: construct with secret keys of length 0, 32 and 4096, generate a key pair, and check that sign/verify or encaps/decaps works.
  • testNativeGenerateKeypair*RejectsShortBuffers: call the private native methods through reflection with each buffer one byte short (expect -1) and with exact sizes (expect 0).

mvn package passes locally on macOS: 290 tests, all passing.

🤖 Generated with Claude Code

@jimouris jimouris left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the quick fix!

The Signature and KeyEncapsulation constructors that take a secret key
store it at whatever length the caller passes. generate_keypair() then
handed that array to liboqs, which writes length_secret_key bytes into
it, so a secret key shorter than that overflowed the native buffer and
corrupted the heap. KeyEncapsulation.generate_keypair(seed) had the
same problem.

generate_keypair() now replaces a wrong-length secret key with one of
the right length before generating, since the old key is overwritten
anyway. The native keypair functions also refuse key buffers and seeds
that are too short, so liboqs is never handed a buffer it would overrun.

Fixes #40

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Douglas Stebila <dstebila@uwaterloo.ca>
@dstebila
dstebila force-pushed the ds-fix-keygen-short-secret-key branch from 8310e02 to 74940b5 Compare November 18, 2026 16:00
@dstebila
dstebila merged commit cd555d4 into master Nov 18, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

open-quantum-safe/liboqs-java — a short secret_key array is passed straight to OQS_SIG_keypair as the native output buffer (out-of-bounds write)

2 participants