Skip to content

open-quantum-safe/liboqs-java — a short secret_key array is passed straight to OQS_SIG_keypair as the native output buffer (out-of-bounds write) #40

Closed

Description

@jiliheng0123

Summary

In the JNI wrapper, Signature.generate_keypair() pins the caller-supplied secret_key byte array
and hands it directly to OQS_SIG_keypair() as the native secret-key output buffer, with no check
that the array is large enough for the algorithm's required secret-key size. The sign() path does
validate the secret-key length, but the key-pair generation path does not, so an array that is
shorter than sig->length_secret_key lets liboqs write the full secret key past the end of the
buffer. For ML-DSA-44 the native write is 2560 bytes; if the caller passes, e.g., a 32-byte array, the
write overruns the buffer by ~2.5 KiB — a native heap out-of-bounds write (CWE-787 / CWE-120) and
memory corruption. Severity: 4.8 (Medium).

Details

In
src/main/c/Signature.c#L92-L107,
Java_org_openquantumsafe_Signature_generate_1keypair copies both output arrays into native pointers
and calls the real key-pair routine without any size check:

jbyte *public_key_native = (*env)->GetByteArrayElements(env, jpublic_key, 0);
jbyte *secret_key_native = (*env)->GetByteArrayElements(env, jsecret_key, 0);

OQS_SIG *sig = (OQS_SIG *) getHandle(env, obj, "native_sig_handle_");

// Invoke liboqs sig keypair generation function
OQS_STATUS rv_ = OQS_SIG_keypair(sig, (uint8_t*) public_key_native, (uint8_t*) secret_key_native);

OQS_SIG_keypair writes sig->length_public_key bytes into the public-key buffer and
sig->length_secret_key bytes into the secret-key buffer. Neither array length is compared against
those values before the call. The Java side makes this reachable: the Signature(alg, byte[] secret_key)
constructor copies the caller's array as-is into the instance
(Arrays.copyOf(secret_key, secret_key.length)), and generate_keypair() forwards that array to the
native method without checking secret_key_.length >= sig.getDetails().getLengthSecretKey(). The
sign() method performs exactly that length check, but the key-pair generation path does not — the
gap that triggers this defect.

Fix. Validate the array capacity against the algorithm requirement before the native write, and
reject an undersized buffer:

OQS_SIG *sig = (OQS_SIG *) getHandle(env, obj, "native_sig_handle_");
size_t sklen = OQS_SIG_length_secret_key(sig);
jsize skcap  = (*env)->GetArrayLength(env, jsecret_key);
if (skcap < sklen) {
    (*env)->ReleaseByteArrayElements(env, jsecret_key, secret_key_native, JNI_ABORT);
    /* throw an exception / return an error here */
    return -1;
}
OQS_STATUS rv_ = OQS_SIG_keypair(sig, (uint8_t*) public_key_native, (uint8_t*) secret_key_native);

The same capacity check must apply to jpublic_key against OQS_SIG_length_public_key(sig), and to
the Java-side generate_keypair() before it forwards the arrays. With a correctly sized (≥2560-byte)
secret-key array, the identical call completes without an out-of-bounds write.

Impact

Out-of-bounds write / memory corruption in a native JNI helper (CWE-787 / CWE-120). The write size is
under the caller's control only through the choice of a short input array; the defect is local to the
library and its JNI boundary. This is a memory-safety defect in the Java/native transition, not a claim
of remote code execution or a cryptographic break of the signature scheme. Users affected are
applications and developers calling Signature(alg, byte[]).generate_keypair() with an undersized
secret-key array. Whether the out-of-bounds write is externally reachable and whether it escalates to
code execution is not demonstrated and is left for maintainer confirmation.

Activity

  1. dstebila commented on Nov 18, 2026

    @dstebila
    Member

    Thanks for the report. Could take a look at #41 and see if it addresses your concern?

  2. self-assigned this
    on Nov 18, 2026
  3. added a commit that references this issue on Nov 18, 2026
    74940b5
  4. jiliheng0123 commented on Nov 19, 2026

    @jiliheng0123
    Author

    I think it works. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions