Skip to content

DecodeStream.readString strips a leading U+FEFF from UTF-8 and UTF-16 strings #69

Closed

Description

@trackoor

Decoding a String field in 'utf8', 'utf16le', 'ucs2', or 'utf16be' strips a leading U+FEFF code point, including when decoding bytes produced by toBuffer() or reading binary string fields such as OpenType name table records in fontkit.

// repro.mjs
import * as r from 'restructure';

const show = s => JSON.stringify(s).replace(/\uFEFF/g, '\\uFEFF');

for (const enc of ['utf8', 'utf16le', 'ucs2', 'utf16be']) {
  const str = new r.String(r.uint8, enc);
  const buf = str.toBuffer('\uFEFFA');
  console.log(`${enc}:`, show(str.fromBuffer(buf)));
}

console.log('utf16be bytes:', show(new r.String(4, 'utf16be').fromBuffer(new Uint8Array([0xfe, 0xff, 0x00, 0x41]))));
console.log('utf8 bytes:', show(new r.String(4, 'utf8').fromBuffer(new Uint8Array([0xef, 0xbb, 0xbf, 0x41]))));

Run with npm install restructure@3.0.2 && node repro.mjs:

Actual output:

utf8: "A"
utf16le: "A"
ucs2: "A"
utf16be: "A"
utf16be bytes: "A"
utf8 bytes: "A"

Expected output (matching restructure@2.0.1):

utf8: "\uFEFFA"
utf16le: "\uFEFFA"
ucs2: "\uFEFFA"
utf16be: "\uFEFFA"
utf16be bytes: "\uFEFFA"
utf8 bytes: "\uFEFFA"

DecodeStream.readString() in src/DecodeStream.js calls new TextDecoder(encoding) with default options (ignoreBOM: false), which removes a leading U+FEFF instead of preserving it as new TextDecoder(encoding, { ignoreBOM: true }) does.

Versions: restructure@3.0.2 and master (21346d5) on Node.js v26.13.0 (fontkit@3.0.0 also returns "Fam" instead of "\uFEFFFam" for a platform 3 UTF-16BE name record).

Credit: https://ucsbplse.github.io/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions