Skip to content

fix: always return a Buffer from buffer() - #505

Open
officialasishkumar wants to merge 1 commit into
vercel:mainfrom
officialasishkumar:fix/buffer-ignore-request-charset
Open

officialasishkumar wants to merge 1 commit into
vercel:mainfrom
officialasishkumar:fix/buffer-ignore-request-charset

Conversation

@officialasishkumar

Copy link
Copy Markdown

Closes #504 (originally reported in #419)

buffer() passed the request's Content-Type charset to raw-body as encoding. So a request with Content-Type: application/json; charset=utf-8 made buffer() resolve with a decoded string instead of a Buffer, and that string was then cached for the request. Handlers that need the raw bytes, such as webhook signature checks, behaved differently depending on what the client sent.

Changes

  • buffer() always reads and caches the raw body as a Buffer, which matches its type and the README.
  • text(), and so json(), decodes that buffer using encoding, or the Content-Type charset if encoding isn't set. Decoding uses iconv-lite@0.4.24, the version raw-body already uses internally, so the supported charsets and BOM handling stay the same. It adds no new package to the lockfile.
  • Unsupported charsets and malformed Content-Type headers still return 400 from text() and json() before the body is read.
  • README and a changeset are updated.

Note: buffer(req, { encoding }) used to return a string. It now returns the Buffer. To get a string, use text(req, { encoding }).

Request buffer() before buffer() now
no charset Buffer Buffer
charset=utf-8 string Buffer
json() called first, charset=utf-8 string Buffer

How to test

module.exports = async (req) => ({ isBuffer: Buffer.isBuffer(await buffer(req)) });
curl -d 'foo' -H 'Content-Type: application/json; charset=utf-8' localhost:3000
# before: {"isBuffer":false}, after: {"isBuffer":true}

In test/suite/index.ts:

  • New buffer() tests: it returns a Buffer when Content-Type has a charset, it still returns one after text() has decoded the body, and it never parses Content-Type. All three fail without this change.
  • New text() tests: it decodes iso-8859-1 bodies and returns 400 for an unsupported charset.
  • The two existing tests that passed encoding to buffer() now call text().

pnpm test, type-check and eslint-check pass.

buffer() passed the charset from the request's Content-Type header to
raw-body as `encoding`, so any request with e.g.
`Content-Type: application/json; charset=utf-8` made buffer() resolve
with a decoded string instead of the Buffer it is typed and documented
to return. The decoded string was also what got cached for the request,
so calling text() or json() first made a later buffer() call return a
string as well. Code that needs the raw bytes, such as webhook
signature verification, broke depending on what the client sent.

buffer() now always reads and caches the raw body as a Buffer. text()
(and therefore json()) decodes that Buffer with `encoding`, falling
back to the Content-Type charset as before. Decoding uses iconv-lite,
the same version raw-body already uses internally, so supported
charsets and BOM handling are unchanged. Unsupported charsets and
malformed Content-Type headers are still rejected with a 400 before the
body is read.

Tests:
- buffer() returns a Buffer when Content-Type has a charset, and after
  text() has decoded the body.
- buffer() doesn't parse Content-Type at all.
- text() decodes with the Content-Type charset (iso-8859-1), rejects an
  unsupported charset with 400, and skips parsing Content-Type when an
  encoding is given.

Closes vercel#504
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

buffer() returns a string instead of a Buffer when Content-Type has a charset

1 participant