Lexer rejects a UTF-8 BOM, and a lone \r does not end a line #9108

Found by a bug hunt over the 16 commits from eb087382 to cd641cbc.

UTF-8 BOM. A file that starts with a BOM is rejected by both bsc and bsc -format, with one "Not sure what to do with this character" error per BOM byte (\239, \187, \191):

printf '\xef\xbb\xbflet x = 1\n' > A.res
bsc A.res

Some Windows editors write a BOM by default.

Lone \r. A \r without a \n ends a // comment but does not count as a newline:

printf 'let x = 1 // c\rlet y = 2\n' > B.res
bsc B.res   # Consecutive statements on a line must be separated by ';' or a newline

Both reproduce at b123b7c3.

🤖 Generated with Claude Code