Repository navigation
Keep string literals byte-exact in the SQL console parser - #58
Merged
Merged
Conversation
The parser upper-cased the whole select list, FROM, GROUP BY and ORDER BY
text, so a literal inside an expression changed its content:
CASE WHEN carrid = 'lh' THEN 'Lufthansa' END became 'LH' / 'LUFTHANSA',
which changed both the comparison and the output. condense( ) in
normalize( ) and on every clause also collapsed blanks inside literals
(WHERE carrname = 'Air Berlin' lost a blank), and the select-list
tokenizer split at blanks and commas inside literals ('a, b' became
'a b', a literal 'END' ended a CASE early).
split_literals( ) now separates the text into literals ('...' and `...`,
a doubled delimiter being an escaped one) and the text between them.
upper_outside_literals( ), condense_outside_literals( ) and tokenize( )
build on it and replace to_upper( ), condense( ) and the
replace-commas-and-split tokenizing in the parser. Identifiers and
keywords are still upper-cased. A plain WHERE literal was not affected
before (WHERE is never upper-cased), only its inner blanks were.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014YZFmF4CrXfWRsmrwPJTRL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
z2ui5_cl_sql_queryupper-cased the whole select list, FROM, GROUP BY and ORDER BY text, so a literal inside an expression changed its content:SELECT CASE WHEN carrid = 'lh' THEN 'Lufthansa' ELSE 'other' END AS name FROM scarrran as'LH'/'LUFTHANSA'/'OTHER'. That changed both the comparison and the output.The same parser also damaged literals in two more ways:
condense( )innormalize( )and on every clause collapsed blanks inside literals.WHERE carrname = 'Air Berlin'lost a blank.'a, b'became'a b', and a literal'end', once upper-cased, closed a CASE early.A plain WHERE literal (
WHERE carrname = 'Lufthansa') was not affected before, because WHERE is never upper-cased. Only blanks inside its literals were collapsed.The fix
split_literals( )splits SQL text into literals and the text between them. It handles'...'and`...`, and a doubled delimiter counts as an escaped one.upper_outside_literals( ),condense_outside_literals( )andtokenize( )build on it. They replaceto_upper( ),condense( )and the replace-commas-and-split tokenizing innormalize,parse_segment,get_items,get_sources,join_by_commaandjoin_order_by.Not part of this change: the keyword search (
FROM,WHERE,ORDER BY,UNION SELECT,INTO,UP TO n ROWS) still runs on the raw text. A literal that contains such a keyword with blanks around it can still split the query in the wrong place.Tests
There are five new tests in
ltcl_parser. All five fail on the old code and pass now:parse_case_literals: the CASE literal exampleparse_where_literal: a plain WHERE literal (this assertion already passed before) and a literal with inner blanksparse_escaped_quote:'o''neil','it''s me'and'O''Neil'parse_mixed_case_identifiers: mixed-case keywords and identifiers are still upper-cased in every clausetarget_case_literal_comma: a comma, a double blank and'end'inside literals, throughbuild_target. The test class is nowLOCAL FRIENDS.npm run checkpasses,npm run unitreports 20 passed, and the 7.02auto_downportlint is clean.Generated by Claude Code