Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,7 @@ Note the following conventions:
Parses SQL string and returns the CST tree. Takes the following options:

- **dialect**: `'sqlite' | 'bigquery' | 'mysql' | 'mariadb' | 'postgresql' | 'plpgsql'` The SQL dialect to parse **(required)**.
- **psql**: `boolean` Enables parsing standalone psql backslash commands and variable references for PostgreSQL. Command text is preserved.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a better approach would be to treat psql as a separate SQL dialect, not as a flag that's only applicable to PostgreSQL.

- **includeRange**: `boolean` When enabled adds `range: [number, number]` field to all CST nodes,
which contains the start and end locations of the node.
- **includeComments**: `boolean` When enabled adds `leading: Whitespace[]` and/or `trailing: Whitespace[]`
Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
"test:plpgsql": "jest --config test/config/plpgsql.json",
"test:all": "yarn test:sqlite && yarn test:bigquery && yarn test:mysql && yarn test:mariadb && yarn test:postgresql && yarn test:plpgsql",
"test": "yarn generate && yarn test:all",
"prepare": "yarn build",
"perf:big": "yarn ts-node perf/perf-test.ts big",
"perf:func": "yarn ts-node perf/perf-test.ts func",
"perf:paren": "yarn ts-node perf/perf-test.ts paren",
Expand Down
1 change: 1 addition & 0 deletions src/ParserOptions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ export type ParserOptions = {
includeSpaces?: boolean;
includeRange?: boolean;
paramTypes?: ParamType[];
psql?: boolean;
/** SQL file name, used when reporting syntax errors */
filename?: string;
};
6 changes: 5 additions & 1 deletion src/cst/Expr.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
WindowDefinition,
} from "./Select";
import { Default } from "./Insert";
import { PsqlIdentifier, PsqlVariable } from "./Psql";
import {
PostgresqlOperator,
PostgresqlOperatorExpr,
Expand All @@ -36,6 +37,7 @@ export type AllExprNodes =
| ArraySliceSpecifier;

export type Expr =
| PsqlVariable
| ListExpr
| ParenExpr
| BinaryExpr
Expand Down Expand Up @@ -436,7 +438,9 @@ export interface ArraySliceSpecifier extends BaseNode {
to?: Expr;
}

export interface Identifier extends BaseNode {
export type Identifier = PlainIdentifier | PsqlIdentifier;

export interface PlainIdentifier extends BaseNode {
type: "identifier";
Comment on lines -439 to 444

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't good. We should keep the type field in sync with type name. Like, when type is called PlainIdentifier, then I'd expect it to have type: "plain_identifier".

In here, we likely don't want to change the type: "identifier" at all.

I'm unsure about the best direction where we should go. I guess the main intention in here is to allow Psql identifiers in all places where previously only SQL identifiers were allowed. But frankly, I'm not really even sure if we should allow that in the first place.

text: string;
name: string;
Expand Down
1 change: 1 addition & 0 deletions src/cst/Node.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ export * from "./ProcClause";
export * from "./Procedure";
export * from "./ProceduralLanguage";
export * from "./Program";
export * from "./Psql";
export * from "./Publication";
export * from "./RenameTable";
export * from "./Role";
Expand Down
1 change: 1 addition & 0 deletions src/cst/Program.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ import { Statement } from "./Statement";
export interface Program extends BaseNode {
type: "program";
statements: Statement[];
psql?: boolean;
}
24 changes: 24 additions & 0 deletions src/cst/Psql.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import { BaseNode } from "./Base";
import { Statement } from "./Statement";

export interface PsqlCommand extends BaseNode {
type: "psql_command";
text: string;
}

export interface PsqlQuery extends BaseNode {
type: "psql_query";
query: Statement;
terminator: string;
}

export interface PsqlVariable extends BaseNode {
type: "psql_variable";
text: string;
}

export interface PsqlIdentifier extends BaseNode {
type: "psql_identifier";
name: string;
text: string;
}
3 changes: 3 additions & 0 deletions src/cst/Statement.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,11 @@ import { MergeStmt } from "./Merge";
import { TruncateStmt } from "./Truncate";
import { UpdateStmt } from "./Update";
import { DoStmt } from "./Do";
import { PsqlCommand, PsqlQuery } from "./Psql";

export type Statement =
| PsqlCommand
| PsqlQuery
| AllAlterTableStatements
| AllBigqueryStatements
| AllCommentStatements
Expand Down
62 changes: 59 additions & 3 deletions src/parser.pegjs
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,64 @@
}

start
= c1:__ program:program c2:__ {
= &{ return options.psql && options.dialect === "postgresql"; } p:psql_program { return p; }
/ c1:__ program:program c2:__ {
return surrounding(c1, program, c2);
}

psql_program
= items:(c1:__ x:(psql_command / psql_query) { return surrounding(c1, x, []); })* c2:__ {
return loc({ type: "program", statements: items, psql: true, trailing: c2 });
}

psql_query
= query:statement c:__ terminator:(";" / psql_query_command / !. { return ""; }) {
return loc({ type: "psql_query", query: trailing(query, c), terminator });
}

psql_query_command
= &psql_line_start text:$( "\\" ("gdesc" / "gexec" / "gset" / "gx" / "g") ![a-zA-Z_] psql_command_arguments) {
return text;
}

psql_command
= &psql_line_start text:$("\\" psql_command_name psql_command_arguments) {
return loc({ type: "psql_command", text });
}

psql_command_name
= [^ \t\r\n\\]+

psql_line_start
= &{ return input.slice(input.lastIndexOf("\n", peg$currPos - 1) + 1, peg$currPos).trim() === ""; }

psql_command_arguments
= (psql_single_quoted_argument / psql_double_quoted_argument / psql_backquoted_argument / !("\\" / end_of_line / "'" / "\"" / "`") .)*

psql_single_quoted_argument
= "'" ("''" / "\\" . / !"'" .)* "'"

psql_double_quoted_argument
= "\"" (!"\"" .)* "\""

psql_backquoted_argument
= "`" (!"`" .)* "`"

psql_variable
= &{ return options.psql && options.dialect === "postgresql"; }
text:$(":" ("'" psql_variable_name "'" / psql_variable_name / "{?" psql_variable_name "}")) {
return loc({ type: "psql_variable", text });
}

psql_identifier
= &{ return options.psql && options.dialect === "postgresql"; }
":" '"' name:$psql_variable_name '"' {
return loc({ type: "psql_identifier", name, text: text() });
}

psql_variable_name
= [a-zA-Z_0-9]+

program
= head:(statement / empty) tail:(__ ";" __ (statement / empty))* {
return loc({
Expand Down Expand Up @@ -8116,7 +8170,8 @@ array_slice_specifier
}

primary
= literal
= psql_variable
/ literal
/ primary_paren_expr
/ paren$compound_select_stmt
/ &bigquery x:(typed_array_expr / array_expr / typed_struct_expr) { return x; }
Expand Down Expand Up @@ -8868,7 +8923,8 @@ explicit_alias_ident
}

ident "identifier"
= quoted_ident
= psql_identifier
/ quoted_ident
/ name:ident_name !{ return isReservedKeyword(name); } {
return loc(createIdentifier(name, name));
}
Expand Down
2 changes: 1 addition & 1 deletion src/showNode/base.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@ export const baseMap: FullTransformMap<
all_columns: () => "*",
empty: () => "",
alias: (node) => show([node.expr, node.asKw, node.alias, node.columnAliases]),
program: (node) => show(node.statements, ";"),
program: (node) => show(node.statements, node.psql ? "" : ";"),
};
2 changes: 2 additions & 0 deletions src/showNode/expr.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ import { AllExprNodes } from "../cst/Node";
import { FullTransformMap } from "../cstTransformer";

export const exprMap: FullTransformMap<string, AllExprNodes> = {
psql_variable: (node) => node.text,
psql_identifier: (node) => node.text,
list_expr: (node) => show(node.items, ","),
paren_expr: (node) => "(" + show(node.expr) + ")",
binary_expr: (node) => show([node.left, node.operator, node.right]),
Expand Down
3 changes: 3 additions & 0 deletions src/showNode/transformMap.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { alterActionMap } from "./alter_action";
import { show } from "../show";
import { alterTableMap } from "./alter_table";
import { analyzeMap } from "./analyze";
import { baseMap } from "./base";
Expand Down Expand Up @@ -48,6 +49,8 @@ import { sqliteMap } from "./dialects/sqlite";
import { mysqlMap } from "./dialects/mysql";

export const transformMap: FullTransformMap<string> = {
psql_command: (node) => node.text,
psql_query: (node) => show(node.query) + node.terminator,

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not good. This file should only import various -Map objects and combine them to one large transformMap.

...baseMap,

// SELECT, INSERT, UPDATE, DELETE, TRUNCATE, MERGE
Expand Down
34 changes: 34 additions & 0 deletions test/psql.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
import { parse, show } from "../src/main";
import { includeAll } from "./test_utils";

describe("psql scripts", () => {
it.each([
"\\set id 1\nSELECT :id, :'name' FROM :\"table\";\n\\quit\n",
"\\if :enabled\nSELECT 1\n\\gset result_\n\\else\nSELECT 2;\n\\endif\n",
"\\restrict abc123\nSELECT 1;\n\\unrestrict abc123\n",
"-- heading\nSELECT 1; -- trailing\n\\echo 'done'\n",
"SELECT 1\n\\gexec\nSELECT 2;",
"SELECT 1",
"\\echo 'two words with ''quotes'''\n",
'\\echo "two words"\n',
"\\echo `printf hi`\n",
"\\unknown value\n",
"SELECT :id, :'name', :\"table\", :{?enabled};",
])("round-trips %s", (sql) => {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test looks kinda horrendous. All these escape codes make my head hurt. Plus, it looks like we're testing all kinds of different things in this single it.each() block.

I think we should split this up to several separate tests, each with a clear description of what exactly we're testing.

Maybe we could use String.raw to avoid writing all these escapes. Or maybe we can adopt the dedent helper to aid us. Not entirely sure, but in the current form, this code is pretty hard to understand.

const tree = parse(sql, { dialect: "postgresql", psql: true, ...includeAll });
expect(show(tree)).toBe(sql);
expect(tree.psql).toBe(true);
});

it.each(["SELECT\n\\echo partial\n1;", "SELECT 1 \\gset", "\\echo hi \\\\ SELECT 1;"])(
"rejects unsupported command placement or syntax: %s",
(sql) => {
expect(() => parse(sql, { dialect: "postgresql", psql: true })).toThrow();
}
);

it("requires opt-in and the PostgreSQL dialect", () => {
expect(() => parse("\\quit", { dialect: "postgresql" })).toThrow();
expect(() => parse("\\quit", { dialect: "sqlite", psql: true })).toThrow();
});
});
Loading