JavaScript: Migrate the TypeScript parser to TypeScript 7.1 (based on Go) by paldepind · Pull Request #22841 · github/codeql

paldepind · 2026-10-19T09:59:23Z

Migrate the TypeScript dependency in the TypeScript extractor to 7.1. This entail:

Fairly large changes to the wrapper around TypeScript. Serializing AST nodes to JSON is now a bit harder, since the new node types allow access to children through getters, and these getters are available on every node kind. So we can't just iterate over properties on the nodes. To handle this pull in dist/api/node/protocol.generated.js which contains a map from node kinds to the children. This is hacky (the API is not public) but it works.
Changes to the bazel build as we now need to include native binaries in the distribution. These binaries are not included in the typescript npm package. Instead they're in sibling packages. For instance @typescript/typescript-linux-x64 contains the binary for Linux on x86.

Note to reviewer: This code was created with heavy LLM assistance and I'm not familiar with the code base, but I've done my best to review and polish up things. I've focused primarily on the core of the TypeScript part (main.ts and ast_extractor.ts) as well as the Bazel changes. I've mostly skimmed the changes to the Java part.

Beware:

The changes to javascript/extractor/lib/typescript/README.md are full of junk that we certainly don't want to merge. I've left them in just in case they're useful for a review, but I propose that we simply undo those changes before merging.
Copilot has added a lot of tests and some of them don't even run in CI. I suspect that we have enough verification as-is and that we don't really need them. But again, if left them in just so that a reviewer has a chance to see them.

Copilot

🔵 Needs a closer look

The production parser and packaging overhaul still lacks Linux and Windows execution coverage and needs human review.

3 open issues

2 files not reviewed

javascript/extractor/lib/typescript/package-lock.json: Generated file
javascript/extractor/lib/typescript/ts7/package-lock.json: Generated file

What changed in this PR

Migrates the JavaScript extractor’s TypeScript parser to native 7.1 while preserving its Java-facing AST protocol.

Changes:

Ports AST projection and manages native resources per file.
Separates common JavaScript files from platform-specific executables.
Adds compatibility, lifecycle, packaging, and diagnostic coverage.

File	Description
`misc/bazel/pkg.bzl`	Supports architecture-specific imported ZIPs.
`javascript/ql/test/library-tests/TypeScript/SourceImports/tst.ts`	Adds source-phase import fixture.
`javascript/ql/test/library-tests/TypeScript/SourceImports/test.ql`	Checks imports and references.
`javascript/ql/test/library-tests/TypeScript/SourceImports/test.expected`	Records expected bindings.
`javascript/ql/integration-tests/diagnostics/internal-error/test.py`	Enables injected native failure.
`javascript/ql/integration-tests/diagnostics/internal-error/fail-native.cjs`	Terminates the spawned child.
`javascript/extractor/tests/ts/output/trap/import-attributes.ts.trap`	Records import-attribute extraction.
`javascript/extractor/test/com/semmle/js/extractor/test/BUILD.bazel`	Adds native runtime test inputs.
`javascript/extractor/test/com/semmle/js/extractor/test/AllTests.java`	Installs ZIPs with executable permissions.
`javascript/extractor/lib/typescript/tsconfig.json`	Updates compilation for ESM.
`javascript/extractor/lib/typescript/ts7/wrapper.test.mjs`	Tests protocol compatibility.
`javascript/extractor/lib/typescript/ts7/probe.mjs`	Probes native API compatibility.
`javascript/extractor/lib/typescript/ts7/package.json`	Pins probe dependencies.
`javascript/extractor/lib/typescript/ts7/package-lock.json`	Locks probe packages.
`javascript/extractor/lib/typescript/test/package_test.py`	Tests installed runtime archives.
`javascript/extractor/lib/typescript/test/lifecycle.test.mjs`	Tests process lifecycle and failures.
`javascript/extractor/lib/typescript/src/virtual_source_root.ts`	Removes legacy source-root mapping.
`javascript/extractor/lib/typescript/src/typescript.js`	Loads native API entry points.
`javascript/extractor/lib/typescript/src/typescript.d.ts`	Declares native API exports.
`javascript/extractor/lib/typescript/src/main.ts`	Implements per-file native parsing.
`javascript/extractor/lib/typescript/src/common.ts`	Removes legacy project management.
`javascript/extractor/lib/typescript/src/ast_extractor.ts`	Projects native ASTs into JSON.
`javascript/extractor/lib/typescript/README.md`	Documents migration and performance tradeoffs.
`javascript/extractor/lib/typescript/package.json`	Updates runtime and build commands.
`javascript/extractor/lib/typescript/package-runtime.mjs`	Selects and stages native executables.
`javascript/extractor/lib/typescript/package-lock.json`	Locks production runtime packages.
`javascript/extractor/lib/typescript/BUILD.bazel`	Builds split archives and packaging tests.
`javascript/BUILD.bazel`	Includes native packages in distributions.

_{Review effort: Max}

asgerf · 2026-10-19T11:09:15Z

+            $end: node.end,
+        };
+        const properties = node as ts.Node & { [property: string]: unknown };
+        for (const property of astProperties) {


Looping over the whole astProperties array for every node seems very slow. Could we replace this with a guarded for..in loop or equivalent?

Allow imported ZIP prefixes in arch_overrides so native dependencies can be distributed separately from architecture-independent files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9f12f7cb-26aa-405b-a1f0-9dcc239778cb

Split the wrapper and native runtimes into common and architecture-specific archives. Select and verify pinned platform packages, preserve executable permissions, and use ZIP response files on Windows. Add installed-package smoke tests covering platform selection, archive layout, runtime compatibility, and the merged distribution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9f12f7cb-26aa-405b-a1f0-9dcc239778cb

Copilot

🔵 Needs a closer look

The cross-platform runtime migration has unresolved correctness and resource-budget issues that require human review.

4 open issues

3 resolved since last review

1 file not reviewed

javascript/extractor/lib/typescript/package-lock.json: Generated file

_{Review effort: Max}

paldepind · 2026-10-27T11:26:35Z

-/** Property names to extract in an AST response. */
-const astPropertySet = new Set([...astProperties, ...astMetaProperties]);
+function getAPI(): API {
+    return api ??= new API();


That's on purpose really. I'm not sure we have to worry about this, now that the extractor only does parsing and not type inferrence. We might go the other way and remove more of the memory limits set on the Java side.

+    return source.substring(start, name.getLoc().getStart().getOffset())
+        .matches("(?U)(?:\\uFEFF|" + WHITESPACE_CHAR + ")*");


+      start = modifier.getAsJsonObject().get("$end").getAsInt();
    }
+    return source.substring(start, name.getLoc().getStart().getOffset())
+        .matches("(?U)(?:\\uFEFF|" + WHITESPACE_CHAR + ")*");


paldepind · 2026-10-27T12:27:06Z

+    copyFileSync(join(root, "build", name), join(directory, name));
+}
+copyFileSync(join(root, "package.json"), join(directory, "package.json"));
+symlinkSync(join(root, "node_modules"), join(directory, "node_modules"), "dir");


I think we'll delete this test anyway.

Pin the native compiler and adopt its ESM API. Project native AST nodes into the existing JSON protocol and adapt Java conversion to native fields, token kinds, namespaces, and import attributes. Reuse one lazy API instance across files and prepare-files batches, disposing each program after serialization. Close the API on reset, quit, and EOF; rely on process-exit cleanup for fatal errors. Add wrapper, lifecycle, and Java regression coverage, teach the Java test harness to install native runtimes, and regenerate import-attribute expectations. Cover source-phase import bindings with a QL fixture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8f505060-5341-43f3-b8e9-1b3d60171a2d

Use the wrapper exit-code classification after IPC failures so a native prefetch crash is reported consistently with malformed responses. Preserve the communication-error fallback when the wrapper is still running. Make the internal-error integration test terminate the native child instead of relying on the legacy parser's stack overflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9f12f7cb-26aa-405b-a1f0-9dcc239778cb

Describe the pinned compiler, build and packaging workflow, AST projection, shared-API lifecycle, compatibility coverage, and performance comparisons. Raise the documented minimum Node.js version to 16.20.0. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9f12f7cb-26aa-405b-a1f0-9dcc239778cb

paldepind · 2026-10-27T12:52:15Z

 For TypeScript extraction on all platforms:

- Node.js 14 or higher must be installed and available on the ``PATH`` as ``node``.
+- Node.js 16.20.0 or higher must be installed and available on the ``PATH`` as ``node``.


This is the minimum version that the updated typescript package requires. I've tested locally with this Node.js version and things seem to work.

paldepind requested review from a team as code owners October 19, 2026 09:59

Copilot AI max review requested due to automatic review settings October 19, 2026 09:59

github-license-compliance Bot found potential problems Oct 19, 2026

View reviewed changes

Comment thread javascript/extractor/lib/typescript/ts7/package-lock.json Fixed

Copilot started reviewing on behalf of paldepind October 19, 2026 09:59

View session

Copilot AI reviewed Oct 19, 2026

View reviewed changes

Comment thread javascript/extractor/lib/typescript/package.json Outdated

Comment thread javascript/extractor/lib/typescript/ts7/probe.mjs Outdated

Comment thread javascript/ql/integration-tests/diagnostics/internal-error/test.py

github-actions Bot added JS documentation labels Oct 19, 2026

paldepind marked this pull request as draft October 19, 2026 11:09

asgerf reviewed Oct 19, 2026

View reviewed changes

paldepind force-pushed the js/typescript-7.1 branch 4 times, most recently from 4b0c1ed to 4a1f6e7 Compare October 22, 2026 11:37

paldepind and others added 2 commits October 27, 2026 08:58

paldepind force-pushed the js/typescript-7.1 branch from 4a1f6e7 to 77c4112 Compare October 27, 2026 09:11

paldepind changed the title ~~JavaScript: migrate the TypeScript parser to native 7.1~~ JavaScript: Migrate the TypeScript parser to TypeScript 7.1 (based on Go) Oct 27, 2026

paldepind force-pushed the js/typescript-7.1 branch from 77c4112 to dd0ae7e Compare October 27, 2026 09:30

paldepind requested a max review from Copilot October 27, 2026 10:11

Copilot started reviewing on behalf of paldepind October 27, 2026 10:11

View session

Copilot AI reviewed Oct 27, 2026

View reviewed changes

paldepind and others added 3 commits October 27, 2026 12:25

paldepind force-pushed the js/typescript-7.1 branch from dd0ae7e to 6493379 Compare October 27, 2026 11:26

paldepind commented Oct 27, 2026

View reviewed changes

paldepind marked this pull request as ready for review October 27, 2026 13:31

paldepind requested a review from a team as a code owner October 27, 2026 13:31

Navigation Menu

Repository navigation

JavaScript: Migrate the TypeScript parser to TypeScript 7.1 (based on Go) - #22841

JavaScript: Migrate the TypeScript parser to TypeScript 7.1 (based on Go)#22841
paldepind wants to merge 5 commits into
mainfrom
js/typescript-7.1

paldepind commented Oct 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asgerf Oct 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

paldepind Oct 27, 2026

Uh oh!

paldepind Oct 27, 2026

Uh oh!

paldepind Oct 27, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		return source.substring(start, name.getLoc().getStart().getOffset())
		.matches("(?U)(?:\\uFEFF\|" + WHITESPACE_CHAR + ")*");

Conversation

paldepind commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Uh oh!

Uh oh!

asgerf Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

paldepind Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

paldepind Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

paldepind Oct 27, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

paldepind commented Oct 19, 2026 •

edited

Loading