Skip to content

ci: stabilize release build memory on hosted agent - #177

Merged
turbobobbytraykov merged 3 commits into
vnextfrom
btraykov/go-memory-options
Nov 2, 2026
Merged

turbobobbytraykov merged 3 commits into
vnextfrom
btraykov/go-memory-options

Conversation

@turbobobbytraykov

@turbobobbytraykov turbobobbytraykov commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Closes #171

Summary

The Azure release pipeline (definition 586) intermittently ran out of memory on the ~7.9 GB Microsoft-hosted agent during npm run build:ci. The build then stalled for about 45 minutes and failed with "We stopped hearing from agent". This PR makes the build pass consistently with three changes:

  1. Limit esbuild's memory (4fe458b): set GOMEMLIMIT=6GiB, GOGC=50 and NODE_OPTIONS=--max_old_space_size=4096 on the build step.
  2. Fix the package-upgrade schematic (b2ff382, cherry-picked from Fix release pipeline after Angular upgrade #170): change projectTemplate in ignite-ui-cli.json from "ng-cli" to "base". Without this, upgrade-packages falls back to the hidden ai-config partial, which fails with Method not implemented. before the build starts.
  3. Build only the browser output (14baeaf): add a browser-only configuration to angular.json (server: false, ssr: false) and build with --configuration production,browser-only.
    • IIS serves only the static browser output, so the server bundle and route prerendering aren't needed.
    • production has to be listed explicitly. Naming a configuration replaces the default one, and on its own browser-only would turn off output hashing and budgets.

Why this works where #170 didn't

Angular's application builder runs esbuild as a separate Go process, and Go's memory use isn't affected by any Node setting. With the default GOGC=100, Go lets its heap grow to about twice the live data before collecting, so esbuild took whatever memory the agent had left.

Every mitigation tried in #170 acted on the Node side: NG_BUILD_MAX_WORKERS=2, NG_BUILD_PARALLEL_TS=false, and moving from Node 24 to Node 26. None of them could limit esbuild. GOGC=50 makes esbuild collect garbage about twice as often, and GOMEMLIMIT gives the Go runtime a soft ceiling that triggers more aggressive collection as memory gets close to it.

The Go options alone improved the success rate a lot but didn't eliminate stalls. Adding the browser-only build cut the bundling work, and the remaining stalls went away in testing.

Results (definition 586, ubuntu-latest, ~7.9 GB)

Configuration Source Succeeded Agent lost build:ci duration (successes)
Node 24, full build or browser-only, worker limits #170 0 / 4 4 -
Node 26, browser-only, worker limits #170 4 / 8 4 3:24 - 6:23
Node 24, full build, Go options (b2ff382) this PR 8 / 10 2 5:35 - 6:17
Node 24, browser-only + production, Go options (14baeaf) this PR 10 / 10 0 3:45 - 4:56

Local verification of the browser-only build

npm run build:ci -- --configuration production,browser-only finished in 3:01. The output in dist/angular-grid-examples/browser:

  • file names are hashed (e.g. main-7E3VDCKY.js), confirming the production settings apply
  • one SPA index.html and no prerendered routes
  • web.config is included
  • 548 files, the same count Fix release pipeline after Angular upgrade #170 recorded when it validated the browser-only artifact under IIS

Notes and remaining risk

  • Memory is still tight. 9 of the 10 final runs still logged "Free memory is lower than 5%" (96.2% - 97.2%). The build now finishes, but there's little headroom. If stalls come back, the next things to tune are GOMEMLIMIT (6 GiB for esbuild plus up to 4 GiB of Node heap exceeds the agent's memory) and the Node heap size.
  • What's lost without prerendering: HTTP calls are already skipped on the server (skipOnServerInterceptor), so the prerendered pages only had empty grids. The main loss is the build-time render of each route, which used to catch route-initialization errors before deploy.
  • Deep links: the existing web.config rewrite sends non-file requests to index.html, so direct links to routes still work.
  • Hydration: provideClientHydration stays on. With no server-rendered HTML, Angular simply renders on the client.
  • Local and SSR builds: the default ng build and the per-app SSR scripts are unchanged.

After merge: staging check

On https://staging.infragistics.com/angular-grid-examples/:

  • Home page and each sample route load when opened by direct URL (refresh on a deep link)
  • Grids load data: inventory, hr-portal, finance, sales, fleet (both at the top level and under home/)
  • Chart samples render under charts/: column, bar, line, pie, step, polar
  • No 404s for hashed JS/CSS chunks after deployment

🤖 Generated with Claude Code

turbobobbytraykov and others added 3 commits November 2, 2026 12:28
Replace the obsolete ng-cli template identifier so the package upgrade schematic resolves the base project instead of the ai-config partial.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
IIS serves only the static browser output, so skip the server bundle and
route prerendering in the release build to cut esbuild's memory peak.
List `production` explicitly so output hashing and budgets still apply.

Also correct the GOMEMLIMIT/GOGC comment, which described a GitHub
Actions runner rather than this Azure agent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@turbobobbytraykov turbobobbytraykov changed the title Try Go memory options ci: stabilize release build memory on hosted agent Nov 2, 2026
@turbobobbytraykov
turbobobbytraykov merged commit c2c0aed into vnext Nov 2, 2026
6 checks passed
@turbobobbytraykov
turbobobbytraykov deleted the btraykov/go-memory-options branch November 2, 2026 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Angular 22 CI build intermittently exhausts hosted-agent memory

1 participant