From 5d4021b3744c2387c222b099a4f4f50191c06254 Mon Sep 17 00:00:00 2001 From: Tejesh Vaish <188414140+tridev-tej@users.noreply.github.com> Date: Tue, 27 Oct 2026 00:57:33 +0400 Subject: [PATCH 1/3] added blog --- .../index.md | 686 ++++++++++++++++++ 1 file changed, 686 insertions(+) create mode 100644 content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md diff --git a/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md b/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md new file mode 100644 index 000000000..18f36b173 --- /dev/null +++ b/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md @@ -0,0 +1,686 @@ +--- +title: "Bounded Pagination in PowerShell Tools for Agents" +description: "Build a PowerShell REST reader with page and item limits, repeated-token detection, and explicit completion and error states. Verify it against a small local API." +author: Tejesh Vaish +authors: + - Tejesh Vaish +date: "2026-10-26T04:00:00+04:00" +categories: + - PowerShell for Developers +tags: + - powershell + - rest-api + - ai-agents +--- + +A REST-reading tool returns four records. A caller still needs to know why it stopped: the API reached its end, the tool used its request allowance, or another page failed. Those states tell an agent or another script whether it has a complete set. + +This example adds request and item budgets to a PowerShell reader and returns the stopping reason with the data. A local API supplies normal pages, broken continuations and failed responses, so the behavior can be reproduced without an account or credentials. + +## The finished result + +The normal fixture has five records across three pages. Reading it within both budgets returns this object: + +```json {linenos=false} +{ + "Items": [ + {"id": 1}, + {"id": 2}, + {"id": 3}, + {"id": 4}, + {"id": 5} + ], + "RequestsAttempted": 3, + "ItemsFetched": 5, + "ItemsReturned": 5, + "Status": "Complete", + "StopReason": "EndOfData", + "IsComplete": true +} +``` + +With a two-page budget, the same reader returns records 1 through 4 with `Partial/PageBudget`. If the second page points back to its own cursor, it returns records 1 and 2 with `Error/RepeatedToken`. The current page is discarded when its continuation is cyclic; records from earlier accepted pages remain available. + +## Define the page and result contracts + +The API returns an object with two required properties: `items`, an array of objects with integer `id` values, and `nextToken`, either a nonempty ASCII token or `null`. Only `null` means the end. An empty page can still have a continuation. + +Tokens match `\A[A-Za-z0-9_-]{1,64}\z`. This API treats `A` and `a` as different cursors. PowerShell's ordinary string comparisons are case-insensitive, so the reader uses an explicit `StringComparer.Ordinal` set for cursor identity. See [PowerShell comparison operators](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_comparison_operators?view=powershell-7.6#common-features). + +| Result | Meaning | +| --- | --- | +| Data | `Items`: retained records, always an array | +| Attempts | `RequestsAttempted`: GET attempts, including a failed attempt | +| Fetched | `ItemsFetched`: records in schema-valid received pages, including a page rejected for cyclic continuation | +| Returned | `ItemsReturned`: exactly `Items.Count` | +| State | `Status`: `Complete`, `Partial` or `Error` | +| Reason | `StopReason`: `EndOfData`, `PageBudget`, `ItemBudget`, `RepeatedToken`, `ReadFailed` or `InvalidArgument` | +| Complete? | `IsComplete`: true only for `Complete/EndOfData` | + +`MaxPages` caps attempted GETs. `MaxItems` caps returned records. Both accept numeric integer values from 1 through `Int32.MaxValue`; strings, Boolean values, enums, null, fractions and non-finite numbers fail before HTTP. Keep these parameters untyped until validation: [parameter binding](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_type_conversion?view=powershell-7.6#parameter-binding-conversions) can convert a value to `[int]` before the function checks it. + +A partial result does not prove another record exists. If a nonterminal page fills a budget exactly, the reader stops without fetching another page to settle that uncertainty. If a terminal page fits exactly, it is complete. If both budgets run out together, `ItemBudget` wins. + +## How the pieces fit + +Start with `api.mjs`, which supplies fixture responses and records requests. `Get-BoundedPage.ps1` implements the reader, and `Show-Examples.ps1` calls it for the three outcomes above. Two process adapters let `verify.mjs` compare returned objects with the HTTP traces. + +The reader makes its decisions in this order: + +1. Validate the endpoint and budgets before HTTP. +2. Before each GET, check the item budget, then the page budget, then register the cursor. +3. Validate the received page, count fetched items, and reject a cyclic continuation before retaining records. +4. Retain the allowed prefix. Omitted records force `Partial/ItemBudget`; otherwise a null continuation completes the result. A nonterminal page returns to the budget checks. + +## Run a local API + +The example ran with PowerShell 7.6.9 and Node.js 26.3.0 on macOS 26.3.2 arm64. Other operating systems were not tested. Use an installed PowerShell 7.6 executable and Node.js; there are no package dependencies. + +In Bash, create a working directory: + +```bash {linenos=false} +mkdir bounded-pages +cd bounded-pages +export PWSH='/absolute/path/to/pwsh' +"$PWSH" -NoLogo -NoProfile -Command '$PSVersionTable.PSVersion.ToString()' +node --version +``` + +Save each named code block below as its filename in that directory. The server binds only to `127.0.0.1` and asks the OS for an available port. Its fixture maps keep the data deterministic. A separate listener records whether a redirect was followed. + +### `api.mjs` + +```javascript {linenos=false} +import http from 'node:http'; +import { pathToFileURL } from 'node:url'; + +const page = (ids, nextToken) => ({ items: ids.map(id => ({ id })), nextToken }); +const prefix = page([1, 2], 'A'); +const normal = { r: prefix, A: page([3, 4], 'B'), B: page([5], null) }; +const bad = { + json: '{', nullRoot: null, scalarRoot: 7, missingItems: { nextToken: null }, + nullItems: { items: null, nextToken: null }, + scalarItems: { items: { id: 3 }, nextToken: null }, + badItem: { items: [{ id: '3' }], nextToken: null }, + nullItem: { items: [null], nextToken: null }, + fractionalId: { items: [{ id: 3.5 }], nextToken: null }, + booleanId: { items: [{ id: true }], nextToken: null }, + missingId: { items: [{}], nextToken: null }, + missingToken: { items: [{ id: 3 }] }, + numberToken: { items: [], nextToken: 3 }, + objectToken: { items: [], nextToken: {} }, + emptyToken: { items: [], nextToken: '' }, + urlToken: { items: [], nextToken: 'http://127.0.0.1/other' }, + newlineToken: { items: [], nextToken: 'A\n' }, + longToken: { items: [], nextToken: 'A'.repeat(65) } +}; +const fixtures = { + normal, empty: { r: page([], null) }, + emptyAfter: { r: prefix, A: page([], null) }, + emptyNext: { r: page([], 'A'), A: page([1], null) }, + oversized: { r: page([1, 2, 3], null) }, + repeat: { r: prefix, A: page([3, 4], 'A') }, + cycle: { ...normal, B: page([5, 6], 'A') }, + caseSensitive: { r: page([1], 'A'), A: page([2], 'a'), a: page([3], null) }, + extraFields: { r: { items: [{ id: -1, label: 'kept' }], nextToken: null } } +}; + +function sendJson(response, body) { + response.setHeader('Content-Type', 'application/json'); + response.end(JSON.stringify(body)); +} + +function sendFault(response, scenario, redirectUrl) { + if (scenario === 'stall') { + response.writeHead(200, { 'Content-Type': 'application/json' }); + response.write('{"items":'); + return; + } + if (scenario === 'redirect') { + response.writeHead(302, { Location: redirectUrl }); + response.end(); + return; + } + response.writeHead(Number(scenario), { 'Retry-After': '0' }); + response.end('synthetic failure'); +} + +function respond(response, scenario, cursor, redirectUrl) { + if (scenario === 'firstFailure') { + sendFault(response, '503', redirectUrl); + return; + } + if (['429', '503', 'redirect', 'stall'].includes(scenario)) { + if (cursor === 'r') sendJson(response, prefix); + else sendFault(response, scenario, redirectUrl); + return; + } + if (Object.hasOwn(bad, scenario)) { + response.setHeader('Content-Type', 'application/json'); + if (cursor === 'r') sendJson(response, prefix); + else if (scenario === 'json') response.end(bad.json); + else sendJson(response, bad[scenario]); + return; + } + const body = fixtures[scenario]?.[cursor]; + if (body === undefined) { + response.writeHead(404); + response.end('unknown fixture or cursor'); + return; + } + sendJson(response, body); +} + +function handler(trace, redirectUrl) { + return (request, response) => { + trace.push({ method: request.method, url: request.url }); + const url = new URL(request.url, 'http://127.0.0.1'); + const scenario = url.pathname.slice(1); + const cursor = url.searchParams.get('cursor') ?? 'r'; + respond(response, scenario, cursor, redirectUrl); + }; +} + +function listen(server) { + return new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(0, '127.0.0.1', resolve); + }); +} + +function stop(server) { + server.closeAllConnections(); + return new Promise((resolve, reject) => { + server.close(error => { + if (error) reject(error); + else resolve(); + }); + }); +} + +export async function startApi() { + const trace = []; + const destinationTrace = []; + const destination = http.createServer((request, response) => { + destinationTrace.push({ method: request.method, url: request.url }); + sendJson(response, page([999], null)); + }); + await listen(destination); + const redirectUrl = `http://127.0.0.1:${destination.address().port}/destination`; + const server = http.createServer(handler(trace, redirectUrl)); + await listen(server); + return { + base: `http://127.0.0.1:${server.address().port}`, + trace, destinationTrace, + async close() { await stop(server); await stop(destination); } + }; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + const api = await startApi(); + process.stdout.write(`${api.base}\n`); + process.on('SIGINT', async () => { await api.close(); }); +} +``` + +In the first Bash terminal, start the server and leave it running: + +```bash {linenos=false} +node api.mjs +``` + +It prints its base URL. Keep that value for the walkthrough. Stop this standalone server with Ctrl+C when finished. + +## Build the bounded reader + +The reader accepts a fixed HTTP loopback endpoint without user information, query or fragment. It adds an encoded `cursor` query value to that same endpoint. It never treats response content as a destination URL. + +Each request explicitly disables redirects and retries, bypasses proxies and uses terminating errors. The [Invoke-RestMethod documentation](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-7.6#-maximumredirection) defines zero as disabling redirection and documents the retry controls. The fixture checks the actual request trace for 429, 503 and redirect responses. + +### `Get-BoundedPage.ps1` + +```powershell {linenos=false} +function Test-PageBudget { + param($Value) + + if ($null -eq $Value) { return $false } + if ($Value.GetType().IsEnum) { return $false } + $numericTypes = @('SByte', 'Byte', 'Int16', 'UInt16', 'Int32', 'UInt32', + 'Int64', 'UInt64', 'Single', 'Double', 'Decimal') + if ([Type]::GetTypeCode($Value.GetType()).ToString() -notin $numericTypes) { + return $false + } + $number = [double]$Value + if (-not [double]::IsFinite($number)) { return $false } + return $Value -ge 1 -and $Value -le [int]::MaxValue -and + $Value -eq [math]::Truncate($number) +} + +function Get-PageEndpoint { + param($Endpoint) + + if ($Endpoint -isnot [string]) { throw 'Endpoint must be a URL string.' } + $uri = $null + if (-not [uri]::TryCreate($Endpoint, [UriKind]::Absolute, [ref]$uri)) { + throw 'Endpoint must be absolute.' + } + if ($uri.Scheme -cne 'http' -or -not $uri.IsLoopback) { + throw 'Use an HTTP loopback endpoint.' + } + if ($uri.UserInfo -or $uri.Query -or $uri.Fragment) { + throw 'Endpoint must not contain user information, query or fragment.' + } + return $uri.AbsoluteUri +} + +function Test-PageItem { + param($Item) + + if ($Item -isnot [pscustomobject]) { return $false } + return $Item.id -is [long] -or $Item.id -is [int] +} + +function Assert-PageShape { + param($Page) + + if ($Page -isnot [pscustomobject]) { throw 'Page must be an object.' } + if ($null -eq $Page.PSObject.Properties['items']) { throw 'Missing items.' } + if ($Page.items -isnot [array]) { throw 'items must be an array.' } + foreach ($item in $Page.items) { + if (-not (Test-PageItem $item)) { throw 'Each item needs an integer id.' } + } + if ($null -eq $Page.PSObject.Properties['nextToken']) { throw 'Missing nextToken.' } + if ($null -eq $Page.nextToken) { return } + if ($Page.nextToken -isnot [string]) { throw 'nextToken must be a string or null.' } + if ($Page.nextToken -cnotmatch '\A[A-Za-z0-9_-]{1,64}\z') { + throw 'Invalid continuation token.' + } +} + +function Read-LabPage { + param([string]$Endpoint, [string]$Cursor) + + $url = $Endpoint + if ($Cursor.Length -gt 0) { + $url += '?cursor=' + [uri]::EscapeDataString($Cursor) + } + $settings = @{ + Uri = $url + Method = 'Get' + MaximumRetryCount = 0 + MaximumRedirection = 0 + NoProxy = $true + ConnectionTimeoutSeconds = 2 + OperationTimeoutSeconds = 2 + ErrorAction = 'Stop' + } + $page = Invoke-RestMethod @settings + Assert-PageShape $page + return $page +} + +function Set-PageStop { + param($State, [string]$Status, [string]$Reason) + + $State.Status = $Status + $State.StopReason = $Reason + return $false +} + +function Test-RepeatedContinuation { + param($Page, $Seen) + + return $null -ne $Page.nextToken -and $Seen.Contains($Page.nextToken) +} + +function Invoke-PageStep { + param($State, [string]$Endpoint, [int]$MaxPages, [int]$MaxItems) + + if ($State.Items.Count -ge $MaxItems) { + return Set-PageStop $State 'Partial' 'ItemBudget' + } + if ($State.RequestsAttempted -ge $MaxPages) { + return Set-PageStop $State 'Partial' 'PageBudget' + } + if (-not $State.Seen.Add($State.Cursor)) { + return Set-PageStop $State 'Error' 'RepeatedToken' + } + $State.RequestsAttempted++ + try { $page = Read-LabPage $Endpoint $State.Cursor } + catch { + [Console]::Error.WriteLine($_.Exception.Message) + return Set-PageStop $State 'Error' 'ReadFailed' + } + $State.ItemsFetched += $page.items.Count + if (Test-RepeatedContinuation $page $state.Seen) { + [Console]::Error.WriteLine('Continuation refers to an already requested cursor.') + return Set-PageStop $State 'Error' 'RepeatedToken' + } + $take = [math]::Min($page.items.Count, $MaxItems - $State.Items.Count) + for ($index = 0; $index -lt $take; $index++) { + $State.Items.Add($page.items[$index]) + } + if ($take -lt $page.items.Count) { + return Set-PageStop $State 'Partial' 'ItemBudget' + } + if ($null -eq $page.nextToken) { + return Set-PageStop $State 'Complete' 'EndOfData' + } + $State.Cursor = $page.nextToken + return $true +} + +function New-PageResult { + param($State) + + return [pscustomobject]@{ + Items = [object[]]$State.Items.ToArray() + RequestsAttempted = $State.RequestsAttempted + ItemsFetched = $State.ItemsFetched + ItemsReturned = $State.Items.Count + Status = $State.Status + StopReason = $State.StopReason + IsComplete = $State.Status -ceq 'Complete' + } +} + +function Get-BoundedPage { + param($Endpoint, $MaxPages = 10, $MaxItems = 100) + + $state = @{ + Items = [Collections.Generic.List[object]]::new() + Seen = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + Cursor = '' + RequestsAttempted = 0 + ItemsFetched = [long]0 + Status = 'Error' + StopReason = 'InvalidArgument' + } + try { + if (-not (Test-PageBudget $MaxPages)) { throw 'MaxPages must be a positive Int32-valued number.' } + if (-not (Test-PageBudget $MaxItems)) { throw 'MaxItems must be a positive Int32-valued number.' } + $endpointUrl = Get-PageEndpoint $Endpoint + } + catch { + [Console]::Error.WriteLine($_.Exception.Message) + return New-PageResult $state + } + while (Invoke-PageStep $state $endpointUrl $MaxPages $MaxItems) {} + return New-PageResult $state +} +``` + +Each step registers the requested cursor before HTTP. After validating the entire response, it counts fetched items and checks the next token against that set. A repeated continuation rejects the current page's items before they can affect the returned prefix. + +Only then does it append the allowed prefix. Omitted items force `Partial/ItemBudget`, even when that page says `nextToken: null`. Otherwise terminal evidence wins. The next iteration checks budgets before another GET, so an exactly full nonterminal page cannot cause an extra request. + +HTTP, decoding and schema failures return `Error/ReadFailed`. Earlier accepted pages remain available. Diagnostics go to stderr; the JSON result has stable fields instead of platform-dependent exception text. A caller must inspect `Status` and `StopReason`; serializing an error object can still leave the PowerShell process with exit code zero. + +## Compare three outcomes + +Save this walkthrough as `Show-Examples.ps1`: + +### `Show-Examples.ps1` + +```powershell {linenos=false} +param([string]$Base) + +. "$PSScriptRoot/Get-BoundedPage.ps1" +Get-BoundedPage "$Base/normal" | ConvertTo-Json -Depth 20 -Compress +Get-BoundedPage "$Base/normal" -MaxPages 2 | ConvertTo-Json -Depth 20 -Compress +Get-BoundedPage "$Base/repeat" | ConvertTo-Json -Depth 20 -Compress +``` + +In a second Bash terminal, change to the same directory. Each terminal needs its own `PWSH` setting, so repeat the export. Replace the example port with the one printed by the server: + +```bash {linenos=false} +export PWSH='/absolute/path/to/pwsh' +"$PWSH" -NoLogo -NoProfile -File ./Show-Examples.ps1 -Base http://127.0.0.1:54321 +``` + +The walkthrough returned these records and counters. `r` denotes the first GET with no cursor; subsequent letters are the exact `cursor` query values. + +- Normal: GET order r, A, B; returned IDs 1, 2, 3, 4, 5. Attempts/fetched/returned: 3/5/5. Result: Complete/EndOfData. +- Normal with MaxPages 2: GET order r, A; returned IDs 1, 2, 3, 4. Counts: 2/4/4. Result: Partial/PageBudget. +- Repeat: GET order r, A; returned IDs 1, 2. Counts: 2/4/2. Result: Error/RepeatedToken. + +The repeating fixture's second page contains records 3 and 4, but its continuation points back to `A`. Those records count as fetched and are discarded. + +## Exercise the boundaries and failures + +Save these two process adapters and the verifier. The ordinary adapter reads JSON requests; the numeric probe also passes values JSON cannot represent and precise decimal fractions that an early Double cast would round. + +### `Invoke-Case.ps1` + +```powershell {linenos=false} +param([string]$RequestFile) + +$ErrorActionPreference = 'Stop' +. "$PSScriptRoot/Get-BoundedPage.ps1" +$requests = Get-Content -Raw -LiteralPath $RequestFile | ConvertFrom-Json -AsHashtable +foreach ($request in $requests) { + Get-BoundedPage @request | ConvertTo-Json -Depth 20 -Compress +} +``` + +### `Invoke-NumericProbe.ps1` + +```powershell {linenos=false} +param([string]$RequestFile) + +$ErrorActionPreference = 'Stop' +. "$PSScriptRoot/Get-BoundedPage.ps1" +$endpoint = Get-Content -Raw -LiteralPath $RequestFile +foreach ($value in @([double]::NaN, [double]::PositiveInfinity, [double]::NegativeInfinity, + [decimal]::Parse('1.0000000000000000000000000001'), + [decimal]::Parse('2147483647.0000000001'))) { + foreach ($key in @('MaxPages', 'MaxItems')) { + $arguments = @{ Endpoint = $endpoint; MaxPages = 10; MaxItems = 100 } + $arguments[$key] = $value + Get-BoundedPage @arguments | ConvertTo-Json -Depth 20 -Compress + } +} +foreach ($value in @([DayOfWeek]::Monday, [decimal]1)) { + foreach ($key in @('MaxPages', 'MaxItems')) { + $arguments = @{ Endpoint = $endpoint; MaxPages = 10; MaxItems = 100 } + $arguments[$key] = $value + Get-BoundedPage @arguments | ConvertTo-Json -Depth 20 -Compress + } +} +``` + +### `verify.mjs` + +The case table gives each scenario an expected result and HTTP request order. `runCase` launches PowerShell and checks both. Separate functions exercise repeated calls, typed budgets, the walkthrough and an isolated mutation of the cycle guard. + +The verifier starts its own listeners and waits for readiness. It compares all seven result fields and actual server-observed method/path/query order. Every child has a 10-second watchdog; a watchdog kill fails the test. It closes its own servers in `finally` and retains a sibling `run-work-*` directory containing inputs and the isolated mutant. + +```javascript {linenos=false} +import assert from 'node:assert/strict'; +import { spawn } from 'node:child_process'; +import { mkdtemp, readFile, writeFile, cp } from 'node:fs/promises'; +import { join, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { startApi } from './api.mjs'; + +const here = dirname(fileURLToPath(import.meta.url)); +const pwsh = process.env.PWSH; +assert.ok(pwsh, 'Set PWSH to the PowerShell executable'); +const ids = values => values.map(id => ({ id })); +const result = (values, requests, fetched, status, reason) => ({ + Items: ids(values), RequestsAttempted: requests, ItemsFetched: fetched, + ItemsReturned: values.length, Status: status, StopReason: reason, + IsComplete: status === 'Complete' +}); +const complete = (values, requests, fetched) => result(values, requests, fetched, 'Complete', 'EndOfData'); +const partial = (values, requests, fetched, reason) => result(values, requests, fetched, 'Partial', reason); +const error = (values, requests, fetched, reason) => result(values, requests, fetched, 'Error', reason); +const cases = [ + ['normal', {}, complete([1,2,3,4,5],3,5), ['r','A','B']], + ['empty', {}, complete([],1,0), ['r']], + ['emptyAfter', {}, complete([1,2],2,2), ['r','A']], + ['emptyNext', {}, complete([1],2,1), ['r','A']], + ['normal', {MaxPages:2}, partial([1,2,3,4],2,4,'PageBudget'), ['r','A']], + ['normal', {MaxPages:3}, complete([1,2,3,4,5],3,5), ['r','A','B']], + ['normal', {MaxPages:4}, complete([1,2,3,4,5],3,5), ['r','A','B']], + ['normal', {MaxItems:3}, partial([1,2,3],2,4,'ItemBudget'), ['r','A']], + ['normal', {MaxItems:4}, partial([1,2,3,4],2,4,'ItemBudget'), ['r','A']], + ['normal', {MaxItems:5}, complete([1,2,3,4,5],3,5), ['r','A','B']], + ['oversized', {MaxItems:2}, partial([1,2],1,3,'ItemBudget'), ['r']], + ['normal', {MaxPages:1,MaxItems:2}, partial([1,2],1,2,'ItemBudget'), ['r']], + ['emptyAfter', {MaxItems:2}, partial([1,2],1,2,'ItemBudget'), ['r']], + ['repeat', {}, error([1,2],2,4,'RepeatedToken'), ['r','A']], + ['cycle', {}, error([1,2,3,4],3,6,'RepeatedToken'), ['r','A','B']], + ['caseSensitive', {}, complete([1,2,3],3,3), ['r','A','a']], + ['firstFailure', {}, error([],1,0,'ReadFailed'), ['r']] +]; +const malformed = ['json','nullRoot','scalarRoot','missingItems','nullItems','scalarItems', + 'badItem','nullItem','fractionalId','booleanId','missingId','missingToken','numberToken', + 'objectToken','emptyToken','urlToken','newlineToken','longToken','429','503','redirect','stall']; +for (const name of malformed) cases.push([name, {}, error([1,2],2,2,'ReadFailed'), ['r','A']]); +for (const key of ['MaxPages','MaxItems']) { + for (const value of [0,-1,1.5,true,null,'2',2147483648,[],{}]) { + cases.push(['normal', {[key]:value}, error([],0,0,'InvalidArgument'), []]); + } +} +for (const endpoint of ['https://example.invalid/normal','http://127.0.0.1/normal?x=1', + 'http://u@127.0.0.1/normal','http://127.0.0.1/normal#x','not a uri',null]) { + cases.push(['normal', {Endpoint:endpoint}, error([],0,0,'InvalidArgument'), []]); +} +cases.push(['normal', {MaxPages:2147483647,MaxItems:2147483647}, complete([1,2,3,4,5],3,5), ['r','A','B']]); +cases.push(['extraFields', {}, {...complete([],1,1), Items:[{id:-1,label:'kept'}], ItemsReturned:1}, ['r']]); + +function launch(directory, requestFile, entry = 'Invoke-Case.ps1', parameter = '-RequestFile') { + return new Promise((resolve, reject) => { + const child = spawn(pwsh, ['-NoLogo','-NoProfile','-NonInteractive','-File',join(directory,entry),parameter,requestFile]); + let stdout = ''; + let stderr = ''; + const timer = setTimeout(() => { child.kill('SIGKILL'); reject(new Error('10-second watchdog expired')); }, 10000); + child.stdout.on('data', data => { stdout += data; }); + child.stderr.on('data', data => { stderr += data; }); + child.on('error', reject); + child.on('close', code => { clearTimeout(timer); resolve({code,stdout,stderr}); }); + }); +} + +function expectedTrace(scenario, cursors) { + return cursors.map(cursor => ({method:'GET',url:`/${scenario}${cursor === 'r' ? '' : `?cursor=${encodeURIComponent(cursor)}`}`})); +} + +async function runCase(api, directory, scratch, test, index) { + const [scenario, budgets, expected, cursors] = test; + const requestFile = join(scratch,`request-${index}.json`); + await writeFile(requestFile,JSON.stringify([{Endpoint:`${api.base}/${scenario}`,MaxPages:10,MaxItems:100,...budgets}])); + const start = api.trace.length; + const destinationStart = api.destinationTrace.length; + const observed = await launch(directory,requestFile); + const record = {index,scenario,budgets,expected,expectedTrace:expectedTrace(scenario,cursors),...observed, + trace:api.trace.slice(start),destinationTrace:api.destinationTrace.slice(destinationStart)}; + process.stdout.write(`${JSON.stringify(record)}\n`); + assert.equal(observed.code,0,`case ${index} native exit`); + const actual = JSON.parse(observed.stdout); + assert.deepEqual(actual,expected,`case ${index} complete result`); + assert.deepEqual(record.trace,record.expectedTrace,`case ${index} actual HTTP trace`); + assert.deepEqual(record.destinationTrace,[],`case ${index} redirect destination untouched`); + if (expected.Status === 'Error') assert.notEqual(observed.stderr.trim(),'','error diagnostic'); + else assert.equal(observed.stderr,'','successful stderr'); +} + +async function repeatedCalls(api, scratch) { + const requestFile = join(scratch,'twice.json'); + const request = {Endpoint:`${api.base}/normal`,MaxPages:10,MaxItems:100}; + await writeFile(requestFile,JSON.stringify([request,request])); + const start = api.trace.length; + const observed = await launch(here,requestFile); + process.stdout.write(`${JSON.stringify({kind:'same-process',...observed,trace:api.trace.slice(start)})}\n`); + assert.equal(observed.code,0); + assert.deepEqual(observed.stdout.trim().split('\n').map(line => JSON.parse(line)),[complete([1,2,3,4,5],3,5),complete([1,2,3,4,5],3,5)]); + assert.deepEqual(api.trace.slice(start),[...expectedTrace('normal',['r','A','B']),...expectedTrace('normal',['r','A','B'])]); +} + +async function numericProbes(api, scratch) { + const requestFile = join(scratch,'numeric.txt'); + await writeFile(requestFile,`${api.base}/normal`); + const start = api.trace.length; + const observed = await launch(here,requestFile,'Invoke-NumericProbe.ps1'); + process.stdout.write(`${JSON.stringify({kind:'typed-budget-probes',...observed,trace:api.trace.slice(start)})}\n`); + assert.equal(observed.code,0); + const expected = [ + ...Array.from({length:12}, () => error([],0,0,'InvalidArgument')), + partial([1,2],1,2,'PageBudget'),partial([1],1,2,'ItemBudget') + ]; + assert.deepEqual(observed.stdout.trim().split('\n').map(line => JSON.parse(line)),expected); + assert.deepEqual(api.trace.slice(start),[...expectedTrace('normal',['r']),...expectedTrace('normal',['r'])]); + assert.notEqual(observed.stderr.trim(),''); +} + +async function examples(api) { + const start = api.trace.length; + const observed = await launch(here,api.base,'Show-Examples.ps1','-Base'); + process.stdout.write(`${JSON.stringify({kind:'article-examples',...observed,trace:api.trace.slice(start)})}\n`); + assert.equal(observed.code,0); + assert.deepEqual(observed.stdout.trim().split('\n').map(line => JSON.parse(line)),[ + complete([1,2,3,4,5],3,5),partial([1,2,3,4],2,4,'PageBudget'),error([1,2],2,4,'RepeatedToken')]); + assert.deepEqual(api.trace.slice(start),[ + ...expectedTrace('normal',['r','A','B']),...expectedTrace('normal',['r','A']),...expectedTrace('repeat',['r','A'])]); + assert.match(observed.stderr,/Continuation refers/); +} + +async function mutation(api, scratch) { + const mutant = join(scratch,'mutant'); + await cp(here,mutant,{recursive:true}); + const sourcePath = join(mutant,'Get-BoundedPage.ps1'); + const source = await readFile(sourcePath,'utf8'); + const site = 'if (Test-RepeatedContinuation $page $state.Seen) {'; + assert.equal(source.split(site).length,2,'single verified mutation site'); + await writeFile(sourcePath,source.replace(site,'if ($false) {')); + await runCase(api,mutant,scratch,cases[0],'mutant-control'); + await assert.rejects(runCase(api,mutant,scratch,cases[13],'mutant-repeat'), /complete result|actual HTTP trace/); + await assert.rejects(runCase(api,mutant,scratch,cases[14],'mutant-cycle'), /complete result|actual HTTP trace/); + process.stdout.write(`${JSON.stringify({kind:'mutation',ordinaryControl:'passed',repeat:'rejected',cycle:'rejected'})}\n`); + await runCase(api,here,scratch,cases[13],'restored-repeat'); +} + +async function main() { + const scratch = await mkdtemp(join(here,'..','run-work-')); + const api = await startApi(); + process.stdout.write(`${JSON.stringify({kind:'ready',base:api.base,node:process.version,pwsh,scratch})}\n`); + try { + for (const [index,test] of cases.entries()) await runCase(api,here,scratch,test,index); + await runCase(api,here,scratch,cases[0],'fresh-process-replay'); + await repeatedCalls(api,scratch); + await numericProbes(api,scratch); + await examples(api); + await mutation(api,scratch); + process.stdout.write(`${JSON.stringify({kind:'summary',cases:cases.length,passed:true})}\n`); + } finally { + await api.close(); + } +} + +await main(); +``` + +From Bash, run: + +```bash {linenos=false} +node verify.mjs > results.jsonl +``` + +The executed suite passed 65 individual cases, a fresh-process replay, two calls in one process, fourteen typed budget probes and the three walkthrough calls. The probes include ten non-finite/precise-decimal negatives, enum rejection for each budget and Decimal 1 positive controls for both budgets. Cases cover empty pages, exact budgets, oversized pages, distinct `A/a`, a longer cycle, malformed bodies, HTTP failures, redirects and a stalled body. The redirect destination received no requests. The stalled response returned `Error/ReadFailed` through the HTTP timeout, without a watchdog kill. + +The verifier also changes the repeated-continuation guard to `if ($false)` in a separate copy. Ordinary completion must still pass, while repeat and cycle checks must reject the changed reader. This checks that the assertions detect incorrectly retained page data, even when the reader still terminates. The production reader has no switch to disable this guard. + +The final verifier line from the run was: + +```json {linenos=false} +{"kind":"summary","cases":65,"passed":true} +``` + +## What these bounds leave open + +`MaxItems` limits what the caller retains. It does not limit downloaded bytes or the memory used to deserialize one response. A server can send a large page before the reader trims its prefix. + +The connection and operation timeouts are two seconds each. Microsoft documents [OperationTimeoutSeconds](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-7.6#-operationtimeoutseconds) as a read-stall timeout. These settings do not form a total wall-clock deadline; a continuously progressing response is outside that guarantee. + +This lab omits retries and resume cursors. A cursor beyond a partly retained page would skip its omitted records. For a caller that requires a complete set, accept only `IsComplete: true` and retain the stopping reason with every partial or error result. From 544e7637998d280114b4b3565cb741178690f9b6 Mon Sep 17 00:00:00 2001 From: Tejesh Vaish <188414140+tridev-tej@users.noreply.github.com> Date: Tue, 3 Nov 2026 18:25:46 +0400 Subject: [PATCH 2/3] docs: clarify bounded pagination walkthrough --- .../index.md | 118 ++++++++++-------- 1 file changed, 64 insertions(+), 54 deletions(-) diff --git a/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md b/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md index 18f36b173..84ae97924 100644 --- a/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md +++ b/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md @@ -13,13 +13,13 @@ tags: - ai-agents --- -A REST-reading tool returns four records. A caller still needs to know why it stopped: the API reached its end, the tool used its request allowance, or another page failed. Those states tell an agent or another script whether it has a complete set. +I want a PowerShell tool to tell the caller when a read is incomplete. Returning the records collected so far is useful, but the caller shouldn't have to guess why the tool stopped. -This example adds request and item budgets to a PowerShell reader and returns the stopping reason with the data. A local API supplies normal pages, broken continuations and failed responses, so the behavior can be reproduced without an account or credentials. +The reader below limits attempted GETs and returned records. It carries completion and error information with the data, so an agent or another script can decide whether to use the result. You can check the behavior against a local API without an account or credentials. -## The finished result +## Read the result -The normal fixture has five records across three pages. Reading it within both budgets returns this object: +The normal fixture has five records across three pages. With enough room in both budgets, the reader returns this object: ```json {linenos=false} { @@ -39,13 +39,15 @@ The normal fixture has five records across three pages. Reading it within both b } ``` -With a two-page budget, the same reader returns records 1 through 4 with `Partial/PageBudget`. If the second page points back to its own cursor, it returns records 1 and 2 with `Error/RepeatedToken`. The current page is discarded when its continuation is cyclic; records from earlier accepted pages remain available. +You can use the records already collected, but check the status fields before treating them as a complete set. + +With a two-page budget, the reader returns records 1 through 4 with `Partial/PageBudget`. If the second page points back to its own cursor, it returns records 1 and 2 with `Error/RepeatedToken`. The reader discards a page with a cyclic continuation and keeps records from earlier accepted pages. ## Define the page and result contracts -The API returns an object with two required properties: `items`, an array of objects with integer `id` values, and `nextToken`, either a nonempty ASCII token or `null`. Only `null` means the end. An empty page can still have a continuation. +Each API page must be an object with two required properties: `items`, an array of objects with integer `id` values, and `nextToken`, a nonempty ASCII token or `null`. An empty page doesn't mean the read is complete. Only `null` ends pagination, so the reader follows a continuation from an empty page if the budgets allow it. -Tokens match `\A[A-Za-z0-9_-]{1,64}\z`. This API treats `A` and `a` as different cursors. PowerShell's ordinary string comparisons are case-insensitive, so the reader uses an explicit `StringComparer.Ordinal` set for cursor identity. See [PowerShell comparison operators](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_comparison_operators?view=powershell-7.6#common-features). +Tokens match `\A[A-Za-z0-9_-]{1,64}\z`. The API treats `A` and `a` as different cursors, so the reader uses a `StringComparer.Ordinal` set to track them. PowerShell's ordinary string comparisons are case-insensitive; see [PowerShell comparison operators](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_comparison_operators?view=powershell-7.6#common-features). | Result | Meaning | | --- | --- | @@ -57,24 +59,11 @@ Tokens match `\A[A-Za-z0-9_-]{1,64}\z`. This API treats `A` and `a` as different | Reason | `StopReason`: `EndOfData`, `PageBudget`, `ItemBudget`, `RepeatedToken`, `ReadFailed` or `InvalidArgument` | | Complete? | `IsComplete`: true only for `Complete/EndOfData` | -`MaxPages` caps attempted GETs. `MaxItems` caps returned records. Both accept numeric integer values from 1 through `Int32.MaxValue`; strings, Boolean values, enums, null, fractions and non-finite numbers fail before HTTP. Keep these parameters untyped until validation: [parameter binding](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_type_conversion?view=powershell-7.6#parameter-binding-conversions) can convert a value to `[int]` before the function checks it. - -A partial result does not prove another record exists. If a nonterminal page fills a budget exactly, the reader stops without fetching another page to settle that uncertainty. If a terminal page fits exactly, it is complete. If both budgets run out together, `ItemBudget` wins. - -## How the pieces fit - -Start with `api.mjs`, which supplies fixture responses and records requests. `Get-BoundedPage.ps1` implements the reader, and `Show-Examples.ps1` calls it for the three outcomes above. Two process adapters let `verify.mjs` compare returned objects with the HTTP traces. - -The reader makes its decisions in this order: - -1. Validate the endpoint and budgets before HTTP. -2. Before each GET, check the item budget, then the page budget, then register the cursor. -3. Validate the received page, count fetched items, and reject a cyclic continuation before retaining records. -4. Retain the allowed prefix. Omitted records force `Partial/ItemBudget`; otherwise a null continuation completes the result. A nonterminal page returns to the budget checks. +`MaxPages` caps attempted GETs, and `MaxItems` caps returned records. Both accept numeric integer values from 1 through `Int32.MaxValue`. Strings, Boolean values, enums, null, fractions and non-finite numbers fail before HTTP. I keep the budgets untyped until validation because [parameter binding](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_type_conversion?view=powershell-7.6#parameter-binding-conversions) can convert a value to `[int]` before the function checks it. ## Run a local API -The example ran with PowerShell 7.6.9 and Node.js 26.3.0 on macOS 26.3.2 arm64. Other operating systems were not tested. Use an installed PowerShell 7.6 executable and Node.js; there are no package dependencies. +The example ran with PowerShell 7.6.9 and Node.js 26.3.0 on macOS 26.3.2 arm64. Other operating systems were not tested. You'll need an installed PowerShell 7.6 executable and Node.js; there are no package dependencies. In Bash, create a working directory: @@ -86,9 +75,11 @@ export PWSH='/absolute/path/to/pwsh' node --version ``` -Save each named code block below as its filename in that directory. The server binds only to `127.0.0.1` and asks the OS for an available port. Its fixture maps keep the data deterministic. A separate listener records whether a redirect was followed. +Save each named code block as its filename in that directory. `api.mjs` serves fixture responses and records requests. `Get-BoundedPage.ps1` contains the reader, and `Show-Examples.ps1` calls it for the three outcomes above. Two process adapters let `verify.mjs` compare returned objects with the HTTP traces. + +The server binds only to `127.0.0.1` and asks the OS for an available port. Fixture responses are deterministic. A separate listener records whether the reader followed a redirect. -### `api.mjs` +### Fixture server: `api.mjs` ```javascript {linenos=false} import http from 'node:http'; @@ -223,21 +214,28 @@ if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) } ``` -In the first Bash terminal, start the server and leave it running: +Start the server in the first Bash terminal and leave it running: ```bash {linenos=false} node api.mjs ``` -It prints its base URL. Keep that value for the walkthrough. Stop this standalone server with Ctrl+C when finished. +It prints its base URL. Keep that URL for the walkthrough, and stop this standalone server with Ctrl+C when you're finished. ## Build the bounded reader -The reader accepts a fixed HTTP loopback endpoint without user information, query or fragment. It adds an encoded `cursor` query value to that same endpoint. It never treats response content as a destination URL. +I keep the endpoint fixed for the read. The reader accepts an HTTP loopback URL without user information, query or fragment, then adds an encoded `cursor` query value to that endpoint. Response content never becomes a destination URL. + +Each GET disables redirects and retries, bypasses proxies and uses terminating errors. The [Invoke-RestMethod documentation](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-7.6#-maximumredirection) defines zero as disabling redirection and documents the retry controls. The fixture checks the actual request trace for 429, 503 and redirect responses. -Each request explicitly disables redirects and retries, bypasses proxies and uses terminating errors. The [Invoke-RestMethod documentation](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-7.6#-maximumredirection) defines zero as disabling redirection and documents the retry controls. The fixture checks the actual request trace for 429, 503 and redirect responses. +### Reader: `Get-BoundedPage.ps1` -### `Get-BoundedPage.ps1` +The order of these checks determines which records the caller can keep: + +1. Validate the endpoint and budgets before HTTP. +2. Before each GET, check the item budget, then the page budget, then register the cursor. +3. Validate the received page, count fetched items, and reject a cyclic continuation before retaining records. +4. Retain the allowed prefix. Omitted records force `Partial/ItemBudget`; otherwise a null continuation completes the result. A nonterminal page returns to the budget checks. ```powershell {linenos=false} function Test-PageBudget { @@ -410,17 +408,19 @@ function Get-BoundedPage { } ``` -Each step registers the requested cursor before HTTP. After validating the entire response, it counts fetched items and checks the next token against that set. A repeated continuation rejects the current page's items before they can affect the returned prefix. +I check for a cyclic continuation before keeping any records from the page because that page must not change the returned prefix. Each step registers the requested cursor before HTTP. After validating the entire response, it counts fetched items and checks the next token against those cursors. A repeated continuation rejects the current page's records. + +After that check, the reader appends the allowed prefix. Leaving out any records forces `Partial/ItemBudget`, even if the page says `nextToken: null`. When the whole page fits and its continuation is null, the result is complete. -Only then does it append the allowed prefix. Omitted items force `Partial/ItemBudget`, even when that page says `nextToken: null`. Otherwise terminal evidence wins. The next iteration checks budgets before another GET, so an exactly full nonterminal page cannot cause an extra request. +A partial result doesn't prove there's another record. If a nonterminal page fills a budget exactly, the reader stops before another GET, so it can't settle that question. A terminal page that fits exactly is complete. If both budgets run out together, `ItemBudget` wins because the item check comes first. -HTTP, decoding and schema failures return `Error/ReadFailed`. Earlier accepted pages remain available. Diagnostics go to stderr; the JSON result has stable fields instead of platform-dependent exception text. A caller must inspect `Status` and `StopReason`; serializing an error object can still leave the PowerShell process with exit code zero. +HTTP, decoding and schema failures return `Error/ReadFailed`. You still get records from earlier accepted pages. Diagnostics go to stderr; the JSON result keeps stable fields instead of platform-dependent exception text. Check `Status` and `StopReason` even if the PowerShell process exits with code zero, because serializing an error result can still produce that exit code. ## Compare three outcomes -Save this walkthrough as `Show-Examples.ps1`: +The walkthrough calls the normal fixture with default budgets, then with a two-page limit, then calls the repeating fixture. -### `Show-Examples.ps1` +### Walkthrough: `Show-Examples.ps1` ```powershell {linenos=false} param([string]$Base) @@ -431,26 +431,30 @@ Get-BoundedPage "$Base/normal" -MaxPages 2 | ConvertTo-Json -Depth 20 -Compress Get-BoundedPage "$Base/repeat" | ConvertTo-Json -Depth 20 -Compress ``` -In a second Bash terminal, change to the same directory. Each terminal needs its own `PWSH` setting, so repeat the export. Replace the example port with the one printed by the server: +Open a second Bash terminal and change to the same directory. Set `PWSH` again because each terminal needs its own export. Replace the example port with the port printed by the server: ```bash {linenos=false} export PWSH='/absolute/path/to/pwsh' "$PWSH" -NoLogo -NoProfile -File ./Show-Examples.ps1 -Base http://127.0.0.1:54321 ``` -The walkthrough returned these records and counters. `r` denotes the first GET with no cursor; subsequent letters are the exact `cursor` query values. +These are the walkthrough results. `r` denotes the first GET without a cursor; subsequent letters are the exact `cursor` query values. Counts are `RequestsAttempted` / `ItemsFetched` / `ItemsReturned`, in that order. -- Normal: GET order r, A, B; returned IDs 1, 2, 3, 4, 5. Attempts/fetched/returned: 3/5/5. Result: Complete/EndOfData. -- Normal with MaxPages 2: GET order r, A; returned IDs 1, 2, 3, 4. Counts: 2/4/4. Result: Partial/PageBudget. -- Repeat: GET order r, A; returned IDs 1, 2. Counts: 2/4/2. Result: Error/RepeatedToken. +| Call | Observed result | +| --- | --- | +| Normal | GET order: r, A, B; Returned IDs: 1, 2, 3, 4, 5; Counts: 3/5/5; `Complete/EndOfData` | +| Normal with `MaxPages` 2 | GET order: r, A; Returned IDs: 1, 2, 3, 4; Counts: 2/4/4; `Partial/PageBudget` | +| Repeat | GET order: r, A; Returned IDs: 1, 2; Counts: 2/4/2; `Error/RepeatedToken` | -The repeating fixture's second page contains records 3 and 4, but its continuation points back to `A`. Those records count as fetched and are discarded. +The repeat case fetched four records but returned two because the second page contains records 3 and 4 and points back to `A`. The reader counts those records as fetched, then discards them when it rejects the continuation. ## Exercise the boundaries and failures -Save these two process adapters and the verifier. The ordinary adapter reads JSON requests; the numeric probe also passes values JSON cannot represent and precise decimal fractions that an early Double cast would round. +The two adapters below expose different inputs to the verifier. Save them and `verify.mjs` in the same working directory. + +### JSON adapter: `Invoke-Case.ps1` -### `Invoke-Case.ps1` +This adapter reads a JSON array of requests and writes one JSON result per request. Passing multiple requests lets the verifier check two calls in the same PowerShell process. ```powershell {linenos=false} param([string]$RequestFile) @@ -463,7 +467,9 @@ foreach ($request in $requests) { } ``` -### `Invoke-NumericProbe.ps1` +### Typed budget probe: `Invoke-NumericProbe.ps1` + +JSON can't carry every budget value the function can receive. The numeric probe passes non-finite numbers and enums directly, along with precise Decimal fractions. An early Double conversion would round those fractions, so the probe checks that validation rejects the original fractional values. ```powershell {linenos=false} param([string]$RequestFile) @@ -489,11 +495,11 @@ foreach ($value in @([DayOfWeek]::Monday, [decimal]1)) { } ``` -### `verify.mjs` +### Verifier: `verify.mjs` -The case table gives each scenario an expected result and HTTP request order. `runCase` launches PowerShell and checks both. Separate functions exercise repeated calls, typed budgets, the walkthrough and an isolated mutation of the cycle guard. +I check the server's request trace because the returned JSON can't show whether the reader made an extra GET, retried or followed a redirect. Each case specifies an expected result and method/path/query order; `runCase` launches PowerShell and compares both. Other functions check repeated calls, typed budgets, the walkthrough and an isolated mutation of the cycle guard. -The verifier starts its own listeners and waits for readiness. It compares all seven result fields and actual server-observed method/path/query order. Every child has a 10-second watchdog; a watchdog kill fails the test. It closes its own servers in `finally` and retains a sibling `run-work-*` directory containing inputs and the isolated mutant. +It starts its own listeners and waits for readiness, then compares all seven result fields and the actual request order. Every child has a 10-second watchdog, and a watchdog kill fails the test. The verifier closes its servers in `finally`. It retains a sibling `run-work-*` directory with the inputs and isolated mutant so you can inspect them. ```javascript {linenos=false} import assert from 'node:assert/strict'; @@ -661,26 +667,30 @@ async function main() { await main(); ``` -From Bash, run: +Run the verifier from Bash: ```bash {linenos=false} node verify.mjs > results.jsonl ``` -The executed suite passed 65 individual cases, a fresh-process replay, two calls in one process, fourteen typed budget probes and the three walkthrough calls. The probes include ten non-finite/precise-decimal negatives, enum rejection for each budget and Decimal 1 positive controls for both budgets. Cases cover empty pages, exact budgets, oversized pages, distinct `A/a`, a longer cycle, malformed bodies, HTTP failures, redirects and a stalled body. The redirect destination received no requests. The stalled response returned `Error/ReadFailed` through the HTTP timeout, without a watchdog kill. +The recorded run passed 65 individual cases, a fresh-process replay, two calls in one process, fourteen typed budget probes and the three walkthrough calls. Of the fourteen probes, ten reject non-finite numbers or precise Decimal fractions, two reject enums (one for each budget), and two use Decimal 1 as positive controls for the budgets. -The verifier also changes the repeated-continuation guard to `if ($false)` in a separate copy. Ordinary completion must still pass, while repeat and cycle checks must reject the changed reader. This checks that the assertions detect incorrectly retained page data, even when the reader still terminates. The production reader has no switch to disable this guard. +The cases cover empty pages, exact budgets, oversized pages, distinct `A/a` cursors, a longer cycle, malformed bodies and HTTP failures. They also check redirects and a stalled body. The redirect destination received no requests. The stalled response returned `Error/ReadFailed` through the HTTP timeout, without a watchdog kill. -The final verifier line from the run was: +Stopping the loop isn't enough if the reader has already kept records from a cyclic page. To check that failure, the verifier changes the repeated-continuation guard to `if ($false)` in a separate copy. Ordinary completion must still pass, while the repeat and cycle checks must reject the changed reader. The production reader has no switch to disable this guard. + +The recorded run ended with this verifier line: ```json {linenos=false} {"kind":"summary","cases":65,"passed":true} ``` -## What these bounds leave open +## What the budgets don't limit + +`MaxItems` limits the records you retain. A server can still send a large page before the reader trims it, so the budget doesn't limit downloaded bytes or the memory needed to deserialize one response. -`MaxItems` limits what the caller retains. It does not limit downloaded bytes or the memory used to deserialize one response. A server can send a large page before the reader trims its prefix. +The connection and operation timeouts are two seconds each. Microsoft documents [OperationTimeoutSeconds](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-7.6#-operationtimeoutseconds) as a read-stall timeout. Together, these settings still don't impose a total wall-clock deadline. They don't bound a response that keeps progressing. -The connection and operation timeouts are two seconds each. Microsoft documents [OperationTimeoutSeconds](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-7.6#-operationtimeoutseconds) as a read-stall timeout. These settings do not form a total wall-clock deadline; a continuously progressing response is outside that guarantee. +The lab doesn't retry or return a resume cursor. A cursor beyond a partly retained page would skip the records left out of that page. -This lab omits retries and resume cursors. A cursor beyond a partly retained page would skip its omitted records. For a caller that requires a complete set, accept only `IsComplete: true` and retain the stopping reason with every partial or error result. +If your caller needs a complete set, accept only `IsComplete: true`. Keep the stopping reason with every partial or error result. From 14dec9a509a1a93363522daf2097cb27067975db Mon Sep 17 00:00:00 2001 From: Tejesh Vaish <188414140+tridev-tej@users.noreply.github.com> Date: Wed, 4 Nov 2026 02:31:45 +0400 Subject: [PATCH 3/3] added blog Signed-off-by: Tejesh Vaish <188414140+tridev-tej@users.noreply.github.com> --- .../index.md | 378 +++++++++++++++--- 1 file changed, 317 insertions(+), 61 deletions(-) diff --git a/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md b/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md index 84ae97924..1a3ea1859 100644 --- a/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md +++ b/content/articles/2026/10/bounded-pagination-in-powershell-tools-for-agents/index.md @@ -13,23 +13,25 @@ tags: - ai-agents --- -I want a PowerShell tool to tell the caller when a read is incomplete. Returning the records collected so far is useful, but the caller shouldn't have to guess why the tool stopped. +Suppose a script is building an inventory report from a paginated API. The first response contains records 1 and 2, plus a token for the next page. If the script returns only those records, a caller can mistake a successful request for a complete inventory. The local fixture here contains five records across three pages. -The reader below limits attempted GETs and returned records. It carries completion and error information with the data, so an agent or another script can decide whether to use the result. You can check the behavior against a local API without an account or credentials. +A read can also stop at a budget or encounter a repeated cursor. A script or agent using those records needs to know which happened before it treats them as the whole inventory. -## Read the result +The reader below caps attempted GETs and returned records, then carries the stopping reason with the data. You can check it against a local API without an account or credentials. -The normal fixture has five records across three pages. With enough room in both budgets, the reader returns this object: +## Compare the one-page read with a bounded result + +The intentionally incomplete read in the walkthrough makes one GET and returns only its items: + +```json {linenos=false} +[{"id":1},{"id":2}] +``` + +That output loses the continuation and says nothing about completeness. With enough room in both budgets, the bounded reader reaches the final page and returns: ```json {linenos=false} { - "Items": [ - {"id": 1}, - {"id": 2}, - {"id": 3}, - {"id": 4}, - {"id": 5} - ], + "Items": [{"id":1},{"id":2},{"id":3},{"id":4},{"id":5}], "RequestsAttempted": 3, "ItemsFetched": 5, "ItemsReturned": 5, @@ -39,9 +41,21 @@ The normal fixture has five records across three pages. With enough room in both } ``` -You can use the records already collected, but check the status fields before treating them as a complete set. +With a two-page budget, it returns the records it can keep and makes the stopping condition explicit: -With a two-page budget, the reader returns records 1 through 4 with `Partial/PageBudget`. If the second page points back to its own cursor, it returns records 1 and 2 with `Error/RepeatedToken`. The reader discards a page with a cyclic continuation and keeps records from earlier accepted pages. +```json {linenos=false} +{ + "Items": [{"id":1},{"id":2},{"id":3},{"id":4}], + "RequestsAttempted": 2, + "ItemsFetched": 4, + "ItemsReturned": 4, + "Status": "Partial", + "StopReason": "PageBudget", + "IsComplete": false +} +``` + +You can use the records in either result, but only the complete result establishes that you reached the end. If a later page points back to an already requested cursor, the reader instead returns `Error/RepeatedToken` and keeps the earlier accepted records. ## Define the page and result contracts @@ -61,25 +75,34 @@ Tokens match `\A[A-Za-z0-9_-]{1,64}\z`. The API treats `A` and `a` as different `MaxPages` caps attempted GETs, and `MaxItems` caps returned records. Both accept numeric integer values from 1 through `Int32.MaxValue`. Strings, Boolean values, enums, null, fractions and non-finite numbers fail before HTTP. I keep the budgets untyped until validation because [parameter binding](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_type_conversion?view=powershell-7.6#parameter-binding-conversions) can convert a value to `[int]` before the function checks it. -## Run a local API +## Set up the local example -The example ran with PowerShell 7.6.9 and Node.js 26.3.0 on macOS 26.3.2 arm64. Other operating systems were not tested. You'll need an installed PowerShell 7.6 executable and Node.js; there are no package dependencies. +Start in a PowerShell 7.6 console with Node.js installed. These instructions target PowerShell 7.6, including the HTTP settings below. Node installation instructions are on the [official download page](https://nodejs.org/en/download); the lab uses only built-in Node modules, so there's no package installation step. -In Bash, create a working directory: +The example was tested with PowerShell 7.6.9 and Node.js 26.3.0 on macOS 26.3.2 arm64 and Ubuntu 22.04.5 LTS x64. The numeric probes and full verifier passed under both `en-US` and `fr-FR`. Windows has not been tested. -```bash {linenos=false} -mkdir bounded-pages -cd bounded-pages -export PWSH='/absolute/path/to/pwsh' -"$PWSH" -NoLogo -NoProfile -Command '$PSVersionTable.PSVersion.ToString()' +Create a directory and record the runtimes. `PWSH` points to the PowerShell executable running this console, so the verifier can launch the same runtime: + +```powershell {linenos=false} +New-Item -ItemType Directory -Path './bounded pages' +Set-Location './bounded pages' +$env:PWSH = (Get-Process -Id $PID).Path +$PSVersionTable.PSVersion.ToString() node --version ``` -Save each named code block as its filename in that directory. `api.mjs` serves fixture responses and records requests. `Get-BoundedPage.ps1` contains the reader, and `Show-Examples.ps1` calls it for the three outcomes above. Two process adapters let `verify.mjs` compare returned objects with the HTTP traces. +Save all six files below in this directory. Files 1 through 3 are enough for the walkthrough; files 4 through 6 add verification. + +1. `api.mjs`: Local pages, request recorder and separate redirect destination +2. `Get-BoundedPage.ps1`: Bounded reader and its validation, request and result helpers +3. `Show-Examples.ps1`: Three serialized complete, page-limited and repeated-token results +4. `Invoke-Case.ps1`: JSON request-array adapter, including multiple calls in one process +5. `Invoke-NumericProbe.ps1`: Fourteen typed budget probes +6. `verify.mjs`: Process and HTTP checks using its own listeners -The server binds only to `127.0.0.1` and asks the OS for an available port. Fixture responses are deterministic. A separate listener records whether the reader followed a redirect. +The fixture binds to `127.0.0.1` on an available port and prints its base URL. `page` builds deterministic bodies; `respond` selects normal or faulty responses, and `handler` records each method and URL before responding. `startApi` starts both the fixture and a separate redirect destination, then returns their traces and a `close` method. The verifier uses that lifecycle to inspect real requests and release both listeners. -### Fixture server: `api.mjs` +### 1. Save `api.mjs` ```javascript {linenos=false} import http from 'node:http'; @@ -214,31 +237,45 @@ if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) } ``` -Start the server in the first Bash terminal and leave it running: +Start the fixture in this first PowerShell terminal and leave it running: -```bash {linenos=false} -node api.mjs +```powershell {linenos=false} +node ./api.mjs ``` -It prints its base URL. Keep that URL for the walkthrough, and stop this standalone server with Ctrl+C when you're finished. +Keep the printed loopback URL for the second terminal. Stop this standalone fixture with Ctrl+C in its owning terminal when you're finished. ## Build the bounded reader -I keep the endpoint fixed for the read. The reader accepts an HTTP loopback URL without user information, query or fragment, then adds an encoded `cursor` query value to that endpoint. Response content never becomes a destination URL. +The reader accepts a fixed HTTP loopback endpoint without user information, query or fragment. Each subsequent request adds an encoded `cursor` value. Response content cannot choose a host or path. Each GET disables redirects and retries, bypasses proxies and uses terminating errors. The [Invoke-RestMethod documentation](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-7.6#-maximumredirection) defines zero as disabling redirection and documents the retry controls. The fixture checks the actual request trace for 429, 503 and redirect responses. -### Reader: `Get-BoundedPage.ps1` - -The order of these checks determines which records the caller can keep: +The validation helpers check arguments and whole pages. `Read-LabPage` performs one GET; `Invoke-PageStep` decides what to retain and when to stop. `New-PageResult` builds the envelope. `Get-BoundedPage` keeps the state separate for each call and runs the checks in this order: 1. Validate the endpoint and budgets before HTTP. 2. Before each GET, check the item budget, then the page budget, then register the cursor. 3. Validate the received page, count fetched items, and reject a cyclic continuation before retaining records. 4. Retain the allowed prefix. Omitted records force `Partial/ItemBudget`; otherwise a null continuation completes the result. A nonterminal page returns to the budget checks. +Each function includes [comment-based help](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_comment_based_help?view=powershell-7.6). Its example shows the setup it needs; HTTP examples ask for the running fixture's base URL. + +### 2. Save `Get-BoundedPage.ps1` + ```powershell {linenos=false} function Test-PageBudget { + <# + .SYNOPSIS + Checks a budget without coercing its original numeric value. + .PARAMETER Value + Numeric integer from 1 through Int32.MaxValue. Null, Boolean, enum, string, + fractional and non-finite values are rejected. + .OUTPUTS + System.Boolean. True when the value is an accepted budget. + .EXAMPLE + . ./Get-BoundedPage.ps1 + Test-PageBudget -Value ([decimal]1) + #> param($Value) if ($null -eq $Value) { return $false } @@ -255,6 +292,17 @@ function Test-PageBudget { } function Get-PageEndpoint { + <# + .SYNOPSIS + Validates the fixed destination before any request. + .PARAMETER Endpoint + Absolute HTTP loopback URL with no user information, query or fragment. + .OUTPUTS + System.String. The absolute URL. Invalid input throws. + .EXAMPLE + . ./Get-BoundedPage.ps1 + Get-PageEndpoint -Endpoint 'http://127.0.0.1:54321/normal' + #> param($Endpoint) if ($Endpoint -isnot [string]) { throw 'Endpoint must be a URL string.' } @@ -272,6 +320,17 @@ function Get-PageEndpoint { } function Test-PageItem { + <# + .SYNOPSIS + Checks that a deserialized record has an integer id. + .PARAMETER Item + PSCustomObject whose id must be Int32 or Int64. Other properties are retained. + .OUTPUTS + System.Boolean. True when the record has the required shape. + .EXAMPLE + . ./Get-BoundedPage.ps1 + Test-PageItem -Item ([pscustomobject]@{ id = [int]1; label = 'kept' }) + #> param($Item) if ($Item -isnot [pscustomobject]) { return $false } @@ -279,6 +338,22 @@ function Test-PageItem { } function Assert-PageShape { + <# + .SYNOPSIS + Validates every record and the continuation before a page changes state. + .PARAMETER Page + PSCustomObject with array items and a required nextToken: null or an ASCII + token matching \A[A-Za-z0-9_-]{1,64}\z. Empty arrays may have a continuation. + .OUTPUTS + No output on success. Invalid pages throw. + .EXAMPLE + . ./Get-BoundedPage.ps1 + $page = [pscustomobject]@{ + items = @([pscustomobject]@{ id = [int]1 }) + nextToken = $null + } + Assert-PageShape -Page $page + #> param($Page) if ($Page -isnot [pscustomobject]) { throw 'Page must be an object.' } @@ -296,6 +371,21 @@ function Assert-PageShape { } function Read-LabPage { + <# + .SYNOPSIS + Fetches and validates one page without retries or redirects. + .PARAMETER Endpoint + Fixed URL already validated by Get-PageEndpoint. + .PARAMETER Cursor + Empty string for the first page; otherwise a validated continuation token. + The token is encoded as a cursor query value. + .OUTPUTS + PSCustomObject. A validated page. HTTP, decoding or schema failures throw. + .EXAMPLE + . ./Get-BoundedPage.ps1 + $Base = Read-Host 'Paste the base URL printed by the running api.mjs' + Read-LabPage -Endpoint "$Base/normal" -Cursor '' + #> param([string]$Endpoint, [string]$Cursor) $url = $Endpoint @@ -318,6 +408,23 @@ function Read-LabPage { } function Set-PageStop { + <# + .SYNOPSIS + Records the stop state and signals the caller to leave the loop. + .PARAMETER State + Per-call mutable hashtable whose Status and StopReason will be replaced. + .PARAMETER Status + Complete, Partial or Error, chosen by the caller's stop condition. + .PARAMETER Reason + EndOfData, PageBudget, ItemBudget, RepeatedToken, ReadFailed or InvalidArgument. + .OUTPUTS + System.Boolean. Always false; the supplied state is also updated. + .EXAMPLE + . ./Get-BoundedPage.ps1 + $state = @{ Status = ''; StopReason = '' } + Set-PageStop -State $state -Status 'Partial' -Reason 'PageBudget' + $state + #> param($State, [string]$Status, [string]$Reason) $State.Status = $Status @@ -326,12 +433,54 @@ function Set-PageStop { } function Test-RepeatedContinuation { + <# + .SYNOPSIS + Checks whether a page points to an already requested cursor. + .PARAMETER Page + Validated page with a nextToken property. Null is terminal, not repeated. + .PARAMETER Seen + HashSet of requested cursors using StringComparer.Ordinal for case sensitivity. + .OUTPUTS + System.Boolean. True when the non-null continuation is in Seen. + .EXAMPLE + . ./Get-BoundedPage.ps1 + $seen = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + $null = $seen.Add('A') + Test-RepeatedContinuation -Page ([pscustomobject]@{ nextToken = 'A' }) -Seen $seen + #> param($Page, $Seen) return $null -ne $Page.nextToken -and $Seen.Contains($Page.nextToken) } function Invoke-PageStep { + <# + .SYNOPSIS + Attempts one bounded page step and updates this read's state. + .PARAMETER State + Per-call hashtable with Items list, Seen ordinal set, Cursor, counters, + Status and StopReason. Registering the cursor precedes the GET. + .PARAMETER Endpoint + Fixed URL already validated by Get-PageEndpoint. + .PARAMETER MaxPages + Validated positive Int32-valued attempt limit. Checked after the item limit. + .PARAMETER MaxItems + Validated positive Int32-valued retained-record limit. A cyclic page is rejected + before its records are appended; omitted records force ItemBudget. + .OUTPUTS + System.Boolean. True to continue; false when State contains the stop result. + .EXAMPLE + . ./Get-BoundedPage.ps1 + $Base = Read-Host 'Paste the base URL printed by the running api.mjs' + $state = @{ + Items = [Collections.Generic.List[object]]::new() + Seen = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + Cursor = ''; RequestsAttempted = 0; ItemsFetched = [long]0 + Status = 'Error'; StopReason = 'InvalidArgument' + } + while (Invoke-PageStep -State $state -Endpoint "$Base/normal" -MaxPages 2 -MaxItems 100) {} + New-PageResult -State $state + #> param($State, [string]$Endpoint, [int]$MaxPages, [int]$MaxItems) if ($State.Items.Count -ge $MaxItems) { @@ -369,6 +518,24 @@ function Invoke-PageStep { } function New-PageResult { + <# + .SYNOPSIS + Builds the caller's envelope from the retained records and stop state. + .PARAMETER State + Per-call hashtable with Items list, RequestsAttempted, ItemsFetched, Status + and StopReason. ItemsReturned is derived from the list count. + .OUTPUTS + PSCustomObject. Seven fields, with Items always an array. IsComplete is true + only for the Complete state selected on EndOfData by the reader. + .EXAMPLE + . ./Get-BoundedPage.ps1 + $state = @{ + Items = [Collections.Generic.List[object]]::new() + RequestsAttempted = 0; ItemsFetched = [long]0 + Status = 'Error'; StopReason = 'InvalidArgument' + } + New-PageResult -State $state | ConvertTo-Json -Depth 20 -Compress + #> param($State) return [pscustomobject]@{ @@ -383,6 +550,28 @@ function New-PageResult { } function Get-BoundedPage { + <# + .SYNOPSIS + Reads local pages with explicit request, record and completion limits. + .PARAMETER Endpoint + Absolute HTTP loopback URL without user information, query or fragment. + Response tokens cannot change this destination. + .PARAMETER MaxPages + Numeric integer from 1 through Int32.MaxValue; defaults to 10. Includes failed + GET attempts. Invalid types or values return InvalidArgument before HTTP. + .PARAMETER MaxItems + Numeric integer from 1 through Int32.MaxValue; defaults to 100. Bounds retained + records, not response bytes or deserialization memory. ItemBudget wins when + both budgets are exhausted on a nonterminal page. + .OUTPUTS + PSCustomObject. Items, RequestsAttempted, ItemsFetched, ItemsReturned, Status, + StopReason and IsComplete. Errors retain earlier accepted records and write + diagnostics to stderr. A zero process exit does not establish completeness. + .EXAMPLE + . ./Get-BoundedPage.ps1 + $Base = Read-Host 'Paste the base URL printed by the running api.mjs' + Get-BoundedPage -Endpoint "$Base/normal" -MaxPages 2 | ConvertTo-Json -Depth 20 -Compress + #> param($Endpoint, $MaxPages = 10, $MaxItems = 100) $state = @{ @@ -416,13 +605,25 @@ A partial result doesn't prove there's another record. If a nonterminal page fil HTTP, decoding and schema failures return `Error/ReadFailed`. You still get records from earlier accepted pages. Diagnostics go to stderr; the JSON result keeps stable fields instead of platform-dependent exception text. Check `Status` and `StopReason` even if the PowerShell process exits with code zero, because serializing an error result can still produce that exit code. -## Compare three outcomes -The walkthrough calls the normal fixture with default budgets, then with a two-page limit, then calls the repeating fixture. +## Run the before and after -### Walkthrough: `Show-Examples.ps1` +The walkthrough takes the fixture's base URL and prints three JSON envelopes: a complete normal read, a two-page normal read and a repeated-token error. Its diagnostic stays on stderr. + +### 3. Save `Show-Examples.ps1` ```powershell {linenos=false} +<# +.SYNOPSIS +Prints complete, page-limited and repeated-token results from the local fixture. +.PARAMETER Base +Base URL printed by the running api.mjs, with no trailing slash. +.OUTPUTS +Three JSON lines in call order. The repeat-case diagnostic goes to stderr. +.EXAMPLE +$Base = Read-Host 'Paste the base URL printed by the running api.mjs' +& ./Show-Examples.ps1 -Base $Base +#> param([string]$Base) . "$PSScriptRoot/Get-BoundedPage.ps1" @@ -431,14 +632,28 @@ Get-BoundedPage "$Base/normal" -MaxPages 2 | ConvertTo-Json -Depth 20 -Compress Get-BoundedPage "$Base/repeat" | ConvertTo-Json -Depth 20 -Compress ``` -Open a second Bash terminal and change to the same directory. Set `PWSH` again because each terminal needs its own export. Replace the example port with the port printed by the server: +Open a second PowerShell 7.6 terminal. From the same parent directory used above, enter the directory containing your saved files and set `PWSH` again. Each terminal has its own environment. Paste the first terminal's actual printed URL at the prompt; don't reuse a port from an earlier run. -```bash {linenos=false} -export PWSH='/absolute/path/to/pwsh' -"$PWSH" -NoLogo -NoProfile -File ./Show-Examples.ps1 -Base http://127.0.0.1:54321 +```powershell {linenos=false} +Set-Location './bounded pages' +$env:PWSH = (Get-Process -Id $PID).Path +$Base = Read-Host 'Paste the base URL printed by api.mjs' ``` -These are the walkthrough results. `r` denotes the first GET without a cursor; subsequent letters are the exact `cursor` query values. Counts are `RequestsAttempted` / `ItemsFetched` / `ItemsReturned`, in that order. +First, run the deliberately incomplete one-page read. It omits continuation handling and outputs only the received items: + +```powershell {linenos=false} +$page = Invoke-RestMethod -Uri "$Base/normal" -Method Get -NoProxy -ErrorAction Stop +$page.items | ConvertTo-Json -Depth 20 -Compress +``` + +This returns IDs 1 and 2 after one GET and leaves records 3 through 5 on later pages. Run the bounded examples against the same fixture. The `&` operator invokes the executable path even if it contains spaces: + +```powershell {linenos=false} +& $env:PWSH -NoLogo -NoProfile -File ./Show-Examples.ps1 -Base $Base +``` + +In the walkthrough, `r` denotes the first GET without a cursor; subsequent letters are the exact `cursor` query values. Counts are `RequestsAttempted` / `ItemsFetched` / `ItemsReturned`, in that order. | Call | Observed result | | --- | --- | @@ -448,15 +663,36 @@ These are the walkthrough results. `r` denotes the first GET without a cursor; s The repeat case fetched four records but returned two because the second page contains records 3 and 4 and points back to `A`. The reader counts those records as fetched, then discards them when it rejects the continuation. -## Exercise the boundaries and failures -The two adapters below expose different inputs to the verifier. Save them and `verify.mjs` in the same working directory. +You can also read the helper's full help or just its examples from this directory: -### JSON adapter: `Invoke-Case.ps1` +```powershell {linenos=false} +. ./Get-BoundedPage.ps1 +Get-Help Get-BoundedPage -Full +Get-Help Read-LabPage -Examples +``` -This adapter reads a JSON array of requests and writes one JSON result per request. Passing multiple requests lets the verifier check two calls in the same PowerShell process. +## Exercise the boundaries and failures + +`Invoke-Case.ps1` reads a JSON array and writes one envelope per request. Supplying two requests lets the verifier check that state doesn't leak between calls in the same PowerShell process. + +### 4. Save `Invoke-Case.ps1` ```powershell {linenos=false} +<# +.SYNOPSIS +Runs a JSON array of reader requests in one PowerShell process. +.PARAMETER RequestFile +Path to a UTF-8 JSON array of objects with Endpoint and optional MaxPages and +MaxItems. Each object is passed to Get-BoundedPage without a type-changing cast. +.OUTPUTS +One JSON envelope per request. Reader diagnostics go to stderr. +.EXAMPLE +$Base = Read-Host 'Paste the base URL printed by the running api.mjs' +@(@{ Endpoint = "$Base/normal"; MaxPages = 2 }) | + ConvertTo-Json -Depth 10 -AsArray | Set-Content -LiteralPath ./requests.json +& ./Invoke-Case.ps1 -RequestFile ./requests.json +#> param([string]$RequestFile) $ErrorActionPreference = 'Stop' @@ -467,19 +703,34 @@ foreach ($request in $requests) { } ``` -### Typed budget probe: `Invoke-NumericProbe.ps1` +JSON can't carry every budget value the function can receive. `Invoke-NumericProbe.ps1` reads a text file containing the endpoint and passes non-finite numbers, enums and precise Decimal fractions directly to the reader. An early Double conversion would round those fractions, so the probe checks that validation rejects the original values. -JSON can't carry every budget value the function can receive. The numeric probe passes non-finite numbers and enums directly, along with precise Decimal fractions. An early Double conversion would round those fractions, so the probe checks that validation rejects the original fractional values. +Both Decimal strings use an explicit `InvariantCulture` provider because [Decimal.Parse](https://learn.microsoft.com/en-us/dotnet/api/system.decimal.parse?view=net-10.0) uses the caller's number format when no provider is supplied. Under `fr-FR`, the old adapter threw before its probe loop could test budget validation. The explicit provider keeps the exact fractions without changing the reader's culture or converting them through Double. + +### 5. Save `Invoke-NumericProbe.ps1` ```powershell {linenos=false} +<# +.SYNOPSIS +Exercises typed budgets that JSON cannot represent faithfully. +.PARAMETER RequestFile +Path to a UTF-8 text file containing the local normal-fixture endpoint. +.OUTPUTS +Fourteen JSON envelopes: twelve rejected budgets, then Decimal 1 page and item +controls. Rejection diagnostics go to stderr; only the controls issue GETs. +.EXAMPLE +$Base = Read-Host 'Paste the base URL printed by the running api.mjs' +Set-Content -LiteralPath ./endpoint.txt -Value "$Base/normal" -NoNewline +& ./Invoke-NumericProbe.ps1 -RequestFile ./endpoint.txt +#> param([string]$RequestFile) $ErrorActionPreference = 'Stop' . "$PSScriptRoot/Get-BoundedPage.ps1" $endpoint = Get-Content -Raw -LiteralPath $RequestFile foreach ($value in @([double]::NaN, [double]::PositiveInfinity, [double]::NegativeInfinity, - [decimal]::Parse('1.0000000000000000000000000001'), - [decimal]::Parse('2147483647.0000000001'))) { + [decimal]::Parse('1.0000000000000000000000000001', [Globalization.CultureInfo]::InvariantCulture), + [decimal]::Parse('2147483647.0000000001', [Globalization.CultureInfo]::InvariantCulture))) { foreach ($key in @('MaxPages', 'MaxItems')) { $arguments = @{ Endpoint = $endpoint; MaxPages = 10; MaxItems = 100 } $arguments[$key] = $value @@ -495,11 +746,13 @@ foreach ($value in @([DayOfWeek]::Monday, [decimal]1)) { } ``` -### Verifier: `verify.mjs` +`verify.mjs` starts its own listeners and waits for readiness. These addresses differ from the standalone fixture's address. `launch` runs the saved PowerShell adapters, and `runCase` compares all seven result fields with literal expectations and the server's ordered method/path/query trace. Returned JSON alone can't show whether a reader made an extra GET, retried or followed a redirect. + +The verifier also checks repeated calls, typed inputs and the three walkthrough results. To check that its assertions catch a broken cycle guard, it changes that guard in an isolated copy. Ordinary completion must still pass; repeat and cycle assertions must fail. The production reader has no switch to disable that guard. -I check the server's request trace because the returned JSON can't show whether the reader made an extra GET, retried or followed a redirect. Each case specifies an expected result and method/path/query order; `runCase` launches PowerShell and compares both. Other functions check repeated calls, typed budgets, the walkthrough and an isolated mutation of the cycle guard. +Every child has a 10-second watchdog; a watchdog kill fails the check. The verifier closes its servers in `finally` and retains a sibling `run-work-*` directory containing its request inputs and isolated mutant. -It starts its own listeners and waits for readiness, then compares all seven result fields and the actual request order. Every child has a 10-second watchdog, and a watchdog kill fails the test. The verifier closes its servers in `finally`. It retains a sibling `run-work-*` directory with the inputs and isolated mutant so you can inspect them. +### 6. Save `verify.mjs` ```javascript {linenos=false} import assert from 'node:assert/strict'; @@ -667,24 +920,27 @@ async function main() { await main(); ``` -Run the verifier from Bash: +Run the verifier in the second PowerShell terminal, from the directory containing all six files. Inspect the native exit code and the last result line after it finishes: -```bash {linenos=false} -node verify.mjs > results.jsonl +```powershell {linenos=false} +node ./verify.mjs > ./results.jsonl +$verifyExit = $LASTEXITCODE +$verifyExit +Get-Content -LiteralPath ./results.jsonl -Tail 1 ``` -The recorded run passed 65 individual cases, a fresh-process replay, two calls in one process, fourteen typed budget probes and the three walkthrough calls. Of the fourteen probes, ten reject non-finite numbers or precise Decimal fractions, two reject enums (one for each budget), and two use Decimal 1 as positive controls for the budgets. - -The cases cover empty pages, exact budgets, oversized pages, distinct `A/a` cursors, a longer cycle, malformed bodies and HTTP failures. They also check redirects and a stalled body. The redirect destination received no requests. The stalled response returned `Error/ReadFailed` through the HTTP timeout, without a watchdog kill. - -Stopping the loop isn't enough if the reader has already kept records from a cyclic page. To check that failure, the verifier changes the repeated-continuation guard to `if ($false)` in a separate copy. Ordinary completion must still pass, while the repeat and cycle checks must reject the changed reader. The production reader has no switch to disable this guard. - -The recorded run ended with this verifier line: +A passing run exits with code `0` and ends with: ```json {linenos=false} {"kind":"summary","cases":65,"passed":true} ``` +Those 65 cases are followed by a fresh-process replay, two calls in one process, fourteen typed budget probes, the three walkthrough calls and mutation checks before the summary is written. Ten typed probes reject non-finite numbers or precise Decimal fractions, two reject enums, and two use Decimal 1 as positive controls for the page and item budgets. + +The cases cover empty pages, exact budgets, oversized pages, distinct `A/a` cursors, a longer cycle, malformed bodies and HTTP failures. The redirect destination must receive no requests. The stalled response must return `Error/ReadFailed` through the HTTP timeout before the watchdog expires. Check any nonzero exit or missing summary. The presence of a results file doesn't establish success. + +When finished, use Ctrl+C in the first terminal to stop the standalone fixture. The verifier closes its own independent listeners. + ## What the budgets don't limit `MaxItems` limits the records you retain. A server can still send a large page before the reader trims it, so the budget doesn't limit downloaded bytes or the memory needed to deserialize one response.