Skip to content

[ENHANCEMENT] Retry Trino statement POST and honour 429 Retry-After - #939

Merged
Nexucis merged 3 commits into
perses:mainfrom
dougkirkley:feat/trino-retry
Nov 16, 2026
Merged

Nexucis merged 3 commits into
perses:mainfrom
dougkirkley:feat/trino-retry

Conversation

@dougkirkley

@dougkirkley dougkirkley commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part of #938. The rest waits on perses/perses#4764 (see below).

The Trino client now retries the initial POST /v1/statement as well as result-page GETs, following the client protocol:

  • 502, 503 and 504 are tried up to 3 times per request, with a 100 ms × attempt backoff.
  • A 429 is retried only after a Retry-After the client can read (delta-seconds or HTTP-date) of at most 10 s, never sooner than the backoff. A 429 with no Retry-After, an unparseable one, or one over 10 s fails at once; the error names the requested Retry-After so the panel shows when to retry.
  • The abort signal cancels any wait, the wait removes its abort listener when it ends, and the body of a retried response is cancelled.
  • When retries run out, the error says so: Trino returned 503 after 3 attempts: ….

Trino only dispatches a statement when its queued nextUri is first polled, so retrying the POST cannot run a query twice.

Perses fetch. The datasource uses the fetch Perses passes to plugins, and @perses-dev/client's fetch throws UserFriendlyError/FetchError on any non-OK response instead of returning it. Before this change that meant the existing GET retries never ran inside Perses. Both fetch paths now feed one retry decision: thrown errors with a 502, 503 or 504 .status are retried. They carry no headers, so inside Perses a 429 has no Retry-After and fails at once, as before. perses/perses#4764 asks for the headers to be kept on those errors; once it lands, the thrown path can pass them through and #938 can close.

Known limitation: if a gateway labels a non-JSON error page application/json, the Perses fetch throws a JSON parse error without a status, so that response isn't retried.

The README says which statuses are retried and how a 429 behaves inside Perses.

Tests cover 502/503/504 on POST, 429 and 503 on result-page GETs, the Retry-After forms and the 10 s boundary, 429s that fail at once, the backoff growth and floor, abort during a wait and listener cleanup, body cancellation, and both a fetch that returns non-OK responses and one that throws like the Perses fetch (UserFriendlyError and the empty-body FetchError). The fake server answers by method and URL, and the retry tests run on fake timers.

Screenshots

None, no UI changes.

Checklist

  • Pull request has a descriptive title and context useful to a reviewer.
  • Pull request title follows the [<catalog_entry>] <commit message> naming convention using one of the
    following catalog_entry values: FEATURE, ENHANCEMENT, BUGFIX, BREAKINGCHANGE, DOC,IGNORE.
  • All commits have DCO signoffs.

UI Changes

  • Changes that impact the UI include screenshots and/or screencasts of the relevant changes.
  • Code follows the UI guidelines.

Retry the initial POST /v1/statement as well as result-page GETs on
429/502/503/504, up to three attempts. A 429 waits for Retry-After
(seconds or HTTP-date, capped at 10s), never less than the normal
backoff. The Perses fetch throws on non-OK responses, so thrown errors
with a retryable status are retried too; they carry no headers and use
the normal backoff.

Closes perses#938

Signed-off-by: Douglass Kirkley <dkirkley@enlighten.com>
@dougkirkley
dougkirkley requested a review from a team as a code owner November 16, 2026 15:33
@dougkirkley
dougkirkley requested review from adrianSepiol and removed request for a team November 16, 2026 15:33
The Perses fetch throws on non-OK responses without their headers, so a
thrown 429 can't honour Retry-After; fail it at once instead of retrying
after 100/200 ms. 502/503/504 are still retried.

Route the test fake by method and URL for thrown errors too, and cover
Retry-After being ignored on a 503, backoff growth on returned errors,
and the original error kept as the cause. Document retries and the
double-run risk in the README.

Signed-off-by: Douglass Kirkley <dkirkley@enlighten.com>

@Nexucis Nexucis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dougkirkley, this is careful work: the abort handling during waits, cancelling retried bodies and the DELETE for a pending page are all nicely covered.

Requesting changes for two things (details inline):

  1. 429s follow two policies. A thrown 429 is never retried, but a returned 429 without a usable Retry-After is retried after 100/200 ms, which is what the thrown path is meant to avoid.
  2. The README overstates the double-run risk. Trino only dispatches a statement when its queued nextUri is first polled, so a retried POST doesn't run it twice.

The other comments are nits.

Also checked and fine: the source only has a type import from @perses-dev/client and duck-types .status, so there's no bundle cost and no instanceof across Module Federation copies. The app's QueryClient sets retry: 0, so these retries don't stack with React Query's.

Ran on e1d6ed13: trino vitest (64/64), oxlint, tsc --noEmit and oxfmt --check all pass.

Comment thread trino/src/model/trino-client.ts Resolved
Comment thread trino/src/model/trino-client.ts Resolved
Comment thread trino/src/model/trino-client.ts Resolved
Comment thread trino/README.md Resolved
Comment thread trino/src/model/trino-client.test.ts Resolved
Comment thread trino/src/model/trino-client.test.ts Resolved
Both fetch paths now share one decision: 502/503/504 use the backoff,
and a 429 is retried only after a Retry-After the client can read that
is within 10s. A missing, unparseable or longer Retry-After fails at
once, naming the requested value. Thrown errors pass no headers until
perses/perses#4764, so a thrown 429 still fails at once.

Read the abort signal from the request init, and remove the delay's
abort listener once the wait ends. Correct the README: Trino only
dispatches a statement on its first poll, so a retried POST does not
run it twice. Move the retry tests to fake timers and cover the
FetchError the Perses fetch throws for an empty error body.

Signed-off-by: Douglass Kirkley <dkirkley@enlighten.com>

@Nexucis Nexucis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates! All six points are addressed in c2f498e. I also ran the new tests against e1d6ed1: the 429, Retry-After cap and abort-listener tests fail there, so they guard these fixes. LGTM.

@Nexucis
Nexucis merged commit cd13aa1 into perses:main Nov 16, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants