Repository navigation
[ENHANCEMENT] Retry Trino statement POST and honour 429 Retry-After - #939
Merged
Merged
Conversation
Retry the initial POST /v1/statement as well as result-page GETs on 429/502/503/504, up to three attempts. A 429 waits for Retry-After (seconds or HTTP-date, capped at 10s), never less than the normal backoff. The Perses fetch throws on non-OK responses, so thrown errors with a retryable status are retried too; they carry no headers and use the normal backoff. Closes perses#938 Signed-off-by: Douglass Kirkley <dkirkley@enlighten.com>
dougkirkley
requested review from
adrianSepiol
and removed request for
a team
November 16, 2026 15:33
The Perses fetch throws on non-OK responses without their headers, so a thrown 429 can't honour Retry-After; fail it at once instead of retrying after 100/200 ms. 502/503/504 are still retried. Route the test fake by method and URL for thrown errors too, and cover Retry-After being ignored on a 503, backoff growth on returned errors, and the original error kept as the cause. Document retries and the double-run risk in the README. Signed-off-by: Douglass Kirkley <dkirkley@enlighten.com>
This was referenced Nov 16, 2026
Nexucis
requested changes
Nov 16, 2026
Nexucis
left a comment
Member
There was a problem hiding this comment.
Thanks @dougkirkley, this is careful work: the abort handling during waits, cancelling retried bodies and the DELETE for a pending page are all nicely covered.
Requesting changes for two things (details inline):
- 429s follow two policies. A thrown 429 is never retried, but a returned 429 without a usable
Retry-Afteris retried after 100/200 ms, which is what the thrown path is meant to avoid. - The README overstates the double-run risk. Trino only dispatches a statement when its queued
nextUriis first polled, so a retriedPOSTdoesn't run it twice.
The other comments are nits.
Also checked and fine: the source only has a type import from @perses-dev/client and duck-types .status, so there's no bundle cost and no instanceof across Module Federation copies. The app's QueryClient sets retry: 0, so these retries don't stack with React Query's.
Ran on e1d6ed13: trino vitest (64/64), oxlint, tsc --noEmit and oxfmt --check all pass.
Both fetch paths now share one decision: 502/503/504 use the backoff, and a 429 is retried only after a Retry-After the client can read that is within 10s. A missing, unparseable or longer Retry-After fails at once, naming the requested value. Thrown errors pass no headers until perses/perses#4764, so a thrown 429 still fails at once. Read the abort signal from the request init, and remove the delay's abort listener once the wait ends. Correct the README: Trino only dispatches a statement on its first poll, so a retried POST does not run it twice. Move the retry tests to fake timers and cover the FetchError the Perses fetch throws for an empty error body. Signed-off-by: Douglass Kirkley <dkirkley@enlighten.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of #938. The rest waits on perses/perses#4764 (see below).
The Trino client now retries the initial
POST /v1/statementas well as result-pageGETs, following the client protocol:Retry-Afterthe client can read (delta-seconds or HTTP-date) of at most 10 s, never sooner than the backoff. A 429 with noRetry-After, an unparseable one, or one over 10 s fails at once; the error names the requestedRetry-Afterso the panel shows when to retry.Trino returned 503 after 3 attempts: ….Trino only dispatches a statement when its queued
nextUriis first polled, so retrying thePOSTcannot run a query twice.Perses fetch. The datasource uses the
fetchPerses passes to plugins, and@perses-dev/client'sfetchthrowsUserFriendlyError/FetchErroron any non-OK response instead of returning it. Before this change that meant the existingGETretries never ran inside Perses. Both fetch paths now feed one retry decision: thrown errors with a 502, 503 or 504.statusare retried. They carry no headers, so inside Perses a 429 has noRetry-Afterand fails at once, as before. perses/perses#4764 asks for the headers to be kept on those errors; once it lands, the thrown path can pass them through and #938 can close.Known limitation: if a gateway labels a non-JSON error page
application/json, the Persesfetchthrows a JSON parse error without a status, so that response isn't retried.The README says which statuses are retried and how a 429 behaves inside Perses.
Tests cover 502/503/504 on
POST, 429 and 503 on result-pageGETs, theRetry-Afterforms and the 10 s boundary, 429s that fail at once, the backoff growth and floor, abort during a wait and listener cleanup, body cancellation, and both a fetch that returns non-OK responses and one that throws like the Persesfetch(UserFriendlyErrorand the empty-bodyFetchError). The fake server answers by method and URL, and the retry tests run on fake timers.Screenshots
None, no UI changes.
Checklist
[<catalog_entry>] <commit message>naming convention using one of thefollowing
catalog_entryvalues:FEATURE,ENHANCEMENT,BUGFIX,BREAKINGCHANGE,DOC,IGNORE.UI Changes