Skip to content

fix: report the HTTP status when retries run out, and jitter the retry backoff - #1349

Open
mbevc1 wants to merge 7 commits into
mainfrom
20261106_fix_error_msgs
Open

mbevc1 wants to merge 7 commits into
mainfrom
20261106_fix_error_msgs

Conversation

@mbevc1

@mbevc1 mbevc1 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

When the shared HTTP client exhausts its retries on a 409, 429 or 5xx, the CLI reports only giving up after N attempt(s). That makes a lock conflict, a rate limit and a server failure indistinguishable, as in the v2.55.0 release failure.

  • Status in the error: a custom ErrorHandler appends the last response's status, e.g. Post "<url>": giving up after 4 attempt(s): HTTP 409. The method and URL that the default handler repeated are dropped, since the *url.Error wrapper already names them. Transport errors keep their underlying cause.
  • Upward jitter: each retry wait is stretched by a random 0-50%, capped at the maximum, so parallel jobs that fail together no longer retry in lockstep. The total time before giving up never shrinks, and a Retry-After wait is kept as is. This is a separate commit.

Closes #1348

Checklist

@github-actions github-actions Bot added go Pull requests that update go code fix labels Nov 6, 2026
@claude

claude Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @mbevc1's task in 1m 11s —— View job


Follow-up review of commit 59ecb6cc (460d4da7..59ecb6cc)

  • Fetch previous review comments/threads on this PR
  • Check resolution status of each previous finding
  • Review the new commit's changed lines for bugs/regressions
  • Post summary

Scope: this commit only exports httpStatus → HTTPStatus in internal/requests/requests.go and reuses it in cmd/kosli/evaluateHelpers.go, internal/bitbucket/bitbucket.go, and s3ErrorMessage. It does not touch jitteredBackoff or giveUpErrorHandler's logic.

Previous findings:

  1. Edge case — 409/500 with Retry-After skipping jitter: ✅ Resolved (thread confirmed resolved; untouched by this commit).
  2. Nit (optional) — use resp.Status/http.StatusText instead of bare status code: ✅ Resolved via the httpStatus/HTTPStatus helper (thread confirmed resolved; untouched by this commit).

Both inline threads are already marked resolved — nothing left to close.

New issues in the changed lines: none found. Verified:

  • No stale lowercase httpStatus( references remain anywhere in the repo; all call sites were updated to the exported HTTPStatus.
  • s3ErrorMessage, fetchRemotePolicy, serverSideRequestError, and both Bitbucket PR error paths build their format strings correctly (no doubled "HTTP" prefix, no mismatched verbs).
  • The Bitbucket wording change ("got %s instead of HTTP 200 OK") and its new table-driven test (TestOddReplies_ASuccessOtherThanOKIsNamed) line up with the fake server's new per-route status override.
  • The evaluateHelpers_test.go additions correctly pin the HasServerMessage: true + empty-message branch to the status-only message.

Note: no build/test tooling was available in this environment (go build/go test require approval that wasn't granted), so verification was by static review of the diff only.

Comment thread internal/requests/requests.go Resolved
Comment thread internal/requests/requests.go Resolved
@claude

claude Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Follow-up review of commits bb80eb83..97dd63cd

Previous findings:

  1. Edge case — 409/500 with Retry-After skipping jitter: ✅ Resolved. jitteredBackoff now compares the actual wait against the exponential wait computed with resp=nil; only a wait that differs (i.e. truly came from a parsed Retry-After) is passed through unjittered. This also incidentally fixes the smaller gap the original review called out (an unparseable Retry-After on 429/503 falling back to exponential but still skipping jitter) — since the fallback value equals the nil-resp exponential, it's jittered too. New tests cover a 409 with Retry-After and a 429 with an unparseable Retry-After. Resolved the inline thread.
  2. Nit (optional) — use resp.Status instead of bare status code in giveUpErrorHandler: Not addressed — this code wasn't touched by the new commits, and the suggestion was explicitly optional. Leaving the thread open in case the author wants to pick it up.

New issues in the changed lines: none found.

(Reviewed only the hunks in internal/requests/requests.go and internal/requests/requests_test.go touched by commits 164ec3d6, 753f835c's follow-ups bb80eb83 and 97dd63cd.)

Comment thread internal/requests/requests.go Resolved
@claude

claude Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Follow-up review of commit 8eeaef15 (97dd63c..8eeaef1)

Previous findings:

  1. Edge case — 409/500 with Retry-After skipping jitter: ✅ Already resolved (prior to this commit; untouched by it).
  2. Nit (optional) — use resp.Status/http.StatusText instead of bare status code: ✅ Resolved. A new httpStatus(code) helper renders "HTTP 409 Conflict" via http.StatusText, falling back to the bare "HTTP %d" form for codes with no standard text — matching the defensive suggestion. The exact-match test assertions (requests_test.go:353, :372) were updated accordingly, and a new TestHTTPStatus pins both the known-code and fallback cases. Closed the inline thread.

New issues in the changed lines: none found. The diff is a small, self-contained refactor (giveUpErrorHandler + new httpStatus helper) with no behavioural edge cases beyond what's covered by the updated tests.

When the retry client exhausted its retries on a 409, 429 or 5xx, its
default error handler drained the last response and returned only
"giving up after N attempt(s)". A lock conflict, a rate limit and a
server failure were indistinguishable from the CLI output, unlike every
other non-2xx path, which already reports the status.

A custom ErrorHandler now appends "HTTP <status>" to the giving-up
error. It also drops the method and URL the default handler repeated,
because the *url.Error that http.Client wraps the error in already
names them. Transport errors keep their underlying cause.
Retries used retryablehttp's default exponential backoff, so clients
that failed together waited exactly as long as each other and collided
again on every retry. Parallel CI jobs attesting to one trail are the
common case: the v2.55.0 release failed when about half of its matrix
jobs gave up at once.

Each wait is now stretched by a random 0-50%, capped at the maximum
wait. Stretching rather than shrinking keeps the total time before
giving up at least what it was, so a server blip the CLI recovers from
today is still recovered from. A wait the server sets through
Retry-After is left untouched, and the number of attempts and the
statuses retried are unchanged.
The handler had three return paths that each rebuilt the same message.
It now builds the message once, adding the status when there is a last
response and the cause when there is an error. The comments on the
handler and the backoff are cut to the facts the code cannot show.
The backoff skipped the jitter whenever a response carried a
Retry-After header. retryablehttp only honours that header on a 429 or
503, and only when it parses, so a 409 lock conflict from a proxy that
adds the header, or a 429 with an unparseable value, got the plain
exponential wait with no jitter.

The backoff now treats a wait as server-set only when it differs from
the plain exponential wait, which matches the library's own rules
without copying them.
"HTTP 409" leaves the reader to know what the code means. The error now
reads "HTTP 409 Conflict" or "HTTP 429 Too Many Requests", which tells a
lock conflict from a rate limit at a glance.

The text comes from http.StatusText rather than resp.Status, so the
server's reason phrase never reaches the output and a hand-built
response without a status line still renders.
@mbevc1
mbevc1 force-pushed the 20261106_fix_error_msgs branch from 8eeaef1 to ede8bf2 Compare November 6, 2026 20:53
@mbevc1
mbevc1 enabled auto-merge (squash) November 6, 2026 20:55
* fix: report non-JSON error responses by their HTTP status

An error response whose body is not JSON, such as a proxy's HTML page
or an empty 401 or 404, was reported by the JSON decoder's complaint
alone, e.g. "invalid character '<' looking for beginning of value". The
status was carried on the APIError but never reached the message, so
the user could not tell a firewall block from a missing resource.

The message is now the status, such as "HTTP 403 Forbidden". When the
body is not empty it adds that the response is not JSON and, unless
--debug is already on, that rerunning with --debug shows it, since the
body is logged at debug level only.

* fix: treat a whitespace-only error body as empty

Some proxies answer a 401 or 404 with a lone newline. That body counted
as non-empty, so the user was told to rerun with --debug to see it, and
--debug then showed nothing. The body is now trimmed before the check.

Also renames the test case for an unrouted GET, which gets an empty 404
rather than the HTML body its old name described.
@claude

claude Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Follow-up review of commit 460d4da7 (ede8bf24..460d4da7)

Scope: this commit only touches internal/requests/requests.go (new nonJSONErrorMessage helper, reusing the existing httpStatus helper) and internal/requests/requests_test.go. It does not touch jitteredBackoff or giveUpErrorHandler.

Previous findings:

  1. Edge case — 409/500 with Retry-After skipping jitter: ✅ Already resolved (prior to this commit; untouched by it — thread already marked resolved).
  2. Nit (optional) — use resp.Status/http.StatusText instead of bare status code: ✅ Already resolved via the httpStatus helper (prior to this commit; untouched by it — thread already marked resolved).

New issues in the changed lines: none found. nonJSONErrorMessage correctly distinguishes an empty/whitespace-only body (status alone), a non-JSON body with --debug on (status + note), and without (status + note to rerun with --debug), reusing httpStatus for consistent formatting. New/updated tests (requests_test.go) cover all three branches and match the implementation; no stale "unexpected end of JSON input"/"invalid character" assertions remain. No build/test tooling was available in this environment, so verification was by static review of the diff only.

…1351)

* refactor: export HTTPStatus for use outside the requests package

Other packages report HTTP errors by bare code and are about to render
them the same way as the API client does, so the helper needs to be
reachable from them.

* fix: name the status in upload errors

An upload refused by the storage bucket read "upload refused with HTTP
403". It now reads "HTTP 403 Forbidden", like the API client's errors.

* fix: name the status in kosli evaluate errors

A refused policy fetch and a server-side evaluation the API turned down
were reported by bare code, e.g. "the Kosli server answered 403". They
now read "HTTP 403 Forbidden", like the API client's errors.

* fix: name the status in Bitbucket pull request errors

A pull request or commit list that came back with a success other than
200 was reported as "got HTTP status 202". It now reads "got HTTP 202
Accepted", like the API client's errors. The fake Bitbucket server in
the tests can now answer a reply with a chosen status, to reach these
paths.

* fix: say what Bitbucket should have answered instead of blaming permissions

The pull request details error reached only on a 201 or 202 told the
user to review repository permissions, which a success status does not
suggest. Both Bitbucket errors on that path now say the reply was not
the 200 OK the code reads. The tests match the whole message.

Also pins the evaluate error for a server message that is present but
empty, which reports the status alone.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix go Pull requests that update go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retry-exhausted errors omit the HTTP status code

1 participant