Skip to content

feat: abort a solve when the caller closes its connection - #35

Merged
Trosper3 merged 1 commit into
mainfrom
feature/abort-on-client-disconnect
Nov 17, 2026
Merged

Trosper3 merged 1 commit into
mainfrom
feature/abort-on-client-disconnect

Conversation

@wrigjl

@wrigjl wrigjl commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

run_with_timeout blocked in process.join(25). A solve whose caller had gone away, such as a Redux worker process that was killed, kept its gunicorn sync worker and a CPU busy until it finished or timed out. Now it:

  • waits in short steps of POLL_INTERVAL (0.2 s) instead of one long join;
  • checks the client socket between steps. gunicorn exposes it as environ["gunicorn.socket"]. A closed connection polls readable and a MSG_PEEK read returns EOF, or the read fails with a reset. When that happens the solve's child process is terminated. Peeking leaves real data unread, so a keep-alive caller's next request isn't mistaken for a close;
  • keeps a backstop timeout, raised from 25 to 30 s (SOLVE_TIMEOUT).

Under the flask dev server and the test client there's no gunicorn.socket, so only the timeout applies, as before.

Timeouts that must each exceed the one inside them:

Setting Before After
routes.SOLVE_TIMEOUT 25 s 30 s
gunicorn.conf.py timeout 30 s 35 s
deploy/quantumsolver.nginx proxy_read_timeout 35 s 40 s

The comments mentioning 25 s are updated, and the README now describes the timeout and the abort.

What it can't detect

If the caller's machine dies or the network drops, no FIN or RST ever arrives, so nothing is detected. The 30 s backstop still covers that case.

Testing

  • New tests/test_run_with_timeout.py (9 tests):

    • client_gone: no socket, an idle connection, unread data (not treated as a close, and still readable afterwards), a closed connection, and a reset (SO_LINGER 0);
    • run_with_timeout: aborts within seconds when the caller closes mid-solve, still times out at SOLVE_TIMEOUT, and returns answers normally with and without a socket. It also checks that no child process is left running.
  • Full suite: 93 passed on 3.12 and 3.13, with 88% coverage (routes.py from 78% to 85%). ruff and black are clean.

  • End to end under gunicorn with the repo's gunicorn.conf.py and a single sync worker, using Shor's with N=391 (takes about 155 s on its own):

    Check Result
    Caller gives up after 2 s /health answered in 0.05 s on the same worker; no solver children left
    Caller waits {"answer":"timeout"}, HTTP 200, at 30.2 s; no gunicorn worker timeout
    N=15 [3,5] as before

    gunicorn logged no broken-pipe or worker-timeout errors.

Deployment notes

  • deploy/quantumsolver.nginx has to be installed on the nginx host for the 40 s proxy_read_timeout to take effect. Until it is, nginx's 35 s still exceeds the 30 s backstop, just with less margin.
  • Redux's QuantumServerAPI HttpClient timeout is 30 s, so Redux now gives up at about the same moment the backstop fires, and will rarely receive the "timeout" answer itself. When it gives up, its closed connection aborts the solve anyway.

This is based on the idea from #4 and
closes #4

🤖 Generated with Claude Code

run_with_timeout blocked in process.join(25), so a solve whose caller
had gone away (e.g. a Redux worker process that was killed) kept its
gunicorn sync worker and a CPU busy until it finished or timed out.
Wait in short steps instead and, between them, check the client socket
gunicorn exposes as environ["gunicorn.socket"]: a closed connection
polls readable and peeks EOF (or raises a reset), and the solve's child
process is then terminated. Under the flask dev server and test client
there is no socket, so only the timeout applies, as before.

Raise the backstop timeout from 25 to 30 seconds (SOLVE_TIMEOUT), and
with it the gunicorn worker timeout to 35 s and nginx's
proxy_read_timeout to 40 s so each still exceeds the one inside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Redux Build System — CI Report

uv · commit d9e401e

Operation Status Summary Time
audit ✅ no known vulnerabilities 7.5s
format-check ✅ all files formatted 0.6s
lint ✅ 0 issues 0.0s
typecheck ⏭️ not implemented —
unit-test · 3.12 ✅ 93 passed · coverage 88% 105.6s
unit-test · 3.13 ✅ 93 passed · coverage 88% 98.3s
build ✅ built local/quantumsolver:ci · 582MB 16.1s
integration-test ✅ /health ready · 7 passed 14.7s
push ⏭️ only on push to main —

Overall: ✅ 7 passed · 0 failed · 2 skipped

@Trosper3 Trosper3 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks Jason. Freeing the worker when the caller hangs up is a real win, and the tests cover the close, reset and keep-alive cases well. Two notes, neither blocking:

  1. Tests on Windows. socket.MSG_DONTWAIT doesn't exist on Windows, so 5 tests in test_run_with_timeout.py fail there (4 with AttributeError, and test_times_out because an earlier failure leaves a child process running). Production and CI are Linux, so it only affects developers on Windows. getattr(socket, "MSG_DONTWAIT", 0) would fix it, and is safe because select has already said the socket is readable.
  2. The 30-second timeout. Redux's quantum request timeout is 30 seconds (QuantumServerAPI), and Redux's compute limit is going to 30 seconds (ReduxISU/Redux#1022). With the service also at 30, Redux always gives up first, so the service's clean "timeout" answer never reaches it. Since this PR already stops abandoned solves, keeping SOLVE_TIMEOUT around 25 seconds (and gunicorn/nginx where they were) would let Redux show a real timeout answer. Your call.

@Trosper3
Trosper3 merged commit 92dd2bf into main Nov 17, 2026
4 checks passed
@Trosper3
Trosper3 deleted the feature/abort-on-client-disconnect branch November 17, 2026 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

no hardcoded timeout for processing

2 participants