Repository navigation
feat: abort a solve when the caller closes its connection - #35
Merged
Merged
Conversation
run_with_timeout blocked in process.join(25), so a solve whose caller had gone away (e.g. a Redux worker process that was killed) kept its gunicorn sync worker and a CPU busy until it finished or timed out. Wait in short steps instead and, between them, check the client socket gunicorn exposes as environ["gunicorn.socket"]: a closed connection polls readable and peeks EOF (or raises a reset), and the solve's child process is then terminated. Under the flask dev server and test client there is no socket, so only the timeout applies, as before. Raise the backstop timeout from 25 to 30 seconds (SOLVE_TIMEOUT), and with it the gunicorn worker timeout to 35 s and nginx's proxy_read_timeout to 40 s so each still exceeds the one inside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Redux Build System — CI Report
Overall: ✅ 7 passed · 0 failed · 2 skipped |
Trosper3
approved these changes
Nov 17, 2026
Trosper3
left a comment
There was a problem hiding this comment.
Looks good, thanks Jason. Freeing the worker when the caller hangs up is a real win, and the tests cover the close, reset and keep-alive cases well. Two notes, neither blocking:
- Tests on Windows.
socket.MSG_DONTWAITdoesn't exist on Windows, so 5 tests intest_run_with_timeout.pyfail there (4 with AttributeError, andtest_times_outbecause an earlier failure leaves a child process running). Production and CI are Linux, so it only affects developers on Windows.getattr(socket, "MSG_DONTWAIT", 0)would fix it, and is safe becauseselecthas already said the socket is readable. - The 30-second timeout. Redux's quantum request timeout is 30 seconds (
QuantumServerAPI), and Redux's compute limit is going to 30 seconds (ReduxISU/Redux#1022). With the service also at 30, Redux always gives up first, so the service's clean"timeout"answer never reaches it. Since this PR already stops abandoned solves, keepingSOLVE_TIMEOUTaround 25 seconds (and gunicorn/nginx where they were) would let Redux show a real timeout answer. Your call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
run_with_timeoutblocked inprocess.join(25). A solve whose caller had gone away, such as a Redux worker process that was killed, kept its gunicornsyncworker and a CPU busy until it finished or timed out. Now it:POLL_INTERVAL(0.2 s) instead of one long join;environ["gunicorn.socket"]. A closed connection polls readable and aMSG_PEEKread returns EOF, or the read fails with a reset. When that happens the solve's child process is terminated. Peeking leaves real data unread, so a keep-alive caller's next request isn't mistaken for a close;SOLVE_TIMEOUT).Under the flask dev server and the test client there's no
gunicorn.socket, so only the timeout applies, as before.Timeouts that must each exceed the one inside them:
routes.SOLVE_TIMEOUTgunicorn.conf.pytimeoutdeploy/quantumsolver.nginxproxy_read_timeoutThe comments mentioning 25 s are updated, and the README now describes the timeout and the abort.
What it can't detect
If the caller's machine dies or the network drops, no FIN or RST ever arrives, so nothing is detected. The 30 s backstop still covers that case.
Testing
New
tests/test_run_with_timeout.py(9 tests):client_gone: no socket, an idle connection, unread data (not treated as a close, and still readable afterwards), a closed connection, and a reset (SO_LINGER0);run_with_timeout: aborts within seconds when the caller closes mid-solve, still times out atSOLVE_TIMEOUT, and returns answers normally with and without a socket. It also checks that no child process is left running.Full suite: 93 passed on 3.12 and 3.13, with 88% coverage (
routes.pyfrom 78% to 85%). ruff and black are clean.End to end under gunicorn with the repo's
gunicorn.conf.pyand a singlesyncworker, using Shor's with N=391 (takes about 155 s on its own):/healthanswered in 0.05 s on the same worker; no solver children left{"answer":"timeout"}, HTTP 200, at 30.2 s; no gunicorn worker timeout[3,5]as beforegunicorn logged no broken-pipe or worker-timeout errors.
Deployment notes
deploy/quantumsolver.nginxhas to be installed on the nginx host for the 40 sproxy_read_timeoutto take effect. Until it is, nginx's 35 s still exceeds the 30 s backstop, just with less margin.QuantumServerAPIHttpClienttimeout is 30 s, so Redux now gives up at about the same moment the backstop fires, and will rarely receive the"timeout"answer itself. When it gives up, its closed connection aborts the solve anyway.This is based on the idea from #4 and
closes #4
🤖 Generated with Claude Code