Repository navigation
Fix race between SendMessage and server-initiated key re-exchange - #1823
Conversation
|
A note on CI coverage: the new ProFTPD rekey test currently runs on the Linux job only. On the Windows jobs it reports inconclusive, because Docker on the Windows runners is in Windows containers mode and cannot run the Linux ProFTPD image. If you'd like it to run there too, I'm happy to extend the "Setup SSH Server" step of the Windows Integration Tests .NET job to also build and start the ProFTPD container with Podman in WSL2 on a second fixed port (mirroring how the OpenSSH server is provided), and to switch the test to the same Windows-CI detection |
|
@Rob-Hague @WojciechNagorski @drieseng — gentle ping on this one. The branch is up to date with Could one of you let me know whether this is worth keeping open? Specifically:
If there's no appetite for it, just say so and I'll close it — no hard feelings. Otherwise I'll keep it rebased and ready. |
ff84d50 to
f400dd5
Compare
Fixes a race condition during key re-exchange where the message listener thread receives and responds to KEXINIT between the time that another thread sending a (non-KEX/transport) message checks _keyExchangeCompletedWaitHandle.IsSet and enters the write lock. This is a protocol error since neither party may send such messages between KEXINIT and NEWKEYS (RFC 4253 section 7.1) Fix by checking the handle state within the lock. Because sending the KEXINIT takes the same path with the same lock, it cannot preempt the other message and either it will be sent afterwards or it is sent first and the other message will correctly wait for the kex to complete. We also relax the restriction on non-KEX messages during re-exchange under strict kex, since the spec intends this just for the initial exchange. This is not strictly necessary for this fix.
Rob-Hague
left a comment
There was a problem hiding this comment.
I believe this change is a sufficient fix: Rob-Hague@76ab503?w=1
| /// ProFTPD mod_sftp keeps sending channel messages (SSH_MSG_CHANNEL_WINDOW_ADJUST, | ||
| /// SSH_MSG_CHANNEL_DATA) after it has sent its SSH_MSG_KEXINIT. The same load which passes | ||
| /// against the OpenSSH test server therefore fails against ProFTPD with either | ||
| /// "Message type 93 is not valid in the current context." or a connection drop |
There was a problem hiding this comment.
Could you please double check this? I do not believe that is what is happening. RFC 4253 section 7.1 says
Once a party has sent a SSH_MSG_KEXINIT message for key exchange or
re-exchange, until it has sent a SSH_MSG_NEWKEYS message (Section
7.3), it MUST NOT send any messages other than:
...
It would be against spec if ProFTPD were to be sending channel messages after it has sent KEXINIT, not something for this library to solve. Note that error message in #1764 is Message type 93 is not valid., not Message type 93 is not valid in the current context.. The former message pattern I do not see in this library and I believe it is a message from the server.
I think this is rather a problem in this library where responding to the KEXINIT can preempt other messages that have already checked a kex is not in progress.
There was a problem hiding this comment.
Thanks Rob — you’re right. I re-ran the ProFTPD test with trace logging on the send-side-only state of this branch: across 1138 server-initiated re-exchanges ProFTPD sent nothing but SSH_MSG_KEX_ECDH_REPLY between its KEXINIT and NEWKEYS, so the comment in the test is wrong and the failure is purely our send side preempted by the KEXINIT reply. Sorry for the noise.
Your 76ab503 is functionally the same fix as this PR, just cleaner, and I agree with the strict-kex relaxation.
Do you want to push it to develop yourself (then I’ll close this PR), or should I finish it here — take your commit, drop my Session.cs change and the receive-side commit, fix the test comments, keep the ProFTPD test? Either works for me.
There was a problem hiding this comment.
Thanks, I would take my commit as is, without the ProFTPD test for now (it was helpful to validate and I like the idea of testing against multiple servers, but don't think there is enough value-add for one test).
If you would like contribution credit for the investigation and the validation, you can reset your branch to mine or I can add you as a co-author on the commit.
There was a problem hiding this comment.
Sounds good. I’ve reset this branch to your 76ab503 as-is, so the PR is now just your commit — feel free to add me as co-author if you like, but no need. I’ll keep the ProFTPD test around in my fork in case multi-server testing becomes a thing later.
f400dd5 to
76ab503
Compare
Fixes #1764
The race
Session.SendMessagechecks_keyExchangeCompletedWaitHandlebefore acquiring_socketWriteLock. When the server initiates a key re-exchange in that window, the message loop thread resets the wait handle and sends the client'sSSH_MSG_KEXINIT(under the same write lock). A data message that already passed the check then acquires the lock and goes out on the wire after our KEXINIT, violating RFC 4253 section 7.1 ("Once a party has sent a SSH_MSG_KEXINIT message ... it MUST NOT send any messages other than: Transport layer generic messages ...; Algorithm negotiation messages ...; Specific key exchange method messages").Strict servers such as ProFTPD mod_sftp then fail the exchange or drop the connection, producing the two failure variants reported in #1764 (
Message type 93 is not valid/ connection drop withKey exchange failed).The fix
SendMessagenow re-checks the wait handle while holding the write lock and goes back to waiting when a re-exchange has started in the meantime. Re-checking under the lock is required: making the reset + KEXINIT send atomic is not sufficient, because a sender can already be blocked on the lock with a stale check result.The packet is now also built entirely under the write lock, so a completing re-exchange can no longer swap
_clientCipher/_clientMac/compression in the middle of building a packet.The diff looks bigger than it is because the packet-building body moved unchanged into
SendMessageWithinWriteLock— reviewing with whitespace changes hidden shows the actual change is the retry loop.Why #1774 could not reproduce it
The repro attempt in #1774 ran against the OpenSSH test server. OpenSSH queues non-key-exchange output while a re-exchange is in progress and tolerates client data that slips in, so the race stays invisible there. ProFTPD mod_sftp does not tolerate it, so this PR adds an integration test running concurrent SFTP uploads against a ProFTPD container (
alpine:3.24, ProFTPD 1.3.9c) configured to re-key every 1 MB (SFTPRekey required 3600 1), giving the race many trials per upload.Evidence
Same environment (Docker Desktop/WSL2, net10.0), 4 concurrent uploads of 128 MB files, 3 attempts per run:
SshConnectionException: Key exchange failed