Skip to content

Fix race between SendMessage and server-initiated key re-exchange - #1823

Merged
Rob-Hague merged 1 commit into
sshnet:developfrom
radekvermirovsky:fix/rekey-race-1764
Nov 3, 2026
Merged

Rob-Hague merged 1 commit into
sshnet:developfrom
radekvermirovsky:fix/rekey-race-1764

Conversation

@radekvermirovsky

Copy link
Copy Markdown
Contributor

Fixes #1764

The race

Session.SendMessage checks _keyExchangeCompletedWaitHandle before acquiring _socketWriteLock. When the server initiates a key re-exchange in that window, the message loop thread resets the wait handle and sends the client's SSH_MSG_KEXINIT (under the same write lock). A data message that already passed the check then acquires the lock and goes out on the wire after our KEXINIT, violating RFC 4253 section 7.1 ("Once a party has sent a SSH_MSG_KEXINIT message ... it MUST NOT send any messages other than: Transport layer generic messages ...; Algorithm negotiation messages ...; Specific key exchange method messages").

Strict servers such as ProFTPD mod_sftp then fail the exchange or drop the connection, producing the two failure variants reported in #1764 (Message type 93 is not valid / connection drop with Key exchange failed).

The fix

SendMessage now re-checks the wait handle while holding the write lock and goes back to waiting when a re-exchange has started in the meantime. Re-checking under the lock is required: making the reset + KEXINIT send atomic is not sufficient, because a sender can already be blocked on the lock with a stale check result.

The packet is now also built entirely under the write lock, so a completing re-exchange can no longer swap _clientCipher/_clientMac/compression in the middle of building a packet.

The diff looks bigger than it is because the packet-building body moved unchanged into SendMessageWithinWriteLock — reviewing with whitespace changes hidden shows the actual change is the retry loop.

Why #1774 could not reproduce it

The repro attempt in #1774 ran against the OpenSSH test server. OpenSSH queues non-key-exchange output while a re-exchange is in progress and tolerates client data that slips in, so the race stays invisible there. ProFTPD mod_sftp does not tolerate it, so this PR adds an integration test running concurrent SFTP uploads against a ProFTPD container (alpine:3.24, ProFTPD 1.3.9c) configured to re-key every 1 MB (SFTPRekey required 3600 1), giving the race many trials per upload.

Evidence

Same environment (Docker Desktop/WSL2, net10.0), 4 concurrent uploads of 128 MB files, 3 attempts per run:

Result
Without the fix 7 of 8 runs failed with SshConnectionException: Key exchange failed
With the fix 4 of 4 runs passed

@radekvermirovsky

Copy link
Copy Markdown
Contributor Author

A note on CI coverage: the new ProFTPD rekey test currently runs on the Linux job only. On the Windows jobs it reports inconclusive, because Docker on the Windows runners is in Windows containers mode and cannot run the Linux ProFTPD image.

If you'd like it to run there too, I'm happy to extend the "Setup SSH Server" step of the Windows Integration Tests .NET job to also build and start the ProFTPD container with Podman in WSL2 on a second fixed port (mirroring how the OpenSSH server is provided), and to switch the test to the same Windows-CI detection InfrastructureFixture uses instead of skipping. Just let me know if that's wanted, or if the Linux coverage is sufficient.

@radekvermirovsky

Copy link
Copy Markdown
Contributor Author

@Rob-Hague @WojciechNagorski @drieseng — gentle ping on this one.

The branch is up to date with develop and CI is green on the Linux integration job. I'm happy to keep maintaining it (rebases, review feedback, extending the ProFTPD rekey test to the Windows runners as offered above), but I don't want it sitting in the queue if it isn't a direction you want to take.

Could one of you let me know whether this is worth keeping open? Specifically:

  1. Is the approach (re-checking _keyExchangeCompletedWaitHandle under _socketWriteLock and building the packet inside the lock) acceptable in principle, or would you prefer the fix somewhere else in the send path?
  2. Is the Linux-only ProFTPD integration test enough, or should I wire it up for the Windows CI jobs too?

If there's no appetite for it, just say so and I'll close it — no hard feelings. Otherwise I'll keep it rebased and ready.

@radekvermirovsky
radekvermirovsky force-pushed the fix/rekey-race-1764 branch 2 times, most recently from ff84d50 to f400dd5 Compare October 21, 2026 15:24
Fixes a race condition during key re-exchange where the message listener thread receives
and responds to KEXINIT between the time that another thread sending a (non-KEX/transport)
message checks _keyExchangeCompletedWaitHandle.IsSet and enters the write lock. This is a
protocol error since neither party may send such messages between KEXINIT and NEWKEYS
(RFC 4253 section 7.1)

Fix by checking the handle state within the lock. Because sending the KEXINIT takes the
same path with the same lock, it cannot preempt the other message and either it will be
sent afterwards or it is sent first and the other message will correctly wait for the
kex to complete.

We also relax the restriction on non-KEX messages during re-exchange under strict kex,
since the spec intends this just for the initial exchange. This is not strictly necessary
for this fix.

@Rob-Hague Rob-Hague left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this change is a sufficient fix: Rob-Hague@76ab503?w=1

/// ProFTPD mod_sftp keeps sending channel messages (SSH_MSG_CHANNEL_WINDOW_ADJUST,
/// SSH_MSG_CHANNEL_DATA) after it has sent its SSH_MSG_KEXINIT. The same load which passes
/// against the OpenSSH test server therefore fails against ProFTPD with either
/// "Message type 93 is not valid in the current context." or a connection drop

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please double check this? I do not believe that is what is happening. RFC 4253 section 7.1 says

Once a party has sent a SSH_MSG_KEXINIT message for key exchange or
re-exchange, until it has sent a SSH_MSG_NEWKEYS message (Section
7.3), it MUST NOT send any messages other than:
...

It would be against spec if ProFTPD were to be sending channel messages after it has sent KEXINIT, not something for this library to solve. Note that error message in #1764 is Message type 93 is not valid., not Message type 93 is not valid in the current context.. The former message pattern I do not see in this library and I believe it is a message from the server.

I think this is rather a problem in this library where responding to the KEXINIT can preempt other messages that have already checked a kex is not in progress.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Rob — you’re right. I re-ran the ProFTPD test with trace logging on the send-side-only state of this branch: across 1138 server-initiated re-exchanges ProFTPD sent nothing but SSH_MSG_KEX_ECDH_REPLY between its KEXINIT and NEWKEYS, so the comment in the test is wrong and the failure is purely our send side preempted by the KEXINIT reply. Sorry for the noise.

Your 76ab503 is functionally the same fix as this PR, just cleaner, and I agree with the strict-kex relaxation.

Do you want to push it to develop yourself (then I’ll close this PR), or should I finish it here — take your commit, drop my Session.cs change and the receive-side commit, fix the test comments, keep the ProFTPD test? Either works for me.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I would take my commit as is, without the ProFTPD test for now (it was helpful to validate and I like the idea of testing against multiple servers, but don't think there is enough value-add for one test).

If you would like contribution credit for the investigation and the validation, you can reset your branch to mine or I can add you as a co-author on the commit.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I’ve reset this branch to your 76ab503 as-is, so the PR is now just your commit — feel free to add me as co-author if you like, but no need. I’ll keep the ProFTPD test around in my fork in case multi-server testing becomes a thing later.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍

@Rob-Hague
Rob-Hague merged commit 3476f73 into sshnet:develop Nov 3, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[SFTP] Race Condition during Rekeying: "Message type 93 is not valid" or Connection Drop

2 participants