Skip to content

Add URMA runtime recovery and TCP fallback - #92

Open
Spicy-cream wants to merge 21 commits into
integration/rdma-urmafrom
feature/rdma-runtime-recovery
Open

Spicy-cream wants to merge 21 commits into
integration/rdma-urmafrom
feature/rdma-runtime-recovery

Conversation

@Spicy-cream

@Spicy-cream Spicy-cream commented Oct 26, 2026 •

Copy link
Copy Markdown

What problem does this PR solve?

URMA can fall back to TCP during connection setup, but established failures do not control subsequent admission. A failed server SEND can close the control socket while the client immediately reconnects over URMA to the same peer.

What is changed and the side effects?

  • Add process-level LEGACY, AUTO and TCP_ONLY modes. AUTO applies a configurable peer cooldown, defaulting to 5 seconds; LEGACY remains the default.
  • Report completion, JFC and WR posting failures once per connection, filter earlier epochs, count access errors separately and preserve EAGAIN for backpressure.
  • Report established client control-socket EOF/read failures so server-side faults also trigger peer cooldown. Normal server-side EOF from short clients closes the connection without quarantining the peer.
  • Select CTP/RTP paths from device capabilities, release short-connection resources, serialize TCP reads and initialize/validate send SGEs. Optional aligned copied sends add one payload copy.
  • Add a recovery client, process-local SEND injection, independent regressions and a URMA mock workflow for pull requests, including non-master targets.

Mode changes affect new connections. Short connections can resume URMA after cooldown; established single/pooled TCP connections can continue serving RPCs. Existing v2/v3 wire formats remain compatible.

Validation

  • Synchronized with Apache master at 2a3a186. The CI configure/build/test commands pass locally with RDMA disabled and URMA enabled: 15 regression groups passed.
  • The server-SEND regression fails before the control-socket fix. All 8 CTP/RTP, v2/v3 and event/polling cases pass after it, covering normal short closes, TCP cooldown, URMA recovery and 64 KiB attachment echoes.
  • C++14 compilation, the injection runner and fetched-SDK-header configuration checks passed.
  • Prior URMA RM/RTP hardware runs passed TCP_ONLY, v2/v3 AUTO, mode switching, 64 KiB/1 MiB attachments and client SEND_POST injection across short/single/pooled and event/polling cases.

Related upstream work: send-buffer reclamation, polling lifetime and common URMA handshake.

Check List

  • Build the library and run URMA regressions with mock CI.
  • Add independent tests, preserve community UT sources and update documentation.
  • Follow four-space indentation and C++ nullptr conventions.

@Spicy-cream Spicy-cream changed the title Design RDMA runtime recovery and controlled TCP fallback Add initial RDMA runtime recovery and TCP fallback Oct 26, 2026
@Spicy-cream Spicy-cream changed the title Add initial RDMA runtime recovery and TCP fallback Add RDMA and URMA runtime recovery and TCP fallback Oct 28, 2026
@Spicy-cream Spicy-cream mentioned this pull request Nov 2, 2026
3 tasks done
@Spicy-cream
Spicy-cream changed the base branch from master to integration/rdma-urma November 2, 2026 03:49
@Spicy-cream Spicy-cream changed the title Add RDMA and URMA runtime recovery and TCP fallback Add URMA runtime recovery and TCP fallback Nov 2, 2026
@Spicy-cream
Spicy-cream marked this pull request as ready for review November 2, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant